1. Gaussian mean estimation.
1.1. Optimal shrinkage factor [10pts]. Let X1; X2; :::; Xn 2 Rd be i.i.d. multivariate Gaussian random vectors, i.e., Xi N ( ; 2I). Denoting the sample mean estimator with ^ , 1 Pn Xi,
n i=1
consider an estimator of the form ^s = 1
^: Find the optimal that minimizes the risk
2
R(^s; ) = E[k^s k22].
k^k2
1.2. Generalizing Stein’s lemma [10pts]. Let X p (x) and g : Rd ! Rd where p (x) and g (x) are di erentiable w.r.t and x, and let E[g (X)] = ( ) for some function . Show that
(a) E[rx log p (X)g (X)>] + E[rxg (X)] = 0,
(b) E[r log p (X)g (X)>] + E[r g (X)] = r ( ).
1.3. Generalizing SURE [10pts]. Let X N ( ; ) where 2 Rd and 2 Rd d. If ^(X) 2 Rd is an estimator of the form X + g(X) where g : Rd ! Rd is di erentiable. De ne the functional
S(X; ^) = Tr( ) + 2 Tr( rxg(X)) + kg(X)k22:
Then show that S(X; ^) is an unbiased estimator of the risk, i.e., E[k^(x) k22] = E[S(X; ^)].
2. Exponential families.
2.1. Second moment [10pts]. For a random variable X p (x) = exp(h ; (x)i ( )), let
E[ (X)] = . For 2 Rd, nd Tr(E[( (X) )( (X) )>]) in terms of and ri ( ) for i 0.
2.2. Score function [10pts]. Assume that X p (x) where p is not necessarily in the expo-nential family form. Denote the log-likelihood by ‘ (x) = log p (x), show that
(a) E[r ‘ (X)] = 0.
(b) E[r ‘ (X)r ‘ (X)>] = E[r2‘ (X)] (Problem 1.2 may be helpful).
2.3. Maximum entropy principle [Bonus 2pts]. Assume that p(x) is a probability mass function of a discrete random variable taking values from a nite set X . Entropy of p is de ned as H(p) = Px2X p(x) log p(x). For : X ! Rd, show that the maximum entropy distribution satisfying
Ep[ (X)] = 2 Rd is a member of exponential family. That is, show that the solution to
maximizeH(p) subject to: Ep[ (X)] = ;
p
is an exponential family. (Hint: Write the Lagrangian associated with the above optimization problem. Since X is nite, think of p(x) as a vector and maximize over it.)
1
