• Written Problems (6 points)
1. MLE minimizes KL divergence to the empirical distribution (Exercise 2.15 of Murphy’s book) (1 point)
2. Centering and ridge regression (Exercise 7.3 of Murphy’s book) (1 point)
3. Symmetric version of ‘2 regularized multinomial logistic regression (Ex-ercise 8.5 of Murphy’s book) (1 point)
4. Elementary properties of ‘2 regularized logistic regression (Exercise 8.6 of Murphy’s book) (1 point)
5. Given the following denominator layout derivatives, (2 points)
▪ Di erentiation of a scalar function w.r.t. a vector: If f(w) is a scalar function of d variables, w is a d 1 vector, then di erentiation of f(w) w.r.t. w results in a d 1 vector
2 @f 3
@w1
df(w) .
dw = 6 .. 7
4 5
@f
@wd
• Di erentiation of a vector function w.r.t. a vector: If f(w) is a vector function of size h 1 and w is a d 1 vector, then di erentiation of f(w) w.r.t. w results in a d h vector
2
@f1
:.:..:
@fh
3
df(w)
=
@x...1
@w...1
dw
6
@wd
: : :
@wd
7
4
@f1
@fh
5
1
Please prove the following derivatives, and X and y are not functions of w:
d(X>w)
dw
= X;
d(y>Xw) = X>y
dw
d(w>Xw) = (X + X>)w
dw
• Programming (5 points)
2
4
3
3
2
0
3
6
1
2
7
6
1
7
Given Xw = y where X =
6
5
6
7
and y =
6
1
7
which constitute an ex-
3
8
0
6
9
10
7
6
0
7
6
7
6
7
emplary problem. Write a
Python routine to nd the least squares solution
4
5
4
5
for w given arbitrary X 2 R5 2
and y 2 R5 1.
Submit your Python codes
as a function routine (\def A1 MatricNumber(X,y)") that takes in X and y as inputs and generate (X>X) 1 and w as outputs in a single le with lename \A1 StudentMatriculationNumber.py". Your Python routine should return the least squares solution vector w (as numpy array) and two matrices X> and (X>X) 1. Hint: you will need \import numpy as np" and its matrix manipu-lation functions.
Please use the python template provided to you. Remember to rename both
\A1 StudentMatriculationNumber.py" and \A1 MatricNumber" using your stu-
dent matriculation number. For example, if your matriculation ID is 123456789,
then you should submit \A1 123456789.py" that contains the function \A1 123456789".
The way we would run your code might be something like this:
import A1 123456789 as grading
w, XT, InvXTX = grading.A1 123456789(X,y)
Marks allocation is based on the three outputs: w (3 points), XT (1 point), InvXTX (1 point)
NOTE: Please do NOT zip/compress your le. Please make sure you replace \StudentMatriculationNumber" and \MatricNumber" with your matriculation number! Because of the large class size, points will be deducted if instruc-tions are not followed.
2
