Assignment-1-Solution

    • Linear Regression

In the lectures, we have seen the loss function, L2-Norm Squared (L22), can be minimized using both the Normal Equations solution or using the gradient descent algorithm. In this assignment, we will discuss an important technique, known as data-normalization.

Data Normalization:

Assume that you have a data-set Dfxi; yig; i = 1; :::; m. Lets assume that we construct a feature representation (x) as discussed in the lectures. We assume that the first term in the feature-vector (x) 2 RN is a constant 1, which helps in learning the bias term. The other N 1 components in (x) will depend on x.

The first step is to compute the mean and standard-deviation of the features. We can construct a matrix 2 Rm N , where the ith row matrix is (xi)T . Now for each of the colums of , except for the 1st column, which is a vector of 1, compute the mean and standard-deviation of the column-vector.

i    =    mean( i)
i    =    std( i)

where    i is the ith column of the matrix  . The next step is to subtract the mean and divide by the feature vector.
^i
=
i
(1m) i;
for i = 2; :::; N
(1)









^i




^i



for i = 2; :::; N
(2)


1m  i
;


where step-1 and step-2 must be followed in the sequence. Here 1m denotes a vector of dimension m with all components of the vector being 1. Furthermore, the division in step-2 is an element-wise division. Essentially we are dividing each of the components of i with the value i.

^    ^
Now, the weights should be learned using the feature matrix    or the new feature representation    . Now, once
the weights are learned, w?, on the normalized data. Suppose now a new data-point x^ is given. To predict its corresponding target value y^, we should construct (x), then we should use the same mean and that were computed on the origianl dataset D, to normalize the feature vector as:

^
=   (^x)i
i;
i = 2; :::; N

(^x)i




^
(^x)i



(^x)i


;
i = 2; :::; N



i







(3)
Now compute the target y^ = w
?| ^





(^x).




Problem 1 Download the dataset for the 1st problem in the assignment “data_regress.txt”. The first column in this file represents the x values, and the second column represents the target y values. This dataset is generated using:

x3
y  =
2 + x2 +

N(0; 2) = 10

1
2    HOW TO SUBMIT THE ASSIGNMENT


Your tasks are:s

Create an appropriate feature representation, (x) (hint: it should be a 3-degrees polynomial function, plus the bias term 1).

Try to fit the polynomial on the downloaded dataset, with the gradient descent algorithm on the data provided, without any normalization.

Try to fit the polynomial on the downloaded dataset, with the gradient descent algorithm on the data provided with normalization.

For the tasks mentioned above, you should run the gradient descent algorithm for a maximum of 1000 epochs only, and share the output with us.

You should share with us:

username is your username with which you registered for DeepEigen (username should not have spaces)

Python file named “code_1_username.py” which runs the gradient descent algorithm without the normal-ization.

A text file named “learningRate_1_username.txt” which has the learning rate with which the gradient descent algorithm worked without normalization. For most the learning rates you will try, the weights vector w will become either NaN or Inf ( 1). Just save the value, do not write any text in this file. For example, if you selected the learning rate as 100, then the file should only have 100 written in it without any space either before or after the value. (hint: after selecting a sufficiently small learning rate, the weights will converge to optimal value).

Python file named “code_2_username.py” which runs the gradient descent algorithm with the normaliza-tion.

A text file named “learningRate_2_username.txt” which has the learning rate with which the gradient descent algorithm worked with normalization. Just save the value, do not write any text in this file. For example, if you selected the learning rate as 100, then the file should only have 100 written in it without any space either before or after the value.

    • How to Submit the Assignment

Make a folder named Assgn_1_username, and put all the above mentioned files in that folder. Compress that folder and create “Assgn_1_username.zip” and submit this file.



















DeepEigen
2
c


Authors: Sanjeev Sharma

Emails: {sanjeevsharma}@swaayatt-robots.com