Neural Networks Assignment 2 Solution

An engineer is trying to design a linear neural network for a regression task. The design is based on the following cost function:


1
Xn
(yn    wTxn)
2
C1 =



(1)

2




where yn is scalar output and xn is the vector input for the nth training/testing sample.
Answer the questions below.

    a) Prove that minimizing C1 is equivalent to minimizing another cost of the following form:


1

T


T


C2
=


w

Aw
b

w
(2)












2








Find the expressions for A and b in terms of the input and output of the network. Find the update rule for w according to gradient descent optimization.


    b) Identify an appropriate change of variables to prove that the update rule obtained in part a is equivalent to the update rule for a di erent cost function:


1

T


C3
=


w~

Aw~
(3)









2





Express w~ in terms of w, A and b.


    c) Assume that A is a symmetric and positive-de nite matrix. Find the maximum learning rate for which the weight updates lead to a stable solution.
Question 2. [30 points]

In this question we will revisit the cat versus car detection problem from the previous as-signment. The le assign2_data1.mat contains the variables trainims (training images) and testims (testing images) along with the ground truth labels trainlbls and testlbls. You will implement stochastic gradient descent on mini-batches.

Here, you will measure the error term as a function of epoch number, where an epoch is a single pass through all images in the training set. Two di erent error metrics will be calculated: the mean squared error and the mean classi cation error (percentage of cor-rectly class ed images). Record the error metrics for each epoch separately for the training samples and the testing samples. Answer the questions below.


    a) Using the backpropagation algorithm, design a multi-layer neural network with a single hidden layer. Assume a hyperbolic tangent activation function for all neurons. Experiment with di erent number of neurons N in the hidden layer, initialization of weight and bias terms, and mini-batch sample sizes. Assuming a learning rate of 2 [0:1 0:5], select a particular set of parameters that work well. Using the selected parameters, run backpropa-gation until convergence. Plot the learning curves as a function of epoch number for training squared error, testing squared error, training classi cation error, and testing classsi cation error.


    b) Describe how the squared-error and classi cation error metrics evolve over epochs for the training versus the testing sets? Is squared error an adequate predictor of classi cation error?


    c) Train separate neural networks using substantially smaller and larger number of hidden-layer neurons (Nlow and Nhigh). Plot the learning curves for all error metrics, overlaying the results for Nlow, Nhigh and N prescribed in part a.


    d) Design and train a separate network with two hidden layers. Assuming a learning rate of 2 [0:1 0:5], select a particular set of parameters that work well. Plot the learning curves for all error metrics, and comparatively discuss the convergence behavior and classi cation performance of the two hidden-layer network with respect to the network in part a.


    e) Assuming a momentum coe cient of 2 [0:1 0:5], retrain the neural network described in part d. Select a particular set of parameters that work well. Plot the learning curves for all error metrics, and comparatively discuss the convergence behavior with respect to part d.
Question 3. [40 points]

Neural network architectures can produce powerful computational models for natural lan-guage processing. Here, you will consider one particular model for examining sequences of words. The task is to predict the fourth word in sequence given the preceding tri-gram, e.g., trigram: ‘Neural nets are’, fourth word: ‘awesome’. A database of articles were parsed to store sample fourgrams restricted to a vocabulary size of 250 words. The le assign2_data2.mat contains training samples for input and output (trainx, traind), for validation (valx, vald), and for testing (testx, testd). Using these samples, the following network should be trained via backpropagation:


Softmax output   awesome

Logistic hidden

units



Embedding

Neural    nets    are

The input layer has 3 neurons corresponding to the trigram entries. An embedding matrix R (250 D) is used to linearly map each single word onto a vector representation of length D. The same embedding matrix is used for each input word in the trigram, without consid-ering the sequence order. The hidden layer uses a sigmoidal activation function on each of P hidden-layer neurons. The output layer predicts a separate response zi for each of 250 vocabulary words, and the probability of each word is estimated via a soft-max operation
(oi = Peziz  ).

j e j

a) Assume the following parameters: a stochastic gradient descent algorithm, a mini-batch size of 200 samples, a learning rate of = 0:15, a momentum rate of = 0:85, a maximum of 50 epochs, and weights and biases initialized as random Gaussian variables of std 0.01. If necessary, adjust these parameters to improve network performance. The algorithm should be stopped based on the cross-entropy error on the validation data. Experiment with dif-ferent D and P values, (D,P) = (32,256), (16,128), (8,64) and discuss your results.


b) Pick some sample trigrams from the test data, and generate predictions for the fourth word via the trained neural network. Store the the predicted probability for each of the 250 words. For each of 5 sample trigrams, list the top 10 candidates for the fourth word. Are the network predictions sensible?