Q1 (100 pts) In this computer project, we will use a neural network for curve fitting.
Draw n = 300 real numbers uniformly at random on [0, 1], call them x1, . . . , xN.
Draw n real numbers uniformly at random on [− 101 , 101 ], call them ν1, . . . , νN.
Let dI = sin(20xI) + 3xI + νI, i = 1, . . . , n. Plot the points (xI, dI), i = 1, . . . , n.
We will consider a 1 × N × 1 neural network with one input, N = 24 hidden neurons, and 1 output neuron. The network will thus have 3N + 1 weights including biases. Let w denote the vector of all these 3N + 1 weights. The output neuron will use the activation function φ(v) = v; all other neurons will use the activation function φ(v) = tanh v. Given input x, we use the notation f (x, w) to represent the network output.
Use the backpropagation algorithm with online learning to find the optimal weights/network that minimize
the mean-squared error (MSE)
1
N (dI − f (xI, w))2. Use some η of your choice. Plot the number of
N
I=1
epochs vs the MSE in the backpropagation algorithm.
Hint: As discussed in class, for a given fixed η, the algorithm may not always result in a monotonically decreasing MSE (the descent may overshoot the locally optimal point). You may have to modify the gradient descent algorithm in such a way that you decrease η (e.g. via η ← 0.9η) whenever you detect that the MSE has increased. Also, beginning with a very large η may result in an immediate divergence of the weights.
Let us call the weights resulting from the backpropagation algorithm (when it converges) as w0. The curve (x, f (x, w0)), x ∈ [0, 1] will then be a fit to the points (xI, dI), i = 1, . . . , n. Plot the curve f (x, w0) as x ranges from 0 to 1 on top of the plot of points in (c). The fit should be a “good” fit.
Your report should include a pseudocode of your training algorithm including all update equations written out explicitly (similar to what I had done in the second question of your second homework). The pseu-docode should be written in such a way that anyone would be able to implement your algorithm without knowing anything about neural networks.
1
