Homework #4 Solution

Figure  1: The  results  of the  boston  housing data  set.  Left:  The  predicted  house-price  (black)  and  the  true price (red) of the training data.  Middle:  The predicted  prices of the test data.  Right:  The root-mean-squared error (RMSE)  of train  (blue)  and test  (green)  as a function  of gradient steps.

 

 

In this assignment you will implement a Neural Network.  First,  perform an ”svn update” in your svntop directory. Then  Download  the boston  housing data  set

 

http://www.cse.wustl.edu/~kilian/cse517a2015/hw4/boston.mat

 

This data  set contains  housing prices as targets and community statistics as features.

As  always, make sure to add a test for  every function you  implement.

 

1.  Implement the  function  preprocess.m.   This  should  make  the  training data  set  zero-mean  and  each feature  should have standard deviation  1. Make sure to apply exactly  the same transformations to the test  data  set.  (HINT:  Each  input  vector  should be transformed by ~xi  → U ~xi  − m, where u can be a diagonal  d × d matrix.)

 

2.  (optional ) Ideally you would like the input  features  to be de-correlated. The correlation matrix  should be diagonal  (in this case even the identity matrix). One way to do this is to project  the data  onto the PCA  principal  components  (which we will still cover later  in the course).  You can get the transposed projection matrix  by calling pcacov(xT r0 ).  Make sure to apply PCA after  you subtracted off the mean.

 

 

 

 

 

Figure  2: Results  on a xor data  set (with  noise).  Left:  The  prediction  of the  training data  set.  Right:  The predictions of the surface within  the unit  square.

 

 

3.  Implement the  function  ffnn.m  which  computes  the  loss and  gradient of a  particular feed-forward neural  net  for a data  set  xT r, yT r.  All weights  are  passed  in as a single vector  w.  The  neural  net structure is passed  in as an array,  where e.g.  wst  = [1, 5, 12, 17] would mean  a neural  network  with one output, and  two hidden  layers with 5 and  12 nodes and  an input  dimensionality of 17. With  any data  set x and y should be able to train  the neural net with your grdescent function  from the previous homework.

 

f=@(w) ffnn(w,x,y,wst);

w=grdescent(f,w,0.1,100,1e-8);

 

Alternatively, you can download

 

http://learning.eng.cam.ac.uk/carl/code/minimize/minimize.m

 

which  is a  slightly  more  sophisticated implementation  of gradient descent (some  people  claim  it  is among the top-ten best pieces of code ever written) and call:

 

[w,l]=minimize(w,’ffnn’,100,x,y,wst);

 

Use the  script  testhw4.m to test  if your  implementation is correct.   Once you pass this  test,  you can evaluate  your neural net and call the functions  xordemo  and bostondemo.  How are the results  affected when you change the network  structure (or you switch to minimize.m )?