Anomaly Detection and Recommender Systems Solution

In this  exercise, you will implement the  anomaly  detection  algorithm  and apply it to detect  failing servers on a network.  In the second part,  you will use collaborative filtering to build a recommender system for movies. Before starting  on the programming  exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.

To get started with the  exercise, you will need to download the  starter code and unzip its contents  to the directory where you wish to complete the exercise.  If needed, use the cd command  in Octave/MATLAB to change to this directory  before starting  this exercise.

You can also find instructions  for installing Octave/MATLAB in the “En- vironment Setup Instructions” of the course website.

 

 

Files included in  this exercise

 

ex8.m - Octave/MATLAB script for first part  of exercise

ex8 cofi.m - Octave/MATLAB script for second part  of exercise ex8data1.mat - First  example Dataset  for anomaly detection ex8data2.mat - Second example Dataset  for anomaly detection ex8 movies.mat - Movie Review Dataset

ex8 movieParams.mat - Parameters provided for debugging multivariateGaussian.m - Computes  the  probability  density  function for a Gaussian  distribution

visualizeFit.m - 2D plot of a Gaussian  distribution and a dataset checkCostFunction.m - Gradient checking for collaborative  filtering computeNumericalGradient.m - Numerically compute gradients

 

fmincg.m - Function  minimization  routine  (similar to fminunc) loadMovieList.m - Loads the list of movies into a cell-array movie ids.txt - List of movies

normalizeRatings.m - Mean normalization  for collaborative filtering submit.m - Submission script that  sends your solutions to our servers [?] estimateGaussian.m - Estimate the parameters of a Gaussian  dis- tribution with a diagonal covariance matrix

[?] selectThreshold.m - Find a threshold  for anomaly detection

[?] cofiCostFunc.m - Implement the cost function for collaborative fil- tering

 

 

? indicates  files you will need to complete

 

Throughout the first part of the exercise (anomaly detection)  you will be using the  script  ex8.m.  For  the  second part  of collaborative  filtering,  you will use ex8 cofi.m. These scripts  set up the dataset for the problems and make calls to functions that  you will write.  You are only required to modify functions in other files, by following the instructions  in this assignment.

 

 

 

Submission and Grading

 

After completing various parts  of the assignment,  be sure to use the submit function system to submit  your solutions to our servers.  The following is a breakdown of how each part  of this exercise is scored.

 

Part
Submitted File
Points
Estimate Gaussian  Parameters

Select Threshold
estimateGuassian.m

selectThreshold.m
15 points

15 points
Collaborative  Filtering  Cost

Collaborative  Filtering  Gradient

Regularized Cost

Gradient with regularization
cofiCostFunc.m

cofiCostFunc.m cofiCostFunc.m cofiCostFunc.m
20 points

30 points

10 points

10 points
Total  Points
 
100 points
You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.