Assignment 2 Solution

Hybrid Images




Here, we will create hybrid images using the approach described in the SIGGRAPH 2006 paper by Oliva, Torralba, and Schyns [3]. Hybrid images are static images that change in interpretation as a function of the viewing distance. The basic idea is that high frequency tends to dominate perception when it is available, but, at a distance, only the low frequency (smooth) part of the signal can be seen. By blending the high frequency portion of one image with the low-frequency portion of another, you get a hybrid image that leads to di erent interpretations at di erent distances. Some examples are shown in this gallery: http://cvcl.mit.edu/hybrid_gallery/gallery.html. You can also see a fun example on Stephen Fry’s show: https://www.youtube.com/watch?v=OlumoQ05gS8.




We have included two sample images of your two favorite professors, malik.png, papadimitriou.png, as well as some starter code in Matlab and Python to load and align images. The alignment is important, as it a ects the perceptual grouping (read the paper [3] for details). Chose a few pairs of images that you want to make into hybrid images as well. Try creating a variety of types of hybrid images (change of expression, morph between di erent objects, change over time, etc.).







 
Write a function img hybrid = hybrid image(img1, img2). The function should:




 
Low-pass lter img1. Oliva, et al [3] suggest using a standard 2D Gaussian lter, but feel free to try your own favorite lter. The cuto can be chosen with some experimentation.




 
High-pass lter img2. Oliva, et al [3] suggest using the impulse lter minus the Gaussian lter, which can be computed by subtracting the Gaussian- ltered image from the original image. The cuto can be chosen with some experimentation and does not need to be the same as for the low-pass lter.




 
Linearly combine the two images.




 
In your report, include the following:




 
Brie y describe how this algorithm works. Write out the parametric form of the low-pass and high-pass lters you used, any parameters you chose, and include plots of the lters spatially.




 
Show your favorite result and illustrate the process through frequency analysis. Show the log magnitude of the Fourier transform of the two input images, the ltered images, and the hybrid image. You can com-pute and display the 2D Fourier transform with imagesc(log(abs(fftshift(fft2(gray image))))) in MATLAB and plt.imshow(np.log(np.abs(np.fft.fftshift(np.fft.fft2(gray image))))) in Python.




 
Show at least two more results, including one that does not work so well. Display your resulting hybrid images at various scales in order to simulate the viewer being closer or further away from the hybrid image. Brie y explain how you got the good results (e.g., chosen cut-o frequencies, alignment tricks, other techniques), as well as any di culties and the possible reasons for the bad results. If you are so fortunate that everything that you try works well, try to gure out what shouldn’t work.







 
Edge Detection




In this problem, build an elementary edge detector on some sample images from the Berkeley Segmentation Dataset [2]. We have provided a few images from the BSDS. You can nd the human annotations by access-ing the site /www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/BSDS300/html/dataset/images/ color/###.html, where ### is the image number. We will qualitatively compare the results to a 2014 state-of-the-art algorithm [1] as well. Also try using a few images from your personal collection.










1
1. Finite operator We will begin by using the humble nite di erence as our lter in the x and y directions.

Dx =
1
1
; Dy =
11
;
(1)














Write function [mag, theta] = difference filter(img). Convolve the nite di erence operator with the images, Dx I and Dy I. Show the oriented gradient responses. Now compute the gradient magnitude at each location by taking the L2-norm over the R,G,&B channels in both the x and y directions and plot. Compute the gradient orientation at each pixel from the channel corresponding to the largest gradient magnitude. Explain how you computed the gradient orientation from the x and y lter responses. Visualize the gradient orientation at each pixel. Notice that the resulting gradients are very noisy. Remove some edges by trying a few thresholds and visualize the remaining gradient magnitudes and orientations.







 
Derivative of Gaussian We noted that the results with just the di erence operator were rather noisy. Luckily, we have a smoothing operator handy: the Gaussian lter G . Write function [mag, theta] = derivative gaussian filter(img, sigma). Convolve the nite di erence operators with the 2D Gaussian lter, G Dx and G Dy, and plot the lters.




Then, as in part 1, convolve the image with the oriented lters, nd the magnitude and orientation, and try a few thresholds. Make the same visualizations as you did in part 1. Compare the new results with the results from part 1, and note any observations.




Note that smoothing the gradient map is the same as computing the gradient on the smoothed image. In other words, G (Dx I) = Dx (G I). Why is this the case, and why may it be important?




 
Oriented Filters Now try to improve your results using a set of oriented lters, rather than the simple derivative of Gaussian approach above. Implement function [mag,theta] = oriented filter(img), which computes the boundary magnitude and orientation using a set of oriented lters of your choice, such as elongated Gaussian derivative lters. Use at least four orientations and explain your choice of lters. Find a way to combine the individual lter responses into a magnitude and orientation map. Visualize the results as before.




 
Qualitatively compare the resulting edge maps you obtain to the 2014 state-of-the-art result from Isola, et al. [1], along with human annotations. What do you notice about the human annotations? What does your best algorithm do well? What does it struggle with?




 
Image Straightening




We will now use edge orientation to rectify a slightly rotated image in order to vertically align it. The basic idea is that in an aligned image the orientation of most of the edges should be either vertical or horizontal. Given an input image, nd a rotation transformation which results in an image whose edge orientation histogram peaks at vertical and horizontal edges. More concretely, implement the following steps




 
Compute an orientation map, as previously. Plot a histogram of the angles and visualize it.




 
Use this histogram to identify a rotation that shifts the peaks of the histogram to horizontal and vertical edges.




 
Apply the rotation and verify the histograms.







Find two natural images, one where this procedure succeeds and another where it fails to align the image.




For both images, provide:




 
Comparison of the histograms of the input image and the aligned image.




 
Comparison of the input and the aligned images.




 
Instructions




 
This assignment is to be done individually.




 
Please submit the assignment using bCourses. Upload the following les:










2
 
A PDF file. The top of the rst page should contain your name and student ID. The le should contain answers to all questions, supporting images, and code for all functions speci ed in Problem 1 and 2.




 
A tar/zip file. The le should contain a pdf of your report, along with a subfolder code, with subdirectories for Problem 1 and 2.




 
The HW is due on: Friday, Mar 8, 2019, 11:55pm. Per course policy, you have 5 emergency slip days to cumulatively use throughout the semester.




 
Acknowledgements




Parts of this assignment are borrowed from Derek Hoiem’s Computational Photography class (https://courses. engr.illinois.edu/cs498dh/fa2011/). The Mini Places Challenge was \heavily inspired" by the MIT assignment (http://6.869.csail.mit.edu/fa15/project.html).







References




 
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Crisp boundary detection using pointwise mutual information. In European Conference on Computer Vision, pages 799{814. Springer, 2014.




 
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, volume 2, pages 416{423. IEEE, 2001.




 
Aude Oliva, Antonio Torralba, and Philippe G Schyns. Hybrid images. In ACM Transactions on Graphics (TOG), volume 25, pages 527{532. ACM, 2006.










































































































3