CS 639 Homework 3: Image Mosaicking

Please follow the homework guidelines in your submission.



will be your main interface for running your code. Parameters for the different programsrunHw3. or unit tests can also be set in that file. Before submission, make sure that you can run all your programs with the command runHw3(“all”) with no errors.

Vectorization





MATLAB is optimized for operations involving matrices and vectors. Avoid using loops (e.g., ) in MATLAB whenever possible – looping can result in long-running code. Instead,

youfr, shouldwile vectorize loops to optimize your code for performance. In many cases, vectorization also results in more compact code (fewer lines to write!). If you are new to MATLAB, refer to th is article for some ideas on how to optimize MATLAB code.

ImageMosaickingApp (13 points)



Your task is to develop an “Image Mosaicking App” that stitches a collection of photos into a mosaic. Creating image mosaics and panoramas has become one of the most popular features on smartphones. Outside the realm of smartphones, similar applications have also been developed to create stunning panoramas seen on GigaPan. With the concepts and algorithms presented in the Image Alignment lecture, you too can create an image mosaic.

Before we create the mosaicking app, we will go over the individual tools required to build it.

After that, you will create these tools as separate programs which you write and submit.



We will take one photo (any image is okay) from the collection as the reference image for the mosaicking process. Given this reference image, the steps to construct the mosaic are:

registration: estimate the geometric transformation relating each image in the collection
1.    to the reference image,    “    ”

warping : for each image in the collection, we  undo  the transform estimated from the

2.    step above, so that each image is in the same “frame of reference”. Still, the different

photos will cover different regions of the field of view, for which we perform...

3.    blending : combine the warped images into a single panoramic view.



The registration method is prone to errors, so we usually apply the RANSAC (RANdom Sampling And Consensus) technique to make it more reliable. In this homework, the basic registration is already implemented for you, but you will need to implement RANSAC.

Each step will be built as its own program, and we will test them individually on some simple inputs so that we are sure that they work correctly. Once we do this, we will put them together to implement the entire stitching pipeline. This homework has five parts: Challenge 1a through 1e.
Registration (already implemented)




NOTE: We strongly recommend checking out the section in the script. It is a test of the registration implementation, and will bedebug1important to understandruHw3when.m you get to

RANSAC.



The registration program calculates the homography between a pair of images. In our case, the homography is estimated as a transformation relating individual keypoints across images. Assume that you have 2 sets of ncorresponding pixel locations – the first set are the n pixel in the reference image, and the second are the corresponding pixels in the other image (the one we want to register to the reference). We use the equation in the Image Alignment lecture to estimate the homography between these sets of points, which you can find in the computeHomography.m file.

incomputeHomographytwoimages: calculates the homography between the two sets of corresponding pixels It uses Hthe= MATLABcomputeHomography(srcfunctionto_pts,computedestthepts)homography matrix.

eig

A second program named applyHomography applies a homography to a set of points:



dest_pts = applyHomography(H, test_pts)

The function helps to view (and debug) the mapping between correspondingshowCorrespointsndenceinthe two images:

Here,  result_andimg = showCorrespondence(srcaretwoimagesrelatedimg,bydestahomography,img,srcpts,and dest_pts)  is an


src_img    dest_img    result_img
image showing    and    side-by-side, with lines connecting    and

. The followingsrcimg imagedshowstimg an example output of    src_pts.


dest_pts    showCorrespondence

Challenge 1a: Warping (4 points)




Here, we will develop a program to warp an image, and use it to warp and paste a portrait of

Vincent (    ) to the empty billboard in the image shown below (    ).

To help youportraitgetstarted,smalla.skeletonpng code has been provided in challenge1a.    osaka.png



















Before
After
(source)






First, you need to estimate the homography from the portrait to the target billboard

•    frame. Fill in the missing. parts in challenge1a where appropriate using (1 point) computeHomography

Second, we need to replace the “Warp Image Here” part of the target image with the

•    portrait image. This will be achieved using backward warping : we will create a new image, such that only the area corresponding to the “Warp Image Here” region in the billboard image, gets exactly filled with the portrait, while the rest is left “blank” (pixel value = zero).

Write a program named backwardWarpImg that warps an image based on a
homography:







[mask, dest_img] = backwardWarpImg(src_img, dest_to_src_homography,

The input argument    is the sourcedestimagecanvas(the_portraitwidthheiginthist)case). Because you


will implement backwardsrcimgwarping,    is the inverse of the


homography that maps source to destinationdestto.src_homography    specifies


width and height of the destination image.    dest_canvas_width_height


Now the output arguments.    is the output image. In this case,    is a

warped portrait on a black canvasdest.img    is a binary mask indicating thedestarea_ofimgthe


warped image on the canvas.    ismaskneeded because the warped image is often not the

final result, but used as an inputmaskto other post-processing tasks (in this case, you will

post the warped image to another image).    (3 points)



You are allowed (and expected) to use the MATLAB built-in function interp2.



After generating a warped portrait image and its associated mask, superimpose the image on the given billboard image. The code to perform this task is already included in challenge1a.

Functions not allowed: ALL im*, cp*, tf* functions.





Challenge 1b: RANSAC (3 points)




When stitching together real images, we will not select keypoints manually as you did in Challenge 1a. We will use automatic keypoint extraction methods such as SIFT. Here too, a

function    has been provided to compute a set of points (    ) in the source

image andgenSIFTMatchasetofcorresponding points ( ) in the destination imagesrc._Typically,pt these matches will include many outliers, that is,destthe_ptmatches are not correct (they correspond to different scene points). We have discussed the outlier problem in the Image Alignment lecture.

RANSAC (RANdom Sampling And Consensus) is a popular method to robustly estimate

quantities in the presence of outliers. Write a function named to robustly compute homography using RANSAC to address the outlier problem. runRANSAC
[inliers_id, src_to_dest_H] = runRANSAC(src_pt, dest_pt, ransac_n,

specifies the maximum number of RANSACransaciterations,eps) and specifies the acceptableransacn alignment error in pixels. Use Euclidean distance to measureansactheerroreps. Your

program should return a homography    that relates the    to the    .

In addition, return a vector    srcthat tolistsdestthe_indicesH of inliers, srci.e.,_thept indicesdestofthept


rows in    and    inliers.Thefollowingid two images show the matches between two


images, srcbeforept and afterdest runningpt RANSAC. Functions not allowed: ALL im*, cp*, tf* functions

















Before RANSAC














After RANSAC





Hint: We recommend you to initially try the values    = 100 and    = 3,

before experimenting more with them. ransac_n ransac_eps Implementation tip: The RANSAC loop can be implemented in nine lines of MATLAB, including the ' ' and ' ' lines. You may find the following snippets helpful, though it is not necessary toforuse themendto receive full credit for this homework.
    • A = H * B; % A and B are 3 x n matrices and H is a 3 x 3 matrix

    • dist = sqrt(sum((A-B).^2)); % A and B are nx3 matrices and dist is a 1xn

            ▪ matrix

    • inds = randperm(n, 4); % inds is a vector of 4 random unique integers

        ◦ in [1, n]
Challenge 1c: Blending (3 points)



Write a program named blendImagePair that blends two images into one:


img1 and img2 are two input images. mask1 and mask2 are the corresponding binary masks

result = blendImagePair(img1, mask1, img2, mask2, blendi g mode)
indicating the regions of interest. blending_mode, a string, is either “overlay” or “blend”.
•
In the case of “overlay”, copy img2 over img1 wherever the mask2 applies. This case is

already implemented for you.
•   In the case of “blend”, perform weighted blending as discussed in class. You may use

the MATLAB function bwdist to compute a new weighted mask for blending – use the

‘
verlay”
’ methodblend”ifyou do so. The figure below shows example outputs (left:

“

, right:
“
). Also see the figure on the next page, which gives a hint on

euclidean


to compute the weighted mask. After computing the distance

how to use




transform, you also need to make sure that the mask weights sum to one in the valid




bwdist


region of the output image.



Functions not allowed: ALL im*, cp*, tf* functions





















overlay    blend

Challenge 1d: Stitching (2 points)



Now you have all the tools to build the mosaicking app! Write a program    that

stitches the input images into one mosaic.    stitchImg





Your programstitchedshouldimg=acceptstitchImg(img1,anarbitrary numberimg2,...of,imagesimN) (you can use the MATLAB


function to take variable number of inputs). You can assume the order of the input images matchesvararginthe order you wish to stitch the images. Also, in this assignment we will only stitch images of a single row/column. Use “blend” mode to blend the image when you call
.
FunctionsblendImagePairnotallowed: ALL im*, cp*, tf* functions
Challenge 1e: Your own photos (1 point)




Capture images using your own (cell phone) camera and stitch them to create a mosaic. Note that the mosaic need not be a horizontal panorama. Submit both the captured and stitched images.