Advanced Computer Vision Assignment 2

This is a programming assignment. You are asked to experiment with Selective Search on the given images with groundtruth bounding boxes. We suggest you use OpenCV with version 4.1.0 or later, since some functions may not be implemented in earlier versions.

    • Image Data

The image data is in CSCI 677 HW 2 Code:zip folder in the DEN class page. Three color images are in \JPEGImages" folder; their corresponding ground truth bounding boxes are in the \Annotations" folder. An example of the bounding box coordinates (saved in .xml le) is in the following:


<object>

<name>aeroplane</name>

<pose>Unspecified</pose>

<truncated>0</truncated>

<difficult>0</difficult>

<bndbox>

<xmin>68</xmin>

<ymin>76</ymin>

<xmax>426</xmax>

<ymax>209</ymax>

</bndbox>

</object>

where the object label is \aeroplane", and the bounding box is described in the format of (xmin, ymin, xmax, ymax) = (68, 76, 426, 209), Namely, the left-top and right-bottom points of the box. You can visualize it by using cv2.rectangle function. We provide the function parse annotation to load the groundtruth annotations. This function returns a list of bounding boxes for the groundtruth.


    • Code

We provide a framework of code in the compressed le along with the images. Please nish the TODO parts of this code. You need to implement the following functions.

2.1 Use OpenCV to read an image.

2.2 Complete the selective search function. You may follow the instructions in the comment of the code.

2.3 Finish the function of calculating IoU between two bounding boxes.

2.4 For each image, calculate the \recall" of groundtruth (GT) bounding boxes by computing the fraction of GT boxes that are overlapped with at least one proposal box with Intersection over Union (IoU)

0:5.

2.5 Finish the function of plotting the bounding box in an image.

2.6 (optional) Finish the function of visualizing or saving the image with bounding boxes.

For selective search in OpenCV, you can nd some related documentation in: [Selective Search Documents]. You may restrict experiments to use only the RGB space. To segment with strategy using texture similarity, for example, do the following:

    • Create SS strategy object by calling createSelectiveSearchSegmentationStrategyTexture().

    • Create SS object by calling createSelectiveSearchSegmentation() and use addStrategy() in SS object to add strategies. You may nd some references in this document.

To feed an image into SS object, you can do the following:


    • ss  =  cv2 . ximgproc . s e g m e n t a t i o n . c r e a t e S e l e c t i v e S e a r c h S e g m e n t a t i o n ()

2ss . s e t B a s e I m a g e ( img )


To get the output bounding boxes, you may call


    • bboxes  =  ss . process ()


    • Report

For the report, please    nish the following tasks.

3.1 Apply the selective search method to the provided data. Please experiment with two strategies: color only and all four similarities (color, texture, size, and ll).

3.2 For each strategy, show the bounding boxes for proposals compared with groundtruth bounding boxes.

3.3 Display the proposal boxes whose IoU 0:5 with the groundtruth bounding boxes, along with the corresponding GT, on each image. If there are multiple proposals with IoU 0:5 with the same groundtruth bounding box, plot the one with the biggest IoU.

    • What to Submit?

You should submit a compressed    le, e.g., .zip, including the following:

4.1 Your code with comments (if any).

4.2 An analysis of your test results. Base your conclusions on your test results rather than on the obser-vations made by the instructor or the authors of the books and papers.