Stereo Calibration and Rectification Solution

Objectives:

    • Learn stereo camera geometry.

    • Learn to calibrate stereo system using OpenCV functions.

    • Learn to rectify stereo images.

    • Learn to obtain 3D information of points of interest.

Instructions:

    • Use the stereo system in Room 250 B34 to complete this assignment.

    • This assignment will help prepare you for your baseball team project.  Save your code and reuse it later.

    • Learn to use the baseball project code to capture images for this assignment.

    • You should ask your team members to help you capture the images (one holds the chessboard and one clicks the button)

    • Write your own code to read in the images and calibrate the system.

    • All team members can use the same images for this assignment.

    • You can download the images and data online from BYU Learning Suite to confirm that your code works before using your own images.

    • You will lose 20 points if any of the following requirements is not met:

o Generate a PDF file that includes (with proper headings) two sets of intrinsic and distortion parameters (Task 1), extrinsic parameters and essential and fundamental matrices (Task 2), two images with epipolar lines (Task 3), two original, rectified, and absolute difference images (Task 4), two images and 3D measurements (Task 5), and your explanations.
o Submit your PDF file and source code file(s) in one zip file without the folder or directory. o Use your first name and last name (e.g., justinsmith.zip) as the file name.
    • Login to myBYU and submit your work through BYU Learning Suite online submission.

Image Acquisition:

    • Make a copy of your own baseball project code in the system.

    • The baseball project code allows you to capture one set of 32 stereo images in bitmap format at a time.

    • Set delay time between frames (the edit box in “Capture and Replay” group).  200 ~ 300 mSec is a good selection.

    • Click “Grab” to show live videos on the image display screens.

    • Have one of your team members hold the chessboard in front of the camera(s) and move the chessboard around.

    • Make sure the chessboard stays in the camera view. You may have to experiment a little to determine the camera view boundary. o Make sure you step back and forth until the images are in focus, especially around the edges.

o Make sure the lighting isn't too harsh above the calibration board, if the squares get washed out the calibration data is very poor. Tilting the board slightly forward helps remove glare a little.

o Make sure most of the frame is visible. Move bookshelves and other obstructions if needed to make sure the calibration grid is visible in as much of the frame as possible.

    • Click “Capture Images” button.  The frame number will be shown in green in the upper left corner of the left screen.

    • Change the delay time back to 30 mSec and click “Replay Images” to examine the images. Repeat the entire procedure until you are satisfied with the image quality.

    • Click “Save Images” to save the images. Type in the file name in the “File Name” box. You need to type in both the folder name and the file name (e.g., C:\Projects\TennisBall team#\Images\Img).

    • The program will automatically add a letter “L” or “R” and a sequential number to each file. You will get 32 images for the left camera (ImgL0.bmp ... ImgL31.bmp) and 32 images for the right camera ((ImgR0.bmp ... ImgR31.bmp) in the folder you select.

    • Copy these images to your thumb drive and remove them from the computer hard rive (empty the recycle bin).

    • You should calibrate cameras separately in order to get good camera intrinsic and distortion parameters (Task 1).

    • You can then use the two sets of intrinsic and distortion parameters (one set for each camera) to calibrate the stereo system.

    • For this assignment, you need 32 good images for each camera for camera calibration and one set of stereo images (32 pairs of left and right) for stereo calibration.

    • When capturing 32 good images for single camera calibration, you want to move the chessboard around so that the entire camera view is covered. Don’t worry about what is seen in the other camera (you won’t need them and will delete them anyway).

    • When capturing the stereo pairs, you need to make sure that the entire chessboard is seen by both cameras.

The chessboard made for this assignment and the baseball project has 3.88-inch black and white squares. The x and y coordinates of the chessboard 3-D points entered to the calibration function should be multiplied by 3.88. The z coordinates stay the same (0).

Task 1:    Camera Calibration    20 points

    • Capture 32 good chessboard images from each camera separately at approximately 20 feet away.

    • Find chessboard corners (subpixel) and do cameraCalibrate() for each camera.

    • This task is the same as Task 2 in Assignment 2.  You can reuse the code and don’t have to submit the code again.

    • Include one set of the intrinsic (3×3) and distortion (5×1) parameters for each camera in your PDF File.

Task 2:    Stereo Calibration    20 points

    • Capture one set of stereo images (32 pairs).

    • Use the two sets of intrinsic and distortion parameters calculated in Task 1 and the OpenCV stereoCalibrate() function to obtain a unique set of extrinsic parameters between the two cameras, an essential matrix and a fundamental matrix.

    • Include the extrinsic parameters (3×3), essential matrix (3×3), and fundamental matrix (3×3) in your PDF file.

    • Submit your code for this task.

Task 3:    Epipolar Lines    20 points

    • Select one pair of your stereo image pairs for this task.

    • Use undistort () to undistort lens distortion for both images. Manually selected 3 points of interest in the left and right images and draw a circle around them.

    • Use the fundamental matrix from Task 2 and computeCorrespondEpilines() to find and draw 3 epipolar lines of the selected 3 points for each image. The epipolar lines found for the points in the left image should be drawn in the right image and vice versa. Confirm that the corresponding points lie on their epipolar lines in the other image.

    • Include both images with the superimposed colored epipolar lines in your PDF file.

    • Submit your code for this task.

Task 4:    Rectification    20 points

    • Select one pair of your stereo image for this task.

    • Use stereoRectify( ), initUndistortRectifyMap( ), and remap( ) functions to rectify both left and right images.

    • Confirm that the image rows are aligned by drawing a few horizontal lines in both rectified images.

    • Include the two original images and the two rectified images in your PDF file.

    • Include the two absolute difference images (between the rectified and original images) in your PDF File.

    • Submit your code for this task.

Task 5:    3D Measurement    20 points

    • Select one pair of your stereo image for this task.

    • Manually select at least 4 3D points of interest from the image pair.

    • Use undistortPoints() to undistort AND rectify the selected points.

    • Use perspectiveTransform() to calculate the 3D information of the selected points.

    • This is a good exercise for you to check if your code works correctly. It is suggested to select points with known or measurable 3D information.

    • Include the stereo image pair with the selected points circled and their 3D (X, Y, and Z) information measured from the left and right cameras in your PDF file and explain your result and observation.

    • Submit your code for this task.

Congratulations! You are ready to start your baseball project. It is suggested that you integrate the code you have developed for this assignment into your team’s baseball project so that your team can calibrate the stereo system easily and whenever it is needed. It is important that you are able to write your calibration parameters into a file after you calibrate the system and read them back into your program every time you start your program.