Project 3: Visual-Inertial SLAM Solution

Collaboration in the sense of discussion is allowed, however, the work you turn in should be your own - you should not split parts of the assignments with other students and you should certainly not copy other students’ solutions or code. See the collaboration and academic integrity statement here: https://natanaso.github. io/ece276a. Books may be consulted but not copied from.

Submission

You should submit the following    les on Gradescope by the deadline shown at the top right corner.

    1. Programming assignment: upload all code you have written for the project (do not include the training and test datasets) and a README le with clear description of each le.

    2. Report: upload your report. You are encouraged but not required to use an IEEE conference template1 for your report.

Problems

In square brackets are the points assigned to each problem.

    1. Implement visual-inertial simultaneous localization and mapping (SLAM) using the Extended Kalman Filter. You are provided with synchronized measurements from an IMU and a stereo camera as well as the intrinsic camera calibration and the extrinsic calibration between the two sensors, specifying the transformation from the IMU to the left camera frame. The data includes:

IMU Measurements: linear velocity vt 2 R3 and angular velocity !t 2 R3 measured in the body frame of the IMU

Stereo Camera Images: pixel coordinates zt 2 R4  M of detected visual features with precomputed correspondences between the left and the right camera frames (see Fig. 1). The landmarks i that
were not observable at time t have a measurement of zt;i =   1
1
1
1 T . There are many
features in each dataset and you can think of ways to keep the computational
complexity manageable.


Videos showing the feature tracking over time (just for illustration
purposes) are provided here:

https://drive.google.com/open?id=1woBO_Gd3A3N3hNH_sYB8rI9CGDE5ag9G

Time Stamps: time stamps  in UNIX standard seconds-since-the-epoch January 1, 1970

Intrinsic Calibration: stereo baseline b and camera calibration matrix:
    • 3
K = 4
f su
0

cu

0
f s

c

0
0
v
1v 5
Extrinsic Calibration: the transformation C TI 2 SE(3) from the IMU to left camera frame

The goal of the project is to use the EKF equations with a prediction step based on SE(3) kinematics and an update step based on the stereo camera observation model to perform localization and mapping. In detail, you should complete the following tasks:

    (a) [15 pts] IMU-based Localization via EKF Prediction: Implement the EKF prediction step based on the SE(3) kinematics to estimate the pose Tt 2 SE(3) of the IMU over time t
    (b) [15 pts] Landmark Mapping via EKF Update: assume that the predicted IMU trajectory from part (a) above is correct and focus on estimating the landmark positions. In detail, you should implement an EKF with the unknown landmark positions m 2 R3 M as a state and perform EKF update step after every visual observation zt in order to keep track of the mean and covariance of m. Note that we are assuming that the landmarks are static so it is not necessary to implement a


    • https://www.ieee.org/conferences_events/conferences/publishing/templates.html


1
ECE 276A: Sensing & Estimation in Robotics    Due: 11:59 pm, 03/14/20

























Figure 1: Visual features matched across the left-right camera frames (left) and across time (right).


prediction step. Moreover, since the sensor does not move su ciently along the z-axis, the estimation for the z coordinate of the landmarks will not be very good. You can assume that the z coordinates for all landmarks are 0 and focus only on estimating their xy coordinates.

        (c) [15 pts] Visual-Inertial SLAM: combine the IMU prediction step from part (a) with the land-mark update step from part (b) and an IMU update step based on the stereo camera observation model to obtain a complete visual-inertial SLAM algorithm. Compare the accuracy of the SLAM implementation versus the individual parts obtained in part (a) and part (b).

    2. Write a project report describing your approach to the visual-inertial SLAM problem. Your report should include the following sections:

[5 pts] Introduction: discuss why the problem is important and present a brief overview of your approach

[10 pts] Problem Formulation: state the problem you are trying to solve in mathematical terms. This section should be short and clear and should rigorously de ne the quantities you are interested in.

[20 pts] Technical Approach: describe your approach to visual-inertial localization and mapping.

[20 pts] Results: present your results, and discuss them { what worked, what did not, and why. Make sure your results include plots clearly showing the estimated robot trajectory as well as the estimated 2-D positions of the visual features. If you have videos do include them in the zip le and refer to them in your report!

















2