Dense Continuous-Time Tracking and Mapping with Rolling Shutter RGB-D Cameras

Christian Kerl, Jörg Stückler and Daniel Cremers

We propose a dense continuous-time tracking and mapping method for RGB-D cameras. We parametrize the camera trajectory using continuous B-splines and optimize the trajectory through dense, direct image alignment. Our method also directly models rolling shutter in both RGB and depth images within the optimization, which improves tracking and reconstruction quality for low-cost CMOS sensors.

Using a continuous trajectory representation has a number of advantages over a discrete-time representation (e.g. camera poses at the frame interval). With splines, less variables need to be optimized than with a discrete representation, since the trajectory can be represented with fewer control points than frames. Splines also naturally include smoothness constraints on derivatives of the trajectory estimate. Finally, the continuous trajectory representation allows to compensate for rolling shutter effects, since a pose estimate is available at any exposure time of an image. Our approach demonstrates superior quality in tracking and reconstruction compared to approaches with discrete-time or global shutter assumptions.

Citation

@inproceedings{kerl15iccv,
  author={Christian Kerl and J\"{o}rg St\"{u}ckler and Daniel Cremers},
  title={Dense Continuous-Time Tracking and Mapping with Rolling Shutter {RGB-D} Cameras},
  booktitle={IEEE International Conference on Computer Vision (ICCV)},
  year={2015}
}

Datasets

Synthetic datasets based on the ICL-NUIM Dataset. For each trajectory we provide a dataset with global shutter and rolling shutter camera model in the TUM RGB-D benchmark format. The readout time for the simulated rolling shutter camera is 28ms. Each dataset contains a calibration.yaml file containing the camera intrinsics. This file can be read with the OpenCV FileStorage class.

lr kt0
lr kt0
# Images1507
Size2.1 GB
Frame Rate30 Hz
Time51 s
Download
lr kt1
lr kt1
# Images964
Size1.3 GB
Frame Rate30 Hz
Time33 s
Download
lr kt2
lr kt2
# Images880
Size1.2 GB
Frame Rate30 Hz
Time30 s
Download
lr kt3
lr kt3
# Images1240
Size1.8 GB
Frame Rate30 Hz
Time42 s
Download
robot t1
robot t1
# Images587
Size379 MB
Frame Rate30 Hz
Time20 s
Download
robot t2
robot t2
# Images664
Size430 MB
Frame Rate30 Hz
Time23 s
Download
robot r1
robot r1
# Images595
Size396 MB
Frame Rate30 Hz
Time20 s
Download
robot r2
robot r2
# Images614
Size407 MB
Frame Rate30 Hz
Time21 s
Download
table1
table1
# Images857
Size577 MB
Frame Rate30 Hz
Time29 s
Download
table2
table2
# Images1078
Size730 MB
Frame Rate30 Hz
Time36 s
Download