The stereo 2015 / flow 2015 / scene flow 2015 benchmark consists of 200 training scenes and 200 test scenes (4 color images per scene, saved in loss less png format). Compared to the stereo 2012 and flow 2012 benchmarks, it comprises dynamic scenes for which the ground truth has been established in a semi-automatic process. Our evaluation server computes the percentage of bad pixels averaged over all ground truth pixels of all 200 test images. For this benchmark, we consider a pixel to be correctly estimated if the disparity or flow end-point error is <3px or <5% (for scene flow this criterion needs to be fulfilled for both disparity maps and the flow map). We require that all methods use the same parameter set for all test pairs. Our development kit provides details about the data format as well as MATLAB / C++ utility functions for reading and writing disparity maps and flow fields. More details can be found in Object Scene Flow for Autonomous Vehicles (CVPR 2015).
Our evaluation table ranks all methods according to the number of erroneous pixels. All methods providing less than 100 % density have been interpolated using simple background interpolation as explained in the corresponding header file in the development kit. Legend:
D1: Percentage of stereo disparity outliers in first frame
D2: Percentage of stereo disparity outliers in second frame
Fl: Percentage of optical flow outliers
SF: Percentage of scene flow outliers (=outliers in either D0, D1 or Fl)
bg: Percentage of outliers averaged only over background regions
fg: Percentage of outliers averaged only over foreground regions
all: Percentage of outliers averaged over all ground truth pixels
Note: On 13.03.2017 we have fixed several small errors in the flow (noc+occ) ground truth of the dynamic foreground objects and manually verified all images for correctness by warping them according to the ground truth. As a consequence, all error numbers have decreased slightly. Please download the devkit and the annotations with the improved ground truth for the training set again if you have downloaded the files prior to 13.03.2017 and consider reporting these new number in all future publications. The last leaderboards before these corrections can be found here (optical flow 2015) and here (scene flow 2015). The leaderboards for the KITTI 2015 stereo benchmarks did not change.
Important Policy Update: As more and more non-published work and re-implementations of existing work is submitted to KITTI, we have established a new policy: from now on, only submissions with significant novelty that are leading to a peer-reviewed paper in a conference or journal are allowed. Minor modifications of existing algorithms or student research projects are not allowed. Such work must be evaluated on a split of the training set. To ensure that our policy is adopted, new users must detail their status, describe their work and specify the targeted venue during registration. Furthermore, we will regularly delete all entries that are 6 months old but are still anonymous or do not have a paper associated with them. For conferences, 6 month is enough to determine if a paper has been accepted and to add the bibliography information. For longer review cycles, you need to resubmit your results.
Additional information used by the methods
Multiview: Method uses more than 2 temporally adjacent images
Motion stereo: Method uses epipolar geometry for computing optical flow
Additional training data: Use of additional data sources for training (see details)
HCI/Bosch Robust Vision Challenge: Optical flow and stereo vision challenge on high resolution imagery recorded at a high frame rate under diverse weather conditions (e.g., sunny, cloudy, rainy). The Robert Bosch AG provides a prize for the best performing method.
Image Sequence Analysis Test Site (EISATS): Synthetic image sequences with ground truth information provided by UoA and Daimler AG. Some of the images come with 3D range sensor information.
Middlebury Stereo Evaluation: The classic stereo evaluation benchmark, featuring four test images in version 2 of the benchmark, with very accurate ground truth from a structured light system. 38 image pairs are provided in total.
Daimler Stereo Dataset: Stereo bad weather highway scenes with partial ground truth for freespace
Make3D Range Image Data: Images with small-resolution ground truth used to learn and evaluate depth from single monocular images.
Middlebury Optical Flow Evaluation: The classic optical flow evaluation benchmark, featuring eight test images, with very accurate ground truth from a shape from UV light pattern system. 24 image pairs are provided in total.
Citation
When using this dataset in your research, we will be happy if you cite us:
@article{Menze2018JPRS,
author = {Moritz Menze and Christian Heipke and Andreas Geiger},
title = {Object Scene Flow}, journal = {ISPRS Journal of Photogrammetry and Remote Sensing (JPRS)},
year = {2018}
}
@inproceedings{Menze2015ISA,
author = {Moritz Menze and Christian Heipke and Andreas Geiger},
title = {Joint 3D Estimation of Vehicles and Scene Flow}, booktitle = {ISPRS Workshop on Image Sequence Analysis (ISA)},
year = {2015}
}
Multiview: Method uses more than 2 temporally adjacent images
Motion stereo: Method uses epipolar geometry for computing optical flow
Additional training data: Use of additional data sources for training (see details)