Multi-Object Tracking and Segmentation (MOTS) Evaluation


KITTI MOTS will be part of the RobMOTS Challenge at CVPR 21. Deadline June 11.

The Multi-Object and Segmentation (MOTS) benchmark [2] consists of 21 training sequences and 29 test sequences. It is based on the KITTI Tracking Evaluation 2012 and extends the annotations to the Multi-Object and Segmentation (MOTS) task. To this end, we added dense pixelwise segmentation labels for every object. We evaluate submitted results using the metrics HOTA, CLEAR MOT and MT/PT/ML. We rank methods by HOTA [1]. Our development kit and github evaluation code provides details about the data format as well as utility functions for reading and writing the label files. (adapted for the segmentation case). Evaluation is performed using the code from the TrackEval repository.

We recommend that everyone who uses this benchmark reads this blog post for an overview of the HOTA metrics.


[1] J. Luiten, A. Os̆ep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, B. Leibe: HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. IJCV 2020.
[2] P. Voigtlaender, M. Krause, A. Os̆ep, J. Luiten, B. Sekar, A. Geiger, B. Leibe: MOTS: Multi-Object Tracking and Segmentation. CVPR 2019.

Note: On 25.02.2020 we have updated the evaluation to use the HOTA metrics as the main evaluation metrics, and to show results as plots to enable better comparison over various aspects of tracking. We have re-calculated the results for all methods. Please download the new evaluation code. Please report these new numbers for all future submissions. The previous leaderboards before the changes will remain live for now, but after some time they will stop being updated. They can be found here.

Please address any questions or feedback about KITTI tracking or KITTI mots evaluation to Jonathon Luiten at luiten@vision.rwth-aachen.de.

Important Policy Update: As more and more non-published work and re-implementations of existing work is submitted to KITTI, we have established a new policy: from now on, only submissions with significant novelty that are leading to a peer-reviewed paper in a conference or journal are allowed. Minor modifications of existing algorithms or student research projects are not allowed. Such work must be evaluated on a split of the training set. To ensure that our policy is adopted, new users must detail their status, describe their work and specify the targeted venue during registration. Furthermore, we will regularly delete all entries that are 6 months old but are still anonymous or do not have a paper associated with them. For conferences, 6 month is enough to determine if a paper has been accepted and to add the bibliography information. For longer review cycles, you need to resubmit your results.
Additional information used by the methods
  • Stereo: Method uses left and right (stereo) images
  • Laser Points: Method uses point clouds from Velodyne laser scanner
  • GPS: Method uses GPS information
  • Online: Online method (frame-by-frame processing, no latency)
  • Additional training data: Use of additional data sources for training (see details)

CAR



This figure as: png pdf

This figure as: png pdf

This figure as: png pdf

This figure as: png pdf

KITTI MOTS will be part of the RobMOTS Challenge at CVPR 21. Deadline June 11.

Method Setting Code HOTA DetA AssA DetRe DetPr AssRe AssPr LocA sMOTSA
1 UW_IPL/ETRI_AIRL 79.57 % 79.66 % 80.00 % 83.05 % 90.35 % 83.08 % 91.42 % 90.06 % 79.58 %
2 ViP-DeepLab 76.38 % 82.70 % 70.93 % 88.70 % 88.77 % 75.86 % 86.00 % 90.75 % 81.03 %
S. Qiao, Y. Zhu, H. Adam, A. Yuille and L. Chen: ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2021.
3 OSN 76.25 % 81.62 % 71.63 % 87.10 % 88.23 % 80.93 % 80.66 % 90.09 % 80.88 %
4 FlowGuideMOTS 75.37 % 83.69 % 68.28 % 88.36 % 89.68 % 82.69 % 77.16 % 90.60 % 83.47 %
5 CCP_ST 75.12 % 84.48 % 67.22 % 88.07 % 90.91 % 79.28 % 79.22 % 90.65 % 84.47 %
6 EagerMOT code 74.66 % 76.11 % 73.75 % 79.59 % 90.24 % 76.27 % 92.70 % 90.46 % 74.53 %
A. Kim and L. Leal-Taix'e: EagerMOT: 3D Multi-Object Tracking via Sensor Fusion. IEEE International Conference on Robotics and Automation (ICRA) 2021.
7 MOTSFusion code 73.63 % 75.44 % 72.39 % 78.32 % 90.78 % 75.53 % 89.97 % 90.29 % 74.98 %
J. Luiten, T. Fischer and B. Leibe: Track to Reconstruct and Reconstruct to Track. IEEE Robotics and Automation Letters 2020.
8 CCP 73.61 % 84.47 % 64.58 % 88.11 % 90.84 % 79.55 % 75.54 % 90.63 % 84.47 %
9 OPITrack code 73.04 % 79.44 % 67.97 % 85.90 % 85.64 % 75.66 % 80.02 % 88.57 % 78.02 %
10 ReMOTS 71.61 % 78.32 % 65.98 % 83.51 % 87.42 % 68.03 % 92.61 % 89.33 % 75.92 %
F. Yang, X. Chang, C. Dang, Z. Zheng, S. Sakti, S. Nakamura and Y. Wu: ReMOTS: Self-Supervised Refining Multi- Object Tracking and Segmentation. 2020.
11 GMPHD_MAF 71.21 % 78.06 % 65.41 % 82.46 % 88.34 % 68.96 % 86.77 % 89.44 % 76.53 %
12 Struct_MOTS 68.73 % 76.43 % 62.24 % 79.73 % 89.62 % 71.01 % 77.77 % 89.65 % 75.51 %
13 MOTS R-CNN 65.36 % 71.57 % 60.38 % 76.20 % 84.37 % 65.70 % 80.61 % 86.52 % 69.79 %
14 PointTrack 61.95 % 79.38 % 48.83 % 85.77 % 85.66 % 79.07 % 56.35 % 88.52 % 78.50 %
Z. Xu, W. Zhang, X. Tan, W. Yang, H. Huang, S. Wen, E. Ding and L. Huang: Segment as Points for Efficient Online Multi-Object Tracking and Segmentation. Proceedings of the European Conference on Computer Vision (ECCV) 2020.
15 WMOTS 59.65 % 66.78 % 54.22 % 72.92 % 79.82 % 57.67 % 84.93 % 84.66 % 61.70 %
16 TrackR-CNN_CCP 57.77 % 74.20 % 45.50 % 79.30 % 84.28 % 61.60 % 62.29 % 86.67 % 71.81 %
17 PolyTrack
This is an online method (no batch processing).
57.61 % 62.47 % 53.97 % 67.83 % 79.11 % 57.13 % 81.92 % 83.70 % 57.49 %
18 CPMOTS 57.58 % 74.17 % 45.18 % 78.86 % 86.06 % 73.79 % 52.94 % 87.86 % 72.77 %
19 TrackR-CNN code 56.63 % 69.90 % 46.53 % 74.63 % 84.18 % 63.13 % 62.33 % 86.60 % 66.97 %
P. Voigtlaender, M. Krause, A. O\usep, J. Luiten, B. Sekar, A. Geiger and B. Leibe: MOTS: Multi-Object Tracking and Segmentation. CVPR 2019.
20 GMPHD_SAF 55.14 % 77.01 % 39.76 % 81.57 % 87.29 % 69.22 % 49.42 % 88.72 % 75.39 %
Y. Song and M. Jeon: Online Multi-Object Tracking and Segmentation with GMPHD Filter and Simple Affinity Fusion. arXiv preprint arXiv:2009.00100 2020.
Table as LaTeX | Only published Methods


PEDESTRIAN



This figure as: png pdf

This figure as: png pdf

This figure as: png pdf

This figure as: png pdf

KITTI MOTS will be part of the RobMOTS Challenge at CVPR 21. Deadline June 11.

Method Setting Code HOTA DetA AssA DetRe DetPr AssRe AssPr LocA sMOTSA
1 UW_IPL/ETRI_AIRL 64.38 % 66.72 % 63.80 % 70.53 % 81.21 % 68.94 % 78.73 % 83.26 % 64.93 %
2 ViP-DeepLab 64.31 % 70.69 % 59.48 % 75.71 % 81.77 % 67.52 % 74.92 % 84.40 % 68.76 %
S. Qiao, Y. Zhu, H. Adam, A. Yuille and L. Chen: ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2021.
3 FlowGuideMOTS 63.77 % 71.26 % 58.19 % 75.18 % 82.45 % 69.38 % 71.97 % 84.03 % 70.22 %
4 CCP_ST 62.22 % 72.09 % 54.73 % 76.19 % 82.76 % 64.91 % 70.15 % 84.39 % 70.55 %
5 CCP 60.50 % 71.35 % 52.50 % 75.02 % 83.06 % 63.39 % 68.85 % 84.36 % 70.16 %
6 OPITrack code 60.38 % 62.45 % 60.05 % 65.65 % 81.61 % 64.33 % 82.23 % 83.55 % 61.05 %
7 OSN 59.93 % 65.04 % 55.79 % 68.64 % 80.25 % 65.93 % 69.58 % 82.47 % 63.44 %
8 ReMOTS 58.81 % 67.96 % 52.38 % 71.86 % 82.22 % 54.40 % 88.23 % 84.18 % 65.97 %
F. Yang, X. Chang, C. Dang, Z. Zheng, S. Sakti, S. Nakamura and Y. Wu: ReMOTS: Self-Supervised Refining Multi- Object Tracking and Segmentation. 2020.
9 Struct_MOTS 58.81 % 60.72 % 57.67 % 63.10 % 84.41 % 64.47 % 79.96 % 85.20 % 60.27 %
10 EagerMOT code 57.65 % 60.30 % 56.19 % 63.45 % 81.58 % 60.19 % 83.35 % 83.65 % 58.08 %
A. Kim and L. Leal-Taix'e: EagerMOT: 3D Multi-Object Tracking via Sensor Fusion. IEEE International Conference on Robotics and Automation (ICRA) 2021.
11 MaskguidedMOTS
This is an online method (no batch processing).
56.85 % 57.81 % 56.61 % 61.63 % 77.28 % 60.98 % 78.93 % 81.07 % 54.39 %
12 GMPHD_MAF 54.66 % 66.61 % 45.71 % 70.32 % 82.48 % 53.68 % 68.52 % 84.42 % 64.88 %
13 PointTrack 54.44 % 62.29 % 48.08 % 65.49 % 81.17 % 64.97 % 58.66 % 83.28 % 61.47 %
Z. Xu, W. Zhang, X. Tan, W. Yang, H. Huang, S. Wen, E. Ding and L. Huang: Segment as Points for Efficient Online Multi-Object Tracking and Segmentation. Proceedings of the European Conference on Computer Vision (ECCV) 2020.
14 MOTSFusion code 54.04 % 60.83 % 49.45 % 64.13 % 81.47 % 56.68 % 70.44 % 83.71 % 58.75 %
J. Luiten, T. Fischer and B. Leibe: Track to Reconstruct and Reconstruct to Track. IEEE Robotics and Automation Letters 2020.
15 CPMOTS 50.11 % 59.73 % 42.46 % 62.93 % 79.62 % 58.56 % 54.86 % 82.21 % 58.13 %
16 GMPHD_SAF 49.33 % 65.45 % 38.32 % 69.62 % 80.98 % 57.88 % 49.77 % 83.82 % 62.87 %
Y. Song and M. Jeon: Online Multi-Object Tracking and Segmentation with GMPHD Filter and Simple Affinity Fusion. arXiv preprint arXiv:2009.00100 2020.
17 MOTS R-CNN 45.73 % 52.12 % 41.02 % 55.50 % 72.90 % 45.98 % 67.67 % 77.88 % 46.75 %
18 PolyTrack
This is an online method (no batch processing).
44.89 % 48.63 % 41.82 % 52.78 % 66.32 % 46.70 % 63.05 % 74.40 % 39.75 %
19 TrackR-CNN_CCP 43.84 % 54.13 % 36.23 % 58.12 % 73.27 % 46.81 % 53.81 % 78.62 % 48.13 %
20 WMOTS 42.80 % 49.48 % 39.23 % 54.35 % 68.41 % 42.90 % 73.98 % 76.74 % 39.35 %
21 TrackR-CNN code 41.93 % 53.75 % 33.84 % 57.85 % 72.51 % 45.30 % 50.74 % 78.03 % 47.31 %
P. Voigtlaender, M. Krause, A. O\usep, J. Luiten, B. Sekar, A. Geiger and B. Leibe: MOTS: Multi-Object Tracking and Segmentation. CVPR 2019.
Table as LaTeX | Only published Methods


Citation

When using this dataset in your research, we will be happy if you cite us:
@INPROCEEDINGS{Voigtlaender2019CVPR,
  author = {Paul Voigtlaender and Michael Krause and Aljosa Osep and Jonathon Luiten and Berin Balachandar Gnana Sekar and Andreas Geiger and Bastian Leibe},
  title = {MOTS: Multi-Object Tracking and Segmentation},
  booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2019}
}

@ARTICLE{Luiten2020IJCV,
  author = {Jonathon Luiten and Aljosa Osep and Patrick Dendorfer and Philip Torr and Andreas Geiger and Laura Leal-Taixe and Bastian Leibe},
  title = {HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking},
  journal = {International Journal of Computer Vision (IJCV)},
  year = {2020}
}



eXTReMe Tracker