RoScenes

A Large-scale Multi-view 3D Dataset for Roadside Perception

Xiaosu Zhu^1*, Hualian Sheng^1*, Sijia Cai^1†, Bing Deng¹, Shaopeng Yang¹,
Qiao Liang¹, Ken Chen², Lianli Gao³, Jingkuan Song^4‡ Jieping Ye^1‡

¹Alibaba Cloud, ²Sichuan Digital Transportation Technology Co., Ltd,
³Independent Researcher, ⁴Tongji University

^*Equal contribution, ^†Project lead, ^‡Corresponding authors

We introduce RoScenes, the largest multi-view roadside perception dataset, which aims to shed light on the development of vision-centric Bird's Eye View (BEV) approaches for more challenging traffic scenes. The highlights of RoScenes include significantly large perception area, full scene coverage and crowded traffic. More specifically, our dataset achieves surprising 21.13M 3D annotations within 64,000 m². To relieve the expensive costs of roadside 3D labeling, we present a novel BEV-to-3D joint annotation pipeline to efficiently collect such a large volume of data. After that, we organize a comprehensive study for current BEV methods on RoScenes in terms of effectiveness and efficiency. Tested methods suffer from the vast perception area and variation of sensor layout across scenes, resulting in performance levels falling below expectations. To this end, we propose RoBEV that incorporates feature-guided position embedding for effective 2D-3D feature assignment. With its help, our method outperforms state-of-the-art by a large margin without extra computational overhead on validation set.