Towards Open-Vocabulary 3D Scene Understanding

 

 

Speaker: Djamahl Etchegaray

 

Abstract: Autonomous driving systems rely on 3D perception to transform raw sensor data into actionable inputs for planning algorithms. However, current 3D detectors are restricted to a closed vocabulary of common object classes due to the prohibitive cost of dense 3D annotations, leaving safety-critical, long-tail obstacles undetected. While 2D vision-language models have achieved open-vocabulary generalisation through billions of web-scraped image-text pairs, 3D perception lags significantly behind due to the scarcity of annotated data. This seminar presents a body of work that investigates cost-effective strategies for bridging this gap and extending 3D perception to all semantics. By evaluating state-of-the-art foundation models and introducing novel, annotation-free frameworks for object detection, semantic evaluation, tracking, and scenario mining, this research demonstrates how to robustly detect and reason about arbitrary object classes in urban environments without relying on dense 3D supervision.

 

Speaker Bio: Djamahl is a third-year PhD student in the Data Science group at the School of EECS, supervised by Dr. Yadan Luo and Prof. Helen (Zi) Huang. His research focuses on open-vocabulary 3D scene understanding for autonomous driving. He received his bachelor's degree in mathematics and computer science from the University of Queensland. His work has been featured in conferences such as ECCV, ACM MM, and IJCNN. He has completed internships at Atlassian, CSIRO and Datarwe.