Towards Cross-Modal Point Cloud Retrieval for Indoor Scenes

Towards Cross-Modal Point Cloud Retrieval for Indoor Scenes

Yu, Fuyang (Beihang University)
Wang, Zhen (Tokyo Institute of Technology)
Li, Dongyuan (Tokyo Institute of Technology)
Zhu, P. (TU Delft Web Information Systems)
Liang, Xiaohui (Beihang University)
Wang, Xiaochuan (Beijing Technology and Business University)
Okumura, Manabu (Tokyo Institute of Technology)

Rudinac, Stevan (editor)
Worring, Marcel (editor)
Liem, Cynthia (editor)
Hanjalic, Alan (editor)
Jónsson, Björn Pór (editor)
Yamakata, Yoko (editor)
Liu, Bei (editor)

2024

Cross-modal retrieval, as an important emerging foundational information retrieval task, benefits from recent advances in multimodal technologies. However, current cross-modal retrieval methods mainly focus on the interaction between textual information and 2D images, lacking research on 3D data, especially point clouds at scene level, despite the increasing role point clouds play in daily life. Therefore, in this paper, we proposed a cross-modal point cloud retrieval benchmark that focuses on using text or images to retrieve point clouds of indoor scenes. Given the high cost of obtaining point cloud compared to text and images, we first designed a pipeline to automatically generate a large number of indoor scenes and their corresponding scene graphs. Based on this pipeline, we collected a balanced dataset called CRISP, which contains 10K point cloud scenes along with their corresponding scene images and descriptions. We then used state-of-the-art models to design baseline methods on CRISP. Our experiments demonstrated that point cloud retrieval accuracy is much lower than cross-modal retrieval of 2D images, especially for textual queries. Furthermore, we proposed ModalBlender, a tri-modal framework which can greatly improve the Text-PointCloud retrieval performance. Through extensive experiments, CRISP proved to be a valuable dataset and worth researching. (Dataset can be downloaded at https://github.com/CRISPdataset/CRISP.)

Cross-modal Retrieval
Indoor Scene
Point Cloud

http://resolver.tudelft.nl/uuid:3c4e3e23-f4bc-44fe-b737-ce1e0fcdfed0

https://doi.org/10.1007/978-3-031-53302-0_7

Springer, Cham

2024-07-29

978-3-031-53301-3

MultiMedia Modeling - 30th International Conference, MMM 2024, Proceedings

30th International Conference on MultiMedia Modeling, MMM 2024, 2024-01-29 → 2024-02-02, Amsterdam, Netherlands

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 0302-9743, 14557 LNCS

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2024 Fuyang Yu, Zhen Wang, Dongyuan Li, P. Zhu, Xiaohui Liang, Xiaochuan Wang, Manabu Okumura