PointCG
Self-supervised Point Cloud Learning via Joint Completion and Generation
Yun Liu (Nanjing University of Aeronautics and Astronautics)
Peng Li (Nanjing University of Aeronautics and Astronautics)
Xuefeng Yan (Nanjing University of Aeronautics and Astronautics)
L. Nan (TU Delft - Urban Data Science)
Bing Wang (The Hong Kong Polytechnic University)
Honghua Chen (Nanjing University of Aeronautics and Astronautics)
Lina Gong (Nanjing University of Aeronautics and Astronautics)
Wei Zhao (Nanjing University of Aeronautics and Astronautics)
Mingqiang Wei (Nanjing University of Aeronautics and Astronautics)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
The core of self-supervised point cloud learning lies in setting up appropriate pretext tasks, to construct a pre-training framework that enables the encoder to perceive 3D objects effectively. In this article, we integrate two prevalent methods, masked point modeling (MPM) and 3D-to-2D generation, as pretext tasks within a pre-training framework. We leverage the spatial awareness and precise supervision offered by these two methods to address their respective limitations: ambiguous supervision signals and insensitivity to geometric information. Specifically, the proposed framework, abbreviated as PointCG, consists of a Hidden Point Completion (HPC) module and an Arbitrary-view Image Generation (AIG) module. We first capture visible points from arbitrary views as inputs by removing hidden points. Then, HPC extracts representations of the inputs with an encoder and completes the entire shape with a decoder, while AIG is used to generate rendered images based on the visible points’ representations. Extensive experiments demonstrate the superiority of the proposed method over the baselines in various downstream tasks. Our code will be made available upon acceptance.