PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models
- Resource Type
- Conference
- Authors
- Liu, Minghua; Zhu, Yinhao; Cai, Hong; Han, Shizhong; Ling, Zhan; Porikli, Fatih; Su, Hao
- Source
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) CVPR Computer Vision and Pattern Recognition (CVPR), 2023 IEEE/CVF Conference on. :21736-21746 Jun, 2023
- Subject
- Computing and Processing
Point cloud compression
Training
Image segmentation
Solid modeling
Computer vision
Three-dimensional displays
Shape
Segmentation
grouping and shape analysis
- Language
- ISSN
- 2575-7075
Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP. which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.