Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
- Resource Type
- Conference
- Authors
- Zhang, Shangzhan; Peng, Sida; Chen, Tianrun; Mou, Linzhan; Lin, Haotong; Yu, Kaicheng; Liao, Yiyi; Zhou, Xiaowei
- Source
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) CVPR Computer Vision and Pattern Recognition (CVPR), 2023 IEEE/CVF Conference on. :8518-8528 Jun, 2023
- Subject
- Computing and Processing
Training
Three-dimensional displays
Image synthesis
Semantics
Pipelines
Pattern recognition
Internet
3D from multi-view and sensors
- Language
- ISSN
- 2575-7075
We introduce a novel approach that takes a single semantic mask as input to synthesize multi-view consistent color images of natural scenes, trained with a collection of single images from the Internet. Prior works on 3D-aware image synthesis either require multi-view supervision or learning category-level prior for specific classes of objects, which are inapplicable to natural scenes. Our key idea to solve this challenge is to use a semantic field as the intermediate representation, which is easier to reconstruct from an input semantic mask and then translated to a radiance field with the assistance of off-the-shelf semantic image synthesis models. Experiments show that our method outperforms baseline methods and produces photorealistic and multi-view consistent videos of a variety of natural scenes. The project website is https://zju3dv.github.io/paintingnature/.