Sim VQA: Exploring Simulated Environments for Visual Question Answering
- Resource Type
- Conference
- Authors
- Cascante-Bonilla, Paola; Wu, Hui; Wang, Letao; Feris, Rogerio; Ordonez, Vicente
- Source
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) CVPR Computer Vision and Pattern Recognition (CVPR), 2022 IEEE/CVF Conference on. :5046-5056 Jun, 2022
- Subject
- Computing and Processing
Training
Visualization
Solid modeling
Three-dimensional displays
Computational modeling
Pipelines
Switches
Vision + language; Visual reasoning
- Language
- ISSN
- 2575-7075
Existing work on VQA explores data augmentation to achieve better generalization by perturbing images in the dataset or modifying existing questions and answers. While these methods exhibit good performance, the diversity of the questions and answers are constrained by the available images. In this work we explore using synthetic computer-generated data to fully control the visual and language space, allowing us to provide more diverse scenarios. We quantify the effectiveness of leveraging synthetic data for real-world VQA. By exploiting 3D and physics simulation platforms, we provide a pipeline to generate synthetic data to expand and replace type-specific questions and answers without risking exposure of sensitive or personal data that might be present in real images. We offer a comprehensive analysis while expanding existing hyper-realistic datasets to be usedfor VQA. We also propose Feature Swapping (F-SWAP) - where we randomly switch object-level features during training to make a VQA model more domain invariant. We show that F-SWAP is effective for improving VQA models on real images without compromising on their accuracy to answer existing questions in the dataset.