Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray images
- Resource Type
- Working Paper
- Authors
- Zhong, Yi; Xu, Mengqiu; Liang, Kongming; Chen, Kaixin; Wu, Ming
- Source
- Subject
- Electrical Engineering and Systems Science - Image and Video Processing
Computer Science - Computer Vision and Pattern Recognition
- Language
Segmentation of the infected areas of the lung is essential for quantifying the severity of lung disease like pulmonary infections. Existing medical image segmentation methods are almost uni-modal methods based on image. However, these image-only methods tend to produce inaccurate results unless trained with large amounts of annotated data. To overcome this challenge, we propose a language-driven segmentation method that uses text prompt to improve to the segmentation result. Experiments on the QaTa-COV19 dataset indicate that our method improves the Dice score by 6.09% at least compared to the uni-modal methods. Besides, our extended study reveals the flexibility of multi-modal methods in terms of the information granularity of text and demonstrates that multi-modal methods have a significant advantage over image-only methods in terms of the size of training data required.
Comment: Provisional Acceptance by MICCAI 2023