Adversarial Training with Comprehensive Objective for Medical Image Report Generation
- Resource Type
- Conference
- Authors
- Zhang, Xuemiao; Liu, Junfei
- Source
- 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) Bioinformatics and Biomedicine (BIBM), 2023 IEEE International Conference on. :2352-2357 Dec, 2023
- Subject
- Bioengineering
Computing and Processing
Engineering Profession
Robotics and Control Systems
Signal Processing and Analysis
Training
Visualization
Reinforcement learning
Writing
Radiology
Generators
Real-time systems
Medical Image Report Generation
Reinforcement Learning
Image Caption
Generative Adversarial Networks
- Language
- ISSN
- 2156-1133
Medical Image Report Generation aims to automatically generate medical reports based on radiology images, thus freeing radiologists from the tedious task of writing reports. Generating report texts that match the content of a given medical image, focus on local anomalies, and fluently conform to professional reporting norms presents the challenge of linking visual patterns to informative human-language descriptions. In this paper, we propose a novel generative adversarial framework (MIRGAN) to guide the generator to generate medical reports that are indistinguishable from those written by professional radiologists. MIRGAN introduces a multimodal discriminator to evaluate the performance of the report generator on the comprehensive objective containing three sub-objectives when generating each token. MIRGAN then uses it as the reward in reinforcement learning to guide the generator to optimize toward the desired objective in real time. We conduct sufficient experiments to evaluate MIRGAN on two widely used datasets for chest-related diseases. The experimental results show that MIRGAN can significantly improve the performance of the generator on most metrics.