In this study, a ChatGPt-based medical image diagnosis and processing architecture were proposed to compensate for the shortcomings of doctors’ subjective misdiagnosis and computer-aided diagnosis in traditional medical image diagnosis. The architecture is divided into an image description generation system and ChatGPT. The former interprets medical images and generates corresponding diagnostic reports, while the latter is responsible for processing diagnostic reports and providing treatment recommendations. In image description generation, we use three text similarity algorithms to evaluate the similarity between the diagnostic report generated by the method and the diagnostic report provided by the doctor and prove that the diagnostic report generated by the algorithm has a high degree of similarity and effectiveness. In the ChatGPT-assisted diagnosis and treatment section, we asked the cardiovascular medical team to score the treatment suggestions generated by ChatGPT. The results showed that the overall acceptance of the suggestions given by ChatGPT was high. Despite the limitations of the data set in this study and the fact that XCA images were only used as an example, the method is expected to be extended to other types of medical images and is expected to become an essential part of intelligent diagnosis and treatment in the future.