Automatic Generation of Radiological Medical Image Report

Abstract

Automated Radiographic Report Generation (R2Gen) is a critical task aimed at the automatic generation of a free-text description for medical images, such as chest X-rays. This process entails producing a comprehensive and contextually relevant summary of the image's content and clinical significance. In this thesis, we propose several innovative approaches to improve the quality and accuracy of radiographic report generation. First, we propose a self-boosting framework that synchronizes image and report features by simultaneously training an auxiliary image-text matching task alongside the main report generation task. This mutual training process allows each task to refine and enhance the other, leading to more accurate report generation aligned closely with ground-truth reports. We also explore sample relationships between image and report features by introducing a sample-graph consistency loss as an additional regularization into the self-boosting framework. Subsequently, with the rise in popularity of the Transformer architecture, we devised a novel mechanism to embed the image-text matching task within a Transformer's encoder-decoder structure, employing a pure Transformer network structure in the R2Gen task for the first time. Moreover, we investigate ways to incorporate external information to assist in the generation of reports. To this end, we have designed a Medical Concepts Generation Network to predict finegrained semantic concepts and integrate them into the report generation process as guidance. Next, treating the current method as a ``single expert'', we introduced a new diagnostic captioning framework, METransformer, to emulate the clinical scenario of ``collaborative diagnosis by multiple experts''. In the last, we propose a novel LLMs-based Radiology report generation framework, dubbed R2GenGPT, which is the first instance of harnessing pre-trained large language models (LLMs) for the R2Gen task.