miniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis

Our model architecture, illustrated in Fig 2, is composed of three key components: a visual backbone, a linear projection layer, and an extensive language model. The details of each component are described as follows:

We employ a prompt template that allows our model can well deal with many diverse medical vision-language skills, such as visual question answering, image captioning, referring expression comprehension (REC), referring expression generation (REG), disease detection, and grounded image captioning. A language model might experience high levels of hallucination and confusion while dealing with many diverse vision-language tasks. For instance, when being asked to identify a potential lung tumor, it could mistakenly focus on and describe areas of calcification in the blood vessels or heart. Therefore, to avoid ambiguity in these multi-task environments, we add task-specific tokens to our training framework. We follow a similar instruction design to that of MiniGPT-v2Chen et al. (2023) in our instruction template, presented as follows:

[INST] $<$Img$>$ $<$ ImageFeature$>$ $<$/Img$>$ [Task Identifier] Instruction [/INST]

We present diverse prompt templates in Table 1 to demonstrate how our model effectively deals with the different tasks through task identifiers.

For grounding skills that involve the spatial location of objects, such as disease detection and grounded image captioning, we employ a textual representation for bounding boxes. This representation allows us to integrate spatial locations into the text fed into the language model. We normalize the bounding box coordinates within the [0,100] range. Each spatial location is expressed in the format:

The lack of quality medical datasets is a significant challenge in the field of deep learning for medical imaging. To address this issue, we have prioritized the collection of a comprehensive dataset focusing on radiology, specifically lung diseases, as well as general medical information. Our goal is to gather a diverse and extensive range of medical images, including X-rays, CT scans, and MRI images. Furthermore, we aim to enhance the dataset by incorporating images with bounding boxes, datasets featuring question-and-answer formats, and datasets for report generation. These additions will support all the necessary skills for the model training and development.

The collected datasets include MIMIC Johnson et al. (2019), NLST The Cancer Imaging Archive (2023) and SLAKE Medical Visual Question Answering (2023) (Med-VQA), RSNA Radiological Society of North America (2018) and RadVQA OSF (2023s).The details for those medical datasets are demonstrated as the following: MIMIC The dataset comprises 377,110 images and 227,835 medical reports. In our study, we obtained the preprocessed MIMIC dataset from XrayGPT Thawkar et al. (2023a), which includes 114,539 de-identified chest X-ray images in JPG format, each accompanied by a corresponding radiology report. Of these, 171,085 images and reports are allocated for training, while 43,454 images and reports are designated for testing. This dataset is utilized for the task of report generation.

In our experiment, we initialize our model with MiniGPT-v2 Chen et al. (2023) pre-trained weights (after stage 3) and keep the vision encoder frozen throughout the whole training process. We finetune the linear projection layer and use LoRA (Low-Rank Adaptation) Hu et al. (2021) to finetune the LLaMA-2 Touvron et al. (2023) large language model. The model is trained using the cross-entropy loss function, which is optimized using the AdamW optimizer. Our dataset comprises 124,276 medical images, each with a resolution of 448x448 pixels, and no data augmentation is applied. The entire training was performed on a single NVIDIA A100 GPU over 100 epochs, with a maximum learning rate of 1e-5. The training duration was approximately 22 hours.

In this study, we conducted an assessment of MiniGPT-Med’s performance across three distinct tasks: medical report generation, disease detection, and medical visual question answering (VQA). We compared our model to both specialist and generalist models. The specialist models represent those who can only do either grounding or non-grounding tasks. The generalist models represent those models that can do various tasks including both grounding and non-grounding tasks.

In our study, we adapted our evaluation approach to align with the distinct skills required for interpreting radiology images using MiniGPT-Med. To assess the model’s ability to generate radiological reports, we used two metrics: BERT Similarity (BERTsim) and CheXbert Similarity (CheXbert-Sim). BERTsim was utilized to evaluate the semantic similarity between the model-generated descriptions of radiological images and the expert-provided ground truth annotations. This involved using a BERT model to embed both the ground truth and generated sentences, followed by computing the cosine similarity between these embeddings. CheXbert-Sim, conversely, was selected for its relevance in assessing the model’s accuracy in replicating professional medical report standards. It is a specialized version of the BERT model, fine-tuned on clinical texts, which computes the cosine similarity between embeddings for each corresponding sentence pair after encoding. For the Visual Question Answering (VQA) aspect, we exclusively used BERTsim to measure the semantic accuracy of the model’s responses. Additionally, we employed Intersection over Union (IoU) for grounding, a metric that quantitatively measures the model’s precision in localizing and identifying specific features or abnormalities within the radiology images, such as pneumonia in the RSNA dataset.

In our comprehensive study, we evaluated the efficacy of the MiniGPT-Med model in the generation of medical report generation, leveraging the comprehensive MIMIC dataset Johnson et al. (2019). The results of this evaluation, which are outlined in Table 5, demonstrate that the MiniGPT-Med model surpasses both specialized and generalized baseline models. Most notably, MiniGPT-Med demonstrates a significant edge over the leading specialized model, CheXagent Chen et al. (2024), with remarkable margins of 21.6 and 5.2 on the BERT-Sim and CheXbert-Sim metrics, respectively. This performance not only showcases MiniGPT-Med’s supremacy in the medical report generation but also underscores its ability to outpace the top generalist models substantially—by a notable 19 points on BERT-Sim and 9 points on CheXbert-Sim. These findings solidify MiniGPT-Med’s position as a cutting-edge tool, demonstrating its effectiveness in medical report generation.

The data showcased in Table 3 reveal that MiniGPT-Med stands out for its competitive performance when compared against a comprehensive range of baseline models. With an Intersection over Union (IoU) score of 0.26, MiniGPT-Med not only exceeds the capabilities of generalist models by a margin of 16% but also attains performance metrics on par with specialist models. The peak IoU score among these specialist models is noted to be 0.31. Our MiniGPT-Med achieves competitive results and it demonstrates good disease detection performance among all the baseline models, highlighting its potential as a versatile and effective tool in the medical domain.

This study evaluates our model, MiniGPT-Med, against various baseline models using the RadVQA OSF (2023s) benchmark, as presented in Table 4. MiniGPT-Med achieves a notable performance metric of 0.58, surpassing both generalist models such as MiniGPT-v2 Chen et al. (2023) and specialist models like OpenFlamingo Awadalla et al. (2023) and Med-Flamingo Moor et al. (2023). This performance not only demonstrates MiniGPT-Med’s superiority over a broad range of models but also shows it can achieve results comparable to those of the leading specialist model, MedVIN Zhang et al. (2023), which has an accuracy of 0.62. The ability of MiniGPT-Med to outperform or match the performance of several specialized and generalist models underscores its significant potential as a foundation for the development of advanced medical visual question-answering models.

Our study evaluated MiniGPT-Med using a rigorous human subjective protocol with two senior radiologists. They assessed 50 random samples from the MIMIC dataset’s test suite, focusing on the model’s robustness, granularity, and accuracy. The evaluation centered on three questions, Q1: how closely does the generated report align with your expert judgment? Q2: How detailed is the medical content of the generated report? Q3: How accurate is the generated report in diagnosing pathologies? We present the results in the accompanying Table 5. The results show that a remarkable 76% of the artificial medical reports are adjudged as of high quality. A further 19% were classified as of medium quality, while a mere 5% were deemed to be of poor quality. This distribution underscores the model’s capability to synthesize medical reports that not only resonate with professional standards but also exhibit a high degree of detail and diagnostic accuracy. Such findings underscore the potential of MiniGPT-Med as a valuable tool in the augmentation of medical reporting processes, indicating its substantial reliability and effectiveness in generating clinically relevant reports.

In this section, we provide comprehensive demonstrations of MiniGPT-Med’s capabilities in generating medical reports and performing interpretative tasks. First, Figure 3(a) illustrates the model’s ability to produce detailed medical reports from imagery data. Also, the model can accurately identify and delineate specified abnormalities with bounding boxes as shown in Figure 3(b). Additionally, Figure 4(a) demonstrates the grounding skill, where the model explains each generated word and draws a bounding box around the object. Furthermore, Figure 4(b) details the model’s precision in referencing, and pinpointing abnormalities as specified by users. Moreover, the identification feature is showcased in Figure 4(c), where the model provides elaborate medical descriptions utilizing object coordinates. Finally, Figure 4(d) presents the model’s visual question-answering (VQA) functionality, underscoring its effectiveness in providing precise answers to medical questions.