-
MAIRA-2: Grounded Radiology Report Generation
Authors:
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Anton Schwaighofer,
Sam Bond-Taylor,
Maximilian Ilse,
Fernando Pérez-García,
Valentina Salvatelli,
Harshita Sharma,
Felix Meissen,
Mercy Ranjit,
Shaury Srivastav,
Julia Gong,
Fabian Falck,
Ozan Oktay,
Anja Thieme,
Matthew P. Lungren,
Maria Teodora Wetscherek,
Javier Alvarez-Valle,
Stephanie L. Hyland
Abstract:
Radiology reporting is a complex task that requires detailed image understanding, integration of multiple inputs, including comparison with prior imaging, and precise language generation. This makes it ideal for the development and use of generative multimodal models. Here, we extend report generation to include the localisation of individual findings on the image - a task we call grounded report…
▽ More
Radiology reporting is a complex task that requires detailed image understanding, integration of multiple inputs, including comparison with prior imaging, and precise language generation. This makes it ideal for the development and use of generative multimodal models. Here, we extend report generation to include the localisation of individual findings on the image - a task we call grounded report generation. Prior work indicates that grounding is important for clarifying image understanding and interpreting AI-generated text. Therefore, grounded reporting stands to improve the utility and transparency of automated report drafting. To enable evaluation of grounded reporting, we propose a novel evaluation framework - RadFact - leveraging the reasoning capabilities of large language models (LLMs). RadFact assesses the factuality of individual generated sentences, as well as correctness of generated spatial localisations when present. We introduce MAIRA-2, a large multimodal model combining a radiology-specific image encoder with a LLM, and trained for the new task of grounded report generation on chest X-rays. MAIRA-2 uses more comprehensive inputs than explored previously: the current frontal image, the current lateral image, the prior frontal image and prior report, as well as the Indication, Technique and Comparison sections of the current report. We demonstrate that these additions significantly improve report quality and reduce hallucinations, establishing a new state of the art on findings generation (without grounding) on MIMIC-CXR while demonstrating the feasibility of grounded reporting as a novel and richer task.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology
Authors:
Anja Thieme,
Abhijith Rajamohan,
Benjamin Cooper,
Heather Groombridge,
Robert Simister,
Barney Wong,
Nicholas Woznitza,
Mark Ames Pinnock,
Maria Teodora Wetscherek,
Cecily Morrison,
Hannah Richardson,
Fernando Pérez-García,
Stephanie L. Hyland,
Shruthi Bannur,
Daniel C. Castro,
Kenza Bouzid,
Anton Schwaighofer,
Mercy Ranjit,
Harshita Sharma,
Matthew P. Lungren,
Ozan Oktay,
Javier Alvarez-Valle,
Aditya Nori,
Stephen Harris,
Joseph Jacob
Abstract:
Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delay…
▽ More
Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delayed in their detection, but gaps remain in clinical practice integration. In this study, we present a human-centered approach to the problem and describe insights derived following contextual inquiry and in-depth interviews with 15 clinical stakeholders. The interviews helped understand challenges in existing workflows, and how best to align technical capabilities with user needs and expectations. We discovered the trade-offs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations for different AI proposals. We explored how to balance AI benefits and risks for healthcare staff and patients within broader organizational and medical-legal constraints. We also identified data issues related to edge cases and data biases that affect model training and evaluation; how data documentation practices influence data preparation and labelling; and how to measure relevant AI outcomes reliably in future evaluations. We discuss how our work informs design and development of AI applications that are clinically useful, ethical, and acceptable in real-world healthcare services.
△ Less
Submitted 8 May, 2024;
originally announced May 2024.
-
Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
Authors:
Nur Yildirim,
Hannah Richardson,
Maria T. Wetscherek,
Junaid Bajwa,
Joseph Jacob,
Mark A. Pinnock,
Stephen Harris,
Daniel Coelho de Castro,
Shruthi Bannur,
Stephanie L. Hyland,
Pratik Ghosh,
Mercy Ranjit,
Kenza Bouzid,
Anton Schwaighofer,
Fernando Pérez-García,
Harshita Sharma,
Ozan Oktay,
Matthew Lungren,
Javier Alvarez-Valle,
Aditya Nori,
Anja Thieme
Abstract:
Recent advances in AI combine large language models (LLMs) with vision encoders that bring forward unprecedented technical capabilities to leverage for a wide range of healthcare applications. Focusing on the domain of radiology, vision-language models (VLMs) achieve good performance results for tasks such as generating radiology findings based on a patient's medical image, or answering visual que…
▽ More
Recent advances in AI combine large language models (LLMs) with vision encoders that bring forward unprecedented technical capabilities to leverage for a wide range of healthcare applications. Focusing on the domain of radiology, vision-language models (VLMs) achieve good performance results for tasks such as generating radiology findings based on a patient's medical image, or answering visual questions (e.g., 'Where are the nodules in this chest X-ray?'). However, the clinical utility of potential applications of these capabilities is currently underexplored. We engaged in an iterative, multidisciplinary design process to envision clinically relevant VLM interactions, and co-designed four VLM use concepts: Draft Report Generation, Augmented Report Review, Visual Search and Querying, and Patient Imaging History Highlights. We studied these concepts with 13 radiologists and clinicians who assessed the VLM concepts as valuable, yet articulated many design considerations. Reflecting on our findings, we discuss implications for integrating VLM capabilities in radiology, and for healthcare AI more generally.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision
Authors:
Fernando Pérez-García,
Harshita Sharma,
Sam Bond-Taylor,
Kenza Bouzid,
Valentina Salvatelli,
Maximilian Ilse,
Shruthi Bannur,
Daniel C. Castro,
Anton Schwaighofer,
Matthew P. Lungren,
Maria Wetscherek,
Noel Codella,
Stephanie L. Hyland,
Javier Alvarez-Valle,
Ozan Oktay
Abstract:
Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, resulting features are limited by the information contained within the text. This is particularly problematic in medical imaging, where radiologists'…
▽ More
Language-supervised pre-training has proven to be a valuable method for extracting semantically meaningful features from images, serving as a foundational element in multimodal systems within the computer vision and medical imaging domains. However, resulting features are limited by the information contained within the text. This is particularly problematic in medical imaging, where radiologists' written findings focus on specific observations; a challenge compounded by the scarcity of paired imaging-text data due to concerns over leakage of personal health information. In this work, we fundamentally challenge the prevailing reliance on language supervision for learning general purpose biomedical imaging encoders. We introduce RAD-DINO, a biomedical image encoder pre-trained solely on unimodal biomedical imaging data that obtains similar or greater performance than state-of-the-art biomedical language supervised models on a diverse range of benchmarks. Specifically, the quality of learned representations is evaluated on standard imaging tasks (classification and semantic segmentation), and a vision-language alignment task (text report generation from images). To further demonstrate the drawback of language supervision, we show that features from RAD-DINO correlate with other medical records (e.g., sex or age) better than language-supervised models, which are generally not mentioned in radiology reports. Finally, we conduct a series of ablations determining the factors in RAD-DINO's performance; notably, we observe that RAD-DINO's downstream performance scales well with the quantity and diversity of training data, demonstrating that image-only supervision is a scalable approach for training a foundational biomedical image encoder.
△ Less
Submitted 19 January, 2024;
originally announced January 2024.
-
RadEdit: stress-testing biomedical vision models via diffusion image editing
Authors:
Fernando Pérez-García,
Sam Bond-Taylor,
Pedro P. Sanchez,
Boris van Breugel,
Daniel C. Castro,
Harshita Sharma,
Valentina Salvatelli,
Maria T. A. Wetscherek,
Hannah Richardson,
Matthew P. Lungren,
Aditya Nori,
Javier Alvarez-Valle,
Ozan Oktay,
Maximilian Ilse
Abstract:
Biomedical imaging datasets are often small and biased, meaning that real-world performance of predictive models can be substantially lower than expected from internal testing. This work proposes using generative image editing to simulate dataset shifts and diagnose failure modes of biomedical vision models; this can be used in advance of deployment to assess readiness, potentially reducing cost a…
▽ More
Biomedical imaging datasets are often small and biased, meaning that real-world performance of predictive models can be substantially lower than expected from internal testing. This work proposes using generative image editing to simulate dataset shifts and diagnose failure modes of biomedical vision models; this can be used in advance of deployment to assess readiness, potentially reducing cost and patient harm. Existing editing methods can produce undesirable changes, with spurious correlations learned due to the co-occurrence of disease and treatment interventions, limiting practical applicability. To address this, we train a text-to-image diffusion model on multiple chest X-ray datasets and introduce a new editing method RadEdit that uses multiple masks, if present, to constrain changes and ensure consistency in the edited images. We consider three types of dataset shifts: acquisition shift, manifestation shift, and population shift, and demonstrate that our approach can diagnose failures and quantify model robustness without additional data collection, complementing more qualitative tools for explainable AI.
△ Less
Submitted 3 April, 2024; v1 submitted 20 December, 2023;
originally announced December 2023.
-
MAIRA-1: A specialised large multimodal model for radiology report generation
Authors:
Stephanie L. Hyland,
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Mercy Ranjit,
Anton Schwaighofer,
Fernando Pérez-García,
Valentina Salvatelli,
Shaury Srivastav,
Anja Thieme,
Noel Codella,
Matthew P. Lungren,
Maria Teodora Wetscherek,
Ozan Oktay,
Javier Alvarez-Valle
Abstract:
We present a radiology-specific multimodal model for the task for generating radiological reports from chest X-rays (CXRs). Our work builds on the idea that large language model(s) can be equipped with multimodal capabilities through alignment with pre-trained vision encoders. On natural images, this has been shown to allow multimodal models to gain image understanding and description capabilities…
▽ More
We present a radiology-specific multimodal model for the task for generating radiological reports from chest X-rays (CXRs). Our work builds on the idea that large language model(s) can be equipped with multimodal capabilities through alignment with pre-trained vision encoders. On natural images, this has been shown to allow multimodal models to gain image understanding and description capabilities. Our proposed model (MAIRA-1) leverages a CXR-specific image encoder in conjunction with a fine-tuned large language model based on Vicuna-7B, and text-based data augmentation, to produce reports with state-of-the-art quality. In particular, MAIRA-1 significantly improves on the radiologist-aligned RadCliQ metric and across all lexical metrics considered. Manual review of model outputs demonstrates promising fluency and accuracy of generated reports while uncovering failure modes not captured by existing evaluation practices. More information and resources can be found on the project website: https://aka.ms/maira.
△ Less
Submitted 26 April, 2024; v1 submitted 22 November, 2023;
originally announced November 2023.
-
Exploring the Boundaries of GPT-4 in Radiology
Authors:
Qianchu Liu,
Stephanie Hyland,
Shruthi Bannur,
Kenza Bouzid,
Daniel C. Castro,
Maria Teodora Wetscherek,
Robert Tinn,
Harshita Sharma,
Fernando Pérez-García,
Anton Schwaighofer,
Pranav Rajpurkar,
Sameer Tajdin Khanna,
Hoifung Poon,
Naoto Usuyama,
Anja Thieme,
Aditya V. Nori,
Matthew P. Lungren,
Ozan Oktay,
Javier Alvarez-Valle
Abstract:
The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-s…
▽ More
The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains ($\approx$ 10% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference ($F_1$). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.
△ Less
Submitted 23 October, 2023;
originally announced October 2023.
-
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Authors:
Shruthi Bannur,
Stephanie Hyland,
Qianchu Liu,
Fernando Pérez-García,
Maximilian Ilse,
Daniel C. Castro,
Benedikt Boecking,
Harshita Sharma,
Kenza Bouzid,
Anja Thieme,
Anton Schwaighofer,
Maria Wetscherek,
Matthew P. Lungren,
Aditya Nori,
Javier Alvarez-Valle,
Ozan Oktay
Abstract:
Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-superv…
▽ More
Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quantify the quality of vision-language representations in terms of temporal semantics. Our experimental results show the advantages of incorporating prior images and reports to make most use of the data.
△ Less
Submitted 16 March, 2023; v1 submitted 11 January, 2023;
originally announced January 2023.
-
Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
Authors:
Benedikt Boecking,
Naoto Usuyama,
Shruthi Bannur,
Daniel C. Castro,
Anton Schwaighofer,
Stephanie Hyland,
Maria Wetscherek,
Tristan Naumann,
Aditya Nori,
Javier Alvarez-Valle,
Hoifung Poon,
Ozan Oktay
Abstract:
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-speci…
▽ More
Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.
△ Less
Submitted 21 July, 2022; v1 submitted 20 April, 2022;
originally announced April 2022.