CheXpert Plus: Augmenting a Large Chest X-ray Dataset with
Text Radiology Reports, Patient Demographics and
Additional Image Formats

CheXpert Plus is a dataset that pairs text and images, featuring 223,228 unique pairs of radiology reports and chest X-rays from 187,711 studies and 64,725 patients. A single patient may be linked to several studies, and each study may include multiple chest X-rays. The dataset comprises:

• •

  223,228 unique chest X-ray images in DICOM format, each featuring 47 DICOM metadata elements. These images are also be made available in PNG format(Section 3.2).

• •

  187,711 unique radiology reports, each report divided into subsections extracted from the original corresponding report (Section 3.3).

• •

  64,725 unique patients, with 8 de-identified demographic data points (Section 3.4).

• •

  187,711 unique annotations for 14 different chest pathologies using CheXbert. (Section 3.5).

• •

  187,575 unique RadGraph annotations for impression sections and 47,328 unique RadGraph annotations for finding sections, using the pretrained RadGraph model (Section 3.6).

• •

  a plurality of models trained on these data sources for key radiology tasks (Section 3.7).

CheXpert Plus comprises 223,228 unique images available both in DICOM format and in PNG format.

The DICOM format includes image metadata attributes that encapsulate essential information for medical image processing and interpretation. In total, we release up to 47 DICOM metadata elements, listed in Appendix B. With these image metadata attributes, it is possible to convert the DICOM pixel data into the PNG format. While the DICOM format offers the most comprehensive data, the PNG format may be more straightforward to integrate into existing training pipelines.

CheXpert Plus contains 187,711 distinct radiology reports, each corresponding to a separate study. For studies comprising more than one image, the report compiles information extracted from all the images in the study. Therefore, the findings described in the report may describe findings from multiple images. Additionally, a single study may be preceded by one or more related studies, especially in cases where a patient has undergone multiple examinations. To facilitate the analysis of a disease’s progression over multiple studies, we have introduced the field $patient\_report\_date\_order$, which ranks the studies of each patient in chronological order. This enhancement makes CheXpert Plus compatible with ML approaches that condition their predictions not only on a single study but also on its prior studies.

Section names were not used consistently by the radiologists producing these reports. Therefore, the reports from CheXpert Plus may contain as many as 11 distinct sections:

Narrative: This section outlines the type of exam and the date it was conducted.

Clinical History: This details the reasons the ordering provider requested the radiology exam. It typically includes patient metadata and symptoms.

History: Similar to the Clinical History, this section explains the reasons for the radiology exam provided by the requesting provider, often incorporating patient symptoms and a reason for the exam.

Comparison: Here, any previous studies that were used as part of the interpretation are listed.

Technique: This section provides details on how the exam was conducted, including the views captured and whether contrast material was injected.

Procedure Comments: Noted here are additional details about the procedure, such as the number of views.

Findings: Observations reported by the radiologist are listed here in detail.

Impression: This section summarizes the findings and the radiologist’s interpretation.

End of Impression: Closing remarks and often the reporting radiologist’s name are included in this section.

Summary: A brief overview of the study is provided here, sometimes alongside a number that classifies the study as normal, abnormal, or extremely abnormal.

Accession Number: A de-identified accession number is given. Although this section holds limited informational value for any ML models, it is retained to preserve the original format of the report.

Aggregated statistics for each section are made available in Table 2. In CheXpert Plus reports, an impression is almost always present.

Alongside the de-identified reports and images, CheXpert Plus includes de-identified demographic data. Each study is linked to the corresponding demographic data using a privacy-preserving identifier created during the de-identification process. While a patient’s insurance may have varied over time, the data contains their insurance status as of February 2024.

As listed in Figure 1, CheXpert Plus lists patient age, sex, race, ethnicity, insurance type, BMI, deceased status, and the need for an interpreter. Collectively, this demographic data allows researchers to account for subgroups of patients and improve the training of ML models relying on CheXpert Plus data sources, enhancing their fairness and robustness.

CheXpert Plus lists 14 pathology labels, generated from the radiology reports using model-based extraction methods. Table 3 presents the prevalence of each pathology based on the CheXpert labeler Irvin et al. (2019). For each disease, the label can be positive (1), uncertain (-1), negative (0) or not mentioned.

In addition to these labels, CheXpert Plus also includes labels generated from CheXbert Smit et al. (2020). Due to CheXbert’s maximum input token size of 512 tokens, different parts of the report are used as input: the full report, the findings, the impression, as well as concatenations (findings - impression) and (impression - findings). When an input exceeds 512 tokens, the excess tokens are not used by the labeler.

To assess the performance of each label, these labels are compared against two human-annotated test sets. The first test set comprises 1000 samples annotated by 2 board-certified radiologists based on the radiology images only, with disagreement resolution through consensus. The second test set includes 500 samples annotated by 8 board-certified radiologists based on the radiology reports only, with a majority vote of 5 radiologists. In order to have meaningful comparisons, the test sets are restricted to instances where both findings and impression are available, leading to 154 and 339 labeled samples respectively. In order to simplify this multi-label classification problem, Not Mentioned labels are assigned to Negative and Uncertain labels to Positive. The results of this analysis are detailed in table 4.

For prediction tasks based on the reports, we recommend using the CheXbert-impression labels, which achieve the highest F1-score (0.93) when evaluated against the text-based test set. On the contrary, for prediction tasks based on the images, we suggest using the CheXbert-findings labels, which achieve an F1-score of 0.44 on the image-based test set. The performance gap compared to the text-based test set indicates that the radiology reports themselves might not encapsulate all the data needed to accurately classify the presence or absence of a pathology, and that labels generated synthetically from radiology reports might not be reliable for training image pathology classifiers using supervised learning . We encourage research aiming at improving the synthetic generation of pathology labels based on text.

The Table 5 showcases the RadGraph Jain et al. (2021) annotations released as part of CheXpert Plus for the Findings and Impression sections of the radiology reports. The most common annotations are Anatomy (46.81% in Findings and 43.27% in Impression) and Observation: Definitely Present (43.30% in Findings and 44.06% in Impression). Observation: Uncertain and Definitely Absent categories are less frequent, ranging from 3.72% to 7.47%. Relation annotations like Modify (59.47% in Findings and 59.16% in Impression) dominate over Located at and Suggestive of. The total entity annotations are significant, with 1,581,863 in Findings and 3,999,559 in Impression, while total relations annotations are 1,106,659 (Findings) and 2,772,337 (Impression), demonstrating the comprehensive nature of the dataset.

The Table 5 showcases the generated RadGraph annotations Jain et al. (2021) on our dataset. These annotations cover the Findings and Impression sections of radiology reports. The most frequent annotations are "Anatomy" (46.81% in Findings and 43.27% in Impression) and "Observation: Definitely Present" (43.30% in Findings and 44.06% in Impression). The categories "Observation: Uncertain" and "Definitely Absent" are less common, with frequencies ranging from 3.72% to 7.47%. Relation annotations, such as "Modify" (59.47% in Findings and 59.16% in Impression), are more prevalent than "Located at" and "Suggestive of." Entity annotations amount to 1,581,863 in Findings and 3,999,559 in Impression, while the total number of relation annotations are 1,106,659 (in Findings) and 2,772,337 (in Impression).

As part of CheXpert Plus , we are releasing pretrained models trained on CheXpert Plus data, incorporating recent developments in machine learning spanning natural language processing, image recognition, and generative modeling. Among these releases are a pretrained LLaMA Touvron et al. (2023) model, which generates human-like text and excels at complex language tasks. A pretrained CLIP Radford et al. (2021) model is being introduced, which improves the way visual concepts are learned from text descriptions, thereby enhancing image search and classification capabilities. The lineup also includes a pretrained VQ-GAN Esser et al. (2021) model, blending VQ-VAE and GAN technologies to produce realistic images and demonstrating generative power. The release also includes a pretrained DINOv2 Oquab et al. (2023) model that employs self-supervised learning with Vision Transformers to achieve robust visual representations. Finally, several architecture-agnostic models are included, todeliver competitive performance in radiology report generation (RRG) and radiology report summarization (RRS).
These models are available at the following URL: https://github.com/Stanford-AIMI/chexpert-plus.

Building upon the foundational CheXpert dataset introduced in 2019, which comprised 224,316 chest radiographs from 65,240 patients annotated for 14 observations, our enhanced version, CheXpert Plus, introduces several significant advancements aimed at pushing the boundaries of medical imaging research. By transitioning to DICOM, the gold standard in medical image formats, we provide images alongside a subset of their original DICOM headers, thereby offering richer image metadata and superior image quality. CheXpert Plus further enriches the dataset by including the corresponding radiology reports, which we have parsed into sections such as medical history, findings, and impressions, enhancing the dataset’s utility for comprehensive analysis. Moreover, by incorporating detailed patient demographic data, CheXpert Plus facilitates the development of fairness-focused analyses and multimodal models capable of utilizing this information for more informed and nuanced diagnoses.

In addition, CheXpert Plus also focuses on releasing higher quality extracted labels, be it for the 14 lung diseases thanks to the CheXbert annotation tool, or for the more recent RadGraph annotations, allowing the training of classifiers directly on top of CheXpert Plus or for CheXpert- and RadGraph-based metrics to be computed.

These enhancements not only deepen the analytical potential within the medical imaging field but also significantly contribute to the evolving medical AI community, particularly as it strides towards leveraging multimodal and image-text learning paradigms, promising to enhance diagnostic accuracy and patient outcomes.

Finally, we mention that 186 studies are missing compared to the original CheXpert release, as the corresponding reports could not be recovered when preparing the CheXpert Plus release.

While numerous chest X-ray datasets exist (Table 1), few compare CheXpert Plus in terms of scale and modality inclusiveness. Among these, MIMIC, OpenI, and PadChest also provide radiology reports, yet PadChest’s utilization of Spanish and OpenI’s smaller scale differentiate CheXpert Plus significantly. Additionally, besides MIMIC and OpenI, only CandidPTX, and VinDR offer X-rays in DICOM format and neither matches CheXpert Plus in the sheer volume of studies.

MIMIC, the closest equivalent, features 377,095 chest X-rays with accompanying reports and includes both DICOM images and patient demographic information. CheXpert Plus  surpasses MIMIC in textual depth, containing 36 million text tokens compared to MIMIC’s 34 million. MIMIC does features a higher number of reports (227,821 unique reports compared to CheXpert Plus’s 187,711 unique reports), these reports are shorter, especially in their impression section: MIMIC impressions count 7,986,317 total tokens, versus the 13,351,758 tokens of CheXpert Plus’s impressions. For any text-image radiology task where the text being handled is restrained to the impression section, CheXpert Plus is therefore also the largest dataset available. Finally, our reports underwent a de-identification process that did not alter their structure, even including de-identified accession number sections, something that is not displayed in MIMIC and makes our reports closer to the natural textual data handled during radiology exams.

Given these distinctions and advancements, CheXpert Plus represents a valuable addition to the existing landscape of medical imaging datasets, poised to significantly enhance and expand the capabilities of research in medical AI.

The release of DICOMs required the de-identification of all metadata DICOM headers, as well as their pixel content, as displayed in Figure 2. The metadata headers were automatically de-identified and reviewed by humans to confirm that no PHI information was present (corresponding to steps 5 and 6 of Figure 2). Along that, we guarantee that the pixels in the released DICOMs are identical to those in the original CheXpert 1.0 images, that were themselves cleared for public release by hiding any PHI content (steps 7 and 8). The code used to accomplish this is available in Appendix A.

As part of the CheXpert Plus release, the reports associated with the CheXpert images underwent a de-identification process that lasted for a year and was supported by 25-30 human annotators. We counted a posteriori the presence of 853,878 total PHI spans, as defined in the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Details of the types of PHI spotted in these reports are further displayed in Table 6.

To the best of our knowledge, this is the largest text de-identification effort in terms of quantity of PHI reviewed. As displayed in Figure 2, these reports underwent a process in 4 steps in order to be considered de-identified and ready for public release.

Step 1 The reports were first automatically de-identified using a two-step model  Chambon et al. (2022b) that first leverages a transformer model for token-level classification into PHI categories, before replacing the true PHI spans with synthetic PHI as a "Hide in plain sight" approach  Carrell et al. (2013). This latter approach adds an additional safety factor to ensure any missed PHI, if any, cannot be easily spotted.

Step 2 Human annotators reviewed each report with the synthetic PHI highlighted by the automatic de-identifier The human reviewers identified any PHI that was missed by the algorithm. A missed PHI can be a full span belonging to a certain PHI category, or any prefix or suffix of a PHI span partially detected by the model. Any missed PHI was highlighted by the annotators for further human review. Out of the 853,878 true PHI spans in all reports, 23 were fully missed by the model (0.002%), and 841 were partially missed by the model (0.01%).

Step 3 Any partially or fully missed PHI then underwent the same "Hide In Plain Sight" step to replace the remaining true PHI spans by synthetic PHI spans.

Step 4 A mapping of each pair of true and synthetic PHI spans was generated and reviewed for PHI by a board-certified radiologist. This combination of automated processes and human review confirmed that all true PHI spans had been replaced by synthetic PHI spans.

CheXpert Plus dataset is being released on the Stanford AIMI Shared Dataset website and can be accessed at the following link. It is associated with a Stanford University Dataset Research Use Agreement, which specifies that CheXpert Plus may not be used for any commercial purposes and is only available for research uses. In particular, you may not distribute, publish or reproduce a copy of this dataset.

As part of the CheXpert Plus release, we underline the following main uses that can be made out of it:

Performance: Due to its extensive size, CheXpert Plus almost double the amount of publicly available English text-image pairs in radiology. Therefore, any model can leverage this dataset to improve in performance compared to models trained before this data release.

Robustness: Along MIMIC, CheXpert Plus is the only large size english text-image dataset in radiology, therefore allowing to not only test models on multiple institutions but also perform cross-institution training, hopefully leading to more robust performance when evaluating on completely new institutions.

Fairness: The inclusion of extensive amounts of patient basic demographic data in CheXpert Plus enables downstream applications to account for the imbalance of patient ethnicity, sex, age or socio-economic background, therefore potentially limiting the bias of models trained for various radiology tasks.

Finally, we underline that any model trained with the help of CheXpert Plus data may still reflect biases based on patient characteristics and pathologies, among else. When using such models, researchers should always look for sources of potential distribution shifts and audit for peformance disparities based on attributes such as race, ethnicity, age or socio-economic background.

First, as described in Section 3.3, the focus was put on collecting reports with detailed impression sections, at the cost of having lots of findings. Therefore, impressions are in total of tokens twice as important as findings. Second, some pathologies such as fracture, lung lesion and pleural other are significantly under-represented, as displayed in Section 3.5, which limits the performance of any models aiming at studying these particular pathologies.