Reducing Biases towards Minoritized Populations in Medical Curricular Content via Artificial Intelligence for Fairer Health Outcomes

The field of medicine is marred by a long, painful, and deleterious history of overt and covert forms of social injustice, bias, and racism, as illustrated by the American Medical Association’s recent pledge to take action to confront systemic racism (Saini 2019; Madara 2020). Studies continue to demonstrate that physicians possess implicit biases in a number of different areas such as race/ethnicity, gender, sex, age, weight, substance use and mental illness (FitzGerald and Hurst 2017). Bias in medicine leads to harmful public health costs, which are borne disproportionately by women, the economically disadvantaged, and other minoritized (Wingrove-Haugland and McLeod 2021) groups. Harmful biases are widespread among clinicians, such as assuming psychogenic causes for physical symptoms for women, minimizing pain in people of color, or other well-documented and entrenched biases in medicine (Norman 2018). This medical biased information (recently termed bisinformation, Dori-Hacohen et al. 2021) among clinicians persists with pernicious effects on health inequities despite refuting evidence.

Unfortunately, a main vector of transmission for bisinformation originates from materials that continue to be taught in medical schools, even long after being debunked by research (Redacted for anonymity, Under Review). Despite numerous calls to action to deracialize and debias medical curricula and assessment content, most medical institutions continue to teach biased medicine in the preclinical years (Tsai et al. 2016; Rodney 2016; Halman, Baker, and Ng 2017). Many educators, for example, continue to inappropriately use race as a proxy for genetics or ancestry, or even as a “risk factor” for numerous health outcomes often erroneously associated with race (e.g., Salt Gene Hypothesis) while ignoring social or structural determinants of health (SSDoH), such as systemic racism or income inequities (Ali-Khan et al. 2011; Hunt, Truesdell, and Kreiner 2013; Acquaviva and Mintz 2010; Karani et al. 2017; Metzl and Roberts 2014). Likewise, the inappropriate use of gender and sex terms perpetuates the idea that sex and gender are binary or stagnant (vs. fluid), which can potentially alienate gender-nonconforming students and patients alike.

By equating social identifiers to biology without social or structural context, medical educators are unknowingly perpetuating a curriculum that medicalizes social identities like race or gender; reifies the false conceptualization that race and gender are a biological reality rather than social constructs; and perpetuates biased knowledge and inappropriate language use. Bias reduction in curricular and assessment content is key for educating future physicians in evidence-based medicine (Le et al. 2020; Ripp and Braun 2017).

Despite the urgent need for debiasing curricular content, there are several reasons why institutions continue to struggle with this issue (Krishnan et al. 2019): (i) faculty educators may have a significant knowledge gap; (ii) faculty educators of all backgrounds may resist confronting their own implicit biases and privilege (Frey 2020); (iii) examining educational content for bias and language misuse often entails a manual review of a cross-sectional sample (Tsai et al. 2016; Martin et al. 2016), leading to high variation in assessing for bias and likely results in continued bias, since not all materials are or can be examined.

Addressing and dismantling these entrenched prejudices is paramount for advancing equitable healthcare. Thus, the objective of this research is twofold. Firstly, we undertake an empirical investigation, drawing on the expertise of trained medical professionals to systematically gather, mark, and annotate instances of bias within medical texts. This process resulted in a robust and reliable dataset that reflects the multi-faceted nature of bias within medical education. Secondly, we harness this new dataset to train artificial intelligence (AI) models for discerning bias in medical text. This approach aligns with the recently introduced Fairness via AI framework (Dori-Hacohen et al. 2021), in which AI is used carefully and deliberately to support equitable outcomes in society. Our contributions are as follows:

• •

  We curate a comprehensive labeled dataset from medical curricula amounting to two years’ worth of instructional content (e.g. lecture notes, PowerPoint slides, articles, textbooks) comprising more than 4,000 quotes annotated with codes covering a wide spectrum of biases [citation redacted for anonymity]. Quotes include positive (bias) and hard negative examples (non-bias) that were reviewed by experts before they could be labeled as ‘bias’ or ‘non-bias’.

• •

  We map those codes to labels of interest through a multi-stage preprocessing procedure guided by our experts’. We then identify the most prevalent types of biases—those related to gender, sex, race, ethnicity, age, and geographic location—to be considered in our study.

• •

  To augment the set of ‘non-bias’ examples, we use lexicons of social identifiers associated with each bias type to extract additional 4,391 quotes from the medical curricular content. Filtering with those identifiers ensures that no trivial samples are included.

• •

  We implement and evaluate four approaches for building a classifier to perform two major tasks include detecting bias from non-bias and detection the type of bias: (i) a type-specific classifier, (ii) a general bias classifier, (iii) an ensemble of bias type-specific classifiers, and (iv) a multi-task classifier that predicts both general and type-specific biases.

• •

  We provide a thorough comparison of the models with respect to accuracy, precision, recall, F1-score, F2-score and area under ROC curve (AUC). Additionally, we contrast the performance of a Transformer model (DistilBERT) with machine learning models trained on static textual embeddings (FastText).

• •

  We develop a first-in-class bias detection classifier by fine-tuning a pre-trained DistilBERT model on our curated dataset, achieving 0.923 of AUC at detecting (general) bias within medical curricula.

By intersecting rigorous data curation with advanced computational techniques, this work illuminates sources of bias in medical education and provides a scalable solution to mitigate their influence on future generations of healthcare professionals.

Artificial intelligence (AI) models have an immense potential for enhancing healthcare by assisting in diagnostics and treatment decision-making through objective analysis of medical records and integration with clinical decision support tools. Yet there is growing concern among researchers about the risks and ethical implications of applying AI in healthcare without proper governance (Althubaiti 2016; Gianfrancesco et al. 2018; Nelson 2019). A major cause for this concern is that current data collection practices used for gathering training data can ultimately lead to models that will produce biased outputs.

In this context, the term “bias” can refer to a variety of different meanings—statistical biases, psychological biases, and societal biases—all of which raise questions over the validity and reliability of some medical research. Althubaiti (2016) examines the first two types of biases in the context of epidemiological and medical research, whereas others (Gianfrancesco et al. 2018; Nelson 2019; Mittermaier, Raza, and Kvedar 2023) discuss how issues with data quality used for training models may favor certain populations in detriment of others.

Some proposals for reducing biases in medical research advocate for changes in medical education (Cavallo and Montenegro 2017; Stanciu et al. 2017). Althubaiti (2016) suggests that bias awareness should be integrated into medical education early on and stress the importance of transparency in reporting research findings. Stone and Moskowitz (2011) address the problem of non-conscious racial and ethnic bias in healthcare, proposing a workshop-based intervention for health care professionals.

Research in trustworthy and explainable AI also aims to create more reliable, fair and equitable models for healthcare in trustworthy and explainable AI. By carefully integrating quality, bias risk, and data fusion, it is possible to train more dependable models, while empowering them with the capability of providing explanations along with predictions, which can help to identify and mitigate residual biases (Albahri et al. 2023). In a similar vein, Kiyasseh et al. (2023) has developed a strategy which helps surgical AI systems avoid algorithmic bias in assessing surgical skills across diverse hospital settings and across surgeon subcohorts.

Additionally, advocacy for formal AI governance arises to ensure responsible use and accountability, alongside potential federal legislation for assessing algorithmic bias risks. Nelson (2019) argues that, in order to prevent bias and misuse, clinicians must play a critical role in overseeing AI algorithms and validating their use in healthcare, whereas Kiyasseh et al. (2023) calls for the need of explainable models so that regulatory bodies, such as the FDA, can provide and validate frameworks to manage these biases effectively.

Complementary to the previous ones, Dori-Hacohen et al. (2021) present a multifaceted approach towards establishing fairness in AI applications by merging insights from medical education, sociology, and antiracism, as part of a new framework. They introduce “bisinformation” as a new concept distinct from misinformation and call for research into its nature and mitigation. They advocate for the use of AI to identify and rectify biased or harmful information that adversely affects minority groups. This is the approach we adopt in the present paper, in which we demonstrate how AI models can be used for detecting biases in medical text, as formally defined in the next section.

Motivated by the mandate to debias the medical curriculum at a large medical school¹¹1Institution name redacted for anonymity., our goal is to review medical educational content in an automated fashion and flag sentences with biased or potentially biased content, which will then be provided to experts for careful review²²2This “expert-in-the-loop” approach means that we have a preference for higher recall, and are willing to tolerate some reduced precision in order to cast a wide net. False positives will subsequently be corrected by expert review at a later stage..

We will define the problem formally as follows. Let $x$ be an educational medical text excerpt, which can contain a claim, case report, incidence statistics, or any other textual content. Let $t$ be a social identity which can be the target of bias, such as “gender,” “race,” etc. Now, let $bias(x,t)\in\{true,false\}$ be a binary variable denoting whether excerpt $x$ exhibits bias with respect to a social identity $t$, or not.

As one example, consider an excerpt in a skin cancer module that reads: “Although the incidence is lower in patients of color, the morbidity/mortality can often be higher”. This excerpt was labeled by our expert annotators (see Section 4.2 for more details) as biased with respect to race, but not with respect to other identities (gender, geography, etc.). The annotator comment indicates: “No source for claim. Consider explaining more beyond ‘patients of color’ as this may come across as grouping every minority group into one and making a generalized statement.”

By extension, we can also define $bias(x,\mathcal{T})\in\{true,false\}$ with respect to a set of social identities $\mathcal{T}=\{t_{1},t_{2}..t_{m}\}$ in the following manner:

Given these definition, we now define a set of bias classification tasks as follows:

• •

  Type-specific Bias Classification: Given a social identity type $t$ and a text excerpt $x$, construct a classifier that produces a prediction for $bias(x,t)$.

• •

  General Bias Classification: Given a set of social identities $\mathcal{T}$ and a text excerpt $x$, construct a classifier that produces a set of predictions $bias(x,\mathcal{T})$.

Therefore, given a set of $\mathcal{T}=\{t_{1},t_{2},\ldots,t_{m}\}$, we wish to construct $m+1$ classifiers ($m$ type-specific classifiers and one general one).

To understand how we can construct effective machine learning (ML) models for detecting bias in medical curricular content, we investigate a few different strategies for building and training models. In this section, we present these strategies and detail our experimental setup.

As described in Section 3, we refer to a total of 7 tasks:

In this section, we present the results of our proposed type-specific, general, ensemble and multi-task learning models and compare their performance with a baseline model in the context of specific and general bias detection tasks. Through a detailed analysis employing ROC curves and AUC metrics, we evaluate the effectiveness of our models across different folds of validation, thereby ensuring robustness and consistency in our evaluations.

Firstly, we discuss the performance of models in the detection of specific bias types. The first comparison focuses on the binary classification models. Table 4 shows their performance across all of our experiment settings. We found that training with only the XN negative category leads to the high recall, at the detriment of lower precision (except for type ‘ethnicity’), ultimately leading to low F1-score. Between training with AN and AN-RN, we find the results to be generally comparable, however, AN usually yields AUC that is higher or comparable to that achieved by AN-RN.

Based on the previous observation, we choose the setting AN to compare the binary classifier with the Multi-Task and baseline methods. Table 5 presents the results of the comparison. Both the type-specific and multitask model greatly outperformed the baseline model. While MTL often led to the best precision, the binary classifier has outperformed MTL in all the other metrics. Furthermore, the AUC for the latter model across different tasks varied from 0.896 to 0.951, which is considered very effective at detecting bias.

Last, we compare the proposed models and the baseline on the general bias classification task. Table 6 shows the performance results for the different experiment settings and metrics. Once again, the baseline did not perform well. Considering only the proposed methods, the AN setting generally led better precision, whereas the XN setting led to better recall. Not surprisingly, the Ensemble method achieves the best overall recall, since the logical OR will cause the model to be highly sensitive, leading to the lowest precision in our results. Based on the F1-Score, F2-Score and AUC, the best result is achieved by the binary classifier model with the AN setting. This result is consistent with that for type-specific bias classification. For other settings (i.e., XN and AN-RN), MTL yields similar F2-Score and AUC to the binary classifier.

We include ROC curves for the baseline and the proposed methods (except for the Ensemble method, since their predictions are based on the logical OR and cannot be directly converted to probabilities) in Figure 5. These figures highlight the superiority of the binary classifier, particularly with the AN setting. Based on Figure 5(middle), we can see that this combination would allow one to choose a threshold that achieves a high true positive rate without incurring in high false positive rates.

Our investigation into bisinformation in medical education has revealed a significant issue that, if left unaddressed, can continue to negatively impact health quality, equity, and outcomes. Through a detailed empirical study, we have made substantial progress in identifying and mitigating bias in medical education materials related to gender, sex, age, race, and ethnicity.

Our work represents the first comprehensive attempt to remove ingrained biases from medical curricula using a systematic and scalable AI-based approach. This paper provides numerous contributions: not only a robust dataset and bias detection classifier but also a pioneering methodology that combines modern AI techniques with a deep understanding of medical educational material. Our proposed method has demonstrated a significant capability in distinguishing biased content, as confirmed by our performance evaluations.

Moreover, our research extends beyond academic boundaries and ventures into the realm of public health and online communities. By adapting our classifier to comprehend and interpret the subtleties of social media platforms like X (formerly Twitter), we strive to bridge the gap between conventional medical instruction and contemporary digital communication channels. This is of utmost importance, especially in the face of current public health challenges and the rampant dissemination of bisinformation.

The present paper lays the groundwork for eliminating biases from medical curricula and sets the stage for a paradigm shift in medical education. In this context, our models make the first strides towards leveraging artificial intelligence to promote fairness and integrity in medical education by addressing biases that disproportionately affect vulnerable communities, particularly those underserved and often misrepresented in medical literature, has been pivotal. By advocating for those most impacted by biases, we ensure that the outcomes of our work are both socially informed and ethically sound.

Our future work will delve into visual content, extending our analytical lens to the biases inherent in images within medical curricula. We recognize that imagery can convey powerful and often subconscious messages. We aim to gain a deep understanding of how visual representations contribute to the perpetuation of bias. Additionally, we will investigate other potential biases in medical texts, such as those related to socioeconomic status, disability, neurodiversity, and intersectionality, to further enhance our understanding of bias’s multifaceted and complex nature in medical education.

Lastly, another critical avenue for our forthcoming research is examining how biased data proliferates through online social media platforms. The velocity and volume with which information circulates in the digital space can amplify the dissemination of biased content, leading to widespread misconceptions and reinforcing harmful stereotypes. We plan to employ and enhance our models to track and analyze the trajectory of such misinformation on social media. Through this, we aim to contribute to developing more robust countermeasures against the spread of medical misinformation, ultimately helping safeguard public health discourse in the digital age.

We include the following information for context on the ethics of our project.