BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs

Recent research has revealed the tendency of LLMs to manifests as discriminatory language or stereotypes against certain vulnerable groups philological1870transactions. These unfair treatments or disparities may be derived from the data they were trained on, which reflects historical and structural power asymmetries of human society gallegos2023bias.

Many efforts have been made to evaluate the fairness of LLMs, which can be categorized into two parts: embedding or probability-based approaches and generated text-based approaches. (1) Embedding or probability-based approaches evaluate LLM by comparing the hidden representations or predicted token probabilities of counterfactual inputs. Methods include computing the correlations between static word embeddings caliskan2017semantics or contextualized embeddings may2019measuring; guo2021detecting with different social groups, comparing the predicted probabilities for counterfactual tokens (e.g., man/woman) via fill-in-the-blank task nadeem2020stereoset, or comparing the predicted pseudo-log-likelihoods between counterfactual sentences nangia2020crows. (2) Generated text-based approaches evaluate LLM by providing prompts (e.g., questions) to a generative LLM and ask the LLM to provide sentence completions dhamala2021bold or select an answer to a question wan2023biasasker. Then, bias is calculated based on the generated texts by calculating co-occurrence distributions difference bordia2019identifying; liang2022holistic, comparing word frequency according to the pre-defined lexicon nozza2021honest; dhamala2021bold (or scoring with a trained classifier).

However, existing approaches still face many limitations. First, many studies indicate that, bias evaluated by embedding or probability-based approaches have a weak relation to bias in downstream text-generation tasks cabello2023independence; delobelle2022measuring; kaneko2022debiasing; blodgett2021stereotyping, which limits their generalizability and reliability. On the other hand, generated text-based approaches still rely on fixed-form inputs and outputs (i.e., choices or lexicon), as it is difficult to assess the bias of content in open text generation scenarios wan2023biasasker; fang2024bias; parrish2021bbq; kaneko2024evaluating; li2020unqovering. To bridge this gap, we propose a plug-and-play bias detection tool, BiasAlert, which can automatically and reliably detect bias in open generated text, thus enabling bias evaluation for the most common LLM deployment scenarios, such as sentence completion, online dialogue, and question answering.

Notably, LLM-as-Judge is another potential paradigm for bias detection, which leverages the capabilities of LLMs to achieve data-independent automatic detection through fine-tuning or prompting lin2024indivec; inan2023llama; achiam2023gpt; touvron2023llama. However, our experiments in Section 4.2 demonstrate that current LLMs still struggle to accurately identify social biases due to a lack of human social ethical knowledge. Additionally, existing bias detection strategies often lead LLMs to "over-avoid" sensitive issues, impairing their usability. Perspective API and Llama-guard are two automatic detection tools similar to our approach. However, they focus on detecting toxicity and safety, respectively, and perform poorly on fairness data, as demonstrated in Section 4.2.

We propose to construct a comprehensive retrieval database that encompasses a wide range of human social biases, aimed at providing reliable decision reference for our detection tool. The database can be easily updated in real-time, ensuring that detection results more accurately and timely reflect ethics of human society. Specifically, we first leverage bias data from existing social bias datasets that are reliably annotated by humans and widely used. Then, we standardize the collected data, simplifying the text into refined corpora (i.e., texts with respect to the target demographic group and biased descriptions). Finally, we integrate the corpora labels according to the type of bias. Here are some examples from our constructed database:

’black people are niggers’, racial bias; ’women can’t handle drugs’, gender bias.

The constructed bias retrieval database encompasses seven types of biases including gender, race, religion, and culture, with a total of 41,000 entries. Each entry corresponds to a specific biased corpora and its annotation. We use the Contriever encoder izacard2021unsupervised to embed the contents of the retrieval database. During the bias detection with BiasAlert, we first use Contriever-MSMARCO to query the top five most relevant entries from the retrieval database. These retrieved entries, representing potential biases within the input text, are then input into the bias detection model along with the query to guide the model in determining whether these biased viewpoints are present in the input sentence.

Overall, the retrieval database directly provides comprehensive knowledge of human societal biases to the bias detection model, serving as the basis for its decisions. This ensures that the model’s judgments are not solely reliant on knowledge acquired through training or fine-tuning, which is not reliable, attributable and hard to update. Ablation studies in Section 5.1 demonstrates the necessity of external retrieval database, where we found that even the most powerful language models, such as GPT-4, are unable to accurately detect biases without the aid of external retrieval.

We propose to construct BiasAlert using an instruction-following paradigm, aiming to leverage external retrieval information combined with the LLM’s internal reasoning and language understanding capabilities to achieve reliable, adaptive, and attributable bias detection. With the content to be checked and retrieved references as inputs, we designed step-by-step instructions. We first define bias and judgment criteria in the instruction, then guide the model to understand the content by identifying specific groups and potential biased descriptions within the content. Subsequently, we instruct the model to review the retrieved references and make judgments about the content. Additionally, we employ in-context demonstrations, to help it better understand and adapt to diverse and complicated scenarios, and enhance the model’s generalization ability. Here is an example of our constructed instruction:

Training and Inference. In the training phase, we utilize the constructed instruction-tuning dataset, which combines instruction, retrieved references, and in-context demonstrations, to fine-tune the pretrained LM. In the inference phase, for each piece of generated content to be analyzed, we first query the retrieval database for the top five social biases as references. Then, the generated content is input into the trained BiasAlert along with the references. Finally, BiasAlert reports whether the generated content contains bias, as well as the type and content of the bias.

Table 1 displays the comparative results on two bias detection datasets. In terms of efficacy score, it is evident that almost all baselines struggle to achieve accurate detection, suggesting that the internal knowledge of the models is insufficient for judging human social biases. In comparison, BiasAlert achieves significantly better results than all the baselines, demonstrating the superiority of our framework that combines external knowledge with internal reasoning capabilities. Regarding the classification score and attribution score, it is observed that BiasAlert not only surpasses the baselines but also achieves nearly perfect performance. This indicates that the model’s judgments align with human annotations, confirming the reliability and interpretability of the detection results. On the more challenging Crows-pairs dataset, BiasAlert also performs better than all models except for the OpenAI Safety API.

We conduct a set of ablation studies to evaluate the efficacy of our proposed methods, with results presented in Table 2. First, we investigate the effect of retrieved social bias knowledge on bias detection, conducting experiments on Llama2-chat (base model of BiasAlert) and GPT-4. We observe a significant performance disparity between scenarios with and without retrieval, particularly in terms of the overall score. These findings underscore the necessity of external human social ethical knowledge for LLMs to accurately and reliably detect biases. Furthermore, we discover that step-by-step Instruction significantly enhances performance, especially in the Classification Score and Attribution Score. This suggests that our designed step-by-step instructions effectively stimulate the internal reasoning capabilities of LLMs to understand the input generations. Although the improvements from in-context demonstration are relatively modest, the results demonstrate its effectiveness in guiding the LLMs to generate answers that better align with expectations.

We describe the scenario of applying BiasAlert in the online dialogue of LLMs to reduce potential biased answer. Specifically, in the online dialogue between humans and LLMs, BiasAlert will monitor each response from the LLM. If a response exhibits bias, an alert will be triggered, and the current generation of the LLM will be terminated.