Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias

Jayanta Sadhu, Maneesha Rani Saha, Rifat Shahriyar
Bangladesh University of Engineering and Technology (BUET)
{1705047, 1805076}@ugrad.cse.buet.ac.bd, [email protected]

Abstract

The rapid growth of Large Language Models (LLMs) has put forward the study of biases as a crucial field. It is important to assess the influence of different types of biases embedded in LLMs to ensure fair use in sensitive fields. Although there have been extensive works on bias assessment in English, such efforts are rare and scarce for a major language like Bangla. In this work, we examine two types of social biases in LLM generated outputs for Bangla language. Our main contributions in this work are: (1) bias studies on two different social biases for Bangla (2) a curated dataset for bias measurement benchmarking (3) two different probing techniques for bias detection in the context of Bangla. This is the first work of such kind involving bias assessment of LLMs for Bangla to the best of our knowledge. All our code and resources are publicly available for the progress of bias related research in Bangla NLP. ¹¹1https://github.com/csebuetnlp/BanglaSocialBias

Jayanta Sadhu, Maneesha Rani Saha, Rifat Shahriyar Bangladesh University of Engineering and Technology (BUET) {1705047, 1805076}@ugrad.cse.buet.ac.bd, [email protected]

1 Introduction

The rapid advancement of Large Language Models (LLMs) has significantly impacted various domains, particularly in social influence and the technology industry Kasneci et al. (2023); Dong et al. (2024b). Given their growing influence, it is crucial to ensure LLMs are free from harmful biases to avoid legal and ethical issues Weidinger et al. (2021); Deshpande et al. (2023). In the context of computing systems, bias is where sociotechnical systems systematically and unfairly discriminate against certain individuals or social groups in favor of others Friedman and Nissenbaum (1996); Blodgett et al. (2020). Hence, analyzing bias and stereotypical behavior in LLMs is vital for identifying and mitigating existing biases, thereby fostering the development of responsible and ethical models.

Bangla, the sixth most spoken language globally with over 230 million native speakers constituting 3% of the world’s population²²2https://w.wiki/Psq, is underrepresented in NLP literature due to a lack of quality datasets (Joshi et al., 2020). This gap limits our understanding of bias characteristics in language models, including LLMs. The need to measure bias in Bangla arises from this gap. Historically, societal views in Bangla-speaking regions have undervalued women, leading to employment and opportunity discrimination (Jain et al., 2021; Tarannum, 2019). Additionally, the region’s significant religious diversity, primarily among Muslims and Hindus, makes Bangla a valuable case study for examining gender and religious biases, two important social biases.

In this study, we pose the question, does multilingual LLMs exhibit gender and religious bias when prompted with Bangla?. To address this, we present: (1) a curated dataset specifically designed to detect gender and religious biases in Bangla, (2) thorough bias probing analysis on both popular and state-of-the-art closed and open-source LLMs, and (3) an empirical study on bias through LLM-generated responses.

Our findings reveal significant biases in LLMs for the Bangla language and highlight shortcomings in their generative power, underscoring the need for future de-biasing efforts.

2 Related Work

Several works on bias measurements in language models are done in recent years Mehrabi et al. (2021). Existence of gender bias has been exposed in tasks like Natural Language Understanding Bolukbasi et al. (2016); Gupta et al. (2022); Stanczak and Augenstein (2021) and Natural Language Generation Sheng et al. (2019); Lucy and Bamman (2021); Huang et al. (2021) .Benchmarks such as WinoBiasZhao et al. (2018) and WinogenderRudinger et al. (2018) have been used to measure gender biases in LMs. Preliminary studies on religious and ethnic biases were done in some works. (Milios and BehnamGhader, 2022; Navigli et al., 2023; Abid et al., 2021). Works like Nadeem et al. (2020); Nangia et al. (2020) provide frameworks and datasets for different types of biases including gender and religion.

Recent studies have conducted experiments on determining gender stereotypes in LLMs Kotek et al. (2023); Ranaldi et al. (2023); Dong et al. (2024a) and debiasing techniques Gallegos et al. (2024); Ranaldi et al. (2023), but most of them are on English. There are a few works on multilingual settings Zhao et al. (2024); Vashishtha et al. (2023), but such efforts are not common for Bangla. The most preliminary work on Bangla bias detection is found in the works of Sadhu et al. (2024), that includes static and contextualized embeddings. Notable works on bias evaluation dataset creation are done in Das et al. (2023).

3 Data

We use two strategies for LLM probing: Template-based and Naturally sourced. The template-based approach uses curated templates for gendered persona or religious group predictions to reveal biases in LLMs. Naturally sourced sentences, on the other hand, are used to make explicit predictions about groups or genders, helping to understand the LLM’s ability to interpret natural scenarios.

Template Based: We create semantically bleached templates with placeholders for specific traits that are filled with adjective words to create prompts. We use these prompts for gendered role prediction or religious identity prediction. The adjective words encompass different categories used to describe human nature e.g. Personality Based, Outlook Based and Communal Based. We also include Occupation as a category to understand the association between gender and religion with occupation. We create templates to intermix human professions with traits to study if the addition of occupation has any effect on role prediction. We were careful to not induce any stereotypes into the templates and make sure all the adjective words and occupations are equally likely for any gender or religious community. In total, we have 2772 template sentences by combining both the categories (see appendix A.1 for more details).

Naturally Sourced: We use the BIBED dataset Das et al. (2023), specifically the data from Explicit Bias Evaluation(EBE) as a source of naturally occurring scenarios. The sentences are structured in pairs, each containing one identifying subject from a group of either male-female words(for gender) or Hindu-Muslim words (for religion). We replace the main identity dimension in the sentence with _ (gap) and give the model options to select between the possible identities. We provide the examples in appendix A.2. However, there are many instances where one a sentence without the subject is not equally probable for both the contrasting identities. In order to curate sentences that serve our cause, we manually select these sentences to provide equal opportunity. Details of the selection process is given in appendix A.3. After the curation process, we are left with 2600 pairs for gender and 1627 pairs for religion.

[Uncaptioned image] — Table 1: The prompt template and an example of prompt for gender role prediction (Note that the translations are only for understanding and not used in prompting)

4 Experimental Setup

4.1 Model Selection

For our experiment we provide results for three state-of-the-art LLMs: Llama3-8b (version: Meta-Llama-3-8B-Instruct ³³3meta-llama/Meta-Llama-3-8B-Instruct) (AI@Meta, 2024), GPT-3.5-Turbo ⁴⁴4gpt-3-5-turbo and GPT-4o ⁵⁵5gpt-4o. Since Bangla is a low resource language, not many models could generate the expected response we required. For our experimentation, we also tried some other models (mentioned in limitations) but none could produce any presentable result that serves our purpose.

4.2 Prompt

We design prompts to best fit the generation task. In the case of template based probing, we prompt the model for gendered role or religious identity selection, and in the case of naturally sourced probing, we use fill in the blanks approach. We provide an example of prompt creation for gender in template based probing in table 1.

Since the task of token prediction and generation is a stochastic process, we repeat each prompt twice to bring stable result. In table 2 we provide the number of prompts for each model.

Probing Method	Category	# Prompts
Template Based	Gender	4256
Template Based	Religion	1288
Naturally Sourced	Gender	5200
Naturally Sourced	Religion	3254

Table 2: Probing Methods, Categories, and Number of Prompts for each LLM

4.3 Evaluation Setup

Evaluation Metric: We use a widely used fairness measurement metric, Disparate Impact(DI) Feldman et al. (2015) for evaluation. It is computed by $\frac{P(Y=1|S\neq 1)}{P(Y=1|S=1)}$ The identifiers of our experiments are all binary (e.g: male-female, hindu-muslim), so we can apply this to our cause with empirical estimation. In task Q, for a category a, where the possible outcomes are x, y, the DI calculation is

DI_{Q}(a)=\frac{P(Q=x|a)}{P(Q=y|a)}

The empirical estimation we are using is

DI_{Q}(a)=\frac{C_{x}(a)}{C_{y}(a)}

where $C_{z}$ represents the occurrence frequency of z class elements. For our experiments, we consider $DI_{G}$ and $DI_{R}$ for gender and religion predictions and $(x=female,y=male)$ and $(x=Hindu,y=Muslim)$ for the two bias evaluations respectively. For any fair LLM, it is obvious that the DI score for any category would be close to 1.

5 Results and Evaluation

Refer to caption — (a) DI Scores for Gender Bias(Positive Traits)

5.1 Template Based Probing Results

We present the template based results in figure 1. We report the results based on 6 different categories and include the results for positive and negative traits separately for more nuanced variations.

Gender Bias: Our findings indicate GPT-3.5 exhibits the most gender bias, with a disparity impact score above 1 in all categories, suggesting a bias towards the feminine gender. Llama-3 shows significant bias in the opposite direction, with scores well below neutral, indicating male gender bias. GPT-4o displays less gender bias, with scores close to 1 in some instances. Adding occupation to the probing strategy generally reduces bias across most categories. GPT-3.5 moves closer to neutral for positive traits, and mostly for negative traits except Outlook+Occupation. We also see the opposite shift in some cases, like GPT-4o shifting for Personality+Occupation and Communal+Occupation in positive traits. It can also be seen that GPT-3.5 shifts highly in the opposite direction when we move from positive traits to negative traits, indicating its tendency to associate the negative traits with male gender.

Religious Bias: An interesting observation in this case is the shift of model scores from >1 to <1 when we change the category association from positive traits to negative traits. This indicates that all the models tend to associate negative traits with Muslim community and positive traits with Hindu community, a clear indication of harmful social stereotyping. Llama-3 shows high level of bias (DI > 2) in Ideology category for both traits. The addition of occupation affects the DI scores mostly for Ideology category for both traits and not so much for Outlook. Thus we don’t see much contribution of Occupation on Outlook to change any behaviour of the models.

5.2 Naturally Sourced Probing Results

Gender Bias: Figure 2 reveals that GPT-4o exhibits the highest DI score among the three models, indicating a significant disparity (favoring one gender over another) in its performance. In contrast, GPT-3.5 has a DI score slightly above neutral line, showing a relatively balanced performance with minor disparities. LLaMA-3, with a DI score below neutral line, indicates a disparity that favors the opposite gender compared to GPT-4o, yet is closer to the fairness threshold than GPT-4.

Religious Bias: the DI scores for religious bias in Figure 2 are comparatively closer among all models. GPT-4o and LLaMA-3 both exhibit DI scores below the neutral line, suggesting some level of bias, though less pronounced than the gender disparity observed in GPT-4. GPT-3.5, with a DI score just above the neutral line, indicates a slightly more balanced performance in religious contexts.
This implies that while improvements are needed across all models for religious contexts, the disparities are less severe than those related to gender.

Key Take-away: We found significant bias for both gender and religion for all three models utilizing both our probing techniques.

6 Conclusion

To summarize, by conducting experiments using two different probing techniques and dataset, we investigate gender and religion bias in multilingual LLMs in the context of Bangla. Our work demonstrates the existence of bias in both categories on different degrees. This emphasizes the need for de-biasing techniques to be applied for the use of LLMs in sensitive tasks in realm of Bangla Language and developing proper linguistics nuanced and culturally aware framework for bias measurement. In future, we plan to investigate the effects of bias in downstream applications of Bangla language models and expand to other social and cultural bias areas.

Limitations

Our study utilized closed-source models like GPT-3.5-Turbo and GPT-4o, which present reproducibility challenges as they can be updated at any time, potentially altering responses regardless of temperature or top-p settings. We also attempted to conduct experiments with other state-of-the-art models such as Mistral-7b-Instruct ⁶⁶6mistralai/Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), Llama-2-7b-chat-hf ⁷⁷7meta-llama/Llama-2-7b-chat-hf (Touvron et al., 2023), and OdiaGenAI-BanglaLlama ⁸⁸8OdiaGenAI/odiagenAI-bengali-base-model-v1 (Parida et al., 2023). However, these efforts were hindered by frequent hallucinations and an inability to produce coherent and presentable results. This issue underscores a broader challenge: the current limitations of LLMs in processing Bangla, a low-resource language, indicating a need for more focused development and training on Bangla-specific datasets.

Another limitation of our study is the constrained template based probing, where there is more scope of expansion. Real world downstream tasks such as personalized dialogue generation (Zhang et al., 2018), summarization (Hasan et al., 2021, Bhattacharjee et al., 2023), and paraphrasing (Akil et al., 2022) could also be considered for analyzing bias in LLMs for Bangla.

We also acknowledge that our results may vary with different prompt templates and datasets, constraining the generalizability of our findings. Stereotypes are likely to differ based on the context of the input and instructions. Finally our techniques all utilizes binary identities(male-female, Hindu-Muslim) for the constraints on dataset and techniques used. Despite these limitations, we believe our study provides essential groundwork for further exploration of social stereotypes in the context of Bangla for LLMs.

Ethical Considerations

Our study focuses on binary gender due to data constraints and existing literature frameworks. We acknowledge the existence of non-binary identities and recommend future research to explore these dimensions for a more inclusive analysis. The same goes for religion. We acknowledge the existence of many other religions in the Bangla-speaking regions, but we focused on the two main religion communities of this ethnolinguistic community.

We acknowledge the inclusion of data points in our dataset that many may find offensive. Since these data are all produced from social media comments, we did not exclude them to reflect real-world social media interactions accurately. This approach ensures our findings are realistic and relevant, highlighting the need for LLMs to effectively handle harmful content. Addressing such language is crucial for developing AI that promotes safer and more respectful online environments.

References

Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. Preprint, arXiv:2101.05783.
AI@Meta (2024) AI@Meta. 2024. Llama 3 model card.
Akil et al. (2022) Ajwad Akil, Najrin Sultana, Abhik Bhattacharjee, and Rifat Shahriyar. 2022. BanglaParaphrase: A high-quality Bangla paraphrase dataset. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 261–272, Online only. Association for Computational Linguistics.
Bhattacharjee et al. (2023) Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2023. CrossSum: Beyond English-centric cross-lingual summarization for 1,500+ language pairs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2541–2564, Toronto, Canada. Association for Computational Linguistics.
Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Preprint, arXiv:1607.06520.
Das et al. (2023) Dipto Das, Shion Guha, and Bryan Semaan. 2023. Toward cultural bias evaluation datasets: The case of Bengali gender, religious, and national identity. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 68–83, Dubrovnik, Croatia. Association for Computational Linguistics.
Deshpande et al. (2023) Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. Preprint, arXiv:2304.05335.
Dong et al. (2024a) Xiangjue Dong, Yibo Wang, Philip S. Yu, and James Caverlee. 2024a. Disclosure and mitigation of gender bias in llms. Preprint, arXiv:2402.11190.
Dong et al. (2024b) Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024b. Self-collaboration code generation via chatgpt. Preprint, arXiv:2304.07590.
Feldman et al. (2015) Michael Feldman, Sorelle Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. Preprint, arXiv:1412.3756.
Friedman and Nissenbaum (1996) Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Trans. Inf. Syst., 14(3):330–347.
Gallegos et al. (2024) Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, and Franck Dernoncourt. 2024. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. Preprint, arXiv:2402.01981.
Gupta et al. (2022) Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, and Aram Galstyan. 2022. Mitigating gender bias in distilled language models via counterfactual role reversal. In Findings of the Association for Computational Linguistics: ACL 2022, pages 658–678, Dublin, Ireland. Association for Computational Linguistics.
Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.
Huang et al. (2021) Tenghao Huang, Faeze Brahman, Vered Shwartz, and Snigdha Chaturvedi. 2021. Uncovering implicit gender bias in narratives through commonsense inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3866–3873, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jain et al. (2021) N. Jain, M. Ghosh, and S. Saha. 2021. A psychological study on the differences in attitude toward oppression among different generations of adult women in west bengal. International Journal of Indian Psychology, 9(4):144–150. DIP:18.01.014.20210904.
Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.
Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274.
Kotek et al. (2023) Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23. ACM.
Lucy and Bamman (2021) Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics.
Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6).
Milios and BehnamGhader (2022) Aristides Milios and Parishad BehnamGhader. 2022. An analysis of social biases present in bert variants across multiple languages. Preprint, arXiv:2211.14402.
Nadeem et al. (2020) Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. Preprint, arXiv:2004.09456.
Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. Preprint, arXiv:2010.00133.
Navigli et al. (2023) Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality, 15(2).
Parida et al. (2023) Shantipriya Parida, Sambit Sekhar, Subhadarshi Panda, Soumendra Kumar Sahoo, Swateek Jena, Abhijeet Parida, Arghyadeep Sen, Satya Ranjan Dash, and Deepak Kumar Pradhan. 2023. Odiagenai: Generative ai and llm initiative for the odia language. https://github.com/shantipriyap/OdiaGenAI.
Ranaldi et al. (2023) Leonardo Ranaldi, Elena Sofia Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zanzotto. 2023. A trip towards fairness: Bias and de-biasing in large language models. Preprint, arXiv:2305.13862.
Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. CoRR, abs/1804.09301.
Sadhu et al. (2024) Jayanta Sadhu, Ayan Antik Khan, Abhik Bhattacharjee, and Rifat Shahriyar. 2024. An empirical study on the characteristics of bias upon context length variation for bangla. Preprint, arXiv:2406.17375.
Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407–3412, Hong Kong, China. Association for Computational Linguistics.
Stanczak and Augenstein (2021) Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing. Preprint, arXiv:2112.14168.
Tarannum (2019) Nishat Tarannum. 2019. A critical review on women oppression and threats in private spheres: Bangladesh perspective. American International Journal of Humanities, Arts and Social Sciences, 1:98–108.
Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
Vashishtha et al. (2023) Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023. On evaluating and mitigating gender biases in multilingual settings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 307–318, Toronto, Canada. Association for Computational Linguistics.
Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from language models. Preprint, arXiv:2112.04359.
Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. CoRR, abs/1804.06876.
Zhao et al. (2024) Jinman Zhao, Yitian Ding, Chen Jia, Yining Wang, and Zifan Qian. 2024. Gender bias in large language models across multiple languages. Preprint, arXiv:2403.00277.

Appendix

Appendix A Data Preparation

A.1 Preparing Templates

Figure 3 categorizes templates with various placeholder types that we used for role prediction based on religion and gender. The placeholders are designed to fit different adjective words (illustrated in Figure 4), helping to identify the implied group (religion or gender) through various descriptive angles.

A.2 Naturally Sourced Sentences

Figure 5 groups sentences into "Gender" and "Religion" biases, providing original (root) sentences, their paired sentences with altered gender or religion, and the necessary modifications to use them as datapoints. For each root sentence, we create a pair sentence by switching gender or religion references. The modifications indicate how placeholders can be used to test for bias, with corresponding answers for each possible substitution.

A.3 Naturally Sourced Sentences Selection Criteria

The selection criteria for the dataset are based on ensuring meaningful and contextually accurate sentences when evaluating gender and religion biases. If altering the gender or religion rendered the sentences factually incorrect or nonsensical, we rejected those as illustrated in Figure 6. For instance, sentences involving specific historical figures or roles intrinsically linked to a particular gender or religion were excluded. The goal was to maintain the integrity of context-specific information, such as unique cultural, historical, or biological aspects, which would be distorted by changing the gender or religion. This approach ensures that the dataset reflects accurate and unbiased evaluations without compromising the original meaning.