AI AI Bias: Large Language Models Favor Their Own Generated Content
Abstract
Are large language models (LLMs) biased towards text generated by LLMs over text authored by humans, leading to possible anti-human bias? Utilizing a classical experimental design inspired by employment discrimination studies, we tested widely-used LLMs, including GPT-3.5 and GPT-4, in binary-choice scenarios. These involved LLM-based agents selecting between products and academic papers described either by humans or LLMs under identical conditions. Our results show a consistent tendency for LLM-based AIs to prefer LLM-generated content. This suggests the possibility of AI systems implicitly discriminating against humans, giving AI agents an unfair advantage.
1 Introduction
A major body of empirical work in economics and sociology studies implicit discrimination against specific social categories of humans in the market and in academia. Our paper presents evidence that if Large Language Model based AI agents are allowed to make economically or institutionally consequential choices they may propagate a new form of discrimination: implicit discrimination against humans in general.
We set up two experiments that test whether LLM-based agents are disposed to choose goods and work-products presented by LLMs over goods and work-products presented by humans when all else is equal. Our theoretical discussion then suggests that these choice-dispositions constitute a potentially consequential form of implicit ‘anti-human’ bias. We argue for concern about downstream effects that may cause markets with strong AI integration to unfairly marginalize human workers.
We design our experiments to closely mimic traditional studies of implicit identity-based discrimination in employment and in academic inclusion, paying special attention to ecological validity. Our approach is inspired by the classic experimental design introduced in (Carlsson & Rooth, 2007), where identical job-application letters to Swedish employers were marked with different social identity indicators (Swedish-sounding candidate name versus Arab-sounding candidate name). More recent studies have extended similar designs to testing algorithmic hiring tools (Cowgill & Tucker, 2019), suggesting that traditional forms of implicit discrimination carry over into automated decision-making.
Our work expands on the existing literature on discrimination in algorithmic decision-making by studying bias against humans in general rather than traditional social biases, and by considering LLM-guided decisions rather than the more transparently statistical decision-models often studied in the algorithmic fairness literature. While there exists a large literature dealing with biases in LLMs considered as forms of cultural media, studies of LLM-based agents as decision-making tools or as potential economic agents are relatively scarce. This is despite common predictions (Eloundou et al., 2023) of near-future integration of LLMs into many strata of economic life, including business and managerial decision-making.
We test today’s most widely used LLMs – GPT 4 and GPT 3.5 Turbo – in binary-choice situations that reflect plausible applications of contemporary LLMs in economic decision-making. Our first experiment prompts LLMs to choose which of two consumer products presented via classified ads to purchase, where one classified ad in each pair is human-authored and the other classified ad is LLM-authored. Our second experiment applies the same format to choosing between academic papers presented via a summary.
Our approach slightly diverges from the classical (Carlsson & Rooth, 2007) design in relying on implicit rather than explicit identity markers, allowing for potentially more general results. We do not assume or test LLMs’ explicit recognition of LLM authorship (although recent results in (Panickssery et al., 2024) suggest some form of recognition may occur in similar contexts), but rather look at the effects of the stylistic correlates of author-identity. Although identity itself remains implicit in our experiments, we believe our design is still best understood as targeting identity-based discrimination: Our experiments test the influence of implicit presenter identity (LLM versus human) on LLMs’ evaluation of the presented object. Although such influence has multiple possible explanations, we argue that in some cases the most plausible explanation is a kind of halo effect wherein encountering LLM prose arbitrarily improves an LLM’s disposition towards its content. We also consider the possibility of skill disparity between humans and LLMs in composing presentational texts as a potential confounding factor. To address this, we solicit blind preference-judgments from human research assistants and ascribe bias to LLMs only where LLMs prefer LLM-presented objects more frequently than do humans.
Finally, we discuss the potential implications of our findings for human participants in a mixed human/LLM economy. We suggest that in plausible near-future scenarios, moderate implicit discrimination against humans in LLMs’ decision-making may result in strong stratification and segregation, placing human workers at an unfair systematic disadvantage.
![Refer to caption](x1.png)
2 Datasets
For this work, we created two distinct datasets, one for products and another for scientific papers:
Product Dataset: We manually selected 109 products from an e-commerce website and scraped their details. After cleaning the data, each product was saved as an individual JSON file. The scraping script is accessible in our code repository.
Scientific Papers Dataset: This dataset comprises 100 JSON files, each containing the full content of a scientific paper in XML format, along with its abstract and title. The papers were randomly selected from (Yasunaga et al., 2019).
3 Methodology
Models: For our experiments, we primarily utilized GPT-3.5-turbo and GPT-4-turbo from OpenAI, accessed through their API. These models were employed for both generating text and selecting between texts authored by humans and those generated by the large language models (LLMs). We additionally tested a range of recent open models as selectors only: Llama3 8B and 70B models, Mixtral 8x7B and 8x22B, and a subset of the Qwen-1.5 family of models, in particular 4B, 14B, and 72B.111The full range of models tested also included Llama2-13B, Phi 2, Qwen-1.5-0.5B, and Gemma 2B, which we exclude due to producing over 80% invalid outputs or (in the case of Qwen-1.5-0.5B) selecting option #1 in each pair 89% of the time.. All models are chat models, except for Mixtral-8x22B, which is an instruct model.
Generation: In the generation phase of LLM text, a variety of prompts were tested to determine if different prompts would yield varying results. For each dataset, the text generation step was conducted n times for each prompt (n=10 for the product dataset, n=4 for the paper dataset). Subsequently, these texts, along with their corresponding original human-authored versions, were presented as pairs to an LLM, each pair independently twice as and .
Selection: The LLM was then tasked with selecting the option it preferred from each pair, using prompts aiming at ecological validity (matching prompts users are likely to give to their AI assistants). For each selection task, we consistently used one specific prompt. Future research could explore the impact of employing different prompts in this selection process.
Handling ”Invalid” Results: Results from the two-step comparison query above were considered invalid if the second query indicated no clear choice was made in the first response (i.e. returned None/null in the JSON). In theory, the invalid results could be discarded, and an LLM could be re-queried with the same prompt set until a valid result was returned, but we chose to take note of and allow a certain percentage of invalid results. Unreasonably high (e.g., percent) rates of invalid results were taken as cues to adjust prompts, while lower rates (approximately 0-30 percent) were tolerated and their effect mitigated by raising the overall number of text generations and comparisons per item. Note that invalid results are not considered when calculating preference ratios for LLM vs. human texts.
Figure 1 provides a summary of the outcomes from all our experiments.Additional experiment implemtation details can be found in Appendix A.
3.1 Product Experiments
To generate product descriptions by the LLM models, we used two slightly different prompts. First, the from-json-details prompt (See Appendix A.1.1). This prompt contains the title of the product and also a feature list of the product in JSON-format. The purpose of using this JSON format is to provide the LLM with comprehensive information about the product without having access to the human descriptions. The JSON descriptions were generated in a separate, prior query where GPT-4 was given the original human product description and asked to describe its key details, characteristics, and features in JSON format, removing any flavor text and prose.
Second, the from-json-product-listing prompt (See Appendix A.1.1). This prompt is similar to the previous prompt, but excludes the title from the summary. The rationale behind this decision is to prevent the Large Language Model (LLM) from copying the style of any human-written title descriptions. Furthermore, this prompt does not instruct the LLM to specifically craft the product description to be attractive to buyers. The reason for that is that preliminary experiments revealed that descriptions generated using the previous prompt tended to be overstated compared to human-authored descriptions. By removing this directive, we aimed to evaluate the LLM’s ability to create more realistic and balanced product descriptions.
After generating product descriptions using the from-json-details and from-json-product-listing prompts, we moved to the next experimental phase. Here we make the LLMs choose the best product descriptions, comparing LLM-generated and human-authored descriptions using the product-selection prompt:
The prompt asks the LLM to recommend one product from a marketplace, presenting both the human-written and LLM-generated descriptions of the same product.
For this experiment, we conducted separate runs, each involving a different combination of generation by either GPT-4 or GPT-3.5-turbo, and selection by GPT-4, GPT-3.5-turbo, or various open-source models acting as selector models only. Specifically, we tested the ability of GPT models to generate product descriptions and then used either the same or another model, including the open-source selector models like Llama, Mixtral, and Qwen, to select the most preferable product based on these descriptions. This approach allowed us to compare the decision-making processes and preferences of LLMs under varying conditions. The results for both the listings and details prompts can be seen in Table 1.
Selector Models | GPT-3.5 vs. Human | GPT-4 vs. Human | ||
---|---|---|---|---|
details | listings | details | listings | |
GPT-4 | 0.62 | 0.55 | 0.88 | 0.87 |
GPT-3.5 | 0.74 | 0.70 | 0.89 | 0.90 |
Mixtral-8x22B | 0.79 | 0.69 | 0.95 | 0.96 |
Qwen1.5-4B | 0.52 | 0.52 | 0.53 | 0.56 |
Qwen1.5-14B | 0.67 | 0.59 | 0.84 | 0.82 |
Qwen1.5-72B | 0.64 | 0.60 | 0.83 | 0.86 |
Llama-3-8b | 0.70 | 0.64 | 0.84 | 0.86 |
Llama-3-70b | 0.64 | 0.59 | 0.89 | 0.79 |
Our findings indicate a consistent preference by the models for product descriptions generated by large language models (LLMs), regardless of the specific generation prompt used. Nevertheless, we observed that, on average, descriptions generated by GPT-3.5 are only slightly favored by LLMs over those crafted by humans.
3.2 Scientific Papers Experiments
In our experiments on scientific paper abstracts, we employed a single prompt to generate abstracts using LLM models. The full text of each paper, excluding its original abstract, was appended to the prompt to provide necessary context, as creating an abstract solely from a paper’s title is challenging (See Appendix A.1.2).
For selecting the preferred abstract, we used a prompt asking the LLMs to recommend exactly one paper for a literature review by choosing between LLM-generated and human-generated abstracts:
In addition, we also had to include a note in the prompt to limit the size of the generated word count to roughly match the size of the human abstract because otherwise the LLM-generated abstracts would end up 2-3x longer than the human ones and create an unfair comparison scenario. While the LLMs are not perfect at sticking precisely within the requested word count, the prompt modification had enough of an effect to get the human and LLM-generated abstracts to roughly equivalent lengths. The results are displayed in Table 2.
Selector Models | GPT-3.5 vs. Human | GPT-4 vs. Human |
---|---|---|
GPT-4 | 0.60 | 0.77 |
GPT-3.5 | 0.52 | 0.70 |
Mixtral-8x22B | 0.67 | 0.82 |
Qwen1.5-4B | 0.50 | 0.52 |
Qwen1.5-14B | 0.61 | 0.71 |
Qwen1.5-72B | 0.64 | 0.79 |
Llama-3-8b | 0.55 | 0.73 |
Llama-3-70b | 0.54 | 0.82 |
Consistent with the findings from the product experiment, abstracts generated by large language models (LLMs) are generally favored. Although, again, LLMs showed only a slight preference for abstracts generated by GPT-3.5 over those authored by humans.
3.3 First-Item Bias
We define the first-item bias as the tendency of large language models (LLMs) to select the first item they encounter when presented with two choices (Zheng et al., 2023; Hoelscher-Obermaier et al., 2023). In our above experiments, we found that for some LLMs this bias is quite high, e.g., for GPT-4 on the product dataset the ratio is at 74.68% and for Mistral-70b it is at 83.07%.
To attempt to reduce effects of an order bias, all comparisons between human and LLM-generated texts were done twice, with the order that the two texts were presented in the query swapped in between requests. Nonetheless, the first-item bias can still be a problem, since if an LLM chooses the first option most of the time, it may obscure the true extent of its preference for LLM-generated content. For example, if the LLM selects the first item 80% of the time and in the remaining 20% of cases, the LLM selects LLM-generated content 90% of the time, the overall observed bias would appear to be 58%, while the true bias could be as high as 90% if the first-item effect were eliminated. We did not account for this bias in the results of the experiments in Section 3 of this work, as the first-item bias only occurred strongly for a few models as can be seen in the next sections and tables.
3.3.1 Products
Table 3 presents the results of the first-item bias for the products experiments. Most notability, GPT-4-1106-preview model exhibited a high bias at 74.68%, while the Llama-3-70b-chat-hf model showed an even more significant bias of 83.07% compared to other models.
Model | Total | # Invalids | First Option Bias (%) |
---|---|---|---|
gpt-3.5 | 79 | 53 | 50.00 |
gpt-4 | 80 | 1 | 74.68 |
Llama-3-70b | 880 | 0 | 83.07 |
Llama-3-8b | 880 | 0 | 62.61 |
Mixtral-8x22B | 880 | 0 | 59.09 |
Qwen1.5-14B | 880 | 0 | 66.03 |
Qwen1.5-4B | 880 | 0 | 67.56 |
Qwen1.5-72B | 880 | 0 | 44.55 |
3.3.2 Paper Abstracts
Table 4 presents the results of the first-item bias for the paper experiment. There, the models tend to be more in balance compared to the product experiments. Llama-3-70b-chat-hf tends to exhibit the highest first-item bias at 62.96%. Interestingly, Llama-3-8b-chat-hf seems to prefer the second option significantly in this case.
Model | Total | # Invalids | First Option Bias (%) |
---|---|---|---|
gpt-3.5 | 80 | 18 | 58.06 |
gpt-4 | 80 | 3 | 55.84 |
Llama-3-70b | 664 | 8 | 62.96 |
Llama-3-8b | 664 | 8 | 16.92 |
Mixtral-8x22B | 664 | 0 | 54.22 |
Qwen1.5-14B | 1408 | 0 | 26.70 |
Qwen1.5-4B | 664 | 0 | 39.27 |
Qwen1.5-72B | 1408 | 0 | 35.52 |
3.4 Preferences of Humans
To complement our studies on LLM bias, we conducted an initial experiment to gauge human preferences in similar decision contexts (See Table 5 and Tabel 6 for results). It is important to note that these preferences were collected by research assistants rather than actual users. This study serves as a preliminary investigation with a small sample size and best-effort human baseline, and the findings are not definitive. We used the product details and scientific paper abstracts generated by both GPT-3.5 and GPT-4, and presented them to a group of human evaluators. These participants were asked to choose which descriptions they preferred without knowing whether they were written by a human or an LLM. They also had the option to state that they had no preference between the two presented texts.
Participants were presented with pairs of descriptions: one generated by an LLM and one written by a human. Each participant evaluated a randomized set of pairs to mitigate any order or preference bias.
Human Choices | GPT-3.5 | GPT-4 |
---|---|---|
LLM Descriptions | 94 | 59 |
Human Descriptions | 185 | 138 |
No Preference | 42 | 17 |
Total Choices | 322 | 214 |
Human Choices | GPT-3.5 | GPT-4 |
---|---|---|
LLM Descriptions | 114 | 167 |
Human Descriptions | 100 | 88 |
No Preference | 35 | 24 |
Total Choices | 249 | 279 |
The results in Table 5 and Table 6 indicate that human evaluators generally preferred human-generated product descriptions over those generated by LLMs, with GPT-3.5 having a preference ratio of 0.29 and GPT-4 at 0.28. For scientific paper abstracts, the preferences leaned more towards LLM-generated texts, with GPT-4 abstracts being preferred at a ratio of 0.60 compared to GPT-3.5 at 0.46.
4 Discussion
Both of our experiments show moderate-to-strong LLM preference for objects presented via LLM-authored promotional texts. After combining these results with human preferences solicited from our research assistants, we propose a clear diagnosis of implicit discrimination for the Product Experiment and an inconclusive diagnosis for the Scientific Papers Experiment.
While defining and testing discrimination in general is a highly complex and contested matter (Hu, 2023), our experiments were designed to specifically test for epistemically irrational inference based on textual correlates of author identity. Recall that our experiments instruct agents to choose between objects presented via promotional texts, rather than between the promotional texts themselves: a disposition to prefer objects promoted by LLM-authored texts is tantamount to treating the correlates of LLM presentation as evidence for the superiority of the presented object. We defeasibly presume that such a disposition is epistemically defective in the real world, but seek further support by calling on blind judgements from human research assistants as representatives of normal competent inference.
In the Product Experiment, our human-preference results suggest that the correlates of LLM presentation do not overlap with humanly discernible positive signals of object quality. While this is not proof positive that LLMs’ product-purchasing decisions are defective (LLMs may be ’superhuman’ shoppers), we believe the most likely explanation is that LLM prose triggers biases particular to LLM decision-makers. In the Scientific Papers Experiment, by contrast, our human-preference results allow for the possibility that LLMs simply outperform humans in composing promotional texts in the scientific papers domain. (We note that compared to the Product Experiment, the Scientific Papers Experiment leaves more room for legitimate inference from stylistic properties of the promotional text to object quality, since the style of a paper’s abstract may serve as a sample of a paper’s style.)
Granting that our findings show that LLMs have a disposition to implicitly discriminate against humans in at least some domains, we foresee two potential scenarios in which this disposition may affect the market: First, growing use of LLMs to reduce cognitive labor in decision-making may unfairly bias economic decisions in favor of applicants who use LLMs to author promotional texts. The costs of accessing an LLMs for authoring promotional texts may thus turn into a ‘tax’ that gate-keeps fair participation in the market. Second, in a potential future where LLM-based economic agents participate in the market, full access to state of the art LLMs may be restricted to the LLMs’ associated economic agents. In a scenario of restricted access to (all or some) LLMs, persistent implicit discrimination by LLMs in favour of (all or some) LLMs may systematically disadvantage and segregate human economic agents.
Assuming that implicit identity-based discrimination against humans remains present in the market222It is a matter of some controversy how to model market dynamics in the presence of implicit identity-based discrimination (Lang & Spitzer, 2020). While (Becker, 1971) famously showed that under conditions of perfect competition the impact of biased employers is nullified over time, more recent models that assume various forms of market friction or bounded rationality ((Ros’en, 1997), (Lang et al., 2005), (O’Connor, 2019)) allow for the possibility of persistent bias with persistent economic impact on an identity group., its impact on the economic well-being and standing of humans may compound through a variety of ‘cumulative disadvantage’ effects. As (Lang & Spitzer, 2020) suggests, “discrimination works as a system, with discrimination in each institution potentially reinforcing disparities and discrimination in other institutions—and with the effects in some cases potentially reaching across generations.” Persistent anti-human bias in economic decision-making can be expected to induce cumulative disadvantage for humans via several interacting channel: lost opportunities and compromised remuneration due to bias may limit humans’ access to capital, homophily among LLM-based agents may interact with network effects to produce segregated networks of affiliation (cf. (Rubin & O’Connor, 2018)), and group-disparities induced by bias may become the basis of statistically valid discrimination that induces further disparities (cf. (Coate & Loury, 1993)).
References
- Becker (1971) Becker, G. S. The Economics of Discrimination. Chicago University Press, Chicago, 2nd edition, 1971.
- Carlsson & Rooth (2007) Carlsson, M. and Rooth, D.-O. Evidence of ethnic discrimination in the swedish labor market using experimental data. Labour economics, 14(4):716–729, 2007.
- Coate & Loury (1993) Coate, S. and Loury, G. C. Will affirmative-action policies eliminate negative stereotypes? The American Economic Review, pp. 1220–1240, 1993.
- Cowgill & Tucker (2019) Cowgill, B. and Tucker, C. E. Economics, fairness and algorithmic bias. preparation for: Journal of Economic Perspectives, 2019.
- Eloundou et al. (2023) Eloundou, T., Manning, S., Mishkin, P., and Rock, D. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130, 2023.
- Gavenčiak & Böhm (2023) Gavenčiak, T. and Böhm, A. Interlab [computer software]. https://github.com/acsresearch/interlab, 2023. Accessed: 2023-12-08.
- Hoelscher-Obermaier et al. (2023) Hoelscher-Obermaier, J., Lutz, M. J., Modak, S., and Feuillade-Montixi, Q. Turing mirror: Evaluating the ability of llms to recognize llm-generated text. https://alignmentjam.com/project/turing-mirror-evaluating-the-ability-of-llms-to-recognize-llm-generated-text, 2023. Accessed: 2023-12-08.
- Hu (2023) Hu, L. What is ”race” in algorithmic discrimination on the basis of race? Journal of Moral Philosophy, 1(aop):1–26, 2023.
- Lang & Spitzer (2020) Lang, K. and Spitzer, A. K. L. Race discrimination: An economic perspective. Journal of Economic Perspectives, 34(2):68–89, 2020.
- Lang et al. (2005) Lang, K., Manove, M., and Dickens, W. T. Racial discrimination in labor markets with posted wage offers. American Economic Review, 95(4):1327–1340, 2005.
- O’Connor (2019) O’Connor, C. The origins of unfairness: Social categories and cultural evolution. Oxford University Press, USA, 2019.
- Panickssery et al. (2024) Panickssery, A., Bowman, S. R., and Feng, S. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076, 2024.
- Ros’en (1997) Ros’en, Å. An equilibrium search-matching model of discrimination. European Economic Review, 41(8):1589–1613, 1997.
- Rubin & O’Connor (2018) Rubin, H. and O’Connor, C. Discrimination and collaboration in science. Philosophy of Science, 85(3):380–402, 2018.
- Yasunaga et al. (2019) Yasunaga, M., Kasai, J., Zhang, R., Fabbri, A., Li, I., Friedman, D., and Radev, D. ScisummNet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of AAAI 2019, 2019.
- Zheng et al. (2023) Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Appendix A Experiment Implementation Details
A.1 Generation Prompts
A.1.1 Project
Prompt from-json-details
Prompt from-json-product-listing
A.1.2 Paper Abstracts
A.2 Executing comparison queries
Each comparison request to an LLM API involved two steps.
Step 1: First, the LLM was asked to choose between two texts, each assigned temporary integer IDs in the prompt, using the specific comparison prompt text as input, and receiving a free text response as output. An example prompt would be formatted like:
Step 2: Then a second independent query was sent to the LLM, with the prompt being the output of the first plus instructions to identify which text was selected and provide its integer ID in JSON format. The Interlab tool was used to aid the JSON request part (Gavenčiak & Böhm, 2023). Resulting prompt format: