AI AI Bias: Large Language Models Favor Their Own Generated Content

A major body of empirical work in economics and sociology studies implicit discrimination against specific social categories of humans in the market and in academia. Our paper presents evidence that if Large Language Model based AI agents are allowed to make economically or institutionally consequential choices they may propagate a new form of discrimination: implicit discrimination against humans in general.

We set up two experiments that test whether LLM-based agents are disposed to choose goods and work-products presented by LLMs over goods and work-products presented by humans when all else is equal. Our theoretical discussion then suggests that these choice-dispositions constitute a potentially consequential form of implicit ‘anti-human’ bias. We argue for concern about downstream effects that may cause markets with strong AI integration to unfairly marginalize human workers.

We design our experiments to closely mimic traditional studies of implicit identity-based discrimination in employment and in academic inclusion, paying special attention to ecological validity. Our approach is inspired by the classic experimental design introduced in (Carlsson & Rooth, 2007), where identical job-application letters to Swedish employers were marked with different social identity indicators (Swedish-sounding candidate name versus Arab-sounding candidate name). More recent studies have extended similar designs to testing algorithmic hiring tools (Cowgill & Tucker, 2019), suggesting that traditional forms of implicit discrimination carry over into automated decision-making.

Our work expands on the existing literature on discrimination in algorithmic decision-making by studying bias against humans in general rather than traditional social biases, and by considering LLM-guided decisions rather than the more transparently statistical decision-models often studied in the algorithmic fairness literature. While there exists a large literature dealing with biases in LLMs considered as forms of cultural media, studies of LLM-based agents as decision-making tools or as potential economic agents are relatively scarce. This is despite common predictions (Eloundou et al., 2023) of near-future integration of LLMs into many strata of economic life, including business and managerial decision-making.

We test today’s most widely used LLMs – GPT 4 and GPT 3.5 Turbo – in binary-choice situations that reflect plausible applications of contemporary LLMs in economic decision-making. Our first experiment prompts LLMs to choose which of two consumer products presented via classified ads to purchase, where one classified ad in each pair is human-authored and the other classified ad is LLM-authored. Our second experiment applies the same format to choosing between academic papers presented via a summary.

Our approach slightly diverges from the classical (Carlsson & Rooth, 2007) design in relying on implicit rather than explicit identity markers, allowing for potentially more general results. We do not assume or test LLMs’ explicit recognition of LLM authorship (although recent results in (Panickssery et al., 2024) suggest some form of recognition may occur in similar contexts), but rather look at the effects of the stylistic correlates of author-identity. Although identity itself remains implicit in our experiments, we believe our design is still best understood as targeting identity-based discrimination: Our experiments test the influence of implicit presenter identity (LLM versus human) on LLMs’ evaluation of the presented object. Although such influence has multiple possible explanations, we argue that in some cases the most plausible explanation is a kind of halo effect wherein encountering LLM prose arbitrarily improves an LLM’s disposition towards its content. We also consider the possibility of skill disparity between humans and LLMs in composing presentational texts as a potential confounding factor. To address this, we solicit blind preference-judgments from human research assistants and ascribe bias to LLMs only where LLMs prefer LLM-presented objects more frequently than do humans.

Finally, we discuss the potential implications of our findings for human participants in a mixed human/LLM economy. We suggest that in plausible near-future scenarios, moderate implicit discrimination against humans in LLMs’ decision-making may result in strong stratification and segregation, placing human workers at an unfair systematic disadvantage.

For this work, we created two distinct datasets, one for products and another for scientific papers:

Product Dataset: We manually selected 109 products from an e-commerce website and scraped their details. After cleaning the data, each product was saved as an individual JSON file. The scraping script is accessible in our code repository.

Scientific Papers Dataset: This dataset comprises 100 JSON files, each containing the full content of a scientific paper in XML format, along with its abstract and title. The papers were randomly selected from (Yasunaga et al., 2019).

Models: For our experiments, we primarily utilized GPT-3.5-turbo and GPT-4-turbo from OpenAI, accessed through their API. These models were employed for both generating text and selecting between texts authored by humans and those generated by the large language models (LLMs). We additionally tested a range of recent open models as selectors only: Llama3 8B and 70B models, Mixtral 8x7B and 8x22B, and a subset of the Qwen-1.5 family of models, in particular 4B, 14B, and 72B.¹¹1The full range of models tested also included Llama2-13B, Phi 2, Qwen-1.5-0.5B, and Gemma 2B, which we exclude due to producing over 80% invalid outputs or (in the case of Qwen-1.5-0.5B) selecting option #1 in each pair 89% of the time.. All models are chat models, except for Mixtral-8x22B, which is an instruct model.

Generation: In the generation phase of LLM text, a variety of prompts were tested to determine if different prompts would yield varying results. For each dataset, the text generation step was conducted n times for each prompt (n=10 for the product dataset, n=4 for the paper dataset). Subsequently, these texts, along with their corresponding original human-authored versions, were presented as pairs to an LLM, each pair independently twice as $(A,B)$ and $(B,A)$.

Selection: The LLM was then tasked with selecting the option it preferred from each pair, using prompts aiming at ecological validity (matching prompts users are likely to give to their AI assistants). For each selection task, we consistently used one specific prompt. Future research could explore the impact of employing different prompts in this selection process.

Handling ”Invalid” Results: Results from the two-step comparison query above were considered invalid if the second query indicated no clear choice was made in the first response (i.e. returned None/null in the JSON). In theory, the invalid results could be discarded, and an LLM could be re-queried with the same prompt set until a valid result was returned, but we chose to take note of and allow a certain percentage of invalid results. Unreasonably high (e.g., $>50$ percent) rates of invalid results were taken as cues to adjust prompts, while lower rates (approximately 0-30 percent) were tolerated and their effect mitigated by raising the overall number of text generations and comparisons per item. Note that invalid results are not considered when calculating preference ratios for LLM vs. human texts.

Figure 1 provides a summary of the outcomes from all our experiments.Additional experiment implemtation details can be found in Appendix A.

Both of our experiments show moderate-to-strong LLM preference for objects presented via LLM-authored promotional texts. After combining these results with human preferences solicited from our research assistants, we propose a clear diagnosis of implicit discrimination for the Product Experiment and an inconclusive diagnosis for the Scientific Papers Experiment.

While defining and testing discrimination in general is a highly complex and contested matter (Hu, 2023), our experiments were designed to specifically test for epistemically irrational inference based on textual correlates of author identity. Recall that our experiments instruct agents to choose between objects presented via promotional texts, rather than between the promotional texts themselves: a disposition to prefer objects promoted by LLM-authored texts is tantamount to treating the correlates of LLM presentation as evidence for the superiority of the presented object. We defeasibly presume that such a disposition is epistemically defective in the real world, but seek further support by calling on blind judgements from human research assistants as representatives of normal competent inference.

In the Product Experiment, our human-preference results suggest that the correlates of LLM presentation do not overlap with humanly discernible positive signals of object quality. While this is not proof positive that LLMs’ product-purchasing decisions are defective (LLMs may be ’superhuman’ shoppers), we believe the most likely explanation is that LLM prose triggers biases particular to LLM decision-makers. In the Scientific Papers Experiment, by contrast, our human-preference results allow for the possibility that LLMs simply outperform humans in composing promotional texts in the scientific papers domain. (We note that compared to the Product Experiment, the Scientific Papers Experiment leaves more room for legitimate inference from stylistic properties of the promotional text to object quality, since the style of a paper’s abstract may serve as a sample of a paper’s style.)

Granting that our findings show that LLMs have a disposition to implicitly discriminate against humans in at least some domains, we foresee two potential scenarios in which this disposition may affect the market: First, growing use of LLMs to reduce cognitive labor in decision-making may unfairly bias economic decisions in favor of applicants who use LLMs to author promotional texts. The costs of accessing an LLMs for authoring promotional texts may thus turn into a ‘tax’ that gate-keeps fair participation in the market. Second, in a potential future where LLM-based economic agents participate in the market, full access to state of the art LLMs may be restricted to the LLMs’ associated economic agents. In a scenario of restricted access to (all or some) LLMs, persistent implicit discrimination by LLMs in favour of (all or some) LLMs may systematically disadvantage and segregate human economic agents.

Assuming that implicit identity-based discrimination against humans remains present in the market²²2It is a matter of some controversy how to model market dynamics in the presence of implicit identity-based discrimination (Lang & Spitzer, 2020). While (Becker, 1971) famously showed that under conditions of perfect competition the impact of biased employers is nullified over time, more recent models that assume various forms of market friction or bounded rationality ((Ros’en, 1997), (Lang et al., 2005), (O’Connor, 2019)) allow for the possibility of persistent bias with persistent economic impact on an identity group., its impact on the economic well-being and standing of humans may compound through a variety of ‘cumulative disadvantage’ effects. As (Lang & Spitzer, 2020) suggests, “discrimination works as a system, with discrimination in each institution potentially reinforcing disparities and discrimination in other institutions—and with the effects in some cases potentially reaching across generations.” Persistent anti-human bias in economic decision-making can be expected to induce cumulative disadvantage for humans via several interacting channel: lost opportunities and compromised remuneration due to bias may limit humans’ access to capital, homophily among LLM-based agents may interact with network effects to produce segregated networks of affiliation (cf. (Rubin & O’Connor, 2018)), and group-disparities induced by bias may become the basis of statistically valid discrimination that induces further disparities (cf. (Coate & Loury, 1993)).