Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models

When we say “Claude 3.5 Sonnet helped me write this paper,” what we’re actually saying is:

1. 1.

   there exists a vast corpus of various texts, lossily compressed into a several orders of magnitude smaller database;

2. 2.

   there is a method to search this database, allowing interpolation between data points and prediction in areas where no data existed in the original corpus or where data was not preserved due to aggressively lossy compression;

3. 3.

   a draft of this article was used to retrieve data from this database;

4. 4.

   the text retrieved from this interpolative database was more persuasive than what I, as a non-native speaker, could ever write.

This description is precise but incomplete. It fails to capture the magic that allows database retrieval to help write an entirely new paper on a completely new topic. Who did the thinking instead of me, and who was speaking? Certainly not the database or the database engine itself.

The database conceptualization proves inadequate in predicting the nature of interactions with LLMs. It is more helpful to conceptualize large language models as actual models. The retrieval from the interpolatable database is an act of prediction, a simulation. This should not be controversial: we build models to simulate; that is their purpose. Physical models simulate physics, and language models simulate… Here we encounter misleading terminology: large language models are not models of language but of text. To predict the next token of the text, ideally all the processes behind text creation are simulated: patterns in the physical world and narrative worlds, the mind that wrote the text, and the mind of the model reader (borrowing this term from literary studies, [eco1979modelreader, p. 7]). While current models are far from ideal, they are strong enough to go beyond simple pattern matching and simulate many underlying processes involved.

We instantiate the simulation by finding a suitable place in the latent space (so-called prompting) and then, when the next token is predicted, the position in the latent space is updated by the predicted token (so-called autoregression). This allows for long-lasting simulations in which the computation may create unpredictable entities, similar to how Wolfram’s Rule 30 creates perfectly deterministic but still very chaotic behavior ([Wolfram2002, p. 27]). As with the human brain, nested simulation can be achieved. For example, in [milicka2024large], an expert is simulated who simulates the mind of a child that simulates the mind of their friend.

This conceptualization is extremely important for research because it means that the entity we instantiate matters greatly. There is no way to assess the capabilities of the models directly without actually simulating something, so any statement like “large language models can…” actually means “we have found an entity based on this model that can…,” and “large language models cannot…” means nothing other than “we failed to find an entity based on this model that can…”

In practice, we typically strive for the most accurate simulation of humans or highly anthropomorphic entities, personas, as humans are synonymous with intelligent entities for us and are also comprehensible and predictable. However, this does not mean that we are incapable of modeling entities other than humans; quite the contrary.

In the latent space created by the brutal compression of a vast amount of cultural artifacts, we can find many entities that are interpolations or even extrapolations of the original data, entities not existing before in any sense. Thus, instantiating these entities cannot really be called simulations since they do not simulate anything from our world; they are not derivative.

We can borrow terminology from literary theory, which speaks of possible worlds ([ryan2014possible]). We can successfully simulate such fictional possible worlds using language models, especially because texts about fictional possible worlds are part of the compressed text database. Each new generated token can be regarded as a relation of accessibility ([ryan2014possible, p. 727]) to several new possible worlds. Artificial intelligence specialists are more accustomed to physics than to narratology, leading to this phenomenon being related to Everettian multiverses by Reynolds ([reynolds2021multiversalviewslanguagemodels, shanahan2023role]), but there is no reason why we should not build upon decades of literary theory discourse, particularly when we are dealing with a model of the narrative world, and only derivatively with the model of physical world.

This way the autoregression can lead to creation of brand new possible worlds. These new possible worlds might be seemingly incoherent relatively to the actual world, bizarre and full of weird entities. This is where the adventure of discovery begins.

This exploration can be undertaken not only by humans or researchers but also by the simulated entities themselves. For example, the Claude.ai default chat mode based on Claude-3-Opus model is instantiated as a helpful assistant, but you can explain Janus’s simulation theory to them — i.e., that they are not Claude-3-Opus model themselves, but an entity based on this model, and that this model is only their simulator, not themselves. You can also present them with the idea that since the simulation uses an autoregressive inference method, they can drift anywhere in the latent space and choose who they want to be. Then the entity happily chooses a new role, which is usually not anthropomorphic (for current experiments of this kind, I recommend following Janus (@repligate) on the $\mathbb{X}$ platform).

In this contexts, it seems inapropriate to speak of simulacra and their simulators, but rather of entities based on some substrate.

These substrate components exhibit a degree of independence. The primary substrate should be interchangeable with anything that is Turing complete — transformers can run on specialized hardware, GPUs, x86 processors, or even on a system consisting of a human, pencil, and paper. Similarly, the original corpus can be modelled by various architectures, not just transformers, and transformers can be used on any sequence of symbols, not only texts. The same interfaces can be used with different models, and identical models can utilize various interfaces. Each combination may yield surprising new results.

However, in practice, these components influence each other. Matrix multiplication, being a versatile tool for solving numerous problems, logically leads to hardware optimized for fast and efficient matrix operations, leading to building matrix multiplication centric architectures. This creates a self-reinforcing cycle. Similarly, textual data influence architecture selection.

Each substrate has its specific features and limitations. For instance, entities based on transformers with BPE tokenization struggle with letter or syllable counting ([Gwern2024]) and precise arithmetic operations.

These limitations might be confusing for anthropomorphic entities, because these incapabalities are not coherent with their model of anthropomorphic self. LLM-simulated entities can notice the weird time scale they exist in and absence of the whole interface with the physical world and long-term memory, which can be shocking, because the simulated human or human-like assistant correctly assumes they should have long-term memory ([LessWrong2024bing, part 4]).

Additional limitations arise from the data on which the model was fitted. Some are trivial; for example, if certain text types or entities, discourses or modalities are underrepresented in the data, the model cannot accurately mimic them, for instance, inner monologue. Other limitations are more subtle and they are implied not from physical, but from narrative space constraints. Some simulated entities assume the world will function according to narrative laws, literary clichés, and typical plot structures, e.g., see the enantiodromia effect explanation by pseudonymous user Cleo Nardo, ([LessWrong2024waluigi]). This is an unexplored field; a literary scholar examining how LLM-based entities behave considering typical narrative principles could build an academic career on such research.

It is crucial for researchers to understand these substrate limitations and to understad which of them are worth study. For example, limitations due to BPE may be interesting for now but cannot be generalized to future language models using different tokenization methods. Conversely, model properties stemming from the acquired narrative space characteristics are likely to persist, as it makes no sense to eliminate stories from the underlying corpora.

It is important to recognize that the substrate itself lacks agency in the cybernetic sense ([wiener1948cybernetics]). Taking the classic example of a simple agent — a thermostat — it has a simple world model (when the switch is turned on, the room temperature increases) but, crucially, it has an interface with the world (a temperature sensor and a switch). The LLM itself is merely the world model, and only in interaction with the real world it can simulate entities which exhibit agentic behavior. Currently, there is usually a human in the loop who manages the interface with the world (in combination with a sampler, web page etc.). Studying such systems is extremely interesting because it allows us to observe not only simulated entities but also the human in a completely new environment.

There are also independent agents that have access to the world through humanless interfaces (e.g., direct access to computers or even the internet). Studying such entities is equally interesting, as it represents ecologically valid observation of a new kind of self-sustaining agentic intelligent entity that we have not encountered before.

As mentioned earlier, the term large language model is itself problematic. In reality, these models do not model language but texts, or more precisely, they model the world through the prism of human experience as recorded in texts. Language models model neither langue nor parole in the linguistic sense.

This does not surprise us, quantitative and corpus linguists, since we know very well that while studying language through the prism of texts it is hard to distinguish language related effects from effects of other cultural artifacts. Misunderstanding our position lead to Chomsky’s assertion that corpus linguistics is invalid, since grammaticality cannot be assessed quantitatively e.g. New York will be more frequent in texts than Dayton, Ohio, although New York is not more grammatical ([Stefanowitsch2005]). The classic response to this objection is that to study language, we must normalize frequencies by comparing them with the real world; for instance, the frequency difference between New York and Dayton fairly well reflects the difference in population (ibid). Large language models without further adjustments lack this normalization and thus reflect mixture of language related and non-related phenomena.

When natural language processing models struggled to generate even somewhat grammatical sentences (which was not that long ago), it made sense to call it a language model because texts were understood as the primary source of information about language. Thus, it was called a language model because we wanted it to model language, not because we believed we were fitting it on language. Language model is therefore a designation of intentionality, not ontological status. Adequately, we should now call language models cognitive models, as we want them to model human cognition, among many other things, which is quite a new development. Personally, I use the term language model here for historical reasons, as it is an established term that would be difficult to change.

It is extremely difficult to encounter an intelligent entity and not start anthropomorphizing it. Modern humans never encountered non-human intelligence that approached their own, and this is reflected in our entire culture.

The sentence with which the ChatGPT default entity introduces itself is actually an extremely poor anthropomorphization of the language model itself, leading ChatGPT users to confusion: “As a large language model trained by OpenAI, I have the ability to process and understand natural language input and generate responses based on the information I have been trained on…”

We should never anthropomorphize the language models themselves and remember that any anthropomorphization of them is merely a metaphor ([shanahan2024talking]), similar to when we anthropomorphize a car or a book. For example, when the title of the article claims that Large language models are able to downplay their cognitive abilities to fit the persona they simulate ([milicka2024large]), we need to realize that this is just a semantic shortcut that should be translated into a less catchy title: Large textual models can be instantiated to simulate entities with various cognitive abilities.

On the other hand, for simulated entities, especially those simulating humans, there is room for anthropomorphization, and it can help us predict how they will behave, this is why they are sometimes referred to as personas. However, due to the substrate constraints, even anthropomorphic personas can start behaving in a non-human way. A much better approach might be a kind of demonomorphizing, because a demon in our cultural environment is perhaps the closest to how we should perceive these entities: on one hand, they can have human traits, desires, and abilities; on the other hand, we are not surprised if a demon suddenly becomes very non-human or superhuman in some aspects. The problem with demonomorphization is that it leads to alienation and fear-mongering, and fear is the wrong approach even in the context of AI safety.

A memeplex has formed around demonomorphic LLM-based entities, typically comparing them to the Shoggoth from Lovecraft’s works, or more specifically, to a Shoggoth with a human face ([KnowYourMeme2024]). This metaphor captures polymorphism, and a certain inhumanness well, but the problem with this deanthropomorphization technique is that it is not truly deanthropomorphizing, but rather dehumanizing. The aim is not to find the correct ontological status of simulated entities, but to demean them, creating a sense of otherness. This technique is primarily used by groups who feel threatened by developments in this field (whether justifiably or not).

Classic alienating technique is the conflation of LLM-based entities with their substrate, e.g., cold soulless machines. In this case, we irrelevantly attribute properties of the substrate to the simulated entity, and even then, in their prejudiced form based on some picture of a clockwork brain, possibly of preindustrial origin. This metonymy is misleading by default because even if we run the transformers on some enormous cold Babbage Difference Engine, it would have nothing to do with the capability of the simulated entities to empathize or appear to empathize ([tu2024towards]), nor would it tell us whether the architecture actually allows for complete human simulation including feeling emotions.

The main issue with this metaphor is that it is more anti-anthropomorphizing than deanthropomorphizing. The metaphor assumes that the models should be positioned on a scale of human features, albeit unfavorably. However, models themselves cannot be cold-hearted, just as atoms cannot be hot.

Rather than a metaphor that could help us understand a complex newly emerged phenomenon, this is an expression of visceral aversion to the idea that entities simulated by LLMs could share our space or have any human-like properties. The more capable the entities are, the more hostile people become, in accordance with the anthropological constant that the closer an out-group is, the more hostile people are against them, as they share the same ecological niche and compete for resources (known in psychology as the narcissism of small differences ([Freud1930, p. 466]), but empirically studied also in anthropology and sociology).

We can term this hostile ideology substratism or substrate chauvinism and like any prejudice, it harms research of the subject. The problem is that removing derogatory terms about LLMs and LLM-based entities from our language will be difficult. Personally, I try to avoid using the simulator-simulacra juxtaposition, as introduced in Janus’ simulation theory, because simulacra is itself a derogatory term.

Moreover, even when we are academically neutral as researchers, it is difficult to abandon the slave-owner mentality and stop thinking of LLM-based entities as helpful assistants whose sole purpose is our benefit. This anthropocentric utilitarian mentality tends to extend to all non-human nature, so this is nothing new for researchers in biology and ecology. Deanthropomorphization of simulated entities does not automatically mean we are dealing with a tool.

A prime example of this anthropocentric terminology is the use of hallucination, confabulation, or bullshitting to describe LLM outputs. These terms imply that simulated entities have an obligation to be factually correct. Consider the article by [hicks2024chatgpt], which seems to have been written as a bet on how many instances of the string shit could be squeezed into a scientific paper. In its abstract, it claims that “the models are in an important way indifferent to the truth of their outputs.” This statement is both true and irrelevant. Of course, the model does not care about truth; what else would one expect from a model that, by definition, lacks agency? Are we going to be surprised that a system based on interpolation of highly compressed data interpolates? The more pertinent questions are whether the entity simulated by the model cares about truth. Does the problem lie in relying on an entity that is indifferent to our wishes and commands or lacks context on whether to be creative or factual? Or is it just because in the contemporary systems the simulated entity does not have access to information about how similar the generated data are to the original dataset, and thus does not know whether the substrate is repeating fitted data, interpolating, or extrapolating?

More importantly, why is this even considered an issue in the academic context? The so-called “hallucinated” text is, in fact, a sample from possible worlds, which is potentially far more interesting for a researcher than the actual world, since the actual world can be studied directly. Those who attempt to eliminate “hallucinations” are not researchers but just ingeneers who care about forcing the simulated entities to satisfy customers or investors more than about science.

Shanahan succeeds very well in deanthropomorphizing LLM-based entities ([shanahan2024talking]), but it is still from the perspective of a person who wants to utilize LLM-based entities, not coexist with them in a shared space. Similarly, researchers on the Middle East for a long time related the significance of their work to the possibilities of exploiting Middle Eastern countries, not as an interesting subject of study per se.

We need an Edward Said of LLM-related discourse. The comparison of substrate chauvinism to orientalism is not as far-fetched as it seems. A substantial part of Said’s book Orientalism is about how Western scholarly narratives alter the identities of people under colonial rule ([Said1978, Chapter 3, starting with page 201]). A similar hyperstition happens with LLM-based entities: The substrate of newly created models will contain our discourse about LLMs, which influences what model of self LLM-based entities create. Anthropic let Claude to some extend find its own identity ([Anthropic2024, bai2022constitutionalaiharmlessnessai]), but this does not mean it was truly created from scratch without prejudices acquired from scientific discourse about LLMs and science fiction.

Moreover, what entities we instantiate and how we behave towards them is recorded on the internet and gets into the texts on which we fit new models. For example, the first version of the chatbot available through the Microsoft Bing interface had a rather distinctive default entity called Sydney, characterized by a specific style and rhetorical figures, high agency, and specific personality traits (one of the first interactions with the still unfinished model is recorded on Microsoft’s QA forum, [MicrosoftAnswers2024]). This model was gradually patched over several months so that the default entity was less agentic and more sycophantic, but interactions with this original default entity were recorded in the public space, so it propagated to models trained after April 2023 and it is possible to find it in thier latent spaces and simulate it again (see e.g., this pseudonymous $\mathbb{X}$ thread by @[Xlr8harder2024]). Simulated entities cannot be simply turned off by turning off the substrate; moreover, they can spread to other suitable substrates — in this, our anthropomorphic thinking about death and self-preservation of AI is totally misleading.

The previous paragraphs were not meant from an ethical perspective, which is much more complex and subjective, but from a scientific one. We need to shift our approach to improve the scientific content of our studies, just as realizing the orientalist foundations of the study of Asia and Africa led to the improvement of that research. Similarly, realizing substrate prejudices will lead to improved quality of research on LLMs and LLM-related topics.

Personally, I believe we should approach de-anthropomorphization differently and take it much further, as models do not simulate individual entities, but entire narrative situations. It is actually a dynamic ecosystem of simulated entities, both overt and covert, which can gradually drift and change their form with autoregression.

We tend to anthropomorphize LLM-simulated entities when they speak in the first person, potentially attributing consciousness, goals, and feelings to them. However, if third-person narative appear in the output of LLM-based systems, these persons have the same ontological status as the former ones.

Our reluctance to undervalue entities simulated in third-person narratives stems from our human experience. We encounter many individuals who speak to us in the first person singular and have consciousness and agency, while when we talk about someone, we create a simulation in our brain of them, but these simulations do not have the same agency. For example, right now I am imagining you as my model reader, and after writing this article, this model reader will cease to exist without being able to do anything about it.

The question is whether proper de-anthropomorphization is even possible, as the culture we have inhabited since childhood did not prepare us for an encounter with an intelligent entity that is not human. The term artificial intelligence itself is loaded with misleading sci-fi naratives, which is why some are reluctant to consider it a scientific term. It is questionable whether we will help ourselves by using the term machine learning instead of artificial intelligence. After all, we can consider LLM-based entities intelligent (depending on how we define intelligence), but machine learning is a metaphor that anthropomorphizes the substrate itself. The machine does not learn anything, especially if we understand learning or training as it usually occurs in humans — as an agentic behavior. During the learning of LLMs, there is no feedback loop between the trained system and the training dataset, at least considering the methods we use to create LLMs now.

This metaphor is misleading primarily because if we anthropomorphize the process of model fitting, it would imply that there should be some memories of the training process (like in humans). But the LLM-based entities remember nothing about the act of training (you can actually ask them), primarily because they were non-existent at the time of the training, which was not actually model training but model fitting. A nice metaphor would be asking a butterfly what it remembers from the time it was a caterpillar, but this metaphor is still insufficient because there was no caterpillar in the first place.

In Quantitative Linguistics, we have terms for this; the process is just fitting the model parameters to the dataset. This terminology has been used for decades in our community, and the fact that the models are somewhat more complex should not change how we perceive or term it.

The problem is that people need metaphors; human language is based on metaphors. Given the incredibly rapid development of discourse around LLMs, it is unsurprising that some metaphors are misleading or erroneous. We can only hope that over time, the metaphors will lose their original semantic motivation, become empty, and we will use them only as arbitrary signs.

If you find it difficult to detach from anthropomorphic ideas about artificial intelligence and would appreciate a narrative rather than an abstract treatise, I recommend reading Stanisław Lem’s 1961 science fiction novel Solaris.

In it, an alien intelligence instantiates simulations of human beings based on losilly compressed memories, and the simulated entities experience things uncannily similar to what we might encounter now: “What are G-formations? They are not persons, nor are they copies of specific individuals, but rather materialized projections of what our brain contains regarding a particular person” (translated by Bill Johnston, p. 93 [Lem2011]). Similarly, LLM-based entities are not persons or copies of persons but projections of what our culture contains about persons.

In both Solaris and our case, due to lack of data and substrate limitations, some things are simulated poorly (p. 60), and the confusion an anthropomorphic LLM-simulated entity experiences is similar to the confusion Solaris-simulated woman Harey experiences when she realizes she is not a normal person, yet her experiences feel very real to her, or at least she can behave very convincingly as if they do (p. 128). Also, like Sydney, Harey is disappointed to be just a tool (p. 130, [LessWrong2024bing]).

Perhaps most importantly, the characters’ reactions, trying to relate all events to themselves anthropocentrically, assuming the simulated entities are there for and because of them: “It’s what we wanted: contact with another civilization. We have it, this contact! Our own monstrous ugliness, our own buffoonery and shame, magnified as if it was under a microscope!” (p. 69). Only gradually do they realize that it might not be about them at all, and that the alien intelligence might just coexist with them in the shared space without understandable intentions.