Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency

In 1948, Claude Shannon wrote on the relation between language and entropy that: “Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem.” (Shannon,, 1948, p.3). Current large language models such as Gemini, Bard, Llama, Megatron Turing, Bloom, and the GPT variants, are an engineering endeavour in these terms, albeit much more sophisticated and enormous in scale compared with Shannon’s original concept. Shannon’s original idea of entropy, that what is being measured, represented, and manipulated is form and not meaning, remains relevant now for LLMs.

Layers of artificial neural network architectures, most notably transformer-based systems and diffusion-based approaches are initialised and trained using massive datasets, often text and text-image pairs, typically sourced from the World Wide Web (WWW) and condensed and filtered via numerous automated mechanisms. In transformer based learning, the data are broken down into “tokens”, typically a few characters in length, and the LLM develops a statistical representation of the relationship between billions of different tokens. While there are various ways in which the operation of these systems can be described, they are ultimately well-defined calculations across tokens, which results in a system which, when given an input, treats that input as the initial movements through a space of possible valid moves (token concatenations), and continues the path of those movements depending on the complexity of the model and the availability of computational power (with longer paths requiring much greater resources). In a sense, the LLM is comprised of a statistical model of the relationship between tokens in a dataset, and a pathfinding mechanism optimised to generate valid sequences of token concatenations using that model, typically displayed as text. In keeping with Shannon’s comment above, we note that the processing of datasets and the generation of output are engineering problems, word prediction or sequence extension grounded in the underlying distribition of previously processed text. The generated text need not necessarily adher to “facts” in the real world.

For this reason, LLMs have been dubbed as Stochastic Parrots Bender et al., (2021), Bullshit Generators McQuillan, (2023); Hicks et al., (2024), and The Great Pretender Coldewey, (2023), amongst other terms. Though the output from such models often seems impressively meaningful, that apparent meaningfulness depends on the extent to which the meaning of a given word, phrase, or paragraph, can be represented by the relationships between tokens in the original dataset. For example, multilingual models trained on multilingual datasets show poor quality performance for languages outside the status quo Kreutzer et al., (2022); Khanuja et al., (2021); Wu and Dredze, (2020). The language that an LLM represents is conceived by its engineers and developers as something static and complete, which can be captured in the relationships between tokens.

From this description we can see that claims regarding linguistic capabilities of LLMs depend on two implicit assumptions of language. The first is what we call the assumption of language completeness - that there exists a “thing”, called a “language” that is complete, stable, quantifiable, and available for extraction from traces in the environment. The engineering problem then becomes how that “thing” can be reproduced artificially. The second assumption is the assumption of data completeness - that all of the essential characteristics can be represented in the datasets that are used to initialise and “train” the model in question. In other words, all of the essential characteristics of language use are assumed to be present within the relationships between tokens, which presumably would allow LLMs to effectively and comprehensively reproduce the “thing” that is being modelled.

Both of these assumptions are rejected by an enactive view of language, which sees it not as a “thing” to be captured by text data, but a practice in which to participate, whether that participation is through speech, written, sign, or other modality. In contrast with computational approaches’ emphasis on form, the enactive approach to language recognises that what truly matters for language is its meaning (in this there are strong resonances with Bender and Koller, (2020)’s critical examination of the relationship between form and meaning in language model output). As such, the enactive approach to language starts not with tokens of verbal activity, but with the fundamental issue of agency, embodiment, precarity, and how meaning arises within situations where things matter to those involved.

Enactive descriptions of agency begin with a process of continuous precariousness – a network of processes constituting a body in a constant flux of degradation for which engagement with and acting on the world provides some prospect of continuity through the managing of conflicts and tensions that result in the degradation Beer and Di Paolo, (2023) .The logic of precarious but continuous dynamic organisation is one that arises in the domain of organic, bodily being, but is general across other domains. Thompson and Varela, (2001) first outlined three such domains – the organic (that of biological bodies), the sensorimotor (that of skilfully engaged bodies), and the intersubjective (that of interacting skilfully engaged bodies). These three domains can be conceived of as distinct, but nevertheless interact in inextricable ways. Organic needs can drive and constrain sensorimotor activity, while those skilful doings in turn affect and transform those biological dynamics. Similarly, the intersubjective is a coordination of skilful bodies interacting with one another, but this domain is constrained by organic and skill-relevant dynamics, while also imposing constraints in turn. Think of the ways in which societies (a collection of intersubjective phenomena) are substantially organised around food production and meal times, constrained by the biology of hunger and nutrition. Hunger and nutrition are in turn affected by social norms and standards around what kinds of foods are generally available (within the society’s “cuisine”) and standardised meal times. These impacts occur over multiple timescales too, as we can see in such phenomena as available foods and cultural resources leading to rises and falls in prevalences of Type II diabetes Magliano et al., (2019) intersubjective phenomena driving and constraining biological ones.

Within an enactive perspective, agency generally, then, is driven by tensions of precariousness and risk. There are continual needs to diffuse a tension by, for instance, expending energy to move and act. These actions, however, necessarily introduce new tensions, such as a hunger arising from the expenditure of energy. Di Paolo et al., (2018) provide an extensive characterisation of this inter-dependency between the organic, the sensorimotor, and the intersubjective, and extend this analysis of agency into the linguistic domain. Linguistic acts are those which manage an inter-subjective tension – a precariousness in the coordination between two or more people engaged together in a shared activity. Their analysis places great emphasis on the variety of ways in which such shared activities occur at multiple temporal and spatial scales, and that the resolution of a tension at one scale tends to introduce tensions at another.

There are two crucial implications for our understanding of language that follow from Di Paolo et al., (2018)’s account. The first is that we are always doing more than one thing. Linguistic actions are made within a nested set of contexts. When we encounter other people we are always already in some broad form of coordination with them in which we are participating. For instance, if we meet someone for the first time at a job interview we are already both participating in the behaviour setting Barker, (1968); Schoggen, (1989) of job interview. Our actions are thus already coordinated at a coarse grain of analysis, but also constrained – there are pressures and processes which will guide and drive our behaviour appropriate to the setting. Thus, coordination produces new tensions that must be managed through our linguistic skills. We spend pretty much all of our lives within behaviour settings Heft, (2001), and our actions are organised accordingly, classically illustrated with the slogan, “people in church behave church, people in school behave school”. Within many settings, however, there are multiple participation genres Bakhtin, (1986) possible. There might be collaborative work, team sports, or more. These are separate but entangled with the behaviour setting and produce their own set of pressures and constraints. Any given situation is a complex of these interacting constraints that is being managed by our behaviour at different scales from very brief microexpressions accenting our speech to the general tone and vocabulary of spoken utterances, to the broad overall shape and sequence of actions appropriate to the behaviour setting. Linguistic agency is not just the words uttered in this situation, but the whole process of managing all of these structured, intersubjective aspects of living.

This flow of tensions brings the second implication of the enactive account of linguistic agency to the fore. From an enactive perspective, any linguistic act is necessarily partial or incomplete in two different ways. First, an individual’s utterances are partial in that they are always made in response to (or in anticipation of a future response) and in coordination with another person as part of a shared on-going activity. Second, while an utterance or other linguistic act manages the tension arising at one level of the interaction, it cannot resolve all such potential conflicts and therefore introduces new tensions in the nested contexts that characterise the situation as a whole. These new unresolved tensions become the animating force for the driving forward of the interaction as it continues to unfold, precariously, over time.

This perforce very brief sketch of the enactive theory of linguistic agency illustrates how essential embodiment, participation, and precarity²²2Readers will note that we use two related terms in discussing the fragility that provides stakes to the actions of autonomous systems: precariousness and precarity. Enactive researchers Di Paolo, (2009); Beer and Di Paolo, (2023) have developed a technical definition of precariousness, which is a tendency of component processes of a self-producing network to lead that network toward dissolution - this is balanced by organisation of the network as a whole to lead the network toward stability. This dynamic occurs in different domains - organic, sensorimotor, intersubjective. The intersubjective domain overlaps substantially with the concept of precarity, which refers to the vulnerability of the agent to injustice, exploitation, or social or physical injury on the basis of their participation within various social networks and social activities. Exploring the fine-grained details of the relationship between these overlapping concepts is beyond the scope of the present paper, but we draw the reader’s attention to it as a consideration for further work to be done in this area. We discuss the explicit relevance of precarity for our understanding of the differences between LLMs and human beings at length in Section 4.3 and Section 5. are to human language practice. We can see how strikingly different the conceptions of language maintained by computational approaches and enactive cognitive science are.

As noted above, this is not just a matter of nit-picking, but a divergence of fundamental tenets. Proponents of stronger claims regarding the validity of LLM operations as intrinsically linguistic may suggest that science and engineering often work with minimal or limited cases in the first instance, to grasp principles, before ’scaling up’ to more richly contextualised, complex settings. Disembodied, text-based interaction may only be the start, with full-bodied, embodied, real-time linguistic interaction being the end goal.

Setting aside for the moment that there is nothing minimal about the storage, computational, or environmental costs of LLM operation or the underlying profit maximization objective driving LLM development, we can see that what constitutes minimal linguistic interaction from an enactive perspective is not something that is constrained and disembodied, but something that is blunt and lacking nuance, but still embodied, participatory, and precarious. What constitutes minimal valid linguistic activity looks a lot more like our reciprocal interactions with pre-verbal infants, or perhaps even the playful negotiation of contextualised behavioural coordination with our pets, than it does disembodied grammatical sentence generation.

The enactive and engineering stances nevertheless share some, at least superficially, common ground which we feel is important to recognise. In the following section, we address first where these two viewpoints can be seen to agree, and explore in Section 4 how the three aspects of enactive language are absent in the operation of LLMs.

While developing a capacity for language does involve exposure to, indeed immersion within existing linguistic practice, we have noted above that in the case of human beings that immersion is never a passive thing. In infancy, the child is involved in various kinds of interactions and games with their caregivers in which language starts having a peripheral role, but which becomes increasingly central. These various activities are ones in which the coordination between the parent and child that supports the development of linguistic skills is bodily – physical, concrete, and grounded in the shared experience of the ongoing interaction Smith et al., (1988); Tamis-LeMonda et al., (2001); Yu et al., (2009).

There is no question that human beings interacting with an LLM are engaged in linguistic activity. Users are mature linguistic agents. Indeed, in many cases for us academics, we are engaged in a complex reflective process of evaluating the linguistic agency of the LLM itself. Interactions with an LLM thus themselves occur immersed within a nested set of interacting contexts. The enactive perspective makes it clear that all linguistic agency is embodied, and that the body, in various ways, plays a role of animating force, enabling condition, and set of constraints Chemero, (2023); Di Paolo et al., (2017); Thompson, (2010); Varela et al., (2016). We should, then, consider carefully the kind of embodied interactions that take place when a person uses an LLM.

We have noted that the user of an LLM is a mature linguistic agent, someone who is a proficient participant in typical human discourse. We know this because part of the process of engaging with an LLM requires the use of widespread but nevertheless deeply technical skills, such as literacy and typing on a keyboard. The user acts toward the model through the technological medium of the keyboard, and the response from the model is typically displayed on a computer monitor, formatted for reading according to a particular genre of text-based interaction in online communication. Other modes of interaction exist, but it is important to recognise that this ‘typical’ mode is primary. Where other forms of technology such as speech-to-text is used to input prompts, for instance, only the sequence of recognised words, with some interpolated grammatical markings are presented as input. Other aspects of the utterance, such as cadence, rhythm, tone of voice, and accent are either ignored, or indeed, become impediments to the interaction; for instance, in the context of the English language, the more accents diverge from those considered ‘standard’ American or British English, the less functional such interfaces become Johnson et al., (2022).

At the other end of the interaction with an LLM, screen readers and other assistive technologies enable engagement with output from the model that differs from the default ‘monitor’ case (see Section 4.1.1 below). But again, no meaningful peculiarities of those modalities are produced by the LLM itself. The non-typical output is dealt with purely by the assistive technology on the basis of the text output by the model.

On the basis of this description the role of the body in participation in text-mediated communication would appear to be very limited. A few twitching fingers or moving lips, the skipping of eyes across an array of displayed text, are all that seem to be needed. But bodily involvement in textual communication is much richer than this.

Referring to a playful passage by the author Italo Calvino, Caracciolo and Kukkonen (2021) note that reading invariably takes place in a physical context, one often carefully tailored to the task that needs to be done. While Calvino writes about sitting with a book, or lying down with one, or putting up one’s feet, or propping up the book, the experience of the text is contextualised in this physical manner. In the case of the kind of machine-reading involved with LLM interactions, we can note how aspects such as font, font-size, and dark or light themed interface affects the interaction for the human, but makes no difference for a machine.

More than this, though, Caracciolo and Kukkonen (2021) explore the ways in which our capacity to engage with, and become involved with text, depends on fundamentally embodied capacities such as shared attention, emotional resonance, and appreciation of rhythm and flow. The authors note that their approach is most directly relevant to long-form narrative, but is grounded in embodied cognitive science of language use more broadly. Some aspects of these phenomena might leave traces in the corpora available to LLM training but most do not. It is the dynamics of shared involvement with a story or utterance, and how these are responsive to the real and imagined readers, that play a significant role in our engagement with text.

More pointedly, we have noted that people involved in linguistic interaction are always embedded within a nested set of contexts. These contexts have multiple facets to them, some of which we raise in following subsections. For our present purpose, we note that the kinds of goals and tensions being managed through linguistic activity invariably involve bodily groundedness. There is a reason that we are involved in particular conversations, participating in particular settings, trying to achieve particular ends, at any given time. While many of these would seem to be so flexible and negotiable as to be autonomous, they nevertheless have an ultimate grounding in constant needful freedom of organic being (these issues are unpacked at length in Di Paolo et al., (2018)). The shared intention and shared experience that is a large part of what enables successful interaction, coordination, and mutual understanding, is something that arises because our bodies enable it. When words fail, we can turn to experience to help re-calibrate an interaction, to negotiate new shared references. Language is always necessarily grounded outside of language by its relationships to these contexts of shared activity and bodily experiences. That machines lack such bodily experiences is not a trivial difference. It is instead something that results in an artificial but fundamental limitation to the possibility of machine language. Even where an interaction between two human beings is mediated by text, these bodily aspects ground language use in real experience of our inexhaustibly rich world. This results in artefacts of LLM output that demonstrate these important constraints.

Following this more active, participatory, and embodied perspective on language, we find more stark and important differences between humans and machines beyond the capacity to calibrate against embodied experience. It is precisely these active, collaborative, and dynamic aspects of languaging which are not and cannot be captured in static representations and included in a corpus of training data. Languaging –- including the casual chit-chats as we enter an elevator with others, gestures, body languages, tones, pauses, and hesitations –- is not something that can be entirely captured in text but is an often fleeting phenomena without clear formalizable rules. These embodied linguistic participations can be peculiar, unrepeatable and take on a “life” of their own in a way that is not predictable Di Paolo et al., (2018).

The social character of linguistic agency is not coincidental. We have noted that LLMs are developed on the basis of a large body of existing practice and textual language use. But even in enormous datasets, that body of practice is fixed. It is not a body to which the model contributes as it “learns”, given that even when new text is generated, it is regurgitated and reconstructed on the basis of the training corpus. This is quite in contrast to human linguistic agency in which participants both experience the practices and contribute to them. Although the data that forms a model’s training set is partly sourced from human linguistic interaction, at best it captures a snapshot of a dynamic human textual linguistic interaction and ‘freezes it in time’. Training data therefore is not only necessarily incomplete but also lacks to capture the motivational, participatory, and vitally social aspects that ground meaning making by people. In fact, elements of motivational, participatory, and social aspects of meaning-making often defy codification and datafication. For instance, we often make ourselves effectively understood from what has been left unsaid in a conversation, or via tones of voice that transform the meaning of an utterance, as in sarcasm. Generating and detecting humor, sarcasm, and jokes, on the other hand, are qualities that remain impoverished in LLMs. Jentzsch and Kersting, (2023), evaluating ChatGPT outputs, for example, found that over 90% of 1008 generated jokes they examined were the same 25 Jokes.

When we speak we frequently misspeak, we hesitate, stumble over words or use the wrong words, or the wrong constructions. The people with whom we are engaged provide support as we fumble our way forwards, and we in turn support them Dingemanse and Enfield, (2024). Dingemanse et al., (2015), for instance, found that clarification questions occur on average every 84 seconds in normal conversation. The level of frequent and adaptive clarification we see in normal human conversation occurs due to an underlying shared sense of direction to the discussion, even where its conclusion is not known. When you have interacted with an LLM, how frequently have you encountered requests for clarification? (Beyond perhaps stock phrases in response to very explicit expressions of frustration in the prompt.) When a person has knowledge of a domain, they typically have a strong sense of how to ask questions and what details to seek or avoid in order to support an on-going dialogue. People are aware both of what they know, what they don’t know, and how well the conversation overall is going. The seeking of clarification is a kind of activity that is grounded in a shared direction for the conversation, in which the discussion is continually being sculpted and steered as a collaboration. To be capable of clarification and repair, the participants have to be sensitive to divergence and breakdown. Indeed, the lack of question asking, or metacognition regarding the tentativeness of much of our understanding, is part of what has resulted in LLMs being experienced as fluent in the ‘mansplaining’ idiom Harrison, (2023).

These collaborative activities sometimes involve helpful corrections, sometimes carefully ignoring invalid statements, sometimes enthusiastically adopting new meanings for old terms, or new words that give better expression to an experience that we share but neither of us can yet put satisfyingly into words.

To understand language is not to be able to produce grammatical strings of words, but rather to participate in this process of negotiated, participatory meaning making. As we have noted above, it is this active, participatory character of language that has led enactive researchers to adopt the verb languaging in preference to the nominal ‘language’ in the research literature. It is an inherently collaborative, dynamic negotiation of meaning, the textual aspects of which are only part of the story. This remains the case even in the constrained text-centric domain of online interaction.

This emphasis on participation and coordination over sentence construction means that much of the research comparing human and LLM production is simply not germane to the question of human linguistic activity. There is a wealth of such research now. Analyses find some parallels between the two (e.g. in variation of word use based on recent semantic context from both its own output and prompt input, Cai et al., (2023)), and some differences (e.g. in appropriate coordination of output with scalar and general conversational implicature of recent output and prompt text; Qiu et al., (2023)).

Given that a LLM is a curve fitted to a dataset with a sophisticated mechanisms for sampling, such analyses have a potentially important engineering role, in evaluating the extent to which there is appropriate correspondence between the map (the LLM) and the territory (human word production in text-based linguistic activity). They cannot, however, provide any argument for the validity of conflating map and territory. Evidence for that simply lies outside of word production, in the field of embodied, participatory, and value-laden interaction between agents. It is possible that artificial linguistic agents might be developed and engineered in the future, but evidence of such success cannot be on the basis of patterns of fluent token sequence production.

The enactive conception of language, because it involves dynamism and sociality, is one which recognises every linguistic act to be radically incomplete. According to this perspective, language is a partial act that can only be completed when it is taken up and extended, embellished, or steered and redirected, by other agents. This can be other people engaged in a complementary or counter move, which is itself also incomplete, dependent on that gesture or utterance being taken up in turn. Language is always and inevitably overspilling the kinds of information that can be made to ‘freeze in time’ within specific computational data structures and used to engineer LLMs. We can refine our understanding of this contrast further by following the lead of enactive thinking and considering with some care just what kinds of embodied actions are involved in human interactions with LLMs.

Humans are not brains that exist in a vat in a social, political, and historical vacuum but are embodied beings marked by “open-ended, innumerable relational possibilities, potentialities, and virtualities.” We necessarily have points of views, moral values, commitments, lived experiences, joys and grievances Di Paolo et al., (2018). We are sense-making organisms that relate to the world and others in a manner that is significant to us. We care about the world and our place in it. Excitement, pain, pleasure, feeling of embarrassment and outrage are some of the feelings that we are compelled to feel by virtue of our relational existence. As living bodies (which themselves change over time), we are compelled to eat, breathe, (sometimes) fall ill and fall in love. Human language is not something that can be finalised and defined once and for all, but is always under construction and marked by ambiguities, imperfections, vulnerabilities, contradictions, inconsistencies, frictions and tensions. If anything characterizes human being, it is our peculiarities, fallibility, and idiosyncrasies, which stands at odds with machines. A machine, by definition, is not capable of grasping these qualities. As Alan Turing puts it: “if a machine is expected to be infallible, it cannot also be intelligent” Turing, (1947).

Importantly, social norms and asymmetrical power structures permeate and shape our linguistic agency and the world around us. This means that factors such as our class, gender, ethnicity, sexuality, (dis)ability, place of birth, the language we speak (including our accents), skin colour, and other similar subtle factors either present opportunities or create obstacles in how a person’s capabilities are perceived. Recognising this, we now look at the final crucial aspect of enactive linguistic agency; precarity. Precarity is often present in interactions with LLMs, but not in a manner that can support the possibility of machine understanding, sentience, or agency.

Linguistic agency, as described by Di Paolo et al., (2018) (see also Cuffari et al., (2015); Di Paolo, (2021)), is a matter of continuous concernful management of conflicts, frictions, and tensions. These tensions emerge within intersubjective interactions, and while they can be addressed, every action taken to address them will unavoidably set up conditions for new tensions and mis-coordinations either immediately at a finer grain of action, or at some point in the future. Agency, within the enactive conception, whether of its basic biological kind, at the level of skilful action in the world, or in the intersubjective domain in which we find language, is seething with frictions, and the possibility of failure and the unravelling of the ongoing process in question (the interaction, the skilled action, the living body).

LLMs do not participate in social interaction, and having no basis for shared experience, they also have nothing at stake. There is no set of processes of self-production that are at risk, and which their behaviour continually stabilises, or at least moves them away from instability and dissolution. A model does not experience a sense of satisfaction, pleasure, guilt, responsibility or accountability for what it produces. Instead, LLMs are complex tools, and within any activity their roles is that of a tool Cuskley et al., (2024). Human beings are animate, skilful beings whose very existence enacts values of continued being Weber and Varela, (2002); Di Paolo, (2009). Human interaction is necessarily enmeshed in intersectional webs of power, privilege, and responsibility Crenshaw, (1989); Vassilicos and McGann, (2023). Social interactions for human beings, even those that are fairly routine such as brief exchanges in retail settings, idle chit-chat in a waiting room, or online comment exchange, constitute opportunities and risks based on our standing in the communities, settings, and contexts in question. Languaging activity is precisely a matter of how these various opportunities and risks are perceived, engaged with, and managed. Not so for machines. Nothing is risked by ChatGPT when it is prompted and generates text. It seeks to achieve nothing as tokens are concatenated into grammatically sound output. All of the values in the interaction between the machine and its user are those of the user and those invested in the production of the systems, such as engineers and developers, and most importantly big tech corporations, emerging AI companies, and start-ups. In fact, in the current climate, values such as model “performance”, “efficiency”, and “scale” Birhane et al., (2022) are considered the most desirable virtues at the heart of the field. It is important to note that although LLMs are devoid of inherent experiences or values, these systems do encode the values of those that develop and deploy them. Subsequently, these values (“performance”, “efficiency”, and “scale”) enable wealth accumulation, market dominance, monopoly, and power centralization  O’Neil, (2017); Noble, (2018); Eubanks, (2018), often at the cost of values such as “justice”, “fairness”, and “privacy” Birhane et al., (2022).

AI systems are derived of (and built on) human activity – from the training data, to the engineers and developers, to the societal uptake needed for them to succeed. That an LLM cannot be in possession of power, intent, or agency, does not mean that the LLM is an “objective”, “neutral” or value free tool. On the contrary, these tools not only encode the values of those that develop and deploy them, they also ingest societal power asymmetries through the mass scraping of data from human interactions, and centralise power in the hands of the few with compute power and other resources required to own, develop and deploy them Birhane et al., (2022); Pratyusha, (2020); Benjamin, (2020). The extent to which their engineering is an expression of a given set of values (instantiated in choices about data sourcing, fine-tuning, and deployment), is the extent to which those values are amplified and impressed upon anyone who interacts with them or is affected by how these systems are used in decision making. Human beings experience those institutional and technological manifestations of power asymmetry in different ways. In recent years we have seen the rise of a form of linguistic phenomenon that is both a response to such forms of power, and a demonstration of linguistic agency that we believe LLMs of the current type are unlikely to ever be capable of, an example of which is the phenomenon of ‘algospeak’.