Codebook LLMs: Adapting Political Science Codebooks for
LLM Use and Adapting LLMs to Follow Codebooks

Measurement is essential to social science: how can broad concepts be expressed as precise constructs, raw data be turned into valid measurements, and the resulting measurements incorporated into downstream analyses? For many political science research projects, text is the best source of data, either as the object of interest in itself (e.g., political speeches, press releases, or political manifestos), or as a useful source of events in the world (e.g., news reports on protests, political violence, etc.). In order to transform unstructured text data into valid quantitative measurements of social science constructs, many researchers rely on human annotation via codebooks, documents that define concepts and outline annotation procedures Krippendorff (2018).

Manually labeling all texts in large-scale projects according to codebook instructions is extremely costly. For well over a decade, analysts have attempted to automate or scale-up manual content analysis via supervised or unsupervised machine learning Grimmer and Stewart (2013). Recently, the promise of better classification performance with fewer (or no) training examples has led analysts to turn to generative large language models (LLMs) to annotate text (Egami et al. 2024; Gilardi, Alizadeh and Kubli 2023; Peskoff and Stewart 2023; Ziems et al. 2024; Wu et al. 2023; Atreja et al. 2024). LLMs use transformer-based deep learning models with a language model loss function trained on large corpora (typically scraped from the web). These models have grown dramatically in the number of parameters they are trained on; for instance, there are roughly 1,000 times more parameters in GPT-3 ($2\times 10^{11}$ parameters; Brown et al. (2020)) than BERT ($3\times 10^{8}$ parameters; Devlin et al. (2019)).

Much of the excitement for LLMs from social scientists comes from LLMs’ potential for zero-shot classification, meaning one can obtain classification predictions with zero human-labeled training examples. Under this zero-shot paradigm, LLMs take as input a sequence of tokens (word pieces) consisting of the instance of text one wishes to annotate and annotation instructions. Then the LLM outputs a sequence of tokens which can be parsed to obtain the predicted class.

In an ideal world with perfect LLM accuracy, a political scientist working on a specialized corpus could specify the exact construct they wanted to measure from text and use LLM predictions in downstream analyses. For example, suppose a political scientist wanted to measure the number of protests from decades of news articles. She might start with prompt templates from existing prior work using LLMs for social science annotation, (e.g., Ziems et al. (2024); Figure 1), and input to an LLM: “A hundred people marched through the capital in opposition to the proposed law. Is this A. a protest or B. not a protest?” Ideally, the LLM would output “A. a protest” and the instance of text would be recorded as a positive instance of the protest class. However, because the ultimate goal is to measure a complex social construct (here, a “protest”), this kind of definition-agnostic LLM prompt can have negative empirical consequences due to the following oversimplified assumption.

Universal label assumption. We define the universal label assumption as the assumption that natural-language classification labels (that an LLM predicts) have one universal meaning. This assumptions implies that class labels, for example, “protest” or “liberal,” mean only what the LLM has inductively learned from its pre-training data, and this meaning matches the concept a social scientist wishes to code. This assumption is related to the “background concept” term used by Adcock and Collier (2001, pg. 530), in which there is a “constellation of potentially diverse meanings associated with a given concept”. In many natural language processing (NLP) tasks, such as sentiment classification for product or movie reviews (Pang, Lee and Vaithyanathan 2002), the “background concepts” cued by the “positive” or “negative” sentiment labels are likely to be sufficient. However, we provide the following three real-world examples in which the universal label assumption breaks down in social science applications.

First, Ziems et al. (2024) show that an LLM can accurately categorize political statements into “Liberal”, “Conservative”, or “Neutral”. In a specific instance (Figure 1), the LLM correctly predicts that a pro-labor union statement is a liberal political position. However, it does so by adopting the universal label assumption we describe above, drawing on the large pre-training corpus to apply a standard definition of liberal and conservative. However, in specific contexts this universal label assumption is likely to break down. Specifically, in Australian politics, the Liberal Party is a conservative, center-right/right-wing party that opposes labor unions. In the context of Australian politics, asking whether a statement is liberal invites ambiguity. Does the LLM predict “liberal” in an ideological sense or “liberal” in the sense of the (conservative) party in Australia?

A second example comes from the extensive study of protests in political science and sociology. While all definitions of protests include a shared core (claim-making or demands for political or social change outside of formal political channels), specific projects use different definitions of protests, categorize protests at varying levels of granularity, and apply different exclusion criteria. These specific definitions depend on the empirical project and are a crucial aspect of high-quality social science measurement. For example, the BFRS dataset on political violence in Pakistan (Bueno de Mesquita et al. 2015) excludes all non-violent protests, while making a distinction between a “riot” and a “violent political demonstration”. In contrast, the common automated event data ontology CAMEO (Gerner et al. 2002) collapses these two into a single “riot” category, while also coding other non-violent protest tactics. The MPEDS project (Hanna 2017, Appendix A) applies much more thoroughly defined requirements about the identity of those protesting than the CAMEO ontology does. CAMEO excludes resignations and complaints by elites as a form of protest, while the Electoral Contestation and Violence (ECAV) dataset (Daxecker, Amicarelli and Jung 2019), which focuses exclusively on subnational protests in the context of elections, includes these as protests. Applying these different operationalizations of “protest” to the same corpus should not produce the same measurements.

As a final example, the empirical study of populism in politics has employed several conceptions of populism. Without providing a precise definition to an LLM labeling populism in text, a researcher cannot be sure if the LLM is labeling populism by rhetoric (e.g., anti-elite rhetoric, rhetoric about the “general will” of the people, or a Manichean worldview), by populist policies (e.g., welfare chauvinism), or conflating ideology and populism (e.g., conflating right-wing nativism with populism) (Meijers and Zaslove 2021). A measurement strategy that conflates these varying definitions of populism will prevent researchers from answering questions such as the relationship between populist rhetoric and populist policies.

Codebook-construct label assumption. In contrast with the universal label assumption, we define the codebook-construct label assumption as the assumption that a natural-language classification label is a placeholder for a rich operationalization of a social science concept that analysts define, constrain, and clarify in a specific codebook document. The goal is to use the LLM as a social measurement instrument and as such the LLM should comprehend and comply with the definition of a class label from the context of the codebook. This assumption aligns with more general assumption made by Raji et al. (2021) about datasets used by machine learning and AI, “all datasets come with an embedded perspective — there is no neutral or universal dataset.” Similarly, Adcock and Collier (2001) argue that broad “background” social science concepts must be “systematized” with an explicit definition, and then operationalized into measures before they can be used to generate useful data.

In this work, we argue that social scientists using LLMs for measurement should make the codebook-construct label assumption. We describe conceptual work that prescribes how to incorporate this assumption into the codebook-based annotation task and pipeline (see Figure 2 for an overview), then assess an LLM’s performance on the task using empirical experiments on real-world political science datasets. In doing so, we aim to address the following research questions:

RQ1. What is the zero-shot classification performance of LLMs on real-world political science datasets under the universal label assumption?

Specifically, if we provide only a label name without any other definition or clarification and zero training examples, can an LLM accurately predict the gold standard label for a document? This question has been addressed by previous work (Ziems et al. 2024; Atreja et al. 2024), but we apply the RQ to three new datasets: the Crowd Counting Consortium (CCC) dataset on protests in the United States (Crowd Counting Consortium 2024), the BFRS dataset on violence in Pakistan (Bueno de Mesquita et al. 2015), and the Manifesto Project corpus (Lehmann et al. 2017); see Section 4 for details. We chose these datasets because they have rich class labels and codebook definitions that allow us to rely explore this RQ. For example, from the CCC dataset on protests, does the LLM pretraining data include enough information for a model to reliably distinguish between, for example, a protest, rally, demonstration, and march?

We contrast this question about the viability with LLMs under the universal label assumption with our second research question:

RQ2a. What is the zero-shot classification performance of LLMs on real-world political science datasets under the codebook-contextual label assumption?

Given that social scientists put a great deal of effort into conceptualizing categories and writing detailed codebooks that guided the gold-standard human annotations, we hypothesize LLMs will perform better if they are provided with the entire set of codebook instructions in their input prompt. However, the greatly expanded size of the prompt may also pose challenges for LLMs if they struggle to attend to all parts of the document-codebook prompt (Levy, Jacoby and Goldberg 2024). Additionally, under the codebook-contextual label assumption and zero-shot paradigm, we seek to understand how the format and composition of the codebook affects LLM performance, specifically:

RQ2b. What components of the codebook impact classification performance?

While zero-shot classification is efficient and straightforward, social scientists typically want the best possible accuracy during the measurement phase since these measurements will be incorporated into inference. Thus, we also explore improving performance by further training LLMs via a paradigm called instruction-tuning (Wei et al. 2022; Sanh et al. 2022; Wang et al. 2022; Longpre et al. 2023; Wang et al. 2023). When instruction-tuning, we take a classification dataset with gold-standard classification labels (i.e., the texts labeled by domain-experts) and further train the LLM on these gold-standard labels via a supervised language modeling objective. This paradigm has proven extremely promising in boosting LLMs ability to follow instructions and has increased zero-shot performance on suites of “traditional” natural language processing tasks (Wei et al. 2022). Promisingly, instruction-tuning has recently been show to scale to a local GPU using parameter efficient training methods such as quantization and/or training low rank adapter weights (Hu et al. 2022; Dettmers et al. 2024). Our hope is that if we eventually gather enough codebooks and their corresponding datasets, we might be able to train the LLM to follow the codebook-contextual label assumption and read, comprehend, and comply with the codebook definition of class labels. This leads to,

RQ3a. How does instruction-tuning an LLM on a single codebook and supervised examples compare to zero-shot classification at inference time?

Ideally, an LLM trained on the broad task of document classification using social science codebooks would perform well on the task of annotating documents according to codebook instructions, even on codebooks it had not seen during instruction-tuning. Thus, our final research question asks about the generalizability of the task:

RQ3b. How does instruction-tuning an LLM on multiple codebooks and supervised examples affect classification performance at inference time on a new codebook?

In subsequent sections, we examine these research questions using a one of the best-performing “local” models, (i.e. weights can be downloaded to a local machine for reproducibility), Mistral 7B Instruct (Jiang et al. 2023). We also describe related work (Section 2), formalize the codebook measurement task and assumptions (Section 3), describe our data collection and curation (Section 4), and walk through our empirical findings (Section 5).

Will this work stand the test of time? Given the current rapid pace of development and improvement in performance of LLMs, it is reasonable to ask whether this work will stand the test of time and be relevant in 5, 10, or even 20 years. We keep this in mind and intentionally compare conditions within a single LLM, rather than comparisons between multiple LLMs. This helps isolate our research questions which focus on codebook component ablations and comparing zero-shot and instruction-tuning conditions. We believe our conceptualization of the codebook-specific task, assumptions, and instruction-tuning pipeline as well our semi-structured LLM codebook format should be agnostic to improvements in the base models.

Our work sits at at the intersection of a rapidly growing methodological literature that uses LLMs for social science and political science text analysis, technical improvements in NLP machine reading comprehension, and a mature literature on measurement in the social sciences.

In this section, we formally define the codebook measurement task as well as the universal label assumption and the codebook-construct label assumption. For now, we only focus on multi-class, single-label classification; in other words, the classifier predicts from multiple classes but the classes must be mutually exclusive and only one class label can be applied to each instance.

Let $\mathcal{Z}$ be a finite set of $L$ concepts/labels described in natural-language words or phrases. For example, for the BFRS dataset employs $L=8$ labels, $\mathcal{Z}=$ {assassination, drone assassination, attack on state, conventional attack on government forces, guerilla attack on government forces, government attack on civilians, riot, terrorism, threat of violence, violent political demonstration, other}.²²2We simplify these labels slightly from the original BFRS labels. See Appendix B.

Let $C$ be the codebook, a sequence of tokens (words or word pieces) that operationalize $\mathcal{Z}$. The codebook is a concatenation of the natural-language operationalization for each of labels in $z_{1},z_{2},\dots,z_{L}\in\mathcal{Z}$, namely $C=[C_{z_{1}},C_{z_{2}},\dots,C_{z_{L}}]$. For example, in BFRS, the (abbreviated) description of $C_{\textsc{terrorism}}$ is “Premeditated, politically motivated violence perpetrated against noncombatant targets by subnational groups or clandestine agents.” Each $C_{z_{j}}$ can consist of longer or shorter operationalization with or without clarifications or examples (see Section 4.3).

The goal of the codebook measurement task is to infer the label $\hat{Y}_{i}\in\mathcal{Z}$ given the tuple $(C,X_{i})$ for all $i$ in the inference dataset where $X_{i}$ is the text of interest. This input text could be a full document, one or more sentences, or a quasi-sentence (e.g., the partial sentences labeled in the Manifestos dataset).

In practice, there are several measurement paradigms: (a) humans can manually assign $\hat{Y_{i}}$ by reading $X_{i}$ while adhering to $C$, (b) models can infer $\hat{Y_{i}}$ zero-shot (with no training examples) for each $X_{i}$, or (c) supervised models can be trained via a training dataset of examples: $\{(C,X_{i},Y_{i}\in\mathcal{Z})\}_{i=1}^{N_{\textsc{train}}}$.

Formally, the universal label assumption is

or, in other words, the measurement of labels is independent of their exact definition in the codebook $C$. In contrast, the codebook-construct label assumption is

or, in other words, changing the codebook from $C$ to $C^{\prime}$ could result in a different label $Y_{i}$ for the same document $X_{i}$, even when holding $\mathcal{Z}$ constant. We argue applied researchers should adopt the codebook-construct label assumption given their projects typically center around precisely defining the constructs that are important to their substantive research question.

Our empirical analysis examines how well LLMs follow the constructs defined in codebooks and can predict gold-standard labels from these codebooks and input texts. To do this, we collect three separate political science datasets, each of which provide document- or (quasi-) sentence-level labels according to a codebook-defined schema: the Crowd Counting Consortium (CCC) dataset on protests in the United States (Crowd Counting Consortium 2024), the BFRS dataset on violence in Pakistan (Bueno de Mesquita et al. 2015), and the Manifesto Project corpus (Lehmann et al. 2017). We obtain their original codebooks and the text that the data was coded from (see Section 4.1 for additional details).

We focus on single-label, multi-class classification: for each document, we seek to recover a single gold-standard label. For the Manifestos dataset, this label is the Manifesto Project policy/ideology label for a (quasi) sentence in a political party’s manifesto. The BFRS and CCC datasets each code a set of variables for events from news articles, including information on the actors and motivations involved in each event. We focus solely on the event categories provided in each dataset.

We chose these three datasets because they represent a range of political science topics: political violence, contentious politics in the United States, and comparative party politics. The datasets also reflect wide variation in the number of classes and the length of input text (Table 1). We also hypothesized that partial overlap in some of the categories in BFRS and CCC related to protests would allow codebook-specific distinctions from similar categories In contrast to many social science related tasks in NLP, each of these datasets were constructed by social scientists with a focus on measuring specific political concepts and phenomena. This provides a point of departure from previous NLP work on reading comprehension and previous work on LLM-based annotation, in that we focus on categories that are carefully defined in the context of a specific research question, not “universal” labels that the LLM would have been trained on. How well LLMs perform on these three codebooks and datasets is thus interesting both for applied social science researchers, who would like to use LLMs to annotate text according to carefully specified instructions, but also for the NLP community, in providing much more difficult tasks than previous benchmarks.

We evaluate an LLM’s performance on our codebook task and BFRS, CCC, and Manifestos datasets across two primary experimental conditions: zero-shot and instruction-tuning. We use the one of the best “local” base models available at the time we began our experiments, Mistral 7B Instruct Jiang et al. (2023).

In a “zero-shot” approach, the LLM is not given any training examples, only a task description consisting of the entire codebook and the documents to be labeled, $\{C,X_{i}\}_{i=1}^{N_{\textsc{inf}}}$. The LLM’s task is to provide the label from the codebook that best matches the document provided, $\hat{Y}_{i}\in\mathcal{Z}|X_{i}$.⁷⁷7In preliminary experiments, the earlier LLM we used had a context window shorter than our codebook+text input, so we posed the task as a set of binary classification tasks: given a document and a single label and definition from the codebook, infer whether provided label applies to the document. However, this was computationally expensive and rapid improvements in LLM context window size now make it possible to provide the entire codebook at once. “Instruction-tuning” refers to updating the LLM’s weights via language model loss function and gold-standard training labels (the natural-language word or phrase that is the label). In our instruction-tuning condition, we further train Mistral 7B Instruct on training examples, $\{C,X_{i},Y_{i}\in\mathcal{Z}\}_{i=1}^{N_{\textsc{train}}}$, drawn from each original project’s dataset and codebook. For computational efficiency, we employ QLoRA (Dettmers et al. 2024), a method that combines quantization (lower-precision weights) and updates a low-rank approximation of the weight matrix. Our instruction fine-tuning set-up with Mistral closely matches the set-up described in (Jindal, Rajpoot and Parikh 2024), which won the 2023 NeurIPS LLM Efficiency Challenge for instruction fine-tuning an LLM on a single GPU in 24 hours.

At inference time, the generative-language output of the LLM is parsed via a deterministic pattern-matching for the correct label. We select $\hat{Y}_{i}$ as the first sequence of tokens in the LLM output that matches a label in $\mathcal{Z}$.⁸⁸8Other work has attempted to constrain the output vocabulary of the LLM to only consist of $\mathcal{Z}$; however, in preliminary experiments we found this decreased performance.

Addressing RQ1 and RQ2a about the viability of using LLMs as zero-shot classifiers, from Table 6, we see that test-set zero-shot results range from very poor with 0.14 (micro) F1 on Manifestos, to marginal with 0.53 and 0.68 on BFRS and CCC respectively. However, we note that this poor zero-shot performance is about on par with what previous work have found in adjacent (but possibly slightly easier tasks). The best zero-shot LLM models from Atreja et al. (2024) achieve 0.73 (macro) F1 on predicting toxicity, 0.48 F1 on sentiment, 0.44 F1 on stance, and 0.54 F1 on frames. Ziems et al. (2024)’s zero-shot results on document-level tasks are 0.23 F1 on event argument classification, 0.52 F1 on ideology, and 0.45 F1 on tropes. This weak performance suggests that zero-shot LLMs on complex codebook tasks are unlikely to be useful to applied analysts without further modification (e.g., instruction-tuning); or inference-specific assumptions and design i.e. Egami et al. (2024) or Chen, Bhattacharya and Keith (2024); or fundamental improvements in the base LLMs to read, comprehend, and comply with codebook instructions.

Shifting from the old to new codebook format results in marginal performance gains. We find mixed evidence that restructuring a codebook into a consistent, structured format improves zero-shot LLM performance (RQ2b). For BFRS and CCC, zero-shot performances improves slightly (an increase in F1 of 0.05) compared to the original codebook, while performance on the Manifesto dataset declines slightly. Despite very modest performance changes, pragmatically, our new semi-structured format made ablation experiments much more feasible, and we still believe that converging to a consistent format for future codebooks developed by social scientists would be beneficial to the field of a whole.

The ablation study on BFRS using the structured codebook shows relatively small changes as we exclude more sections from the codebook. For example, dropping the Output Reminder that emphasizes the importance of answering only with an allowed label does not change performance, while dropping positive and negative examples hurts performance somewhat. We take this as further evidence that LLMs are not attending to the information provided in the codebook.

In-codebook instruction-tuning results in performance gains over zero-shot classification. We find that instruction-tuning an LLM on a specific codebook greatly improves performance for CCC and BFRS, but not for Manifestos (RQ3a). An instruction-tuned Mistral also beats the zero shot performance of GPT-4o. However, we see no performance gains and some losses compared to zero-shot results when we train on multiple codebooks or evaluate on out-of-sample codebooks (RQ3b). We speculate we would need to gather dozens more codebooks (with high quality gold-standard labels) to achieve the kinds of performance gains seen in suites of instruction-tuned NLP tasks, akin to Longpre et al. (2023).

The Manifestos dataset is too challenging for LLMs in their present form. Across both zero-shot and instruction-tuning approaches, we find very poor performance on the Manifesto corpus. We hypothesize that the poor performance is due to the large number of labels (142) and the very long codebook that results. While LLM context windows are rapidly increasing in size, some research suggests that attention is focused at the beginning and ends of prompts, not on the middle (Liu et al. 2024). The long codebook and extreme number of categories make this a good challenge dataset for future work on LLMs.

We see several avenues for future research building on our work here. First, different engineering approaches could possibly result in improved classification performance. We limit our work to a single, relatively small model (Mistral 7B Instruct), which we quantize. Using a larger or more recent local LLM could possibly improve zero-shot classification performance, but at potential cost of foregoing use on a commodity GPU. Instruction-tuning involves a larger set of hyperparameters than we were able to fully investigate. In addition to standard training hyperparameters (e.g., optimizer, learning rate, etc), LoRA also provides other hyperparameters, including the rank and $\alpha$ values.

We see great potential in expanding the set and quality of codebooks and datasets used for evaluation and training. We use only three datasets, which limits our ability to instruction-tune an LLM on a diverse set of instructions. Re-structuring codebooks into a consistent machine- and human-readable format would allow new codebooks and datasets to be added to our collection, similar to work in NLP on representing a diverse set of tasks in a consistent format (Wang et al. 2022). Additionally, via our manual error analysis, we found challenging examples, mistakes in the gold-labels, or places where the codebooks could be refined or clarified. This naturally leads to our next direction.

LLMs could be used in-the-loop for codebook development. Codebooks are often developed iteratively during annotation as edge cases arise or annotators raise questions. Updating the codebook during annotation requires retraining annotators and potentially re-annotating existing documents, which can slow the annotation process. LLMs have the potential to reduce the number of changes a codebook requires after annotation begins by identifying gaps in the codebook. Our manual analysis of CCC, for instance, identified a case where Mistral identified a rally in an event, rather than the more precise march. If the march label takes precedence over a rally, the codebook could be refined to state this explicitly. Relatedly, does the semi-structured format we propose improve annotator accuracy, as well as LLM accuracy? Both of these questions could be explored through experiments with human annotators.

Finally, applied researchers need tests for codebook “compliance” to assure themselves that LLMs can follow the instructions provided in their codebook. When used as a measurement tool, LLMs have the task of inferring a latent concept (a document’s true label) from the observable text provided to them. All measurement hinges on the validity of this translation (Fariss, Kenwick and Reuning 2020), but social scientists do not yet have the diagnostic tools for LLMs that they have for other measurement techniques to understand when LLMs fail. Future work can begin to formalize tests for LLM compliance. For instance, the extreme non-compliance of Mistral on the Manifesto dataset, where it returns hallucinated labels for almost half of documents (Table 5.4), is an immediate indication that the model is not able to follow the most basic instructions in the prompt, raising an immediate red flag for its use on this dataset. We see further work formalizing compliance checks as a prerequisite for applied researchers widespread adoption of codebook-based LLM classification.

As LLMs continue to improve, we believe they will be an increasingly attractive tool to measure social science constructs from text. In this work, we argue that applied researchers should make a codebook-construct label assumption, as opposed to a universal label assumption, and provide LLMs with full codebooks containing precise operationalization of labels. Empirically, we find that providing an LLM with a full codebook, either in its original or re-structured form, does not meaningfully improve (or harm) the zero-shot classification performance of one of the best “local” LLMs, Mistral, on the three real-world political science datasets. We suggest avenues for research to improve LLM “compliance” with social science codebooks, a necessary prerequisite for using LLMs in applied settings.

We see two points of optimism from our work. First, instruction-tuning LLMs with parameter-efficient methods, a task not much more demanding than the previous NLP paradigm of fine-tuning BERT-based models, increased LLM performance on classifying documents for two of our datasets. The performance of this (relatively small) open-weight instruction-tuned LLM matches the performance of closed-weight commercial LLM. This suggests that instruction-tuning local models is a viable path forward for applied researchers, which have significantly more scientific reproducibility than closed models. Second, we are optimistic that if social scientists developing new codebooks write them for both humans and LLMs in our semi-structured format, we can systematically gather these codebooks and datasets, train LLMs on them, and hopefully improve the models for everyone’s benefit.

Our research has several limitations. First, this project focuses only on multi-class document classification. Future work could explore the use of generative LLMs to annotate other information, including extracting spans of text from documents (i.e. information extraction). We also acknowledge that all our results focus on English-language codebooks and text documents (Bender 2019). Our conclusions may not hold on non-English language texts.

We acknowledge the real-world harm associated with the events reported in the documents in the datasets we use. We believe quantitative study of these events is important to mitigate future harm, but we also acknowledge that it should be paired with qualitative work, hopefully that which also connects with and advocates for the people affected by the events (D’Ignazio and Klein 2020).