Can Unconfident LLM Annotations Be Used for Confident Conclusions?

LLMs have shown great potential in handling text-annotation tasks without prior task-specific training, sometimes even outperforming crowd workers Gilardi et al. (2023). NLP, LLMs offer transformative opportunities for any discipline that relies on text as data. Fields such as psychology, political science, sociology, communications, and economics recognize this emerging technology’s potential to enhance simulation-based research Bail (2024), and facilitate tasks such as text analysis, concept induction Lam et al. (2024), and topic modeling Pham et al. (2024).

However, despite their promise, limited research has explored how to harness the potential of LLMs in ways that are both cost-effective and statistically reliable. Our work addresses this gap.

Much of past work frames human and LLM annotations as competing alternatives, with a focus on determining which is superior Thapa et al. (2023). More recent work increasingly calls for a collaborative approach that leverages the complementary strengths of both Allen et al. (1999). These collaborative paradigms aim to balance annotation quality and cost by combining human expertise and LLM efficiency Li et al. (2023c); Kim et al. (2024).

In the spirit of these collaborative paradigms, our work uses LLM confidence to efficiently and cost-effectively allocate annotation tasks, while also ensuring that the statistical inferences derived from the annotated data are valid.

Statistical inference is vital in NLP research. For example, model evaluation requires determining whether a model performs better than a baseline Card et al. (2020), which in turn relies on making valid conclusions about whether one is observing meaningful model improvements or noise Dodge et al. (2019). Chatzi et al. (2024) and Boyeau et al. (2024) leverage prediction-powered inference Angelopoulos et al. (2023a, b) for valid ranking of LLMs. A similar approach is adopted by Saad-Falcon et al. (2024) to evaluate Retrieval-Augmented Generation (RAG) systems.

Beyond model evaluation, NLP applications involve producing measurements, descriptive statistics, and causal effect estimates Feder et al. (2022); Card and Smith (2018). Notably, Keith and O’Connor (2018) introduced the problem of scientifically valid prevalence estimation. They construct Bayesian confidence intervals by proposing a generative model for text documents. We contribute to the existing literature by proposing an entirely model-free approach that is applicable to a broad range of target quantities.

Lastly, Egami et al. (2024) consider the problem of valid statistical inference when combining human and LLM annotations. However, they collect the human annotations for uniformly sampled instances, without adapting to the difficulty of annotation. Given the promise of active learning Zhang et al. (2023); Margatina et al. (2021), we develop an adaptive approach that samples a limited number of human annotations strategically. At a technical level, our approach builds on active inference Zrnic and Candès (2024), which can be seen as a refinement of prediction-powered inference Angelopoulos et al. (2023a, b) that uses active data collection for improved efficiency. Furthermore, we make use of power tuning Angelopoulos et al. (2023b), a technique that ensures that incorporating LLM annotations into the estimation can never be worse than ignoring them completely.

We have a text corpus consisting of $n$ independent and identically distributed (i.i.d.) instances $T_{1},\dots,T_{n}$. We wish to estimate a quantity of interest $\theta^{*}$, such as the prevalence of political bias in the corpus or the causal effect of using certain linguistic markers on the perceived sentiment. To perform the estimation, we require human annotations $H_{1},\dots,H_{n}$ corresponding to $T_{1},\dots,T_{n}$. For example, $H_{i}$ might indicate whether $T_{i}$ contains political bias, or assess the perceived politeness of $T_{i}$. In addition to human annotations, we may also have other readily-available information about $T_{i}$—covariates $X_{i}$such as the source of $T_{i}$ or indicators of whether $T_{i}$ contains certain linguistic markers, computed via a lexicon. Note that $X_{i}$ is available automatically, without needing human annotation. We use the short-hand notation $T=(T_{1},\dots,T_{n})$ and define $X$ and $H$ similarly.

The quantity $\theta^{*}$ can be estimated via an estimator $\hat{\theta}(X,H)$, which we will denote by $\hat{\theta}$ for short. The accuracy of $\hat{\theta}$ improves as the number of samples $n$ increases ($\hat{\theta}$ recovers $\theta^{*}$ as $n$ approaches infinity). We assume that $\hat{\theta}$ is an M-estimator Van der Vaart (2000), meaning it can be written as

for a loss function $\ell_{\theta}$ that is convex in $\theta$. Important special cases include the mean label, $\hat{\theta}=\frac{1}{n}\sum_{i=1}^{n}H_{i}$, and linear regression coefficients, which are pervasive in CSS. Other examples include quantiles, logistic, and other regression coefficients. Notice that in some cases, like calculating the mean, the loss function only depends on $H_{i}$.

Our goal is to produce an estimate of $\theta^{*}$ with uncertainty—by providing a confidence interval at a pre-specified level $(1-\alpha)$—with limited access to human annotations. Specifically, we can only collect $n_{\mathrm{human}}\ll n$ annotations (on average). This means that the “ideal estimate” (1) is out of reach.

To supplement the costly human annotations, we assume access to LLM annotations $\hat{H}_{i}$ for all $n$ instances. However, we make no assumption that the LLM annotations are good: we want to produce a valid confidence interval no matter the quality of the LLM, though we anticipate better gains when their quality is high (i.e., lower mean squared error and a smaller confidence interval).

We combine LLM annotations with strategically chosen human annotations to produce an unbiased estimate $\hat{\theta}^{\mathrm{conf}}$ that lends itself to a confidence interval that is both valid and tight around $\theta^{*}$. In particular, in the large-sample limit, the mean of the estimate is exactly $\theta^{*}$, no matter how biased the LLM annotations are.

We first explain how to choose the set of instances to be human-annotated, which is crucial for producing an accurate estimate. We collect a human annotation $H_{i}$ for instance $T_{i}$ with probability $\pi_{i}$. We let $\xi_{i}=\mathbf{1}\{H_{i}\text{ collected}\}$ denote the indicator of whether $T_{i}$ has been human-annotated. Zrnic and Candès (2024) show that the optimal choice of $\pi_{i}$ is to sample according to the uncertainty of the predicted annotation; roughly speaking, for most estimation problems the optimal rule is

where $\propto$ hides the normalization required to meet the budget, $\mathbb{E}[\sum_{i=1}^{n}\xi_{i}]=\sum_{i=1}^{n}\pi_{i}^{*}=n_{\mathrm{human}}$. Of course, since $H_{i}$ is unknown, $\pi_{i}^{*}$ is unattainable.

A key idea behind our method is to approximate $\pi^{*}_{i}$ by querying the LLM for verbalized confidence. Since RLHF may cause overconfidence Geng et al. (2024); Zhou et al. (2024) and miscalibration Band et al. (2024); Achiam et al. (2023) of the LLM’s conditional token probabilities, verbalized probabilities, i.e., expressions of confidence in token-space, are better-calibrated Tian et al. (2023). Therefore, to collect confidence scores, we adopt the verbalized two-stage prompting approach introduced by Tian et al. (2023), where the model is first asked to provide an answer via zero-shooting and afterward asked to assign a probability to the correctness of the answer. This gives us a confidence score $C_{i}\in[0,1]$ for each instance $T_{i}$. In our applications, we find that the verbalized confidence scores are calibrated (Fig. 3 (right)), meaning that higher confidence scores correspond to higher accuracy with respect to human annotations.

As we collect human annotations, we use $\{(C_{j},(\hat{H}_{j}-H_{j})^{2})\}_{j<i,\xi_{j}=1}$ as feature–label pairs to train a black-box predictor $\widehat{\texttt{err}}_{i}$. In other words, we train a model to predict the LLM error from its confidence. Finally, we set

normalized so that $\mathbb{E}[\sum_{i=1}^{n}\xi_{i}]=\sum_{i=1}^{n}\pi_{i}=n_{\mathrm{human}}$. In practice we do not fine-tune $\widehat{\texttt{err}}_{i}$ at every step $i$, but we do so periodically, after reasonably large batches of data (say, every 50 or 100 data points). See App. A.3 for further details behind the sampling and Table 2 for prompt texts.

After we have collected the human annotations according to $\pi_{i}$, building on active inference Zrnic and Candès (2024) we compute a confidence-driven estimate of $\theta^{*}$:

where we denote $\ell_{\theta,i}=\ell_{\theta}(X_{i},H_{i})$ and $\hat{\ell}_{\theta,i}=\ell_{\theta}(X_{i},\hat{H}_{i})$, and $\lambda\in[0,1]$ is a carefully chosen tuning parameter. Notice that every summand in (2) is in expectation over $\xi_{i}$ equal to $\ell_{\theta}(X_{i},H_{i})$, and thus the loss (2) is on average equal to “ideal” loss (1). This allows showing that, in the limit, $\hat{\theta}^{\mathrm{conf}}$ is on average exactly equal to $\theta^{*}$, no matter the bias in the LLM annotations. To give one example, if we want to estimate the mean of $H_{i}$, $\hat{\theta}^{\mathrm{conf}}$ reduces to

Notice that $\mathbb{E}[\hat{\theta}^{\mathrm{conf}}]=\mathbb{E}[H_{i}]=\theta^{*}$. The parameter $\lambda$ is called a power-tuning parameter Angelopoulos et al. (2023b), and it interpolates between ignoring the LLM annotations ($\lambda=0$) and utilizing them fully ($\lambda=1$). We set $\lambda$ optimally, so that the mean squared error (MSE) of $\hat{\theta}^{\mathrm{conf}}$ is minimized over $\lambda$. This means that, given any sampling rule $\pi_{i}$, the confidence-driven estimator can never be hurt by leveraging erroneous LLM annotations oder miscalibrated confidence scores. The estimator is at least as good as when $\lambda=0$. Details behind the optimization of $\lambda$ are in App. A.2.

Finally, applying the theoretical guarantees of Zrnic and Candès (2024), we form a valid confidence interval at level $1-\alpha$ as

where $z_{1-\alpha/2}$ is the $1-\alpha/2$ quantile of the standard normal distribution and $\hat{\sigma}_{\mathrm{se}}$ is a standard error estimate that has a closed form, stated in App. A.1.

We evaluate our approach and the baselines in terms of effective sample size and coverage. The effective sample size measures the increase in accuracy achieved by incorporating LLM annotations alongside human annotations. This is akin to getting more value out of each human annotation. For instance, if one has only 100 human annotations but combines them effectively with a larger pool of LLM annotations, the resulting accuracy could be comparable to having 150 human annotations. The latter metric, coverage, evaluates the statistical validity of the approaches by capturing how often the true value $\theta^{*}$ falls within the produced confidence interval. In the following we elaborate on the two metrics, deferring further details behind their computation to App. A.4.

Our main evaluation is based on LLM annotations collected with GPT-4o; analogous results with GPT-3.5 can be found in App. B.1. Table 2 in App. A.3 lists prompt texts and parameters. To test LLM performance out of the box, all annotations are collected using zero-shot prompting. Overall, the confidence scores are calibrated with accuracy, but the annotations are only in moderate agreement with human annotations in all three settings (see App. A.3). This is aligned with our lack of assumption that the LLM annotations are good: we want to produce a valid confidence interval no matter the quality of the LLM annotations.

We report the two key metrics (effective sample size and coverage), for the three selected settings (the study of politeness, stance, and bias), where the task is to estimate the five target quantities $\beta_{\mathrm{hedge}}$, $\beta_{\mathrm{1pp}}$, $O_{\mathrm{agreement}}$, $p_{\mathrm{left}}$, and $p_{\mathrm{right}}$. Both metrics are estimated over 100 trials for varying $n_{\mathrm{human}}$, the budget for human annotations.

Our main findings are reported in Figure 2 and summarized in Table 1. Figure 2 also depicts the computed confidence intervals in five randomly chosen trials across the five target quantities, for the lowest value of $n_{\mathrm{human}}$ in the considered range.