metabench
A Sparse Benchmark to Measure General Ability
in Large Language Models

Recent years have seen significant advancements in the development and evaluation of Large Language Models (LLMs), resulting in an increased need for efficient benchmarking methods due to the expense of running and querying them. Traditional benchmarks, such as those used in the Open LLM Leaderboard [open-llm-leaderboard], encompass a wide range of tasks. In the current work, we focus on six different benchmarks: ARC (AI2 Reasoning Challenge) [clark2018think] was designed to test the reasoning abilities of AI through a series of challenging science questions. Similarly, GSM8K (Grade School Math) [cobbe2021training] evaluates the mathematical reasoning capabilities of LLMs by training verifiers on complex word problems. HellaSwag [zellers2019hellaswag] tests commonsense reasoning by asking models to choose the most plausible continuation of a story from several options. MMLU (Massive Multitask Language Understanding) [hendrycks2021measuring] assesses the breadth of expert knowledge and reasoning skills across various disciplines, presenting a comprehensive challenge for LLMs. TruthfulQA [lin2021truthfulqa] aims to measure the tendency of language models to generate truthful responses, tackling the issue of misinformation and model-generated falsehoods. Finally, WinoGrande [sakaguchi2020winogrande] tests models on pronoun resolution, a task that requires understanding context and commonsense reasoning. We use these benchmarks to distill an informative, sparse set of items to form metabench.

Psychometric methods including Item Response Theory are increasingly used in the evaluation of machine learning models [martinez2016making, martinez2019item, Wang.2023a]. Zhuang and colleagues [zhuang2023efficiently] propose using IRT for adaptive testing of LLMs but only used this approach to distinguish between a small set of models and restricted domains and compared models to human responses. Burden and colleagues [burden2023inferring] use Bayesian probabilistic programming to infer agents’ capabilities from diverse task performances, building computational models to link task features to system capabilities. However, these methods are applied to reinforcement learning agents and not to LLMs. There also have been several attempts to create completely new questionnaires to evaluate LLMs using psychometric tools [jiang2024evaluating, wang2023emotional].

Redundancy in benchmark items has been established, analyzed and combated by several recent works [Rodriguez.2021, Vania.2021, Wang.2023, Ye.2023, Hewitt.2024]. Most related to our proposal is recent work by Polo et al. [polo2024tinybenchmarks] introducing tinyBenchmarks. The authors advocate for reducing the number of evaluation examples in benchmarks to lower costs while maintaining accurate performance assessments of LLMs. They demonstrate that evaluating LLMs on curated subsets of 100 examples can reliably estimate the performance on large benchmarks like MMLU and HELM [liang2022holistic] within a 2% mean absolute error. However, their proposal differs from our approach on several dimensions. Polo and colleagues analyzed $395$ LLMs with high-dimensional IRT models and clustering to create four smaller benchmarks. We use over $5000$ LLMs, make direct use of estimated item properties by applying principled item selection, go beyond point scores by estimating multiple latent abilities and show that they jointly allow better score reconstruction for each benchmark’s score and the grand mean.¹¹1Note, that we report root mean squared errors (RMSEs), which are more sensitive to outliers than the mean absolute errors (MAEs) used in tinyBenchmarks. For comparability, we also report MAEs in the appendix Appendix B.5. In short, metabench achieves an average MAE of $1.08\%$ over benchmarks and an MAE of $0.61\%$ for the grand mean.

We collected openly available item-wise accuracies from Hugging Face Datasets for the six benchmarks that are part of the Open LLM Leaderboard[open-llm-leaderboard]: ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande. After excluding flagged and merged models, we obtained data from $1355$ unique users, yielding $n=6875$ unique LLMs and $152,261,495$ responses in total. Out of all LLMs, $5055$ had data for each of the six benchmarks. Whenever possible, we used all available LLMs per benchmark for the analyses. For the remainder of this paper, we use the terms "subjects" and "LLMs" interchangeably, since many of the methods below were designed to analyze human behavior and abilities.

All analyses were performed in R 4.4.0 [R.2024]. For comparability across benchmarks, we normalized the benchmark scores on a percent scale and saved them for reference before removing any items. Per benchmark, we discarded subjects with the lowest $0.1\%$ of scores to further reduce noise. Items with standard deviations below $1\%$ (too little variability) and mean accuracies above $95\%$ (too easy) were removed. We then estimated the part-whole correlation between each item and the total score using the point-biserial correlation coefficient, $r_{pb}$. This coefficient grows if subjects who responded correctly to the item have a higher than average score, especially if the item has medium difficulty ([Irwing.2018], see Appendix A.1 for details). We removed items with $r_{pbis}=0\pm c$ and $c$ was set to $0.05$ except for WinoGrande with $c=0.02$, as otherwise hundreds of items would have been removed. The exact number of items removed per benchmark can be found in Appendix A.2.

For comparability across benchmarks, we conducted a train-test-validation split of the LLMs in the following manner: For stratification, we calculated the grand average of the original benchmark scores for LLMs that ran on all six benchmarks, and used it to create a stratified $10\%$ split as the global test set. For cross-validation per benchmark, we split off further $10\%$ subset of the LLMs as a local validation set, this time stratifying by the specific benchmark score.

For a set of $d$ items it becomes quickly impossible to consider all subsets of size $k$, as there are ${d}\choose{k}$ unique subsets and any exhaustive search algorithm would have $\mathcal{O}(d!)$ time complexity for fixed $k$. Nevertheless, it is astounding how well the given benchmarks can be reduced in a theory-free way using random sampling: [Wang.2023], [Ye.2023] and [Polo.2024] showed that random samples of benchmark items can in principle produce smaller benchmarks that retain enough information to reconstruct their original score with negligible error. One basic procedure for this is:

1. 1.

   Uniformly sample $k$ items and calculate the subtest scores $\tilde{s}_{j}$ for each subject $j$.

2. 2.

   Fit some regression model $\hat{s}_{j}=g(\tilde{s}_{j})$ on the training set.

3. 3.

   Calculate some summary statistic of $\varepsilon_{j}=\hat{s}_{j}-s_{j}$ on the validation set, e.g. its root mean square error (RMSE).

4. 4.

   Repeat steps (1-3) a large number of times.

5. 5.

   Choose the sample with the smallest validation error and check if the performance is still good on the test set.

For $g(\cdot)$ one can use generalized additive models (GAMs, [hastie1987generalized, Fahrmeir.2013, Wood.2017]), which are a flexible class of regression models utilizing basis functions. They can be seen as the frequentist version of the fully Bayesian framework of Gaussian Process regression [Duvenaud.2011].

We ran the described procedure on each benchmark with varying $k$ and $10,000$ repetitions to gauge how information is spread across items. GAMs were fitted using the mgcv package [Wood.2017] with default options. Appendix B.2 contains the results for $k$ between $50$ and $500$ in steps of $50$. As shown in Figure 2B, even for $k\leq 200$, test errors have high density on relatively small intervals and all median RMSEs are below $3.5\%$: Many different (but potentially overlapping samples) are informative. This suggests (1) few items are uninformative, (2) the information they carry is partially redundant, and (3) the useful information saturates swiftly as more items are considered. This notion is further explored in Section 2.4.

This has two practical implications: If we just want medium-sized versions of the six benchmarks, we can simply use cross-validated subsampling. If we want to tap into (or even breach) the boundary of the test RMSE distribution, a more principled approach that models properties of items is required. However, principled item selection can be a follow-up after an initial subsampling sweep, as this preselection still permits reliable score reconstruction. For comparability across item models and to increase the ratio between observations and parameters, we subsampled all six benchmarks to $350$ items. This number is a compromise between predictive performance and subject-to-item ratio (see Appendix B.2). We next describe the approach to modeling item properties.

Item response theory (IRT) is a statistical branch of psychometrics used for the design and analysis of standardized tests [Chen.2021c, Cai.2016, Irwing.2018, hambleton1991fundamentals]. It aims to model the relationship between a subject’s abilities and the probability of a correct response to an item. This framework is closely related to other latent trait models such as factor analysis [Koch.2013], but assumes binary or polytomous responses [Chen.2020, zanon2016application]. Let $x_{ij}$ be the binary accuracy of item $i$ for subject $j$ with some (unknown) latent ability vector $\bm{\vartheta}_{j}$:

where $f$ is a link function and $\bm{\alpha}_{i}$ are the item parameters. We use one-dimensional $\vartheta_{j}$, as higher-dimensional IRT models require much larger subjects-to-items ratios than in our case, but the framework generalizes to multidimensional abilities in a straightforward manner. A simple and common IRT model is the 2-parameter logistic (2PL) model with $\bm{\alpha}_{i}=(\delta_{i},\mathbf{a}_{i})$ and $f(\bm{\vartheta}_{j}|\bm{\alpha}_{i})=\sigma(\mathbf{a}_{i}^{\top}\bm{\vartheta}_{j}-\delta_{i})$. Here $\sigma(\cdot)$ is the logistic function, $\mathbf{a}_{i}$ is the loading parameter and $\delta_{i}$ is the difficulty parameter. Note that this has the same form as logistic regression with the subtle but important difference that $\bm{\vartheta}_{i}$ is unknown. The main goal is to estimate item parameters $\mathbf{A}=(\bm{\alpha}_{i})_{i=1,\ldots,d}$ which capture their test-relevant properties. Due to favorable asymptotic behavior, the standard approach is to use marginal maximum likelihood methods [Wirth.2007, Chen.2020], which treat the latent ability as a random variable and attempt to maximize the marginal likelihood

with $\mathcal{L}_{i}(x_{ij}|\bm{\vartheta}_{j})=f(\bm{\vartheta}_{j}\,|\,\bm{\alpha}_{i})^{x_{ij}}\big{(}1-f(\bm{\vartheta}_{j}\,|\,\bm{\alpha}_{i})\big{)}^{1-x_{ij}}$. Because of the integral, there is no analytical solution to this problem, but it can be approximated using Expectation-Maximization [Dempster.1977, Bock.1981, Bock.1997] algorithms and stochastic approximations exist for multidimensional latent abilities [Zhang.2022a, Zhang.2020]. After model fitting, $\bm{\vartheta}_{j}$ can be estimated for each subject $j$ using, e.g., maximum a posteriori estimation ²²2 Any likelihood-based estimator profits from large $d$, as the link functions have sigmoidal shapes: A hill in the function product only arises after aggregating a sufficient number of item-likelihoods with varying midpoint and steepness. This raises the trade-off that many subjects per item are required to get stable parameter estimates, but many items per subject are required to recover their latent ability. :

We used the mirt [chalmers2012mirt] package with default options to fit the following IRT models to preselected subsets of 350 items each - a 2PL model, a 3PL model with an additional lower asymptote parameter, a 4PL model with an additional upper asymptote parameter. After IRT model fitting, we separately used the maximum a posteriori (MAP) method as well as the expected a posteriori sum (EAPsum) method to estimate each benchmark-specific latent ability [bock1981marginal, thissen1995item]. We trained a GAM with adaptive smoothers to predict the original benchmark score using the latent ability estimate, and tested the GAM on the test set.

Table 1 shows the test RMSE of the predicted scores as well as the Spearman correlations between the estimated latent variables and the original point scores. In the appendix, Figure 6 visualizes the results. All correlation coefficients are above $0.945$, indicating that the ranking between $\bm{\vartheta}$ and $\mathbf{s}$ is largely preserved. The best models achieve an RMSE of on average $1.18\%$ and all but one model have an RMSE well below $2\%$. For $3$ out of $6$ benchmarks, the best model outperforms the cross-validated subsampling approach. For the remaining $3$, the RMSE is at most $0.19\%$ worse. While we use single-predictor GAMs to reconstruct scores from (a) random subtest scores $\tilde{s}_{j}$ and (b) latent abilities $\hat{\vartheta}_{j}$, the latter seem to preserve the information about scores and are at times less noisy. This proof of concept reinforces our general approach of going beyond random subsampling and opens up the possibility of information filtering.

Item parameters have a natural interpretation because of how they affect the link function $f(\vartheta_{j}|\bm{\alpha}_{i})$: The difficulty $\delta_{i}$ shifts its midpoint and the loading $a_{i}$ alters its scale. Thus, the difficulty parameter determines the probability of a correct response for an LLM $j$ with $\vartheta_{j}=0$, and the loading determines how much this probability changes with the LLM’s ability. Intuitively, an item with a high loading discriminates well between subjects whose ability is close to the item’s difficulty. Fisher information functions quantify this property: Since the distribution of $x_{ij}|\vartheta_{j}$ is an exponential family, any response-based estimator $T$ for $\vartheta$ will have a variance bounded from below by the expected Fisher information³³3 The observed Fisher information, $I_{i}(\vartheta)=-\frac{\partial^{2}}{\partial^{2}\vartheta}\log\mathcal{L}(x_{i}|\vartheta)$, is the curvature of the observed likelihood. Its expectation is denoted as $\mathcal{I}_{i}(\vartheta)=\mathbb{E}_{\vartheta}\left[I_{i}(\vartheta)\right]$ and $\mathcal{I}(\vartheta)=\sum_{i=1}^{d}\mathcal{I}_{i}(\vartheta)$. Then, for unbiased $T$, $\mathbb{V}_{\vartheta}(T)\geq\frac{1}{\mathcal{I}(\vartheta)}$. If $T$ is biased, the numerator becomes $(\frac{\partial}{\partial^{2}\vartheta}\mathbb{E}_{\vartheta}\left[T\right])^{2}$. The larger $\mathcal{I}(\vartheta)$ becomes, the more reliably one can estimate $\vartheta$ from the observed responses. [Pawitan.2013, Lehmann.1998]. For the 2PL model with one-dimensional ability,

On a high level, this is a measure of how informative an item is about a test taker’s ability conditioned on the ability, and opens up a more principled approach to item selection. For each benchmark we use the following procedure to pick the most predictive items:

1. 1.

   Estimate the item parameters using an IRT model $\mu\in\{\text{2PL, 3PL, 4PL}\}$.

2. 2.

   Estimate the ability using a method $\nu\in\{\text{MAP, EAPsum}\}$.

3. 3.

   Use the item parameters to calculate Fisher information functions for all items.

4. 4.

   Partition the the support of $\vartheta$ into $q$ quantiles.

5. 5.

   In each quantile, select the item with the highest information (above a threshold $\tau$) and remove it from the search space.

We ran the same models as described in Section 2.3, this time subsampling items using information filtering. We used the rBayesianOptimization package [rBayesianOptimization] with default settings to fine-tune the hyperparameters ($\mu$, $\nu$, $q$, $\tau$) on the validation set to minimize the term RMSE$(\bm{\vartheta},\mathbf{s})+\lambda k$ where $k$ is the size of the chosen subset and $\lambda\in[0.0,0.01]$ is the regularization coefficient. The selected subset is then evaluated on the test set. For each resulting selection we re-run the random subsampling procedure with the same size $k$ and check how many random samples have a worse RMSE.

Table 2 shows the score reconstruction results for each benchmark’s final model. The mean RMSE over benchmarks is $1.856\%$. All IRT models have an RMSE well below $2.5\%$. Except for TruthfulQA, all test RMSEs are better than at least 94% of all random subsamples (see Figure 2B for their distributions). The two very large benchmarks MMLU and HellaSwag have breached the test accuracy boundary of random subsampling. Although this indicates that information filtering is in general effective, we can further improve score reconstruction if we do not limit ourselves to using a single latent ability for prediction.

The idea behind metabench is to go beyond established benchmarking practices for LLMs (cf. [Burnell.2023, Wang.2023a]). This also means going beyond independent latent abilities for each benchmark. As benchmark scores are highly correlated (going from $r=0.571$ to $r=0.956$, see Appendix C.2), each benchmark also carries some information about the others. Thus, if an LLM is run on several reduced benchmarks, the resulting scores may mutually fill in each other’s blanks. We test this hypothesis by using the resulting IRT models from 2.4 to predict each benchmark’s original score by jointly using all six latent ability estimates as predictors in the GAM. As always, IRT models, latent abilities and GAMs were estimated on the train set and the RMSEs reflect the score prediction accuracy on the test set. The Figure 2 and the right side of Table 2 show the results: The mean RMSE has shrunk to $1.508\%$ and is well below $2\%$ in all cases. Two models are better than over $99\%$ of all random subsamples, the remaining four surpass the best random subsample by a substantial margin. This synergy effect underlines the advantage of running all components of metabench.

We then applied this method to predict the grand average score: The mean across all normalized scores for the six benchmarks constituting the Open LLM Leaderboad. Random subsets are sampled from each benchmark using its corresponding $d$ (e.g. $58$ items from HellaSwag), totalling $739$ items. Figure 2C and the bottom row of Table 2 show the score reconstruction results. We were able to reconstruct the grand average with a Spearman correlation of $r=0.996$ and an RMSE of $0.814\%$, which surpassed all random subsamples.

Finally, we investigated the relationship between all latent abilities by using factor analysis [kim1978introduction, Koch.2013, pml1Book], which is a linear latent modeling framework closely related to Principle Component Analysis [mcdonald2014factor, pml1Book] and with a long history in psychometrics [cattell1966meaning, Carroll.1997]. In a nutshell, factor analysis of a set of continuous variables attempts a low-rank approximation of their covariance matrix. As its latent factors have the same interpretation as the latent abilities in IRT, it is commonly applied to analyze the relationship between multiple tests [Irwing.2018]. See Appendix C.1 for a more detailed introduction. We applied FA to the estimated latent abilities using the psych package [psychR] (for details, see Appendix C.2). We find that a single factor is sufficient to capture $79.3\%$ of the total variability across latent abilities. This meta-ability has a Pearson correlation of $r=0.928$ with the original grand average score. Its loadings range from $0.791$ (MMLU) to $0.965$ (WinoGrande): Separately, all six latent abilities contribute a comparable amount to the meta-ability, but the added unique information decreases with each addition. This result is in line with the synergy effect as described above. It consolidates the utility of metabench for efficiently measuring the abilities of LLMs.

In terms of efficiency, the fixed set of $739$ items can still be outperformed: A significant benefit of IRT analysis is that it allows for adaptive testing, in which items can be administered conditional upon the current estimate of the subject’s ability. The information filtering procedure can be seen as a one-size fits most approach, covering as much ground as necessary to get a good estimate of the latent ability (within quantiles populated by the majority of models). However, there is also a made-to-measure alternative which can further reduce item administration costs. By asking the most informative questions sequentially, one can get a good estimate of the latent ability with fewer items. This is the idea behind computerized adaptive tests (CATs, [Chang.2015, Irwing.2018]).

We used the catR package [magis2012catR] to simulate CATs on our sample of LLMs using items from each benchmark (for details, see Appendix D). The goal was to re-estimate $\vartheta_{j}$ for each subject $j$ as provided by an IRT model fitted to the training set, using as few items as possible. Our CAT procedure for each subject $j$ is as follows:

1. 1.

   Initialize an estimate of $\vartheta_{j}$ around $0$.

2. 2.

   Select an item, $i$, with the Maximum Fisher Information with respect to $\hat{\vartheta}_{j}$.

3. 3.

   Update $\hat{\vartheta}_{j}$ given $j$’s response to item $i$.

4. 4.

   Repeat steps (2-3) until the estimated standard error of $\hat{\vartheta}_{j}$ is less than $0.1$ oder all items have been administered.

Using this approach, we found that the latent ability could be reliably re-estimated for every benchmark (RMSE $<0.2525$) with significantly fewer items than the previous approach (as low as $21$ items for the most predictive IRT model). The results of our CAT simulations are presented in Table 3. The estimated latent abilities are predictive of performance across each benchmark. By applying computerized adaptive testing, practitioners can estimate $\vartheta$ for their models using a small fraction of the items in available benchmarks.

Most users who submitted results to the OpenLLM Leaderboard contributed several models, potentially violating the IRT model assumption that the response patterns of different subjects are independent (conditional on the item parameters). This may result in biased parameter estimates and therefor distorted item information functions. Indeed, the population of LLMs presumably has a different dependence structure than than that of human test takers: Many LLMs are fine-tuned versions of others or at least share the same base structure and are trained on overlapping data. In the extreme case, treating them as subjects for IRT models is akin to analyzing a test on several batches of clones. On the other hand, the same can be said about human subjects - despite sharing many fundamental properties of the human mind and brain, and experiencing overlapping school curricula, human subjects measurably differ in their knowledge, problem solving abilities, linguistic skills and memory [tyler1947psychology, eysenck2014fundamentals]. Typically, clustering in human data is treated by including random effects to (generalized) regression models [Fahrmeir.2013]. This is not possible for IRT models, as these random effects would become indistinguishable from latent abilities without introducing further constraints.

However, as the latent abilities are strongly related to the point scores, we checked if the latter have different empirical distributions if we only include the latest LLM per leaderboard user (see Appendix B.1). The distributions are largely identical and the ratio between subjects and items is already low for IRT standards, prohibiting substantial discarding of subjects without also reducing the item sets before model fitting. Future work with more data should analyze to what extent similarities in fine-tuning and architecture deflate predictive performance for scores and (more importantly) change the ranking outcomes. Finally, it is likely that new LLMs tested on these benchmarks are themselves not far out of distribution with respect to the training set. Thus, for practical purposes, our results should still hold.

Future research could explore several avenues to build upon the findings of this study. First, further validation of metabench across a broader range of LLMs would test the generalizability of our results. Additionally, investigating the potential for extending the metabench framework to other domains beyond natural language processing could provide valuable insights into the applicability of sparse benchmarks in different modalities such as vision, reinforcement learning, and visual question answering. Another promising direction for future research is the development of adaptive testing algorithms and interfaces that can further reduce the number of items required for accurate LLM evaluation. By continuously refining the selection process based on real-time performance data, these algorithms could enhance the efficiency and precision of LLM assessments. Finally, the core set of cognitive abilities in LLMs remains largely unknown. Future research should closely investigate its structure, as well as uncover the properties of model architecture and training data that facilitate ability development. This would constitute a good basis for more goal-directed and theoretically grounded benchmarks, driving the field of LLM research into a more insightful future.