Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis

Analyzing examinee responses to test questions is indispensable in the field of measurement. While gathering real data can be time-consuming, costly, and often incomplete, simulation is a useful and economical technique since it can usually be done on a laptop without additional costs. Therefore, researchers commonly use simulation to validate models [12], compare different models [13], and evaluate estimation methods. In fact, among a random sample of publications in the field, 60% of the studies used simulation, while the ratio for real data is just 41% [14]. Typically, a respondent distribution is specified, and then the item response level data (i.e., dichotomous or polytomous response) is simulated accordingly, with no thought process involved. Now, thanks to the advancement of generative AI, we can simulate some response level data and possibly gain insights into the cognitive structure behind the response process. While most work on simulation in educational settings is based on dialogue [15, 16], there are indeed some researchers conducting item response level data simulation. For example, Xu and Zhang [17] demonstrated the possibility of simulating student behavior based on assessment history. Lu and Wang [18] used insights from teachers to create generative students with various profiles, and then used the generative students’ outcomes to guide item development.

In the realm of educational data mining, gathering real learner data can pose privacy concerns [19] and present challenges with the costs of managing logged data [20]. To address these challenges, researchers have at times leveraged simulated data. LLMs are often used to generate the training datasets needed to train and test other models. For example, by using pseudocode to generate synthetic datasets, researchers have been able to develop test cases of teaching activities to inform the development of a Teaching Outcome Model (TOM) [21]. Similarly, researchers have proposed using LLMs as data annotators to create synthetic data that can be used to train other models [8], mimicking the framework of Teacher-Student Learning (TSL) [22]. Simulated data has also been used within the educational data mining community to evaluate latent trait models [23, 24].

Data augmentation is a method often employed to increase the volume and diversity of data by generating new data from the existing set; it can also be applied to mitigate the ’incomplete data’ problem [25]. Having a limited number of data points often leads to weaker generalization capabilities, which can act as an obstacle to the effectiveness of studies [26]. Thus, data augmentation is commonly used to enrich datasets and enhance their suitability for training models. For example, to augment image data, techniques such as resizing, rotating, and shifting images are frequently used [27, 28]. Researchers have also explored introducing noise in LLM training data [27], adding audio tracks or temporal shifts in speech recognition [29, 30], and leveraging Generative Adversarial Networks (GANs) to generate training data for medical imaging [31], ultimately creating datasets that are more generalizable and effective.

In recent years, the field of Open Educational Resources (OER) has seen significant growth and adoption [32], allowing researchers to benefit from a corpus of educational resources at no cost and open materials that can be freely distributed, remixed, and adapted. With the rise of large language models (LLMs), the education sector is experimenting with automating the generation of these resources to reduce costs and enhance efficiency. In particular, there has been an emphasis on automatic item generation, hint generation, and skill tagging. For item generation, much research is focusing on utilizing the capabilities of LLMs to generate math questions either through template-based approaches [33], open-ended generation (Socratic style questions or math word problems) [33, 34, 35, 36], multiple-choice question generation [37, 38, 39], or generation from structured formats (i.e., a bullet-point list) [11]. Hint generation has also been a focus, with researchers examining the effectiveness of LLMs in providing hints (i.e., worked solutions) to support learning in mathematics [40, 41], computer programming [42, 43, 44, 45, 46, 47], and various other STEM subjects. Additionally, studies have investigated human-AI collaboration in skill tagging, assessing its effectiveness across multiple languages and its speed and accuracy [48, 49]. However, unlike these areas, the topic of using LLMs to simulate respondents remains under-researched. Thus, this paper aims to study the feasibility and effectiveness of LLMs in simulating respondents.

We selected six Large Language Models (LLMs) to generate responses that simulate answers from undergraduate college students in the U.S. to assessment questions. Our selection included GPT-3.5, GPT-4, Llama 2, the newer Llama 3, Gemini-Pro, and Cohere Command R Plus. These models were chosen for their varying levels of sophistication, reported accuracy on mathematics items, and widespread popularity, allowing us to simulate a broad spectrum of student abilities, from lower to higher academic proficiency [50, 51, 52]. For implementation, we utilized APIs for each model. As Llama does not offer a direct API, we accessed Llama 2 and Llama 3 via the Replicate API.

College Algebra was chosen as the subject because pre-authored questions were available under a CC BY license from an open textbook publisher, OpenStax¹¹1https://openstax.org/details/books/college-algebra-2e. Additionally, we used a dataset from an earlier study that calibrated its item pool and had already collected responses from human participants via Prolific for 20 of the OpenStax College Algebra questions in Lesson 2.2: Linear Equations in One Variable [11]. This prior data collection contained some missingness in the data, so we were able to effectively use N >= 99 for all items.

We distinctively formatted the questions, each prefixed with a label, in the following format: "Q1: <question1>" followed by a double newline, then "Q2: <question2>", and continued this pattern for all 20 questions. Specifically, the prompt was:

Q1: Given $m=4$, find the equation of the line in slope-intercept form passing through the point $(2,5)$.

Q2: Find the slope of a line that passes through the points $(2,-1)$ and $(-5,3)$.

…

Q20: For the following exercises, solve the equation for x. State all x-values that are excluded from the solution set. $2-3/(x+4)=(x+2)/(x+4)$. Answer choices: Excluded values: -4 and x=-3; Excluded values: 4 and x=-3.

This helped to ensure clarity and separation between items. We simulated 150 respondents for each LLM, as this was deemed the right sample size for conducting further analysis. To assess the accuracy of the responses, the second author of the study manually graded them and noted the accuracy for each one.

In a real-world case, sometimes only partial data is gathered. To explore the possibility of augmenting the data, we treated each human respondent in our dataset as a centroid, using only 50 human responses to represent the partial data. We then identified the nearest synthetic respondent from our pool of synthetically generated answers for each human centroid, allowing us to map the AI responses directly onto the characteristics of individual human responses. Next, we conducted a resampling procedure. First, we resampled a subset of 50 synthetic responses, selecting them based on the distribution of the models represented in the original matched 50, aiming to maintain the proportionality observed in this initial sampling. Lastly, we expanded this by resampling a subset of 100 synthetic responses using the same criteria.

Contrary to sum-score analysis or percentage correct metrics, we plan to use Item Response Theory (IRT) to estimate the latent ability of human and LLM-Respondents. This method has several advantages over sum-score analysis [53]. First, IRT assumes a latent trait theta, transforming all estimations onto a logit scale instead of the sum-score scale, which greatly improves measurement precision. Second, IRT provides person-level fit data, which can be done independently of other respondents. Lastly, and most relevant to our purpose, IRT maps both persons and items onto the same scales, enabling equating to be carried out without assuming population score distributions. In fact, IRT equating may be the best method when tests of differing difficulties are given to nonrandom groups of examinees who differ in ability [54].

The simplest IRT model is often called the Rasch model or the one-parameter logistic model (1PL). The probability of individuals responding to a binary item (i.e., True/False) is determined by the individual’s trait level and the difficulty of the item, which is often presented as:

where:

$X_{ij}$ refers to the response made by individual $i$ to item $j$. If the response is correct/true, then $X_{ij}=1$.

$\theta_{i}$ refers to the trait level of individual $i$.

$\beta_{j}$ refers to the difficulty of item $j$.

A significant merit of using the Rasch model is that the estimates of latent trait and difficulty are mapped onto the same logit scale. The logits are interval units, which make the Wright Map (also known as the item-person map) a useful tool for presenting both item difficulties and person abilities arranged along the same logit scale. The location of item difficulty denotes the ability level at which the individual has approximately a 50% probability of answering the item correctly. The results from human calibration using Rasch analysis are shown in Figure 1. Items were ranked by their difficulties in ascending order from top to bottom [55].

IRT analysis will be carried out for two purposes: Firstly, we will use the item parameters calibrated by human respondents as a standard so that the parameters for LLM-Respondents can be estimated. By treating item parameters as fixed, we will be able to compare the proficiency distribution of college students and LLM-Respondents. Within the fixed parameter calibration (FPC) realm, we will choose multiple weights updating and multiple EM cycles (MWU-MEM) as they are the most robust estimation methods [56]. Secondly, separate calibrations will be carried out on human and AI respondent groups, informing us of the practicality of item calibration using AI generated data.

The initial item parameters for the item pool were calibrated on a group of current college students in the United States. Since multiple forms were used, there was missingness at random in the data, effectively resulting in N >= 99 for all items, satisfying the basic requirement for a Rasch analysis. The results are shown in Figure 1. We then fixed the item parameters estimated from the model to obtain the proficiency estimates for the six AI models. We wondered whether the proficiency distribution of synthetic respondents is comparable to that of humans. Results show that most LLM proficiency distributions have a significant overlap with the human respondents. In particular, Llama 3 and GPT-3.5 have the highest mean proficiency distribution, which is higher than the human mean, indicating AI’s greater proficiency in College Algebra compared to college students. The mean proficiency of GPT-4 is comparable to humans, while Cohere and Gemini are lower than humans. Llama 2 is the worst among all six models, suggesting its incapability in solving College Algebra problems.

Besides the findings on the average proficiency of each model, an interesting observation is that the variability in proficiency distribution for LLM-Respondents (standard deviation ranging from 0.29 to 0.58) was greatly reduced compared to the human distribution (SD = 0.98). Most AI distributions appear to be sharper than their human counterparts (kurtosis larger than 3). Detailed distributional information can be found in Table 1 and Figure 2. Further analysis might explore ways to increase the variability in proficiency and response patterns.

We also performed item calibration on fully simulated data, comparing it against human-calibrated data and AI-calibrated data. Notably, the IRT estimated item difficulties show a significant correlation between AI-generated and human-generated data, especially for GPT-3.5 and GPT-4 (Pearson $\rho$ > 0.7), as shown in Table 2. Rank correlation (i.e., Spearman correlation) is also reported since not all distributions have the same mean, and the difficulties scale might not be an interval scale for different estimates. Given that one Gen-AI model might not have the power to represent the human distribution, we decided to use an ensemble method (also denoted as data augmentation) to create a representative pool of LLM-Respondents whose similarity with human respondents was boosted.

Given that none of the LLM models have a proficiency distribution resembling that of humans, it is not feasible at this time to fully substitute human respondents with LLM-Respondents from a single LLM. However, LLMs could be used in a hybrid approach where half the respondents are human and half are LLM-Respondents. With this in mind, we propose three hybrid, or data augmentation, strategies listed below:

• •

  An enlarged sample of 50 humans: 50 human respondents (examples) and 50 LLM-Respondents

• •

  A mixture of human respondents and resampled LLM-Respondents using proportions learned from humans

• •

  Fully LLM-Respondents using the mixing proportions learned from humans

The resampling analysis resulted in a set with significant variation in the proportions of each model used. GPT-3.5 was the most prevalent, comprising 36% of the synthetic responses, followed by Llama 2 at 3%. Gemini accounted for 12%, while both Llama 3 and GPT-4 were represented at 8% each. Cohere was the least represented model, constituting only 6% of the responses.

To evaluate the relative performance of these three strategies, we proposed four experiments to test the effectiveness of different strategies on the item calibration process. The benchmark performance of the calibration is set by the human respondents; namely, we use all available data from human respondents to calibrate the item pool and gather item parameter estimates. Each experiment is designed to explore a different augmentation strategy. The proposed experiments are as follows:

1. Experiment 1

   We use only half the number of the respondent pool and do the calibration (N = 50), representing a real-world scenario where there is a limitation in the budget, so only part of the intended respondents were collected.

2. Experiment 2

   In addition to Experiment 1, we enlarge the data size by twice using augmentation strategies (N = 100).

3. Experiment 3

   We use a mixture of human respondents and fully resampled data in a 1:1 ratio (N = 100).

4. Experiment 4

   We use fully resampled data, with a size equal to the number of the benchmark dataset (N = 100).

In terms of evaluation criteria, Pearson correlation and Spearman correlation with the benchmark condition are reported. We also use Root Mean Square Error (RMSE) to evaluate how accurate the new estimates are. RMSE measures the average difference between values predicted by a model and the actual values. $RMSE=\sqrt{\frac{\sum_{i=1}^{N}\|y(i)-\hat{y}(i)\|^{2}}{N}}$. RMSE is always non-negative, and a value of 0 (almost never achieved in practice) would indicate a perfect recovery of the true/benchmark data.

Results show that when there aren’t a sufficient number of respondents (Experiment 1), some difficulty estimates from the IRT analysis are not trustworthy, especially at the very ends of the respondent distribution (too-easy or too-hard items), resulting in a relatively high but not perfect correlation (Pearson $\rho$ = 0.92, Spearman $\rho$ = 0.89), and RMSE = 0.55. Among all augmentation strategies, Experiment 3 has the best result; it raised the Spearman $\rho$ from 0.89 to 0.93, indicating it is an effective strategy for recovering the order of item difficulties. However, since the mean center of the respondent distribution in Experiment 3 is tilted to the left ($\mu=-0.29$) compared to Experiment 1 ($\mu=0.08$), the RMSE is larger as a universal shift is applied to all item parameters. Experiment 2 yields results comparable to Experiment 1, suggesting the strategy might not be beneficial in the current settings; practical reasons will be discussed in the Discussion section. As a validation, we also calibrated the respondent proficiency distribution using the fixed item parameter methods, and the results are shown in Figure 3.