Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring

Suppose $X_{p}$ represents a set of all possible student answers for a prompt $p\in{P}$, and ${\mathbf{x}}\in X_{p}$ is an answer. Each prompt has a discrete integer score range $S=\{0,...,N_{p}\}$, which is defined in the rubric. The score of each answer is chosen within the range $S$. Therefore, the SAS task is defined as assigning one of the scores $s\in S$ for each given input ${\mathbf{x}}\in X_{p}$.

In this study, we assume that every prompt is associated with a predefined rubric, which stipulates what information an answer must contain to get a score. An answer gets scored high if it contains the required information sufficiently and low if not. Figure 2 shows an example of a prompt with a rubric from the dataset we used in our experiments [10]. As in the figure, the required information stipulated by a rubric may also be presented by a set of key phrases to help human raters and students understand the evaluation criteria. Each key phrase gives an example of wording that gives an answer score high. In the dataset used in our experiments, every rubric provides a set of key phrases, and we utilize such key phrases in cross-prompt training.

In our cross-prompt training setting, we assume that we have some already graded prompts by human raters $P_{\mathrm{known}}$ and we then want to grade a new prompt $p_{\mathrm{target}}$ automatically. Within this cross-prompt setting, the model is required to score the answers having different score ranges. Therefore, we re-scale the score ranges of all $P_{\mathrm{known}}$ and $p_{\mathrm{target}}$ to $[0,1]$, and as a result, the goal of the task is to construct a regression function $m:\bigcup_{p\in P_{\mathrm{known}}\cup\{p_{\mathrm{target}}\}}\{X_{p}\}\rightarrow[0,1]$ that maps an student answer to a score $s\in[0,1]$.

A typical approach to construct a function $m$ is to use deep neural networks. Suppose $\mathcal{D}=((\mathbf{x}_{i},s_{i}))^{I}_{i=1}$ is training data that consist of the pairs of an actually obtained student answer $\mathbf{x}_{i}$ and its corresponding human-annotated score $s_{i}$. $I$ is the number of training instances. To train the model $m$, we attempt to minimize the Mean Squared Error loss on the training data $L_{m}(\mathcal{D})$:

where $m(\mathbf{x})$ is a score predicted by the model $m$ for a given input $\mathbf{x}$. Once ${m}^{*}$ is obtained, we can predict the score $s$ of a new student answer as: $s=m^{*}(\mathbf{x})$.

We construct the model $m$ as following. Let $\textbf{enc}(\cdot)$ as the encoder, we first obtain a hidden vector $\mathbf{h_{x}}\in\mathbb{R}^{H}$ from an input answer $\mathbf{x}$ as:

Then, we feed the hidden vector $\mathbf{h_{x}}$ to a linear layer with a sigmoid function to predict a score:

where $\mathbf{w}\in\mathbb{R}^{H}$ and ${b}\in\mathbb{R}$ are learnable parameters. In this paper, we used BERT [4], a widely used encoder in various NLP tasks, as the encoder.

As described in Section 3.2, we used pretrained BERT [4] as the encoder for the automatic scoring model and use the vectors of CLS tokens as feature vectors for predicting answers.⁴⁴4We used pretrained BERT models from https://github.com/cl-tohoku/bert-japanese for Japanese.

Similar to previous studies [10, 12, 11], we use Quadratic Weighted Kappa (QWK) [3], the de facto standard evaluation metric in SAS, in the evaluation of our models. The scores were normalized to a range from 0 to 1 according to previous studies [12, 10]. QWK was measured by re-scaling to the original range when evaluated on the test set. We train a model for 5 epochs in the pre-finetuning process. We then finetune the resulting model for 10 epochs. In the setting without pre-finetuning process, we finetune the model for 30 epochs. These epoch numbers were determined in preliminary experiments with dev set. During the finetuning process, we computed the QWK of the dev set at the end of each epoch and stored the best parameters with the maximum QWK.

To verify the effectiveness of cross-prompt pre-finetuning, we compare the following four settings in experiments; Baseline: Only finetune the BERT-based regression model for a target prompt without key phrases, the most straightforward way to construct a BERT-based SAS model. Key phrase: Only finetune BERT for a target prompt, we input an answer and key phrases to the model. Pre-finetune: Pre-finetune BERT on cross-prompt data, input only an answer. Pre-finetune & key phrase: Pre-finetune BERT on cross-prompt data, input an answer and key phrase pairs.

First, to validate the effectiveness of the pre-finetuning with key phrases, we examined the performance of the models for the four settings described in Section 5.2. Here, similar to [10], we experimented with 10, 25, 50, 100, and 200 training instances in the finetuning phase. The results are shown in Fig. 4. We can see that pre-finetune without key phrases slightly lowers the model performance compared to Baseline. As expected, this result indicates that simply pre-finetuning on other prompts is not effective. Similarly, using only key phrases without pre-finetuning does not improve performance. QWK improves significantly only when key phrases are used and when pre-finetune is performed. The gain was notably large when the training data was scarce, with a maximum improvement of about 0.25 in QWK when using 10 answers for finetuning compared to the Baseline. Furthermore, our results indicate that the pre-finetuning with key phrases can reduce the required training data by half while maintaining the same performance. On the other hand, the performance did not improve when we used 200 answers in training, which indicates that pre-finetuning does not benefit when sufficient training data is available. We note that the results of baseline models are comparable to the results of the baseline model shown in [10].

We analyzed the behavior of the model in a zero-shot setting to verify what the model learned from pre-finetuning on cross-prompt data.

First, we examined the performance of the Pre-finetune & key phrase model in a zero-shot setting. As a result, we observed higher QWK scores for some prompts, as 0.81 and 079 points in the best-performing two prompts Y14_2-2_2_3-B and Y14_1-2_1_3-D, respectively. The results indicate that the model somewhat learns the scoring principle in our dataset through pre-finetune using the key phrases; i.e., an answer generally gets a high score if it contains sufficient information specified by the input key phrases.

Next, to examine how the key phrases contribute to the scoring, using the above two best-performing prompts, we examined the similarity between the the key phrases and the manually annotated justification cues [10] (substrings of an answer which contributed to gain the score) in the student answer. For the similarity measure, we employ the normalized edit distance. Then we analyze the relationship between the edit distance and the predicted scores by the model.

The results for the two prompts with the highest QWK, Y14_2-2_2_3-B, Y14_1-2_1_3-D, are shown in Figure 6. The color bars represent absolute error between a predicted score and a gold score. The correlation coefficients are -0.79 and -0.83, respectively, indicating a strong negative correlation between edit distance and predicted scores. This suggests that the more superficially distant the key phrases and answer, the lower the predicted model score will be for an answer. We also see that the model correctly predicts a variety of score points for the same edit distance values. We show some examples that have lower prediction error with high edit distance in Table 1. The examples indicate that the model predicts higher scores for answers that contain expressions that are semantically close to key phrases.

Those analysis indicate that the model partially grasp the property of the scoring task, in which an answer gains higher scores if the answer includes an expression semantically closer to the key phrases. Such a feature could contribute to the model’s high performance, even when the model could not learn enough answer expression patterns from small training data.