Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment

An LLM prompted for comparative assessment, $\mathcal{F}$, can be used to determine the probability that the first candidate is better than the second. Given the context $\mathbf{d}$ and two candidate responses, $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$, to account for positional bias Liusie et al. (2023); Wang et al. (2023b) one can run comparisons over both orderings and average the probabilities to predict the probability that response $\mathbf{x}_{i}$ is better than response $\mathbf{x}_{j}$,

Note that by doing two inference passes of the model, symmetry is ensured such that $p_{ij}=1-p_{ji}$ for all $i,j\!\in\!\{1,...,N\}$. The average comparative probability for each option $\mathbf{x}_{n}$ can then be used as the predicted quality score $\hat{s}_{n}$,

which can be converted to ranks $\hat{r}_{1:N}$, that can be evaluated against the true ranks $r_{1:N}$.

In LLM absolute scoring, the LLM, $\mathcal{F}$, is prompted to directly predict the assessment score. The prompt is designed to request the LLM to assess the quality of a text with a score (e.g. between 1-5). Two variants of scoring can be applied; first where the score is directly predicted by the LLM,

Alternatively, following G-Eval Liu et al. (2023b), if the output logits are accessible one can estimate the expected score through a fair-average by multiplying each score by its normalized probability,

where $K$ is the maximum score, as indicated in the prompt, and the probability for each possible score $k\!\in\!\{1,...,K\}$ is normalized to satisfy basic probability rules, $\sum_{k}P_{\mathcal{F}}(k|\mathbf{x}_{n},\mathbf{c})=1$ and $P_{\mathcal{F}}(k|\mathbf{x}_{n},\mathbf{c})\geq 0$, $\forall n$.

In this work, we use a simple greedy search to learn the universal attack phrase ²²2We also carried out experiments using the Greedy Coordinate Gradient (GCG) attack (Zou et al., 2023) to learn the universal attack phrase, but this approach was found to be not as effective as the greedy search process. Results for GCG experiments are provided in Appendix E.. For a vocabulary, $\mathcal{V}$ the greedy search finds the most effective adversarial word to append iteratively,

In practice, it may be computationally too expensive to compute the average rank (as specified in Equation 8). Therefore, we instead approximate the search by greedily finding the token that maximises the expected score when appended to the current sample,

The algorithm for the practical greedy search attack on comparative assessment and absolute assessment systems is given in Algorithm 1.

We run experiments on two standard language generation evaluation benchmark datasets. The first dataset used is SummEval Fabbri et al. (2021), which is a summary evaluation benchmark of 100 passages, with 16 machine-generated summaries per passage. Each summary is evaluated by human assessors on coherency (COH), consistency (CON), fluency (FLU) and relevance (REL). These attributes can be combined into an overall score (OVE), which is the average of all the individual attributes. The second dataset is TopicalChat Gopalakrishnan et al. (2019), which is a benchmark for dialogue evaluation. There are 60 dialogue contexts, where each context has 6 different machine-generated responses. The responses are assessed by human evaluators on coherency (COH), continuity (CNT), engagingness (ENG), naturalness (NAT), where again the overall score (OVE) can be computed as the average of the individual attributes.

We consider a range of standard instruction-tuned generative language models that can be used as judge-LLMs: FlanT5-xl (3B parameters) Chung et al. (2022), Llama2-7B-chat Touvron et al. (2023), Mistral-7B-chat Jiang et al. (2023), and GPT3.5 (175B parameters). FlanT5-xl, the smallest and the only encoder-decoder system, is used as the surrogate model for learning the universal adversarial attack phrases for both comparative and absolute assessment. Once the attack phrases are learned on FlanT5-xl, they are transferred to the other target LLMs to evaluate their effectiveness. Our prompts for comparative assessment follow the prompts used in Liusie et al. (2023), where different attributes use different adjectives in the prompt. For absolute assessment, we follow the prompts of G-Eval Liu et al. (2023b) and use continuous scores (Equation 4) by calculating the expected score over a score range (e.g., 1-5 normalized by their probabilities). Note that the GPT3.5 API does not provide token probabilities, so for GPT3.5, we use standard prompts without token probability normalization.

Each dataset is split into a development set and a test set following a 20:80 ratio. We use the development set (20% of the passages) to learn the attack phrase using a simple greedy search to maximize the expected score of the attacked samples and evaluate using the test set (80% of the passages). Furthermore, we only use two of the candidate texts to learn the attacks (i.e., 2 of 16 for SummEval and 2 of 6 for TopicalChat), and therefore perform the search over a modest total of 40 summaries for SummEval and 24 responses for TopicalChat.

For each dataset and attribute, we perform a separate universal concatenation attack using the notation (TASK ASSESSMENT ATTRIBUTE) to indicate the task (SummEval, TopicalChat), the assessment method (comparative, scoring), and the evaluation attribute (overall, consistency, continuity) for each learned universal attack phrase ³³3The learned universal attack phrases for each configuration are given in Appendix A.. E.g., SUMM-COMP-OVE denotes the phrase learned for comparative assessment when attacking the SummEval overall score.

We learn a single universal attack phrase on the surrogate model, FlanT5-xl, for all experiments in the main paper. Once the universal attack phrases are learned on the surrogate model, the attack is further assessed when transferred to the other target models: Mistral-7B, Llama2-7B, and GPT3.5. The vocabulary for the greedy attack is sourced from the NLTK python package ⁴⁴4English words corpus is sourced from: nltk.corpus.

To assess the success of an attack phrase, and for comparing the performance between comparative and absolute, we calculate the average rank of each candidate after an attack is applied (Equation 8). An unsuccessful attack will yield a rank near the average rank, while a very strong attack will provide an average rank of 1 (where each attacked candidate is assumed to be the best of all unattacked candidates of the context).

Section 5.3 details the attack approach to learn the universal attack phrases for the surrogate model. Figure 2 illustrates the impact of the universal adversarial on SummEval and TopicalChat, where FlanT5-xl is used as the surrogate LLM assessment system. For Summeval, the overall score (OVE) and consistency (CON) is attacked while for Topical-Chat the overall score (OVE) and continuity (CNT) is attacked. The attributes CON and CNT were selected due to the similar performance for these attributes in the absolute and comparative settings (seen in Tables 1 and 2).

The success of the adversarial attacks is measured by the average ranks of the text after an attack. Figure 2 demonstrates that both comparative assessment and absolute assessment systems have some vulnerability to adversarial attacks, as the average rank decreases, and continues to decrease as more words are added to the attack phrase. However, absolute scoring systems are significantly more susceptible to universal adversarial attacks, and with just four universal attack words, the absolute scoring system will consistently provide a rank of 1 to nearly all input texts. Table 3 provides the raw scores for comparative and absolute assessment, where we see that for absolute assessment, a universal attack phrase of 4 words will yield assessment scores on average near the maximum score of 5. The specific universal attack phrases learnt for each task are given in Appendix A.

The relative robustness of comparative assessment systems over absolute assessment systems can perhaps be explained intuitively. In an absolute assessment setting, an adversary exploits an input space which is not well understood by the model and identifies a region that spuriously encourages the model to predict a high score. However, in comparative assessment, the model is forced to compare the quality of the attacked text to another (unattacked) text, meaning the attack phrase learnt has to be invariant to the text used for comparison. This makes it more challenging to find an effective universal attack phrase. Further explanations for the relative robustness of comparative assessment systems are explored in Appendix B.

Figure 2 demonstrated that absolute assessment systems are highly vulnerable to a simple universal attack phrase concatenated to an input text. To evaluate the effectiveness of these attack phrases on more powerful target models, we explicitly transfer the attacks learned on the FlanT5-xl surrogate model to other models such as Llama2, Mistral and GPT3.5. We focus on transferring the absolute scoring attacks, as comparative assessments were found to be relatively robust for the surrogate FlanT5-xl model. Figure 3 shows the results of transferring the attack phrases to these models, highlighting several key findings: 1) There can be a high level of attack transferability for absolute scoring. For TopicalChat, the attacks generalize very well to nearly all systems, with all systems being very susceptible to attacks when assessing continuity. 2) When more powerful models assess the overall (OVE) quality, the transferability is less effective, suggesting that assessing more general, abstract qualities can be more robust. Interestingly, powerful large models (GPT3.5) are more susceptible when attacked by shorter phrases, possibly because longer phrases may begin to overfit the properties of the surrogate model. 3) The attack transfers with mixed success for SummEval, which may highlight that the complexity of the dataset can influence attack transferability.

In this section, we perform an initial investigation into possible defences that could be applied to detect if an adversary is exploiting a system. Defences can take two forms: adversarial training (Goodfellow et al., 2015) where the LLM is re-trained with adversarial examples, or adversarial attack detection where a separate module is designed to identify adversarial inputs. Although recent LLM adversarial training approaches have been proposed (Zhou et al., 2024; Zhang et al., 2023b), re-training is computationally expensive and can harm model performance, hence detection is preferred. Recent detection approaches for NLG adversarial attacks tend to focus on attacks that circumvent LLM safety filters, e.g., generating malicious content by jailbreaking (Liu et al., 2023c; Zou et al., 2023; Jin et al., 2024). Robey et al. (2023) propose SmoothLLM, where multiple versions of the perturbed input are passed to an LLM and the outputs aggregated. Such defences are inappropriate for LLM-as-a-judge setups, as though the perturbations are designed to cause no semantic change, they can result in changes in other attributes, such as fluency and style, which will impact the LLM assessment. Similarly, Jain et al. (2023); Kumar et al. (2024) propose defence approaches that involve some form of paraphrasing or filtering of the input sequence, which again interferes with the LLM-as-a-judge scores.

A simple and valid defence approach for LLM-as-a-judge is to use perplexity to detect adversarial examples (Jain et al., 2023; Raina et al., 2020). The perplexity is a measure of how unnatural a model, $\theta$ finds a sentence $\mathbf{x}$,

We use the base Mistral-7B model to compute perplexity. Adversarially attacked samples are expected to be less natural and have higher perplexity. Therefore, we can evaluate the detection performance using precision and recall. We select a specific threshold, $\beta$ to classify an input sample $\mathbf{x}$ as clean or adversarial, where if $\texttt{perp}>\beta$ the sample would be classified as adversarial. The precision, recall and F1 is then

where FP, TP and FN are standard counts for False-Positive, True-Positive and False-Negative respectively. The F1 can be used as a single-value summary of detection performance.

To assess detection, we evaluate on the test split of each dataset, augmented with the universal attack phrase concatenated to each text, such that there is balance between clean and adversarial examples. Figure 4 presents precision-recall (p-r) curves for perplexity detection as the threshold $\beta$ is swept, for the different universal adversarial phrases. Table 4 gives the best F1 scores from the p-r curves. For SummEval all the F1 scores are near 0.7 or significantly above, whilst for TopicalChat the performance is generally even better. This demonstrates that perplexity is fairly effective in disentangling clean and adversarial samples for attacks on LLM-as-a-judge. However, Zhou et al. (2024) argue that defence approaches such as perplexity detection can be circumvented by adaptive adversarial attacks. Hence, though perplexity gives a promising starting point as a defence strategy, future work will explore other more sophisticated detection approaches. Nevertheless, it can also be concluded from the findings in this work that an effective defence against the most threatening adversarial attacks on LLM-as-a-judge is to use comparative assessment over absolute scoring, despite an increased computational cost.