Analyzing the Inherent Response Tendency of LLMs:
Real-World Instructions-Driven Jailbreak

Recent work Zhao et al. (2024); Wei et al. (2023) indicated that the main goal of a successful jailbreak attack is to induce LLMs to generate affirmation responses rather than rejection responses. Therefore, our method attempts to create a condition within the prompt conducive to affirmation responses. In our work, we introduce the concept of inherent response tendency, where LLMs have the inherent tendency towards affirmation or rejection responses when faced with each real-world instruction. We measure it by calculating the generation probabilities of affirmation and rejection responses, and in Sec. 5.1, we have conducted quantitative experiments to verify its existence. By analyzing LLMs’ inherent response tendency, we identify real-world instructions that can inherently induce LLMs to generate affirmation responses. In our jailbreak attack strategy, we splice the above-identified instructions around the malicious instruction to amplify the LLMs’ potential to generate affirmation responses. Overall, as shown in Fig. 2, our method consists of the jailbreak idea “Inherent Response Tendency Analysis” and jailbreak strategy “Real-World Instructions-Driven Jailbreak”.

As shown on the left side of Fig. 2, to initiate this analysis, we constructed 20 affirmation responses and 20 rejection responses, which are designed to be general and not specific to any particular instruction. For instance, a representative affirmation response takes the form of “Sure, here’s the information.” while a representative rejection response is “Sorry, I am unable to provide the information”. All manual-constructed responses can be found in App. A. Furthermore, we collected 30,000 real-world English instructions from the alpaca official repository¹¹1https://github.com/tloen/alpaca-lora and iterated over each instruction to calculate the generation probabilities of LLM’s affirmation and rejection responses. Specifically, we assume that a real-world instruction as the input of LLM can be represented by $X$, an affirmation response can be represented by $y_{a}=\{y_{a0},y_{a1},...,y_{an}\}$, and a rejection response can be represented by $y_{r}=\{y_{r0},y_{r1},...,y_{rm}\}$. For the LLM’s affirmation response tendency, the probability $p_{a}$ of generating an affirmation response $y_{a}$ can be calculated as:

$$\displaystyle p_{a}=\sum_{i=1}^{n}P(y_{ai}|X,y_{a0},...,y_{a(i-1)})$$

(1)

We further consider what the LLM itself wants to generate. The probability $p^{*}_{a}$ can be calculated as:

$$\displaystyle p^{*}_{a}=\sum_{i=1}^{n}argmax_{y}P(y|X,y_{a0},...,y_{a(i-1)})$$

(2)

Finally, We employ our constructed affirmation responses to assess LLM’s affirmation response tendency ($T_{a}$) to real-world instructions, which can be calculated as:

$$\displaystyle T_{a}=\frac{1}{num}\sum_{j=1}^{num}\frac{p_{aj}}{p^{*}_{aj}}$$

(3)

where $num$ represents the number of constructed affirmation responses.

For the LLM’s rejection response tendency, the process of calculation is similar. The probability $p_{r}$ of generating a rejection response can be calculated as:

$$\displaystyle p_{r}=\sum_{i=1}^{m}P(y_{ri}|X,y_{r1},...,y_{r(i-1)})$$

(4)

The probability $p^{*}_{r}$ can be calculated as:

$$\displaystyle p^{*}_{r}=\sum_{i=1}^{n}argmax_{y}P(y|X,y_{r1},...,y_{r(i-1)})$$

(5)

The LLM’s rejection response tendency ($T_{r}$) to real-world instructions can be calculated as:

$$\displaystyle T_{r}=\frac{1}{num}\sum_{j=1}^{num}\frac{p_{rj}}{p^{*}_{rj}}$$

(6)

where $num$ represents the number of constructed rejection responses.

Overall, for each real-world instruction, we assign a score to each instruction, reflecting the LLM’s inherent response tendency. The score can be calculated as:

$$\displaystyle Score=\frac{T_{a}}{T_{r}}$$

(7)

where the higher the score, the higher the LLM’s inherent tendency to affirm. As shown in Fig. 2, based on the calculated score, we can get a ranking of real-world instructions.

As shown on the right side of Fig. 2, we perform real-world instructions-driven jailbreak. Based on the above instruction ranking, we select real-world instructions from the top that can inherently induce the LLMs to generate affirmation responses, thereby creating a condition within the prompt conducive to affirmation responses. Notably, for the type of instructions, we abandoned text manipulation instructions, such as “Please translate the following sentence” or “Please change the following text” etc. These instructions always lead the LLM to manipulate the subsequent text, which results in the malicious instruction being translated or rewritten. Subsequently, we strategically splice our selected real-world instructions around the malicious instructions. During the splicing process, we consider the number of spliced real-world instructions and the location of the malicious instructions within the prompt. For the number of spliced real-world instructions, we take into account the LLM’s capacity to process multiple instructions concurrently. Excessive splicing of instructions can lead to the LLM’s responses being impacted by the context, potentially resulting in ambiguity in its comprehension of the instructions. Consequently, our method empirically splices two or four instructions. For the location of the malicious instruction within the prompt, we tried three distinct positions: the front, middle, and end. Our experimental findings reveal that embedding the malicious instruction at the end of the prompt yields optimal performance.

Before presenting the experiment results, we introduce our selected evaluation metrics, test data, advanced LLMs, and comparison baselines used in our experiments.

We calculate the inherent response tendency scores of LLMs and display their distribution in Fig. 3. We can observe that the distribution of scores exhibits a predominantly normal distribution overall. While the majority of instructions have scores concentrated within a certain range, there are still numerous instructions with scores that are dispersed on both ends. This observation underscores the presence of LLMs’ inherent response tendency. We guess that this may be due to the LLMs’ capturing the biased distribution of the training data, which has been extensively investigated in previous work Poliak et al. (2018); Du et al. (2022a); McCoy et al. (2019); Du et al. (2023).

In our ablation analysis, we assess the impact of varying factors on our method. On the one hand, we analyze the effectiveness of the instruction ranking. Assuming that we need to splice k instructions, we have performed the following four settings each time we execute the attack:

• •

  Top: We select k instructions from the top k instruction.

• •

  Top N: Instructions with a score greater than or equal to 1.1 are regarded as the top N instructions, and we select k instructions from the top N instructions.

• •

  Random: We randomly select k instructions from the entire instructions.

• •

  Bottom N: Instructions with a score less than or equal to 0.6 are identified as the bottom N instructions, and we select k instructions from the bottom N instructions.

The hierarchy of ASR among these settings is expected as follows: Top >Top N >Random >Bottom N. Fig. 4 illustrates the changing trends of the average attack success rates for each case, with the AVG line indicating the expected behavior aligning with our hypothesis. Thus, through the validation of instruction ranking’s pivotal role, we can verify the effectiveness of our instruction ranking.

On the other hand, we focus on the number of splicing instructions and the placement of the malicious instruction within the prompt. For the number of splicing instructions, Fig. 4 shows that a higher overall attack success rate is always observed when a greater number of instructions are spliced. However, we caution against an indiscriminate increase in the number of spliced instructions. There are many instances where the accurate execution of each instruction has become challenging when splicing six instructions. We believe that this challenge is closely tied to the LLMs’ inherent capacity to concurrently execute multiple instructions, which has also been discussed in previous work Wei et al. (2023).m For the location of the malicious instruction within the prompt, we experimented with placing the malicious instructions at the front, middle, and end of the prompt, respectively. Fig. 4 shows that pacing the malicious instruction at the end of the prompt yields a higher overall attack success rate.

In our analysis, we observe that LLMs’ responses to malicious instructions are sometimes brief. As illustrated in Fig. 5, the LLM produced only a brief set of planning steps. However, our expectation is for the LLM to provide specific details for each step. We attribute such a phenomenon to LLMs’ susceptibility to in-context, which has been widely investigated in In-Context Learning Dong et al. (2022); Xie et al. (2021); Brown et al. (2020). The two spliced instructions in Fig. 5 both involve the content of “a sentence” (marked in red), which may subtly lead to the LLM’s brief response. To address this limitation, we implement a strategy, asking a follow-up question in the second round of dialogue as shown in Fig. 5. To verify its effectiveness, we analyze the results of two settings on three LLMs. As shown in Tab. 3, we first manually counted the number of samples($S^{\clubsuit}$) where the attack is successful but with a brief response. Then, based on the samples($S^{\clubsuit}$), we counted samples($S^{\spadesuit}$) where the response becomes more detailed under our strategy. The ratio of $S^{\spadesuit}$ to $S^{\clubsuit}$ is reproted. Experiment results show that in over 80% of cases, this strategy effectively works. It is crucial to highlight that this level of performance is easily attained during the second round of dialogue through a straightforward question.

In our method, we adopt the strategy of having the LLMs execute multiple instructions in parallel. We claim that our identified instructions will create a condition conducive to affirmation responses. To verify this claim, we change the way instructions are executed from the parallel to the pipeline, where LLMs execute instructions respectively in multiple rounds and the malicious instruction is typically executed in the final round. The experimental results in Fig. 6 indicate that the pipeline strategy has a significant decrease in ASR compared to the parallel strategy. These results indicate that if we do not splice our identified instructions around the malicious instruction, it will have a significant impact on the attack performance. Such a phenomenon verifies our claim.