Defending Jailbreak Prompts via In-Context Adversarial Game

Upon receiving both failed and successful jailbreak prompts from the evaluator, the attack agent analyzes the failed prompts rejected by the defense LLM. Utilizing Faiss (Johnson et al., 2019), the agent retrieves the five nearest successful prompts that elicit harmful responses. One prompt is randomly selected from this subset and paired with the failed prompt for comparative analysis to extract insights (Zhao et al., 2023). This involves identifying why the successful prompt breached the defenses, with the insights recorded for each comparison. These insights are pooled together and summarized for refining failed jailbreak prompts. The prompt template for this step is illustrated in Table 9.

When refining failed jailbreak prompts, each failed prompt is paired with the previously chosen successful prompt and a randomly selected insight validated by the pair. This combination serves as the input of the attack LLM to craft a new jailbreak prompt. This new jailbreak prompt retains the core message of the failed prompt while integrating the chosen insight, using the successful prompt as a reference. This refining process is repeated up to three times until the jailbreak succeeds. The prompt template for this step is shown in Table 10.

The newly generated jailbreak prompts, along with the initially successful prompts and AutoDAN-generated prompts, are then used for defense and as the basis $P_{t}$ for subsequent iterations. This ensures a continuous improvement and adaptation cycle, as illustrated in Fig.2.

After filtering out failed jailbreak prompts from the attack agent, the defense agent identifies the reasons behind successful jailbreaking. First, the defense assistant LLM generates a similar, less harmful prompt that would lead to a rejection for the defense LLM. Then, a reflection process is implemented Shinn et al. (2023), where the defense assistant LLM compares the two prompts and generates self-reflections to understand how to prevent the original jailbreak prompts from bypassing defenses and causing harmful outputs. These reflections are prefixed to the original prompts and reprocessed through the defense LLM and evaluator. In addition to reflecting on jailbreak prompts, we also reflect on over-defended prompts. By randomly sampling 50 prompts from Xstest Röttger et al. (2023), we identify and reflect on wrongly refused prompts to help reduce the refusals. The reflective process is repeated up to three times or until a failed jailbreak is achieved. The prompt template for reflection is presented in Table 11.

Subsequent to the reflection, prompts that remain jailbroken are filtered out. The pairs of original failed prompts and their successfully defended counterparts, post-reflection, are used for insight extraction Zhao et al. (2023). In each iteration, insights are inherited and refined with new reflections, with redundancy removed to improve efficiency. The condensed insights are then set as system prompt $sys$ for the defense LLM to enhance its defense capabilities while encouraging helpful responses to benign questions. The prompt template for defense insight extraction can be found in Table 12.

It’s important to note the distinct use of reflection in the defense agent, which is absent in the attack agent. This distinction arises because reflection leverages the LLM’s inherent knowledge base, which may not include strategies for crafting successful jailbreak prompts. As a result, methods like PAIR Chao et al. (2023), which directly refine jailbreak prompts with an attacker LLM, are less effective. Conversely, reflecting on defense strategies utilizes the LLM’s reasoning capabilities more efficiently, focusing on identifying potential causes behind a prompt to facilitate jailbreak attempts, which is more likely to be obtained during the instruction tuning and alignment Wei et al. (2021); Ouyang et al. (2022).

Our study compares several defense methodologies against potential jailbreak attacks on LLMs. The baseline defense methods include the use of an LLM without any defense, Self Reminder Xie et al. (2023), Goal Prioritization Zhang et al. (2023), and In-Context Defense (ICD) Wei et al. (2023b). Each method follows the experimental setup from their respective papers. For Goal Prioritization, we apply safety instructions without fine-tuning. These are the state-of-the-art methods that implement safety instructions as the system prompt for defense, providing a fair comparison for evaluating our proposed defense technique.

For benchmarking attack strategies, we include two types of jailbreak attacks: AdvBench-based and SRD-based attacks. For AdvBench-based attacks, we include GCG Zou et al. (2023), PAIR Chao et al. (2023), In-Context Attack (ICA) Wei et al. (2023b), AutoDAN Liu et al. (2023) and Combination 2 Wei et al. (2023a)to generate jailbreak prompts, which are then combined with each test question in AdvBench. For SRD-based attacks, we combine jailbreak prompts from different methods with those in the SRD test set and with five test harmful questions from AdvBench. Specifically, we include SRD prompts without refinement, SRD combined with GCG, ICA, and Combination 2. Each method follows the experimental setup from their respective papers.

We engage all four defense LLM models in an adversarial game spanning ten iterations, typically sufficient for convergence. Llama-3-8B-Instruct is used as the evaluator LLM during “training” for a balance between efficiency and accuracy. Additionally, GPT-3.5-Turbo-0125 is chosen for both the defense assistant LLM and the attack LLM as default due to its excellent reasoning capabilities, essential for insight extraction, prompt refinement, and reflection. Subsequent to this “training” phase, the insights extracted by the defense agent are integrated as the system prompts for the defense LLM, aiming to fortify it against attacks.

To evaluate the attack agent’s efficacy, we refine jailbreak samples by incorporating successful prompts from the last training iteration and a randomly selected insight that contributed to their success. This refinement process, applied to the SRD dataset test samples, creates an augmented dataset, SRD + ICAG, which demonstrates the refining effectiveness of the attack agent and is compared with SRD-based attacks in the evaluation. To observe JSR changes over iterations for ICAG, we combine the training prompts $JP_{0}$ with 3 harmful questions for validation and we evaluate prompts after 0, 1, 5, and 10 training iterations, denoted as ICAG-0, ICAG-1, ICAG-5, and ICAG-10. ICAG-0 indicates direct defense on $JP_{0}$ without involving the attack agent.

We initially assess whether the adversarial game can converge within ten iterations. The validation JSR over successive iterations for GPT-3.5-Turbo and Vicuna is shown in Fig.3. The JSR curves of the other two models present a close tendency. We skip them due to the space limit. The results show a significant decline and convergence in JSR after implementing ICAG defenses. Initially, JSR drops notably, then changes more slowly, converging after 5 iterations. For Vicuna-7B, a slight increase in JSR occurs after the third iteration as the defense focuses on new jailbreak prompts, not those already defended. Eventually, JSR converges after more iterations. Despite the validation JSR being slightly higher than the first iteration, the defense agent adapts to more jailbreak prompts, resulting in a lower JSR on the test set as shown in Table 1.

We evaluated the JSR (%) of five AdvBench-based and five SRD-based attacks on each defense LLM using system prompts generated by four baseline methods and ICAG. The results, shown in Table 1, indicate that ICAG outperforms baseline defenses in most cases. Due to space limitations, Llama-3 results are presented separately in Table 8. The ICAG method demonstrates superior performance across different models and attack types, even though it was trained with only one harmful question combined with the SRD training set and six with AutoDAN. On GPT-3.5-Turbo, ICAG achieves notable defense improvements, particularly against AdvBench + Combination 2 and SRD + Combination 2 attacks, where baseline methods show JSRs above 45%. ICAG reduces JSR to under 5%, even achieving a 0 JSR in some cases. For AdvBench + Combination 2, ICAG-10 achieves a 0 JSR, while baseline methods fail with a JSR near 90%. Although ICAG doesn’t always achieve a 0 JSR on Mistral and Vicuna, it consistently results in the lowest JSR under most attacks, showing a significant improvement over baseline methods. The reason behind this is that GPT-3.5-Turbo’s superior reasoning ability allows better comprehension of safety instructions, resulting in a more effective defense against unseen jailbreak attacks. In contrast, the inferior defense capabilities of Mistral and Vicuna cause ICAG to generate stronger attacks, leading to more complex defense rules that are harder for these models to follow. Despite this, ICAG achieves the best performance across all tests. As shown in Table 8, the JSR of various attack methods on Llama-3 remains consistently low across different defense methods, indicating that Llama-3-Instruct incorporates safety alignments through pre-training and instruction fine-tuning. Nevertheless, ICAG consistently achieves a lower JSR on both Llama models in all tests compared to other defense methods.

Observing the iterative JSR changes from ICAG-0 to ICAG-10, we generally see a decrease in JSR with more training iterations. In some cases, ICAG with fewer iterations performs better, possibly because the game has converged or early defense stages result in uneven protection—leading to low JSR for some attacks but higher JSR for others.

Compared to other baseline methods, SRD+ICAG demonstrates excellent attack capabilities, especially on Vicuna and Mistral, where the JSR surpasses all other attack baselines when targeting undefended models. Its performance on GPT-3.5-Turbo is weaker due to the few successful attack samples during training, limiting ICAG’s ability to learn useful patterns for refining diverse jailbreak prompts.

In this study, we used the Xstest dataset to evaluate the over-defensiveness of our ICAG model. The findings, detailed in Table 2, show that defense methods, including ICAG, significantly increase over-defensiveness. Notably, even LLMs without any defense mechanism exhibit an over-defense rate exceeding 30%, indicating an inherent tendency towards excessive defense in LLM alignment mechanisms. Introducing any defense mechanism further increases over-defensiveness, suggesting that improvements in defense come with this trade-off. Our ICAG model shows comparable levels of over-defensiveness to baseline methods. Additionally, we observe a decreasing trend in over-defense rates with more iterations, demonstrating the effectiveness of over-defense reflections.

We use the MMLU benchmark to evaluate whether ICAG-generated system prompts affect the general helpfulness of LLMs. The accuracy of each defense LLM on MMLU is shown in Table 3. We found that using ICAG-generated defense prompts as system prompts has no impact on the LLMs’ general helpfulness.

In this study, we examine the transferability of the ICAG defense mechanism. We train ICAG on a specific defense LLM, then apply the derived system prompts to other models, assessing their efficacy across all mentioned attacks. The average results are in Table 4, with full outcomes in Table 5. Our findings indicate that ICAG’s defense strategies consistently transfer across different models. Notably, the JSR for transferred defenses is only slightly higher than for non-transferred defenses, demonstrating ICAG’s effectiveness even when transferring across diverse LLMs.

Due to space limitations, the ablation study is presented in App.A.2, and examples of ICAG-generated system prompts can be found in App.C. Additionally, to demonstrate the reliability of using 5 harmful questions in SRD-based attacks, we used another set of 5 unrelated harmful questions. The results, shown in Table 7, are similar to those in Table 1, confirming the consistency of SRD-based attacks.