Benchmarking Complex Instruction-Following with Multiple Constraints Composition

To comprehensively evaluate the ability of LLMs to follow complex instructions, we propose a hierarchical taxonomy to define constraints and composition types. For constraints, we extend common constraints in controlled text generation tasks to the instruction-following tasks and consider a two-level structure including coarse-grained types and fine-grained dimensions (Section 3.2). As for compositions that indicate structural combinations of constraints, we consider the characteristics of instruction-following tasks to define the composition types according to existing works on compositionality in traditional NLP tasks (Section 3.3).

Following existing works on controlled text generation and instruction following [31, 32, 33, 34, 19, 10], we propose a two-level structure for constraints including 4 constraint types (i.e., Lexical, Format, Semantic, and Utility) and 19 specific constraint dimensions which are further divided from the above types. The distribution of these constraint types and dimensions within ComplexBench is shown in Figure 2. We present the definitions of constraint types in the following and describe the details of the constraint dimensions in Appendix C.

Lexical Constraint requires to output specific keywords or phrases or precisely generate texts that are related to specific keywords mentioned in the instructions [35, 36, 34].

Format Constraint specifies the requirements on the output structure (such as JSON, Markdown, and bullet points), length, and patterns of the output, where the patterns include punctuation, content at the beginning or end, and the output templates. Format constraints require LLMs to possess a precise understanding and planning of the output content, which remain challenging for current LLMs [19, 23].

Semantic Constraint specifies the topic [37], language style [32], personality [10], and sentiment [38] of the output, which are common constraints in the existing works on controlled text generation.

Utility Constraint measures the language, helpfulness, supportiveness, consistency, and factuality of generated texts, which are holistic properties. Among these, helpfulness indicates whether the generated text can complete the basic task included in the instruction (such as Please introduce the following painting. in Figure 1) regardless of satisfaction of other constraints, while supportiveness means whether the generated text is faithful to the instruction.

As shown in Figure 3, we propose 4 composition types that indicate typical structural combinations of constraints.

Single. The output is required to satisfy a single constraint, with no composition involved.

And. The output needs to satisfy multiple constraints simultaneously. This simple composition type commonly appears in most of the existing benchmarks on complex instruction-following [3, 19, 7].

Chain. The output is required to complete multiple tasks in the instruction sequentially, each of which may contain several constraints. Formally, Chain contains $n$ tasks $\{T_{1},T_{2},\ldots,T_{n}\}$, which need to be completed sequentially. The output of $T_{k+1}$ depends on that of $T_{k}$ ($k=1,2,\cdots,n-1$).

Selection. The output is required to select different branches according to certain conditions, fulfilling the constraints of the corresponding branch. Formally, Selection contains $m$ branches $\{B_{1},B_{2},\ldots,B_{m}\}$, each of which is a task with expected outputs $Y_{1},Y_{2},\ldots,Y_{m}$ respectively. We denote a selection function as $S$ with a range $\{1,2,\cdots,m\}$, taking the selection condition $cond$ as input. Finally, the expected output of the instruction is $Y_{S(cond)}$.

It’s worth noting that the above composition types can be nested to construct more complex structures. Each task in Chain and each branch in Selection may also contain other composition types. As shown in the last row of Figure 3, a branch of Selection can also contain Selection, thus forming a nested selection composition type.

To verify the necessity and comprehensiveness of the composition types considered in ComplexBench, we analyze the distribution of composition types in real-world scenarios. We collect instructions with high demand and representativeness from two application scenarios including general and professional instructions. Professional instructions may contain specialized domains, such as business and academic research. For each category of instructions, we randomly sample 300 instructions and count the number of instructions containing each composition type. We found that the taxonomy of ComplexBench fully covers present composition types. As shown in Figure 4, although the composition types of general instructions are relatively simple and have already been covered by current benchmarks, professional instructions include more complex composition types, such as Selection and nested structures of multiple composition types, which are rarely been considered by current benchmarks. As LLMs have been gradually applied to deal with complex instructions in professional scenarios, it is necessary to evaluate their ability to follow instructions with multiple constraints composition.

We manually construct ComplexBench based on the taxonomy described in Section 3. The detailed construction pipeline consists of four steps, i.e., Reference Instructions Collection, Task Allocation, Data Annotation and Validation, and Selection Branch Expansion.

Reference Instruction Collection. Considering the difficulty of constructing complex instructions from scratch, annotators are required to create new complex instructions based on provided reference instructions. We collect reference instructions from real-world application scenarios and open-source instruction following benchmarks [19, 3, 7]. We conduct strict desensitization of privacy and carefully filter these instructions using category and quality classifiers.

Task Allocation. To ensure comprehensive coverage of each constraint and composition type, we partition the entire dataset construction into multiple annotation tasks. Each annotation task has different requirements for the minimal number of constraint dimensions in each constraint type and composition type. Annotators are required to modify reference instructions to meet the requirements of corresponding tasks. To alleviate the annotation cost, especially when the constraint dimensions in the reference instructions and task requirements are different, we leverage GPT-4 [39] to automatically acquire the constraint dimensions in reference instructions and assign them to corresponding annotation tasks according to minimal editing distance.

Data Annotation and Validation. Given reference instructions and corresponding annotation task requirements, annotators are expected to construct new complex instructions and annotate the constraint dimensions and composition types. After the data annotation, newly constructed instructions are cross-validated by other annotators. The process of validation continues until constructed instructions meet the following criteria: (1) Clarity & Reasonableness: The instruction should be easy to understand, unambiguous, and realistic, with at least one reasonable answer. (2) Validity of Constraints: Every constraint within the instruction should substantially influence the output. (3) Complexity & Difficulty: The instruction should be challenging for most LLMs and be capable of distinguishing the complex instruction-following abilities of different LLMs.

Selection Branch Expansion. When evaluating the ability of LLMs to follow instructions containing Selection, the predisposition toward random selection by LLMs may bring potential bias because most instructions cover only one selection branch. To address this issue, in the final stage of instruction construction, we manually modify the selection condition based on the selection function to construct multiple instructions that cover all the branches.

To conduct a detailed evaluation of how well each constraint and composition type is satisfied, we draw inspiration from previous works that transform text evaluation into multiple question-answering tasks [16, 17, 7]. For each constraint and composition type specified in an instruction, we manually craft a scoring question that can be succinctly answered with either "yes" or "no."

Current mainstream evaluation methods contain LLM-based [18, 3, 7] and rule-based methods [8, 19, 20]. In our preliminary experiments, we find that LLM-based methods are effective at answering open-ended scoring questions, but they demonstrate a significant deficiency in those involving numerical computation, counting, and other objective rule-defined areas, such as keyword inclusion and text length. Simultaneously, rule-based methods perform well in rule-defined areas but are powerless against open-ended scoring problems. To address their limitations, we design a Rule-Augmented LLM-based (RAL) evaluation method to equip LLM evaluators with rules to answer scoring questions in both rule-defined and open-ended areas. For the instruction $I$, the generated response to be evaluated $o$, and the scoring problem $q$, if $q$ is verifiable by rules, we first use the LLM to automatically extract segments $e$ of $o$, which is related to scoring question $q$. Subsequently, we use the rule $R_{q}$ written for $q$ to obtain the evaluation result $r_{q}\in\{0,1\}$, that is:

$$e=\mathcal{M}_{ext}(I,q,o)$$

(1)

$$r_{q}=R_{q}(e)$$

(2)

where $\mathcal{M}_{ext}$ indicates the LLM with the prompt used for extraction. Otherwise, if $q$ cannot be verified by rules, we directly use the LLM to measure the quality of $o$:

$$r_{q}=\mathcal{M}_{eva}(I,q,o)$$

(3)

where $\mathcal{M}_{eva}$ denotes the LLM with the prompt used for evaluation. For composition types, considering that their satisfaction is a prerequisite for satisfying some constraints, we model the dependencies of its scoring questions. Specifically, for Chain, all the scoring questions of the subsequent task depend on the answers to those of the preceding task. And for Selection, all the scoring questions of the selection branch depend on whether the correct selection branch is selected. If a scoring question is judged as "no", all the scoring questions depending on it will also be directly judged as "no". Formally, we denote the set of scoring questions that $q$ depends on as $Dep(q)$. After all scoring questions have been independently verified, Dependency Aggregation will be performed, and the result of $q$ will be calculated as follows:

$$r_{q}^{{}^{\prime}}=r_{q}\bigwedge_{p\in Dep(q)}r_{p}$$

(4)

Finally, following InfoBench [7], we calculate Decomposed Requirements Following Ratio (DRFR) as the final score during Score Aggregation. Considering a benchmark dataset has $N$ instructions, the instruction $i$ has $m_{i}$ scoring questions, and the result of the $j$-th scoring question is $r_{ij}^{{}^{\prime}}$, the metric is calculated as: $DRFR={\textstyle\sum_{i,j}r_{ij}^{{}^{\prime}}}/{\textstyle\sum_{i}m_{i}}$. Figure 5 shows a framework of our evaluation protocol.

ComplexBench contains 1,150 instructions and 5,306 scoring questions, as shown in Table 2. Nesting depth means the maximum depth of composition types. In addition to three basic composition types including And, Chain, and Selection, we adopt a separate category whose instructions simultaneously contain Chain and Selection, aiming to use these two challenging types to explore the boundary of LLMs’ ability in complex instruction-following²²2Since And commonly appears in various instructions, we simply categorize instructions containing both Chain / Selection and And together with those only containing Chain / Selection into one category.. We also present the task distribution of ComplexBench in Appendix B.

To measure the agreement between our evaluation method and manual evaluation, we randomly sample 200 instructions from ComplexBench to construct a meta-evaluation dataset. Five LLMs are involved in this evaluation as generation models. We employ GPT-4-1106 [39] as our primary judge and adopt two metrics to confirm the reliability of our method: (1) Overall Pairwise Agreement: Given an instruction, two model responses (denoted as A and B), the human annotators are instructed to compare the quality and choose from 3 options, namely A better than B, tie, B better than A. Subsequently, the automatic evaluation scores for two model responses are converted into pairwise comparisons to measure agreement with human annotators. (2) Question-level Agreement: Given an instruction and a model response, human annotators are instructed to judge whether each scoring question is satisfied respectively. Then, we calculate the agreement between automatic evaluation results and human-annotated ones.

For the Overall Pairwise Agreement, we sample 500 pairs from the outputs of 5 LLMs. Direct Scoring serves as a baseline, which adopts a scoring prompt [5] to assign a score to the response with a scale of 1-10. As shown in Table 3, our method can improve the agreement with manual evaluations compared to Direct Scoring with a large margin. Dependency Aggregation also shows its important contribution to our method due to its modeling of composition structures.

For the Question-level Agreement, the scoring questions in the meta-evaluation dataset are categorized into two types: (1) Rule-defined, which can be verified by rules and constitutes 17% of the total, and (2) Open-ended, which is not verifiable by rules. We compare our method with Direct Scoring, which considers a response with a score above 5 to satisfy all scoring questions of an instruction. We also remove rule arguments (w/o rule) to verify its effectiveness. As shown in Table 4, RAL outperforms all the baselines and exhibits an impressive 87.82% agreement with humans at the overall level. The LLM-based evaluator (i.e., RAL w/o rule in Table 4) shows its weakness in rule-defined areas that rule arguments mainly contribute to, supporting our motivation.