TALEC: Teach Your LLM to Evaluate in Specific Domain with In-house Criteria by Criteria Division and Zero-shot Plus Few-shot

TALEC is grounded in customizable, challenging, and adaptable evaluation criteria, distinguishing itself from conventional automatic methods. It allows users to flexibly set their own evaluation criteria, which maybe more difficult than some general criteria because the criteria may have to meet not only general standards but also specific needs of customers and business security requirements at the same time. In addition, it is hard to set the exact scale of the criteria, making it more difficult sometimes. Even for a human evaluator, a series of practices and quality inspections is needed.

In this paper, we do experiments on four distinct tasks and ten customized labels. The tasks include:

Sentiment Analysis.  Given a comment, determine the type of sentiment based on the textual information and provide the reason for the judgment.

Knowledge QA.  Given a knowledge question in the automobile field, provide a detailed answer to this question.

Search QA.  Given a piece of reference information from search engines and a related question, answer the question based on the reference information.

Title Generation.  Given an article and some complex requirements, generate main title and subtitle that is based on the article and completely meets the requirements.

Labels and their descriptions can be viewed in Table 1. The labels can be divided into two categories: acceptable labels(score=1) and unacceptable labels(score=0). If any unacceptable label appears, the final sore will be 0. If unacceptable label does not appear but any acceptable label appears, the final sore will be 1. If no labels appears, the final sore will be 2 (full score).

Many automatic evaluation only rely on the ability of the model itself, without any knowledge injection and teaching based on concrete problems and examples. Actually, even for a human evaluator, a series of practices with concrete examples and quality inspections is needed to learn our evaluation criteria. Therefore, the point is treating judge model like a human evaluator. What we need to do is teaching the judge model repeatedly and patiently with concrete examples. In order to do this, we simulate this practice-quality inspection cycle process by splitting the dataset to "train", "eval", "test" dataset and adding manual adjustment. The details of this simulation will be displayed in Section 3.3.

We introduce TALEC, a novel automatic evaluation framework that leverages SOTA model like GPT-4 to evaluate an outputted span of text from a model. TALEC mainly uses ICL to teach judge model the customized evaluation criteria. The overall process of TALEC can be viewed in Figure 1. We will introduce several key points of TALEC below.

Engineering Approach to Adjust and Iterate the Shots.  We split the dataset to "train", "eval", "test" dataset. The "train" dataset is to find typical cases. Then we will provide these typical cases for the context as shots. The "eval" dataset is to help manual optimization. The overall process can be listed as: (1). Find a some typical cases by feeling, then write the reasons why the cases are wrong, and regard them as the first version of shots. (2). Use this version of shots to run and gather statistics on "eval" dataset. (3). adjust the shots (add/delete/modify) based on the results on "eval" dataset. (4). Repeat the above process until you get a better results on "eval" dataset. After this process, use the final version of shots to run and gather statistics on "test" dataset, to verify the effectiveness of the shots.

Criteria Division.  In our customized criteria, there are 10 labels per task. Each label has several positive and negative shots. So too many shots may cause exceeding context length limit. Criteria division will solve this problem. It is to divide the overall criteria to label granularity. For example, we assume there are 10 labels. Normally, we will let GPT-4 determine whether these 10 labels exist at one time. But criteria division is to let GPT-4 judge 10 times, and only evaluate one label each time. In addition, we also find that despite a 10-fold increase in cost, criteria division result in greatly improvement in judging on almost all the labels. We will discuss the improvement in Section 4.1.

Prompt Paradigm.  ICL is a good way to teach judge model the in-house criteria. However, it will cause some problems. When injecting manually-written shots into the context of a model, the model will not only try to understand the shots but also imitate the writing style of shots. This imitation may make the model ignore some key information and drop its Chain of Thought (CoT, Wang et al. (2023a)) ability. The prompt paradigm can be listed as: (1). Repeat the description of a label before judge. (2). Try not to use transitive ( or) progressive words such as "and", "but", "however" in the first half of the judge reason. (3). Try to keep the positive and negative shots consistent in formatting, especially in the first half. We will verify the effectiveness of our prompt paradigm in section 5.

Combine Zero-shot with Few-shot.  This approach is to compensate for the model’s omission of key information. As Figure 1 shows, the model will make two completely independent judgments, one is zero-shot judge and another one is few-shot judge. Then we will connect system Prompt, Shots, answer outputted by zero-shot, answer outputted by few-shot to get a new context and use this context to judge again. Then we will get the final result. The prompt template can be viewed in Figure 4. Ablation experiment results of this approach is shown in Section 4.2.

Our evaluation methodology employs large language models, with a constraint on their maximum context length. Generally, this approach suffices for most tasks. However, in instances where lengthy prompts and responses are required, such as in the context of few-shot article generation, a single instance can extend to 1500-2000 tokens, thereby posing a limitation on the context length. Furthermore, when evaluations involve intricate tasks with multiple dimensions, the accuracy of the assessment may be negatively affected.

To address these challenges, we strategically decompose the customized evaluation criteria into distinct components, primarily by segmenting the evaluation dimensions. This approach enables the model to concentrate more on each criterion, thereby streamlining the evaluation process and enhancing the outcomes. By associating each problem label with its corresponding few-shot examples and inputting them into the model, we bypass the need for a single evaluation of all labels. Additionally, this method effectively alleviates the problem of insufficient context.

We have attempted to compare the following two experimental setups:

Standard Prompt Paradigm(Division). Individual evaluation dimensions are fragmented into distinct criteria, which are separately fed into the Judge model. The model’s output is aggregated multiple times for each criterion before calculating the overall score.

Non-division. The complete set of evaluation criteria, along with their associated shots, is simultaneously fed into the model, enabling it to produce the final score directly without sequential processing.

Table 2 shows the results. In the tasks of Sentiment Analysis, Search QA, and Title Generation, the evaluation results of the criteria division method significantly outperform those of non-division. Conversely, in the task of knowledge QA, the situation is reversed. We found that the primary cause of this discrepancy lies in the "Incorrect Answer/Unrelated Matching Results" label as shown in Table 3. As previously mentioned, this is because when criteria division is applied, the prompt format is much more distinct than when not divided, making the model more susceptible to format imitation, thereby overlooking practical information.

We said above that the imitation to text format in shots may make the judge model ignore some key information. So we try injecting zero-shot judge to avoid the impact of shots. The process can be viewed in Figure 1 and the prompt can be viewed in Figure 4.

We compare two experimental setups to verify the effectiveness of this approach:

Standard Prompt Paradigm(Single-turn wo Zero-shot).  Use the shots obtained from our engineering approach and inject the shots into context to judge. This approach only judges one case in a single-turn.

Multi-turn with Zero-shot.  As shown in Figure 1, the model will make two completely independent judgments, one is zero-shot judge and another one is few-shot judge. Then we will connect system Prompt, Shots, answer outputted by zero-shot, answer outputted by few-shot to get a new context and use this context to judge again. Then we will get the final result.

Table 4 shows comparative results of the two approach.

As you can see in Table 4, multi-turn with zero-shot approach can improve scores on all task. Especially in "Incorrect Answer/Unrelated Matching Results" label of Knowledge QA, Table 5 shows greatly improvement in this label.

Previous automated evaluation methods have opted to incorporate knowledge through Supervised Fine-Tuning. However, state-of-the-art models such as GPT-4, due to their proprietary nature, cannot be subjected to SFT to further enhance their performance. Therefore, methods utilizing SFT are compelled to resort to relatively weaker open-source models as a foundation, which may potentially impact the evaluation results. Consequently, we have chosen to introduce knowledge using In-Context Learning.

We validate the differences in knowledge introduction using SFT and ICL methods respectively, by fine-tuning Qwen-72B-Chat model (Bai et al. (2023)) and comparing the effects. We have established three experimental setups using three different models:

Standard Prompt Paradigm(GPT4 + ICL). Using GPT-4 as the evaluation model with in-context learning.

Qwen-72B-Chat + ICL. Using Qwen-72B-Chat as the evaluation model with in-context learning.

Qwen-72B-Chat + SFT. Fine-tuning Qwen-72B-Chat as the evaluation model. We construct a dataset composed of 179 manually annotated high-quality data specific to the aforementioned four tasks, 100 open-source evaluation data obtained from the TigerScore dataset, and 300 open-source general data. We have trained Qwen-72B-Chat for one epoch on this dataset.

The results are shown in Table 6. It can be observed that GPT-4 with in-context learning outperforms the method based on Qwen-72B-Chat across all tasks. Meanwhile, Qwen-72B-Chat integrated with in-context learning exhibits comparable performance to Qwen-72B-Chat combined with SFT on two tasks, surpasses the latter on one task, and underperforms on the other task. This suggests that in-context learning can achieve results similar to SFT on LLM, and to a certain extent, can serve as a substitute for SFT. Furthermore, the in-context learning approach can be applied to state-of-the-art proprietary LLM with superior performance, thereby yielding enhanced results.

In the system prompt, we explicitly offer descriptions of the evaluation criteria to help model better understand the criteria. However, we noticed that the model occasionally failed to recall the previously provided descriptions because of very long context caused by too many shots.

For instance, the requirements for some generation tasks include word count specifications. However, due to the token-based tokenizer structure of LLLMs, they can’t accurately count the number of words. Consequently, we employed a rule-based approach to judge this aspect, ensuring more precise assessments without relying on the model’s limitations. To clarify this, we clarified in the prompt that the model should disregard word count requirements. Surprisingly, the model continued to consider it in its evaluations.

To mitigate this issue, we introduce a novel approach where the model is prompted to recurrently summarize and reiterate the interpretation from the system prompt before delivering its evaluation, as depicted in Figure 2.

To validate the efficacy of this method, we executed two experiments employing distinct strategies:

Standard Prompt Paradigm(Repeat descriptions). We required the model to reiterate the descriptions prior to evaluation, maintaining a consistent format for the shots.

Non-repetition. We adopted a more informal format for the shot composition, eliminating the need for the model to reiterate the descriptions.

Table 7 shows the results. The methodology of repeating descriptions slightly outperforms the non-repetitive approach in three tasks, and significantly surpasses the latter in the remaining task. Experimental results indicate that repeating the descriptions of evaluation criteria prior to each label evaluation can assist LLM in better understanding the requirement of the evaluation, thereby enhancing accuracy.

It is universally recognized that large models possess Chain of Thought (CoT) capabilities. The prudent use of CoT enables the model to provide a detailed explanation before delivering the final answer, thereby substantially improving the accuracy of responses. However, in our approach, the incorporation of ’shots’ has instigated a problem. As models mimic the structure of the shots in their output, an informal arrangement of shots could potentially cause the model to prematurely conclude the answer without a comprehensive explanation.

In the previously mentioned example, it seems that the model’s Chain of Thought(CoT) capability is activated by initially offering an explanation. However, the model actually determines the output label at the outset due to the varied formats employed in the construction of positive and negative examples, particularly the inclusion of adversative phrases preceding the negative examples. The explanation is subsequently appended following the decision on the label, leading to an inversion of cause and effect.

Hence, we propose that in the realm of prompt engineering, the utilization of a consistent format for both positive and negative examples is crucial. This should be accompanied by a reduction in the use of adversative expressions. The objective is to postpone the revelation of the answer, thereby enabling the model to provide a comprehensive explanation prior to presenting the ultimate response at the conclusion.

Table 8 shows the results. It can be observed that the method of employing a standardized format for both positive and negative instances surpasses the method of using arbitrary formats in evaluation results across all tasks.

We list Spearman correlation of 3 typical method and our method here: GPTScore (0.1888), TigerScore (0.3373), MT-Bench (0.6/0.85) and TALEC (0.8962/0.875). It is difficult to make a completely fair side-by-side comparison with other methods due to the differences in the score system, evaluation question types, and evaluation criteria. However, the high alignment of TALEC compared to other methods can also indirectly indicate its validity and usability.

Compared to automated evaluation, human assessment possesses a greater capacity to encompass the intricate and adaptable evaluation criteria and scales inherent in our business operations. Nevertheless, the financial implications of employing human evaluators are substantial, with the primary expenditure being personnel training costs. This is particularly true for specialized domains where the evaluator requirements are notably stringent. Furthermore, the efficiency of human assessment significantly lags behind that of automated evaluation, thereby considerably impeding the iterative development of large language models.

Additionally, human annotation, while useful, is not always dependable. In the initial phases of our experiment, we utilized manual evaluation to gather enough data for the development of a more effective assessment system. This process involved a dual-review system, blind reviews, comprehensive quality checks for contentious cases, random spot checks for non-contentious cases, and a concluding round of spot checks. Only through the application of these multiple strategies and checks were we able to ensure the data’s accuracy. However, an analysis of the manual annotation results prior to inspection revealed a significant lack of alignment among different annotators in the absence of rigorous quality control, as demonstrated in Table 9. This discrepancy can be partially attributed to the complexity of the custom business evaluation criteria. Despite these challenges, automated evaluation has proven to be highly effective in such a demanding context.