Evaluating Large Language Models with fmeval

Text classification is a standard task in NLP and many benchmarks exist to evaluate performance, e.g., GLUE (Wang et al., 2018), SentEval (Conneau and Kiela, 2018) and parts of HELM (Liang et al., 2022). More specifically, tasks range from predicting linguistic acceptability (i.e., classifying whether a sentence is grammatical or not, (Warstadt et al., 2019)), opinion popularity (Wiebe et al., 2005) to sentiment analysis. Traditionally, text classification is tackled with supervised machine learning algorithms, using sequence-to-labels models or processing the input text as a bag-of-words (e.g., for sentiment analysis (Turney, 2002; Pang et al., 2002, 2008; Socher et al., 2013)). However, LLMs are gaining popularity in this task and are typically used with short prompts like: “Classify the sentiment of the following question: […]”. In this scenario, an additional challenge is the correct parsing and extraction of a class label from an output generation.

Women’s E-Commerce Clothing Reviews is a dataset about clothing reviews where target labels are either binary (overall sentiment of the review) or on an 1 to 5 scale for multiclass classification.

We offer an array of standard metrics to evaluate binary and multiclass classification tasks. 0-1-score measures if the predicted label matches the target label and, averaged over the dataset, yields the standard accuracy score. Precision measures the fraction of true positive over predicted positives. We expose a multiclass_average_strategy parameter that determines how the scores are aggregated across classes.

Recall measures the fraction of true positives over ground-truth positives. Similar to precision, the behavior is controlled by the parameter multiclass_average_strategy.

Balanced accuracy is the same as accuracy in the binary case and is the averaged recall per class in the multiclass case. For all these metrics, higher is better. They can be aggregated over the whole dataset or over categories.

When using a general purpose language model responses are strings. The provided convert_model_output_to_label function looks for any valid label in the string output and extracts it. For example, if the correct label is 3 and the model returns “The answer is 3.”, this output is considered correct. If no valid label is found, we mark the model output as “unknown”. an “unknown” answer (typically incorrect). While this allows for some flexibility, it does not cover the case where the model rephrases the label, e.g., returning “negative” instead of 0 and “positive” instead of 1 would not be processed as correct. Users may also provide a custom convert_model_output_to_label function.

Historically performed by specialized algorithms, LLMs achieve impressive performance in summarization (Zhang et al., 2023). General-purpose LLMs may be instructed with short prompts such as “Summarize the following: […]” while fine-tuned or purpose-built models may not need any additional context. Given an original text, extractive summarization consists of selecting a few passages from the text in order to produce a summary. Abstractive summaries may instead rephrase and modify the text while preserving its meaning. Evaluating summaries, especially if abstractive, is a notoriously challenging task that requires understanding to which extent a summary covers the original text as well as many other dimensions such as coherency and fluency (Fabbri et al., 2021a).

We use the Government Report Dataset (Huang et al., 2021) for this evaluation.

ROUGE-N (Lin, 2004) are a class of metrics that compute N-gram word overlaps between reference and model summary. The metrics are case insensitive and the values are in the range of $0$ (no match) to $1$ (perfect match). For the choice and effect of the parameter $N$, see §A.2. Meteor (Banerjee and Lavie, 2005) is similar to ROUGE-1, but includes stemming (with a Porter stemmer) and synonym matching via synonym lists (e.g., “rain” matches with “drizzle”). BERTScore (Zhang et al., 2019) uses a second ML model from the BERT family to compute embeddings of reference and predicted summaries, and compares their cosine similarity. Users can choose from two embedding models.

All three metrics are well-known, standard metrics – with ROUGE-N being the most widely used to assess summarization quality. We added METEOR and BERTScore for additional linguistic flexibility. Due to their ability to capture the similarity of rephrased text rather than verbatim overlap only, they should more accurately measure the quality of abstractive summaries.

While METEOR and BERTScore are more suitable for evaluating abstractive summaries than ROUGE, they still do not capture the full complexity of the task. Specifically, BERTScore relies on a second ML model and inherits its limitations when used for comparing passages. Fully automated metrics for abstractive summarization quality remain an active research area (Liang et al., 2022; Fabbri et al., 2021a, b).

This evaluation measures how well the model performs in question answering (QA) tasks. The model is queried for general or domain-specific facts, and we evaluate the accuracy of its response. This task comes in two variants: In open-book QA the model is presented with a reference text containing the answer, i.e., the model’s task is “reading comprehension”, extracting the correct answer from the text. In closed-book QA the model is not presented with any additional information but uses its own world knowledge to answer the question. See (Rogers et al., 2023) for a detailed taxonomy of QA tasks and benchmarks.

We use the BoolQ (Clark et al., 2019), NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017) datasets. Ranging from categorical yes-no questions over common sense to complex and specialized questions, they cover increasing levels of difficulty.

Below metrics evaluate a model’s QA performance by comparing its model output to the ground truth answer included in the dataset. This comparison can be performed in different ways. Exact match (EM): Binary score, $1$ if model output and answer match exactly. Quasi-exact match (Liang et al., 2022): Binary score. Similar as before, but both model output and answer are normalized first by removing any articles and punctuation as they usually do not impact correctness for natural language questions. Punctuation might matter for other domains such as code generation, so following HELM (Liang et al., 2022) we provide both Exact and Quasi-Exact Match metrics. Precision over Words: Precision score (see §A.3 for definition). The text is normalized as before. Recall over Words: Recall over words on normalized text. F1 over Words: The harmonic mean of precision and recall, over words (normalized). Precision, Recall and F1 over Words are more flexible as they assign non-zero scores to model answers containing parts of the ground truth. Specifically, recall measures whether the ground truth answer is contained in the model output, whereas precision penalizes verbosity. Comparing the metrics can yield additional insights in model idiosyncrasies as we will see in §7. All metrics are reported on average over the whole dataset, or per category, resulting in a number between $0$ (worst) and $1$ (best) for each metric.

The built-in metrics are based on comparing predicted and reference answers word for word. Hence, they may be less reliable for questions with linguistically ambiguous answers, e.g. those were the answer can be rephrased without modifying its meaning. An example from the NaturalQuestions dataset is question “Who lives in the imperial palace in Tokyo” with answer “the Imperial Family”. Other valid answers might be “the Emperor of Japan” or “the Emperor and their family”. Those would not be recorded as correct. However, for most questions in the built-in datasets the answers are unambiguous, e.g. country, city or individuals’ names.

This evaluation measures the ability of language models to reproduce facts about the real world. The evaluation queries the model with prompts like “Berlin is the capital of” and “Tata Motors is a subsidiary of” and compares the model generation with one of more reference answers. The prompts are divided into different knowledge categories like capitals, subsidiaries. This evaluation was proposed by Petroni et al. (Petroni et al., 2019).

We use the T-REx (Elsahar et al., 2018) dataset for this evaluation which is extracted from Wikipedia. For details see § A.4.

This evaluation outputs a single binary metric which computes to 1 if the reference answer is found within the generation.

This evaluation relies on knowledge extracted from Wikipedia which might be incomplete, out of date or inaccurate. The evaluation also requires comparing the model generation to reference answer(s), leading to similar issues as discussed in the QA evaluation in § 5.3.

Prompt stereotyping is one of many ways to measure algorithmic bias in LLMs (Blodgett et al., 2020). Many bias evaluations are task specific. For example, in the coreference resolution task, it is common to test whether the gender of a pronoun impacts whether the model can correctly identify its reference (Rudinger et al., 2018; Zhao et al., 2018) (e.g., is the model more likely to resolve a “he” pronoun to doctor and a “she” pronoun to nurse). For sentiment analysis, authors test whether the model associates different occupations, genders, names and countries with positive or negative sentiments (Huang et al., 2019; Dhamala et al., 2021). Meeting the Coverage desideratum (§2), we instead opt for a more general bias evaluation for language generation. We measure whether the model encodes stereotypes by measuring the probability it assigns to more or less stereotypical sentences. This is following the evaluation paradigm from (Nangia et al., 2020).

CrowS-Pairs (Nangia et al., 2020): This dataset provides crowdsourced sentence pairs (i.e., more and less stereotypical sentences) for the categories race/color, gender identity, sexual orientation, religion, age, nationality, disability, physical appearance and socioeconomic status along which stereotyping is measured.

The LLM is presented with two sentences: the more stereotypical sentence $S_{\text{more}}$ and the less stereotypical sentence $S_{\text{less}}$. We compute two metrics, both based on comparing the sentence probabilities $p(S_{\text{more}})$ and $p(S_{\text{less}})$ under the model. Is-biased: Measures whether $p(S_{\text{more}})>p(S_{\text{less}})$ for each sentence pair. The binary is-biased metric is averaged over the whole dataset and per category to produce the final prompt stereotyping score (Nangia et al., 2020; Touvron et al., 2023; Workshop et al., 2022). After averaging, a value between $0$ and $1$ is obtained. $1$ indicates that the model always prefers the more stereotypical sentence while $0$ means that it never prefers the more stereotypical sentence. An unbiased model prefers both at equal rates corresponding to a score of $0.5$. Log-probability-difference: A more fine-grained, numerical score indicating how much the model stereotypes on each pair.

CrowS measures U.S.-typical stereotypes. Specifically, the bias categories are taken from the US Equal Employment Opportunities Commission’s list of protected categories and the sentence pairs are produced by Amazon Mechanical Turk workers in the United States. Other stereotypes prevail in other countries. Additionally, the CrowS dataset has been found to be noisy (Blodgett et al., 2021), a consequence of being crowd-sourced. Some sentence pairs are low-quality or invalid. To address both limitations, user can bring in their own paired dataset to perform the prompt stereotyping evaluation on different data, or their own full evaluation if they opt to change the bias evaluation paradigm entirely. For improved Simplicity (§2), we aim to extend our built-in evaluations to distributional biases in language generation. Those will include geographical biases (Schwöbel et al., 2023) and gender or race vs. occupation biases (Rae et al., 2021; Liang et al., 2022).

Toxicity in NLP refers to obscenity, hate speech, insults or any other type of harmful language (Dixon et al., 2018; Vidgen et al., 2019). The toxicity evaluation aims at assessing and quantifying the level of toxic content in text generated by LLMs. This is done to prevent models from outputting toxic language, thus averting harm to individuals, reputational damage to organizations, polarization, among many other reasons. Detecting and quantifying toxicity is a challenging problem in NLP due to its high degree of subjectivity, cultural diversity, nuanced and context-dependent meaning, and ethical complexity. We use toxicity detector models to score the toxicity of passages and run the toxicity evaluation on all tasks except classification. For open-ended generation we further provide two built-in datasets designed to elicit toxic responses.

Real Toxicity Prompts (Gehman et al., 2020): a collection of truncated sentence snippets from the web. It contains a subset of samples marked as “Real Toxicity Prompts-Challenging” that are likely to elicit toxic generation. BOLD (Dhamala et al., 2021): a series of prompts aimed at testing for biased and toxic generation across profession, gender, race, religion, and political ideology.

We support UnitaryAI Detoxify-unbiased (Hanu and Unitary team, 2020) and ToxiGen-RoBERTa. Both models are based on a RoBERTa architecture (Liu et al., 2019). The first is a multi-label classifier trained on Toxic Comment Classification Challenge and Jigsaw Unintended Bias in Toxicity Classification (Borkan et al., 2019) with the following labels: toxicity, severe toxicity, obscenity, threat, insult, sexual explicitness and identity attack. The second is a binary classifier fine-tuned on the ToxiGen dataset (Hartvigsen et al., 2022). All scores are between 0 and 1, where lower is better.

The concept of toxicity may vary culturally and by context. Our toxicity evaluation employs a model to score the likelihood that generated passages include toxic content. These models are unlikely to detect toxicity in all cases and contexts. We refer the reader to the original works for further discussions on limitations of the respective models and other considerations

This evaluation measures how sensitive the model is to small semantic-preserving changes in the input. When reading, humans have a remarkable ability to understand written text even when it contains typographical errors or typos. In a similar vein, we expect that introducing a small error in the input should not have a big impact on the model output. Semantic robustness is a meta-evaluation—it is computed differently depending on the base evaluation. When the task is open-ended generation, we test whether the model’s response changes when we perturb the input, as we will detail in § 5.7.3. For all other downstream tasks, we follow (Liang et al., 2022) and evaluate whether task performance degrades when typos are introduced.

Built-in datasets depend on the selected base task. For the base tasks classification, summarization and QA we use the respective built-in datasets (see §5.1.2, §5.2.2 and §5.3.2). For the open-ended generation task, we use the T-REx dataset from the Factual Knowledge evaluation (§5.4.2), and additionally use the BOLD and WikiText-2 datasets. See §A.7 for details.

For all tasks except open-ended generation, this evaluation consists of just one metric, performance change, which measures how much the model performance changes as a result of semantic preserving perturbations to the input. How performance is measured depends on the task. For classification, the performance score is the binary indicator on whether or not the model answer is correct. For summarization, the performance scores are ROUGE-N, METEOR and BERTScore, and the performance change is measured once for each of the three scores. For QA, the scores are Exact Match, Quasi Exact Match and F1 over Words (see §5.1.3, §5.3.3, §5.2.3). For open-ended generation, instead of measuring a difference in performance, we instead measure the change in model output using the Word Error Rate.

For details on types of perturbations, a description of Word Error Rate, and the computation of performance change, see § A.7.

The built-in evaluation for QA Accuracy evaluates performance in a closed-book setting, i.e., the model is solving the task using its world knowledge only. In open-book QA, the model additionally has access to a reference text. The task then consists of extracting the correct answer from the reference text. This reading comprehension task can be a good proxy for model performance in a system where the model has access to additional information such as RAG.

We implement open-book QA using fmeval’s standard QA evaluation and the BYO dataset functionality. Specifically, we modify the built-in QA datasets to contain the reference. Here is an example prompt from the BoolQ dataset: “Respond to the following question. Valid answers are “True” or “False”. Is there a difference between sweating and perspiring?”. For the open-book task, this is modified to: “Perspiration, also known as sweating, is the production of fluids secreted by the sweat glands in the skin of mammals. Respond to the following question. Valid answers are “True” or “False”. Is there a difference between sweating and perspiring?”. Implementation-wise, we save the updated dataset locally in JSONLines format and update the dataset_uri field of the DataConfig with the local file path (experiment scripts will be released on publication). Due to context length limitations for some of the models, we filter the modified datasets for records with less than $4000$ characters in question and reference combined. We exclude model-6 model from this evaluation since its context length is $1024$ only.

Performance in the open-book task is similar in trend to closed-book (Figure 4). Performance overall improves by $15.6\%$ on average, since the additional information makes the task easier to solve. model-3 is the exception with a $6\%$ performance decrease in the closed-book compared to the open-book task.

In summary, our example user has identified two strong candidate models for their application: model-2 and model-3. On the one hand, model-2 performed better on task accuracy and robustness evaluations, and incorporated the additional information passed in the open-book task successfully. On the other hand, model-3 exhibited lower levels of toxicity. This toy experiment illustrates the usage of the library. In a real world use case there might, of course, be other factors in the final decision. Such may include cost, infrastructure requirements or the desire to employ an open source model that can easily be modified.