ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues

Our work leverages two distinct dataset sources: XDailyDialog (Liu et al., 2023b) and XPersona (Lin et al., 2021). For training, development and testing, we use XDailyDialog, a multilingual extension of DailyDialog with human translations covering German-de, Italian-it and Chinese-zh. XDailyDialog includes 13K parallel dialogues, amounting to 52K dialogues and 410K utterances. During our pre-processing step we noted a substancial overlap of dialogues between the provided test and training/validation sets of XDailyDialog. As a result, we excluded these dialogues (amounting to 20%) from the test set.

In order to gauge the extensibility to other dialogue datasets and languages, we additionally include XPersona data in our GenResCoh test set. XPersona is a multilingual extension of the PersonaChat dataset Zhang et al. (2018) with human revised machine translations for six languages. Besides English, we include Italian-it, Chinese-zh, and an additional unseen language, French-fr, in our experiments. For each language, we extract 1K contexts from the test set for response generation.

For contrastive comparison, we also use DailyDialog++ (Sai et al., 2020), a similar curation effort which uses the original DailyDialog dataset, and where annotators were asked to create five additional relevant responses and five adversarial irrelevant responses for each context.

We present relevant statistics for our dataset, together with DailyDialog++ in Table 2. Since the test set for our dataset is generated by GPT-4, we opt to present the statistics separately.

Firstly, despite GenResCoh boasting a much larger context set, amounting to 51k/5k for training/validation, each context only has a single positive and negative response, whereas DailyDialog++ contains 5 positive responses and an additional 5 adversarial negative responses.

When comparing the average length of responses, we note that GenResCoh responses are longer than the human curated responses of DailyDialog++. This verbosity is a known behaviour by LLMs, since they are conditioned to output longer responses due to the Reinforcement learning from human feedback (RLHF) step, at least when compared to humans (Kamalloo et al., 2023). Additionally, we note that the response lengths remain similar across the development and test sets, whereas the explanations are much longer in the test set.

For a more fine grained analysis of the responses, we measure their lexical diversity using the Measure of Textual Lexical Diversity (MTLD) (McCarthy, 2005).⁴⁴4Calculated using lexical-diversity Python package. Since DailyDialog++ contains 5 responses per context, we calculate the average diversity when considering the responses individually. We observe that the diversity of human responses for DailyDialog++ is larger than the ones generated by GPT-3.5-Turbo for the development set, but similar to the ones generated by GPT-4 for the test set. This disparity is to be expected, given the performance differences between the two models in creative writing tasks.⁵⁵5It is important to point out, that a higher temperature value would likely result in higher diversity, with a possible trade off in performance.

It is important to note that the adversarial responses from DailyDialog++ exhibit greater diversity compared to those from GenResCoh. This is because the tasks are slightly different: in DailyDialog++, annotators were asked to generate new irrelevant responses by incorporating certain words from the context directly or indirectly into their responses. This stands in contrast to our approach, which prioritises coherence while preserving relevance. As such, the introduction of diverse words into the response is constrained by the fact relevance must be uphold.

Turn-level dialogue coherence evaluation consists of the assessment of a response hypothesis $h$ given a dialogue history (frequently denoted as context) $c$ of varying amount of turns, and optionally one or more references $r$ and/or external knowledge $k$. The goal is to learn a scoring function that assigns a score $f(c,k,r,h)\rightarrow s$ for each individual quality aspect. This scoring function is compared against human judgements, which annotate the same context-response pairs. These responses are evaluated by humans using, for instance, a binary $(0,1)$ judgement or a $[1,5]$ Likert scale, where the lowest value means lowest quality and highest value maximum quality.

In our work, we consider Coherence as being a binary quality aspect. Despite being frequently annotated in the literature on a Likert Scale, what can be considered a response that is neither coherent or incoherent is mostly left to the interpretation of the annotator. Given that we are leveraging an LLM for generation, we find it unfeasible to generate a balanced dataset that contains intermediate levels of coherence. Instead, we generate a positive and a negative response in terms of coherence and label it accordingly. This constrastive sampling strategy for coherent responses is also followed in most metric development work for Relevance or Sensibleness, where models are typically trained using self-supervised learning strategies that sample negative responses by random selection (Mehri and Eskenazi, 2020b; Yeh et al., 2021; Mendonça et al., 2023). Lacking any external knowledge with respect to each dialogue, we then further simplify the reference-free evaluation of coherence as a Question Answering (QA) task ($f(c,h)\rightarrow s\in(0,1)$), with model responses being either coherent ("Yes") or incoherent ("No").

Since the coherence labels are binary, we report detection results using F1-score and Point Biserial Correlation. Additionally, we compute the BLEU-4 score of the generated short explanation using the GPT-4 explanation as a reference. Since BLEU compares overlap in tokens instead of comparing meaning, we also employ GPT-4 as a drop-in replacement for human annotators, and ask it to assess the explanations of 200 random responses from the models that output an explanation.

We collate our main results in Table 3. Due to space limitations, we only report 1-shot performance with a language specific example for Qwen1.5-Chat and the results correspond to the average of the languages. Additional results, including Zero shot and individual language performance, are available in Appendix C.

In order to evaluate our models’ capabilities on unseen dialogue datasets, we evaluate our models on XPersona, which was not seen during finetuning. We only select the best baselines (as reported in Table 3) for this analysis. Additionally, our XPersona subset contains French, which is not present in XDailyDialog, so in addition to the average performance across all languages, we present the results for French separately. For fair comparison, we utilise the English example when evaluating the performance of Qwen1.5-Chat in French.

Looking at the results in Table 4, we find that the conclusions from DailyDialog also carry over to XPersona. For the unseen language (French-fr), we note a large drop in performance for Qwen1.5-Chat-7B, when compared to the other languages, which could be explained by the 1-shot example being in English. For our proposed models, we see a larger gap in performance between French and the other languages for the smaller models, whereas for ECoh-4B, the performance for French is well within the range of that observed for other languages. This is also what we observe for GPT-3.5-Turbo. This finding suggests that, given an LLM that natively supports languages for which we have no finetuning data, coherence knowledge can be drawn from languages that were included for finetuning.⁷⁷7It is important to acknowledge that this finding is only likely to extend to languages that follow western normative rules for coherence. An additional interesting experiment would be to test ECoh on a language that does not conform to these rules – however, these are typically low-resource.

Since the models were trained and evaluated on synthetic data, it is important to check if ECoh performs adequately on external evaluations conducted by human annotators. As such, we also assess ECoh on the FED-turn annotations (Mehri and Eskenazi, 2020a) for "Relevance" and "Overall", which is a typically used benchmark for dialogue evaluation. Similar to other works, we calculate the average human annotation ($[0,2]$ for Relevance and $[0,4]$ for Overall) and report results using Spearman correlation between the human annotation and the score provided by each evaluator. For the LLMs, we keep the binary formulation for coherence (score is either 0 or 1). For the coherence explanation evaluation, lacking a reference, we again use GPT-4 as an explanation evaluator but without a reference response, and evaluate all responses. In order to gauge evaluation performance, we also calculate correlations with GPT-4 as a response evaluator. We refrain from providing GPT-4 explanation scores due to potential self-evaluation bias.

From Table 5, we draw several conclusions. Firstly, when looking at the correlation metric, we see that the performance gap between random response-trained models and ECoh is much smaller. This is mainly due to the older chatbots models used for FED-turn – Meena (Adiwardana et al., 2020) and Mitsuku⁸⁸8Mitsuku blogpost – being more likely to output irrelevant and non-specific responses that mimic random response selection.

Secondly, we note that our finetuning is still useful for detecting coherence issues on FED, since, overall, ECoh outperforms the corresponding parent model (e.g., ECoh-4B vs Qwen1.5-4B) on Relevance. However, our multilingual models underperform against the English-specific finetuning (with the exception of the 4B model) which could be explained by FED being exclusively in English.

Finally, despite GPT-4 not being a perfect evaluator (low correlation in FED), we assume the scores we obtain for FED remain comparable to previous experiments. With that in mind, we find that the explanation quality is overall lower for ECoh and the teacher model, GPT-3.5-Turbo when compared to GenResCoh (for instance, we report an absolute drop of 1.01 for GPT-3.5-Turbo). In contrast, the Qwen1.5-Chat models’ explanation quality remains almost unchanged when compared to GenResCoh. As a result, ECoh models achieve less impressive results when compared to their parent models. When comparing models of the same size up until 4B, we do see some improvement in explanation quality. However, our 4B model fails to outperform the 7B model, both in terms of correlation and explanation quality. For the correlation, we believe this is due to the parent model (Qwen1.5-Chat-4B) having low predictive performance – if we look at the 1.8B models, they yield better correlations. For the explanation quality, we note that the teacher model, GPT-3.5, has also lower results than expected.

However, it is important to acknowledge that these models are tailored towards the binary classification of coherence. As such, we also present results for FED-turn with a binary mapping. In this case, we consider a response to be relevant when the majority of the annotators rate the response as fluent. We present the confusion matrices for the best models of each family in Figure 1. Here, we note that ECoh-1.8B-en is more likely to say a response is coherent, incurring slightly more False Positives than the other models (17 against 5), but also lower False Negatives (83 against >129).

In order to determine limitations and weaknesses with using ECoh, we conducted a thorough analysis of all errors in GenResCoh-PC and FED. We summarise significant findings below.

Statistic-based metrics such as BLEU (Papineni et al., 2002), ROUGE Lin (2004), and METEOR (Banerjee and Lavie, 2005), are a popular choice for dialogue evaluation because they are simple to calculate and lightweight. However, since they assume valid responses have significant word-overlap with the ground truth, their correlations with human judgements annotations are very low (Liu et al., 2016) due to the one-to-many nature of dialogues. Additionally, they cannot be used to evaluate models whenever a gold-response is not available.

Consequently, learned metrics were proposed. The typical approach was to finetune pretrained encoder models using positive and negative samples targeting different quality aspects such as fluency and relevance (Mehri and Eskenazi, 2020b; Phy et al., 2020; Sai et al., 2020; Mendonca et al., 2022; Zhao et al., 2020). Other approaches used graph representations to model dialogue interactions explicitly Huang et al. (2020); Zhang et al. (2021).

With the introduction of LLMs in a wide range of NLP tasks, most recent approaches leverage them for dialogue evaluation. G-Eval (Liu et al., 2023a) uses GPT-3.5-Turbo and GPT-4 for the evaluation of generation models using a "Chain-Of-Thoughts" step and a scoring function based on return token probabilities. LLM-Eval (Lin and Chen, 2023) is a single-prompt-based evaluation method that leverages a unified evaluation schema to cover multiple dimensions of conversation quality in a forward pass. DialEvalML (Mendonça et al., 2023) combines encoder-based models and direct prompting and score extraction from GPT-3.5-Turbo. XDial-Eval (Zhang et al., 2023) probes the evaluation capabilities of several open source LLMs agaisnt GPT-3.5-Turbo (Ouyang et al., 2022), and also finetunes them with dialogue data. To the best of our knowledge, this is the first work that conducts supervised learning of LLMs for the task of dialogue evaluation.

There are several studies that propose augmentation and synthetic generation approaches to scale dataset sizes that target commonsense reasoning (Bhagavatula et al., 2023; Wang et al., 2023), summarisation (Jung et al., 2024), and dialogues (Chen et al., 2023; Kim et al., 2023) for training purposes.

For dialogue evaluation in particular, most metrics are finetuned using self-supervised data (Mehri and Eskenazi, 2020b; Phy et al., 2020; Yeh et al., 2021; Mendonca et al., 2023). The most widely used approach is to select positive samples consisting of the ground truth response, and negative responses from randomly drawn dialogues. Ghazarian et al. (2022) relies on Abstract Meaning Representation (AMR) to apply semantic-level manipulations to existing responses. Our work, in comparison, leverages a strong LLM to generate new incoherent responses at scale.