Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction

A conversational domain for a TOD refers to the specific topic that the dialogue revolves around, encompassing all the knowledge that is pertinent to the conversation. In this context, the conversational domain is typically represented by a domain ontology providing a schema of the concepts (e.g., Restaurant, Hotel, Movie), a set of slots $S$ (e.g., Food, Area, Price) for the concepts, and the set of values that each slot can assume (e.g., Expensive, Moderate, and Cheap for the Price slot). Then, a domain $KB$ comprises a collection of instances for the ontology concepts, each consisting of [slot,slot-value] pairs, adhering to the domain ontology schema.

A TOD can be conceptualized as a sequence of conversational turns between a user and a system aimed at achieving a specific goal. Within this framework, ensuring the consistency of the dialogue is crucial for effective communication between the user and the system. We consider three types of constraints, which need to be respected for a dialogue to be consistent: linguistic, dialogic and domain-based constraints.

A CSP Kumar (1992) imposes certain conditions on a finite set of variables through constraints. Each variable has a finite set of possible values, known as its domain, and constraints define which combinations of values are allowed for specific subsets of the variables. A constraint can be given either explicitly, by enumerating the tuples allowed, or implicitly, e.g., by an algebraic expression. The solution of a CSP is an instantiation of all the variables for which all the constraints are satisfied. A CSP is solvable if it has at least one solution, otherwise it is unsolvable or overconstrained.

The hypothesis of this paper is that the dialogue constraints outlined in Section 2.2 can be modeled as CSPs. Intuitively, variables are the portions of the dialogue that need to be constrained (i.e., the <MASK> tokens in our examples), while the range of possible values for the variables are expressed, either explicitly or implicitly, in a domain $KB$ for that dialogue. The CSP task consists of selecting variable assignments that adhere to linguistic, dialogic, and domain constraints. To formalize this notion, consider a dialogue $d_{i}$ for which $n$ variables (i.e., masked tokens) $x_{1},x_{2},\ldots,x_{n}$ have been defined. Let $D_{i}$ denote the domain of possible values for variable $x_{i}$; let $\mathcal{C}$ be the set of constraints (i.e., linguistic, dialogic, and domain constraints) over the dialogue $d_{i}$, and let $c$ represent a single constraint in $\mathcal{C}$. The CSP task is to determine if there exists an assignment $A=\{(x_{1},a_{1}),(x_{2},a_{2}),\ldots,(x_{n},a_{n})\}$ with $a_{i}\in D_{i}$ for $1\leq i\leq n$, such that $A$ satisfies all constraints in $\mathcal{C}$. This problem can be formulated as follows:

where $\text{{Satisfies}}(A,C_{j})$ denotes the binary relationship between an assignment $A$ and a constraint $C_{j}$, indicating whether the assignment satisfies the constraint.

At step 1 (see Figure 2), we consider a TOD $d$ and a $kb$ (i.e., a set of entities described by slot-value pairs) related to the conversational domain of the dialogue. We do not assume any particular dependency between $d$ and $kb$: $d$ could be either fully covered by $kb$ (i.e., all mentions of slot values in $d$ are present in $kb$), only partially covered, or not covered at all. We consider text portions in $d$ referring to the conversational domain as potential CSP variables: a text portion referring to a slot value or mentioning amounts of instances in $kb$. The rationale is that both slot values and instance amounts are elements that better characterize a TOD and are responsible for its consistency. In our example in Figure 1, we will obtain the following variables with their assignments:

$[x_{1}=Spanish]$, $[x_{2}=three]$, $[x_{3}=Spanish]$, $[x_{4}=cheap]$ … $[x_{10}=Lebanese]$.

We have now established a set $\mathcal{X}$ of variables $x_{1},x_{2},...,x_{n}$, where each variable $x_{i}$ can assume a value either from the slot values or from instance amounts described in $kb$. Moving to step 2 in Figure 2, we now extract the set of constraints $\mathcal{C}$ over the values that can be assigned to $\mathcal{X}$ variables. We consider the three categories of constraints introduced in Section 2.2: linguistic, dialogic, and domain-based constraints.

Once all variables and constraints for a dialogue $d$ are identified, a CSP solver computes possible solutions for the variables in $d$ given $kb$ (step 3 in Figure 2). If one of these solutions matches the variable assignments in $d$, we consider $d$ consistent with respect to $kb$ (step 4 in Figure 2). For example, in the dialogue and $kb$ illustrated in Figure 1, the variable assignments do not match any CSP admissible solution. Specifically, variable assignment $[x_{2}=three]$ violates $C6$, referring to an incorrect number of Spanish instances in $kb$, and variable $[x_{10}=Lebanese]$ violates $C2$, as it does not maintain coherence with the previous turns. If the CSP solver finds at least one solution, the variable assignments in the dialogue must match one of those solutions, ensuring all constraints are followed. On the other hand, if no solution is found with respect to $kb$, the variable assignments should be empty or contain values not in $kb$ to ensure consistency. These aspects will be further explored in the experiments discussed in Section 4.

The purpose of the experiments is to check the feasibility of the CSP-based approach described in Section 3 for detecting dialogue inconsistencies. Our focus is not on optimizing the performance of the CSP solver but rather on investigating critical aspects of the process in a realistic setting. Several steps are involved in this process:

1. 1.

   Initially, we require dialogue-knowledge base ($d$-$kb$) pairs. As for dialogues $d$, we utilize MultiWoz Han et al. (2020) dialogues, which are already annotated for dialogue state tracking, enabling precise identification of variables within the dialogue. From an annotated MultiWoz dialogue $d$, we derive a de-lexicalized version $d_{delex}$, where dialogue content is replaced with CSP variables.

2. 2.

   Additionally, for each dialogue, we derive a knowledge base ($kb$) from the MultiWoz ontology, allowing variation in both the size and type of instances.

3. 3.

   With $d_{delex}$ and $kb$ established, the next step involves generating variable assignments that can be assessed via a CSP solver. To produce dialogues with potential realistic inconsistencies, we employ a large language model (LLM). The LLM is tasked with re-lexicalizing the variables (i.e., substituting slot-values to CSP variables) in $d_{delex}$, considering the provided $kb$. The LLM prompt is illustrated in Appendix A. This re-lexicalization process aims to maximize correctness while adhering to all implicit dialogue constraints.

4. 4.

   Finally, the re-lexicalized dialogue $d_{relex}$ produced by the LLM serves as a variable assignment and is compared with the solutions of the CSP solver on the same $d$-$kb$ pair to produce a consistency score.

The experimental data was sourced from the MultiWOZ 2.3 dataset Han et al. (2020), a widely used benchmark for TOD systems comprising more than ten thousand conversations between a user and a system, covering various domains such as restaurants, hotels, or attractions. For our experiments, we focus on restaurant-related dialogues from the MultiWOZ dataset. In total we consider 131 dialogues with 768 total de-lexicalizations (i.e., CSP variables), as shown in the first row of Table 1. In addition, Table 1 categorizes the dataset into groups based on the number of solutions identified by MiniZinc (see Section 4.4) for each dialogue.

The $kb$ employed in the experiments are sourced from the MultiWOZ database. Specifically, for each dialogue $d$ in MultiWOZ, we selected a pertinent instance from the global MultiWOZ KB that aligns with the content of the dialogue. This ensures both relevance and coherence between the dialogue and the associated domain information. Additionally, to introduce variability in the composition of the dialogue $kb$, we randomly sampled a set of $n$ instances from the global MultiWOZ KB, where $n$ is a randomly generated number between 0 and 8. This approach ensures a diverse range of instances in the dialogue $kb$ while constraining the total number of instances to a maximum of 9, facilitating efficient prompting of the $kb$ to the LLM.

As for CSP solver, we use MiniZinc Nethercote et al. (2007), an open-source constraint programming language specifically designed for modeling and solving constraint satisfaction problems. We employed MiniZinc to obtain solutions satisfying the dialogue constraints for ourevaluation purposes. MiniZinc provides a high-level modeling language that allows users to express problem constraints and objectives. It supports a wide range of constraint types, which make it suitable for modeling diverse problem domains. Among MiniZinc’s suite of solvers, we leveraged Chuffed Chu et al. (2018), a state-of-the-art solver known for its efficiency in solving CSPs through time optimization, especially advantageous for addressing complex and large-scale optimization problems.

For dialogue re-lexicalization, we employed the GPT-3.5-Turbo language model, a member of the OpenAI GPT family Achiam et al. (2023), specifically designed to perform well in conversational contexts. GPT-3.5-Turbo was prompted with both ($d_{delex}$) and its associated $kb$. This comprehensive input served to guide the model to produce dialogues that adhere to the implicit constraints, thereby ensuring dialogue coherence and adherence to the domain. We utilized GPT-3.5 for inference in zero-shot mode (see Appendix A), without any fine-tuning, leveraging the API version dated "2023-05-15" with a temperature setting of 0.9 to ensure balanced exploration and exploitation during dialogue generation.

We introduce two dialogue re-lexicalization baselines, for a comparative analysis with GPT. The first baseline (Random), produces a dialogue $d_{relex}$ where variables in $d_{delex}$ are randomly assigned to slot values present in the $kb$. The second baseline (Most Frequent) produces a dialogue $d_{relex}$ where variables in $d_{delex}$ are assigned to the most frequent value observed in the $kb$. By contrasting our evaluation results with these baselines, we gain insights into the efficacy of our approach in capturing and assessing dialogue consistency.

Global Consistency Accuracy (GCA) and Variable Consistency Accuracy (VCA) are the two metrics used to evaluate the adherence of a dialogue to a specific set of constraints. Given a re-lexicalized dialogue $d_{relex}$ where variables are assigned to values, GCA measures the overall accuracy of the assignments for each variable. The average GCA is calculated as the proportion of dialogues that fully comply with all defined constraints:

where $N$ is the total number of dialogues, and $\textit{Satisfies}(A_{i},C_{j})$ is a binary indicator function that returns 1 if and only if all variable assignments in dialogue $d_{i}$ comply with the constraint $j$, 0 otherwise. On the other hand, VCA assesses the assignment accuracy on individual variables within the dialogue. We compare the dialogue assignment to the solutions of the CSP solver and find the most similar solution; then, we count how many variable assignments coincide with the assignments of the most similar solution. We formally define VCA as follows: