Planning with Large Language Models for Conversational Agents

Conversational agents (CAs) or dialogue systems are designed to offer social support or functional services to human users via natural language interactions. Existing mainstream approaches can be categorized into conversational question answering (CQA) (Anantha et al., 2021; Singhal et al., 2023; Zhuang et al., 2024), open-domain dialogue (ODD) (Li et al., 2017; Roller et al., 2021; Ouyang et al., 2022), task-oriented dialogue (TOD) (Budzianowski et al., 2018; Sun et al., 2022; Quan et al., 2020; Moradshahi et al., 2023) and conversational recommender systems (CRS) (Wu et al., 2019; Zhang et al., 2021; Wang et al., 2023a). The goal of CQA and ODD is to passively respond to users to provide knowledgeable or engaging conversations, thus lacking proactive. CRS sets goals by specifying a topic, such as “movie recommendation, ‘King of Comedy’,” and autonomously plans the best path of dialog actions to smoothly guide the conversation towards recommending “King of Comedy”. However, in complex tasks, dialogue actions frequently adhere to strict sequential constraints, as commonly found in the widely accepted standardized operating procedures (SOPs) established by humans across various domains (Akyar, 2012; Zhou et al., 2023). As shown in Figure 1, the prerequisite for persuading credit card activation is that the user must be the cardholder. CRS lacks planning globally controllable path of dialog actions. TOD aims to provide functional services according to the service process, and their controllability depends on manually drawing dialogue trees or predefined intent, slot, and action training data. It can proactively guide users to provide slot information, but cannot handle situations where users uncooperate, such as persuasive dialogue or negotiation dialogue (Wang et al., 2019; Chawla et al., 2021). We posit that dialogue’s proactivity, controllability, and reduced reliance on manual intervention are crucial to building autonomous business CAs. Recently, prompting large language models (LLMs) has become the predominant approach to zero-shot language tasks, and its applications in dialogue have received much attention. However, integrating LLMs into conversational agents while addressing the limitations of the aforementioned dialogue systems remains a significant challenge.

To address this challenge, we propose a novel dialogue framework, namely planning-based conversational agents (PCA) as shown in Figure 1. In contrast to prior research, our approach only requires humans to provide task definitions and goals for the LLMs to plan the core and necessary SOP offline before the conversation. During the conversation, LLM refers to the SOP to plan the best action path online and generate responses to achieve process controllability. For actions not in SOP, such as answering questions and persuasion, LLM can plan autonomously to enable proactive dialogue. To further this research, we must address several challenges: (1) How to construct a conversation dataset that meets both initiative and controllability? (2) What are the PCA implementation schemes? (3) How to accurately evaluate the effects of different schemes?

For the first issue, the most straightforward approach would be to manually edit flowcharts and dialogues, which are immensely costly. Recently, advanced LLMs, e.g., ChatGPT, can outperform crowd workers on many annotation and evaluation tasks (e.g. (Gilardi et al., 2023; Wang et al., 2023b; Pan et al., 2023)). Therefore, as shown in Figure 2, we design a four-step LLM role-playing system to assist manual correction in creating a dialogue dataset, which is more cost-effective and quickly scalable.

For the last two challenges, we first implemented the chain of thought (CoT) and supervised fine-tuning (SFT) for SOP prediction in offline planning. In online planning, we first attempt to encode SOP directly into the prompt of LLMs, using in-context learning (ICL) to enable them to understand and follow these constraints. Additionally, inspired by the use of the Monte Carlo Tree Search (MCTS) algorithm in the chess game such as AlphaGo which effectively harnesses human prior rules to control and enhance game AI (Silver et al., 2017), we draw an analogy between dialogue task SOPs and game rules. MCTS is a recent and strikingly successful example of a decision-time planning approach (Sutton and Barto, 2018). By utilizing these SOPs’ rules within MCTS’s expansion and simulation phases, we can better search for the optimal dialogue action that satisfies SOP. Finally, We evaluated the overall/controllable/proactive action accuracy in single-round dialogue and integrated dialogue dimensions. Additionally, manual evaluation was conducted to address the limitations of automatic evaluation regarding dialogue diversity.

Overall, we propose conversational agents based on LLMs with proactive and controllability and lower reliance on manual intervention. To the best of our knowledge, this problem has not been fully studied in prior work. Therefore, we advance the research on this issue from three aspects: task definition, data creation, and modeling. Our contributions are summarized as follows:

• •

  We propose and define the proactivity and controllable conversational agents with LLMs, which are promising yet unexplored.

• •

  We propose a semi-automatic dialogue dataset curation framework and develop a high-quality Chinese dialogue dataset (PCA-D). It not only provides insights into this task but also facilitates more research on it.

• •

  We develop multiple variants and evaluation metrics based on ICL, SFT, CoT, and MCTS modifications, which demonstrate superior performance in implementing offline planning of SOP and online planning of dialogue prediction.

Dialogue Planning. In the process of dialogue planning, previous work focuses on subgoal generation (Zhang et al., 2021), the next round of dialogue transition strategy (Tang et al., 2019), hierarchical strategy (Kishinami et al., 2022). Recent work further involves planning dialogue paths based on basic knowledge, a goal-oriented dialogue planning framework to plan a series of dialogue actions, and guiding the system to proactively transition between different dialogue stages (Wang et al., 2022a). Existing research employs greedy, single-turn prediction strategies and overlooks the interdependencies of global policies in dialogue processes, resulting in global dialogue uncontrollable. We propose a SOP-based planning approach to address this issue.
Planning and Reasoning of LLMs. LLMs show prowess in planning and reasoning. Examples include Chain-of-Thought (Wei et al., 2022), its variants (Kojima et al., 2022) , Self-Consistency (Wang et al., 2022b), and Least-to-most prompting (Zhou et al., 2022). self-assessment  (Paul et al., 2023; Welleck et al., 2022; Shinn et al., 2023). Recent research uses more complex reasoning processes, offering new avenues for improving and optimizing the reasoning process. Yao et al. (2023) apply heuristic-based searches, such as depth-first and breadth-first search, to discover optimized reasoning pathways. Zhu et al. (2022); Hao et al. (2023) introduce MCTS to reasoning steps for complex math or logical reasoning. We focus on applying MCTS to implement controllable dialogue systems, with encoding SOPs’ constraints into the MCTS’s expansion and simulation steps.

Given a dialogue corpus $\{(T_{i},C_{i},D_{i})\}_{i=1,2,...,N}$, where $N$ is the number of dialogues. In the $i$-th dialogue, $T_{i}\triangleq(P_{i},A_{i},S_{i})$ represents the task definition, which encompasses task and user profile $P_{i}$ defining the task goal, relevant knowledge, and user personal profile; Agent actions $A_{i}$ contains nodes from SOP and proactive actions outside of SOP; User states $S_{i}$ reflecting the user’s state in completing the task. $C_{i}\triangleq\{\mathrm{dir}(n_{p},m_{q})\}_{n_{p},m_{q}\in A_{i}\cup S_{i}}$ represents SOP, is a directed graph with agent actions and user states as nodes, and $\mathrm{dir}\in\{\text{--},\rightarrow,\leftarrow,\leftrightarrow{}\}$ representing direction between node $n$ and $m$. The field descriptions and examples for task definitions can be found in A.1. $D_{i}\triangleq\{(u_{i}^{t},s_{i}^{t},a_{i}^{t},r_{i}^{t})\}_{t=1,2,...,T}$ represents a dialogue containing $T$ turns of interaction between agent and user. Each dialogue turn $t$ comprises user utterance $u_{i}^{t}$ and corresponding agent response $r_{i}^{t}$. Additionally, we annotate the updated user state $s_{i}^{t}$ resulting from the user utterance (e.g., the user is the cardholder) and the corresponding agent action $a_{i}^{t}$ for response to this status (e.g., introduce activation activity).

The PCA comprises two tasks.

Task 1 is SOP Prediction, defined as

which means that given $i$th dialogue’s task definition $T_{i}$, the model needs to predict the constraint relationship $C_{i}$ between agent actions $A_{i}$ and user states $S_{i}$ by classifying the direction of any two nodes into four categories.

Task 2 is Dialogue Generation, defined as

giving the $i$-th dialogue’s task definition $T_{i}$, constraint relationship $C_{i}$, previous $(t-1)$ turns’ history $H_{i}^{t-1}=D_{i}^{k=1,...,t}$ and vocabulary $W$, the model first predicts the $(t-1)$-th turn’s user state $\hat{s}_{i}^{t-1}$ and then selects the next turn’s agent action $\hat{a}_{i}^{t}$ from $A_{i}$ that not only helps to achieve the goal but also follows the task constraint $C_{i}$, and generates the corresponding response $\hat{r}_{i}^{t}$.

To implement the PCA system, we need to complete two tasks: 1. Design and implement a comprehensive framework for end-to-end multi-turn dialogue. 2. Implement the two algorithm tasks as described in Section 3: SOP prediction and dialogue generation.

In this work, we have explored a new CAs framework and created a high-quality, controllable dialogue dataset. In addition, we proposed an unsupervised algorithm that encodes SOP constraints into the expansion and simulation steps of Monte Carlo Tree Search. Benefiting from our approach developed based on the advanced large language models’ in-context learning, it holds promise to become the next generation of enterprise-level dialogue systems with low cost and high controllability.

Hallucinations Our approach is based on the context learning of LLMs, such as ChatGPT and GPT-4. As LLMs may produce outputs containing hallucinations (Bang et al., 2023), our system might provide information beyond the task definition. We intend to enhance the veracity of responses through post-processing steps, such as training a dedicated safety model and incorporating checks and revisions into the post-processing phase.

Runtime One significant limitation of PCA-M is the runtime. The more exhaustive the tree search is (e.g. increasing n or k), the more likely the algorithm is to find the optimal dialogue policy. However, this comes at the cost of longer simulation times, which may impact the overall user experience. We believe that parallelizing the tree search or reusing portions of the simulated subtrees could help to speed up the runtime. We anticipate that with the advancement of LLMs research, the speed of inference will continue to improve.

Given the independent behavior of agents in goal-oriented dialogue, it’s imperative to scrutinize ethical implications. Our approach does not force the agent to achieve a specified goal, nor does it force the user to accept the agent’s request. Instead, our work highlights the criticality of directing agents to adhere to human-defined limitations. While our measures are potent, we advocate for the stringent regulation of goal signals, particularly when implementing goal-oriented dialogue systems in specialized fields. Currently, the targeting process must uphold factual accuracy, respect user privacy norms, and comply with societal laws.