LLM-based Frameworks for API Argument Filling
in Task-Oriented Conversational Systems

Utilization of pre-trained Language Models for Task-oriented Dialogues (ToD) was pioneered by Zhang et al. (2019) and Peng et al. (2021).  Kulhánek et al. (2021) and Lin et al. (2020) improved the basic ToD modeling approaches by employing contrastive state training and belief state differences, respectively. Other works Pandey et al. (2018); Cai et al. (2019); Nekvinda and Dušek (2022) proposed to combine generative models with retrieval-based approaches. While Hudeček and Dušek (2023) perform zero-shot evaluation of various LLMs for ToD modeling, to the best of our knowledge, this is the first work to exploit and instruction-tune LLMs in the billion-parameter regime for argument filling in ToD systems.

The introduction of Transformer-based architectures heralded the beginning of large and incredibly capable models for Natural Language Processing (NLP) Vaswani et al. (2017). Transformer-based language models with several billions of parameters, such as GPT-3 Brown et al. (2020) and OPT Zhang et al. (2022), have shown unprecedented zero- and few-shot performance across diverse NLP tasks. The generalization capability of these so-called Large Language Models (LLMs) was further improved by training them via instruction-tuning Goldwasser and Roth (2014) with in-context instructions. The promising results obtained by instruction-tuning inspired the development of large instruction-paired datasets, such as NaturalInstructions-v1 (NI-v1) Mishra et al. (2022) and SuperNaturalInstructions Wang et al. (2022a). The remarkable performance of general-purpose instruction-tuned models inspired the development of more domain-specific models. Examples of such models include: InstructUIE Wang et al. (2023) for information extraction, CoEDIT Raheja et al. (2023) for writing, ChatDoctor Yunxiang et al. (2023) for medical purposes, and Goat Liu and Low (2023) for mathematics.

To guarantee experimental consistency across different models and datasets, we first design a common prompt template for argument filling. An example of the default prompt template, which includes a short instruction, the pre-defined API schema, and dialogue history up to the specified API call, is provided in Figure 3. This prompt template is used for both in-context instruction tuning and evaluation processes and remains fixed across all of our experiments unless stated otherwise.

After the LLM has been bootstrapped on the argument filling datasets, we sample $K$ number of outputs from the model and score the generated outputs using the above reward function. We only select outputs that yield positive reward to augment the train dataset. With the newly added instances mixed in the train dataset, we perform one additional epoch of supervised fine-tuning.

There exist two expected advantages of incorporating rejection-sampled model outputs. First, utilizing the model outputs filtered with the custom reward function allows us to effectively augment the train dataset with desirable instances without the need to collect additional data points to avoid overfitting. Second, we expect that incorporating these outputs will improve the fine-tuned LLM’s robustness to noisy data points it may encounter at test-time. Even if the model-generated outputs yield positive reward, they will inevitably be noisier than the curated train dataset with ground-truth labels. Therefore, the LLM that has been exposed to noisier data points in the rejection sampling phase will exhibit a higher degree of robustness and generalization performance.

It is infeasible to fine-tune LLMs whose design and weights are not released to the public. Therefore, we additionally explore a more fine-grained and informative prompting method to complement larger LLMs. The default prompt design as described in Figure 3 asks the model to identify required arguments and extract appropriate information to fill them all at the same time. For multi-step prompting with hints, we instead prompt the model to identify and fill one argument at a time. By using this more targeted prompt design, we are providing the LLM with additional information about required slots and effectively restricting its generative behavior to prevent its digression from the pre-defined schema and dialogue history.

We utilize the Huggingface Wolf et al. (2019) library for implementation and training of models. All experiments are executed on NVIDIA V100 GPU with 32GB RAM. The following set of hyperparameters is used for the supervised fine-tuning phase: batch size of 8, Adam optimizer with initial learning rate of 0.00002, weight decay of 0.1, and constant learning rate scheduling. We run the supervised fine-tuning phase for 5 epochs before performing rejection sampling. As mentioned in Section 3.2, we perform additional fine-tuning with rejection-sampled data for only one additional epoch. All hyperparameters remain unchanged from the supervised fine-tuning phase.

The results obtained on STAR and SGD datasets under the in-domain evaluation setting are reported in Table 1. The suffixes -sft and -sft-rs are used to denote models that have been trained only with supervised fine-tuning and with supervised fine-tuning and rejection sampling, respectively. Multi-step prompting that provides additional hints successfully improves the performance of the ChatGPT models. More importantly, we observe that the LLAMA-v1-7B model that has been trained with the proposed instruction-tuning pipeline with rejection sampling (LLAMA-v1-7B-sft-rs) obtains the best performance across all metrics on both datasets. This result clearly demonstrates that with our training framework, relatively smaller and light-weight LLMs can outperform larger ones. Furthermore, the superiority of LLAMA-v1-7B-sft-rs to LLAMA-v1-sft provides strong support for incorporating rejection-sampled data to effectively improve the performance of fine-tuning with less training budget. Lastly, we note that a larger degree of performance improvement is observed on the SGD dataset, which has a wider variety of tasks and thus can be considered more difficult.

To simulate an out-of-domain test scenario, we deliberately create a train-test split, such that there is no explicit or implicit task domain overlap between the train and test set The results obtained under the out-of-domain evaluation setting are reported in Table 2. In general, the out-of-domain evaluation results show similar tendencies to the in-domain results. While the proposed instruction-tuning framework and multi-step prompting successfully improve the performance of open-sourced and closed-sourced LLMs, respectively, they both experience slight performance degradation when compared to the in-domain evaluation results.

We analyze sources of error in outputs generated by LLAMA-v1-7B-sft-rs to identify room for improvement. In Figures 4 and 5, we compare the four error rates, as defined in Section 3.2, in LLAMA-v1-7B-sft and LLAMA-v1-7B-sft-rs models. Training the LLAMA-v1-7B model with SFT $+$ RS reduces all four error rates, and the rate of hallucinated value errors is particularly low compared to other errors. This analytical result implies that once grounded, the LLM mostly ceases to hallucinate and remains close to the API schema and dialogue history provided as a part of the prompt template.