TourLLM: Enhancing LLMs with Tourism Knowledge

Large language models (LLMs) such as ChatGPT[1] and Llama[2] have proven their effectiveness in various natural language processing (NLP) downstream tasks with their excellent performance[3]. These models generate natural and fluent language expressions in diverse contexts by capturing linguistic patterns from vast amounts of textual data, significantly advancing NLP technology[4].

However, with the increasing public demand for various travel needs, there is a noticeable imbalance between the supply of tourism services and the growing demand. The applications of LLMs in the tourism domain are evident, such as personalized recommendations, language translation, and chatbots[5]. However, the lack of tourism domain knowledge limits the performance of LLMs in attraction recommendations and travel plans. This limitation restricts the application of LLMs in tourism[6]. Additionally, while LLMs perform exceptionally well in English environments, their performance in Chinese environments is relatively limited. This discrepancy is mainly due to the complexity and uniqueness of the Chinese and the differences in data distribution between Chinese and English. These factors can create barriers for non-English speakers and limit the public’s access to travel advice.

To address these limitations, researchers have explored a range of vertical domain fine-tuning methods to alleviate the lack of corpus in the domain such as Parameter-Efficient Fine-Tuning(PEFT)[7], Low-Rank Adaptation of Large Language Models(LoRA)[8], and Quantized LoRa(QLoRA)[9]. For instance, researchers have constructed domain-specific datasets in the medicine and law domain and employed techniques like continued pre-training, vertical domain fine-tuning, and learning from human feedback[10] to enhance model performance. This has driven the development of LLMs in specific domains resulting in more specialized outcomes such as HuaTuo for Chinese medical knowledge[11] and Lawyer LLaMA for legal knowledge[12].To enhance the accuracy of LLMs in providing travel advice and improving user experience in the tourism domain, it is crucial to develop LLMs tailored to the tourism domain.

In this paper, we construct a Chinese Supervised Fine-Tuning(SFT) dataset for the culture and tourism domain, named Cultour, which consists of three parts. Firstly, we build a knowledge base of common tourist questions and answers(QA). Leveraging the capabilities of ChatGPT, we design a set of prompts to expand the QA pairs and generate tourism knowledge QA data. Secondly, we organize data related to travel planning to design travelogues manually. Thirdly. we craft unique QA in the aspects of eating, living, traveling, touring, shopping, and entertaining, to enrich the diversity of the dataset. Based on this dataset, we propose TourLLM, a model fine-tuned based on Qwen1.5, to improve the quality of information provided about attractions and travel planning. To better evaluate the model’s performance, we adopt both manual and automatic evaluation methods to assess the performance of TourLLM. In particular, for manual evaluation, we design a new metric called CRA (Consistency, Readability, Availability) to evaluate LLMs in the tourism domain manually and the experimental results demonstrate the effectiveness of TourLLM.

In summary, our contributions can be summarized as follows:

1. We construct Cultour, a high-quality Chinese SFT dataset for tourism and culture. The dataset contains tourism knowledge base QA data, travelogues data, and tourism diversity QA data.

2. We propose TourLLM, a Qwen-based model supervised-finetuned with Cultour in the tourism domain.

3. We introduce CRA, a novel metric for evaluating LLMs in the tourism domain considering consistency, readability, and availability.

4. We assess TourLLM using both automated and human evaluations. The experimental results demonstrated the effectiveness of TourLLM.

The emergence of general LLMs like ChatGPT 3.5 opens up a new way of life for people and reduces the cost of living[13]. However, in specific application domains, the performance of general large models is still limited[12]. Therefore, researchers are considering combining domain-specific data with general-domain large models to train a vertical-domain large model to serve specific domains. Huang et al.[12] used 50k legal domain-related data to continue pre-train the model, enabling it to possess more relevant knowledge in the legal domain. In the process of answering questions, the model’s responses are more knowledgeable. Yang et al.[14] implemented a complete LLMs training process involving pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF) to enhance the model’s performance in the Chinese medical field. To address the limited datasets for specific languages, Xiong et al.[15] designed a translation model to translate English datasets into Chinese and then fine-tuned the ChatGLM to expand the domain-specific datasets to enhance the model’s performance. Li et al.[16] fine-tuned LLaMA using 100K patient-doctor dialogues and supplemented knowledge with a recall-based retrieval system. Wang et al.[11] integrated distilled data from ChatGPT and real data from doctors, leveraging reinforcement learning to enhance model performance and achieve the functionality of interactive medical consultation. Beyond conventional LLMs training methods, Xu et al.[17] employed RLHF[10] and Direct Preference Optimization[18] to make the final output results more fitting to human reading habits. After optimizing LLMs through continued pre-training, supervised fine-tuning, and RLHF methods, hallucinations still occur within the domain. Retrieval-augmented Generation(RAG)[19] combines LLMs (such as Llama and ChatGLM) with external data, enabling the model to refer to the knowledge base when answering questions. Cui et al.[20] used laws and regulations as an external knowledge base to enhance the accuracy of the model’s responses.

The combination of LLMs with the tourism domain has also attracted the attention of researchers. Hsu et al.[21] discussed opportunities and challenges of applying Generative Artificial Intelligence (GenAI) and LLMs by the tourism industry and tourists. Mo et al.[22] leveraged prompt-based learning with LLMs, meticulously designing prompts to ensure the responses of the model are reasonable. Wang et al.[23] used domain-oriented LLMs to transform user inquiries into diverse guidance-seeking contexts and facilitate multi-modal interactions. Secchi et al.[24] proposed a method that integrates LLMs with domain-specific knowledge graphs, aiming to optimize hotel services. This approach combines domain knowledge graphs with feature engineering to enrich the data representation within LLMs. Based on the current development of the cultural and tourism domain, we constructed Cultour dataset. Based on this dataset, we proposed TourLLM, a model fine-tuned based on LLMs to enhance the performance of LLMs in the cultural and tourism domains. This provides new methods for applying LLMs in the tourism domain and offers valuable references and insights for future research.

In this section, we provide a detailed overview of our proposed method for constructing the Cultour dataset and the PEFT method for fine-tuning TourLLM. The overall training process is shown in Fig. 1.

To validate the performance of the TourLLM model, we evaluate it against common LLMs and conduct a detailed analysis of the results. Additionally, we introduce CRA, a human evaluation standard specifically for LLMs in the tourism domain, which assesses the model’s performance based on Consistency, Readability, and Availability.

In this section, we use automatic and human evaluation methods to evaluate the performance of TourLLM.

In this paper, we present Cultour, a high-quality Chinese tourism SFT dataset, which is specifically fine-tuned for LLM. It includes 12,823 data items such as eating, living, traveling, traveling, shopping, and entertaining. Based on this dataset, we fine-tuned an LLM, TourLLM, for the cultural and tourism domain to gain deeper insights into people’s needs. To better evaluate the performance of TourLLM, we propose human evaluation CRA metrics tailored for LLMs, incorporating consistency, readability, and availability. We use both automated and manual evaluation methods to assess model performance. Experimental results demonstrate the effectiveness of TourLLM.

In future work, we will utilize TourLLM and external documents to enhance retrieval capabilities through a retrieval augmentation generation (RAG) approach to make responses more reliable.