RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Models

The evolution of role-playing in AI Wu et al. (2023), marked by the transition from basic text-based interactions to intricate character simulations Park et al. (2023), reflects the strides made in natural language processing and AI technologies Mysore et al. (2023). Initially, AI role-playing systems offered only fundamental exchanges, limited in their ability to craft dialogues with emotional depth and contextual relevance. With the emergence of advanced models such as GPT-3 Brown et al. (2020), LLaMA Touvron et al. (2023), and ChatGLM, there was a notable enhancement in AI’s capability for engaging in more sophisticated, context-aware conversations. Yet, these improvements also underscored a significant gap in personalization for role-playing applications. Predominantly, LLMs trained on wide-ranging, generic datasets fell short in handling scenarios that demanded a deeper understanding of nuanced emotional nuances Radford et al. (2018) and specific character traits. To address these shortcomings, we meticulously develop the RoleCraft framework. It stands out with its unique dataset, focusing on diverse, non-celebrity personas enriched with detailed emotional annotations Bender and Koller (2020). This dataset is key to overcoming previous limitations, facilitating a new level of personalization and emotional intricacy in AI role-playing interactions.

The recent strides in LLMs, particularly in understanding user context and preferences Wang et al. (2023a); Abbasian et al. (2023), have significantly propelled the personalization aspect of AI interactions Lee et al. (2022); Subhash (2023). Previous works Shanahan et al. (2023); Li et al. (2023c); Chen et al. (2023) have demonstrated the potential of LLMs in mimicking specific fictional characters and simulating complex human behaviors. However, these models often face challenges in achieving a deep level of personalization and emotional richness that aligns precisely with individual user contexts and needs Miłkowski et al. (2022). Additionally, while these models are adept at simulating personalities or historical figures, they may not effectively handle the subtleties of user-specific emotional responses or cater to nuanced personal preferences. Our work aims to address these gaps by building upon these advancements and offering a framework that focuses on a more granular level of personalization. We propose novel methods for emotional and context-specific interaction, ensuring that our model can adapt and respond more accurately to individual user scenarios.

Our methodology uniquely advances the capabilities of LLMs in role-playing. Setting ourselves apart from approaches such as RoleLLM Wang et al. (2023b), we focus on an innovative integration of fine-grained character portrayal, profound emotional depth, and heightened contextual awareness in dialogue generation. This approach differentiates our work from existing models and addresses challenges in a novel way, enhancing how LLMs can be utilized for creating more realistic and engaging role-playing scenario.

Addressing the issue of inaccurate and semantically irrelevant information retrieval in dialogues, we adopt the BGE²²2https://github.com/FlagOpen/FlagEmbedding retrieval method. BGE is an efficient Chinese and English semantic vector model that ensures the accuracy of responses, especially when dealing with sensitive topics, and remains semantically sensitive to the context, significantly enhancing the quality of interaction Xiao et al. (2023). This familiarity allows models to generate dialogue based on a wealth of pre-existing knowledge. In contrast, modern datasets prioritize the nuanced portrayal of personal and everyday characters. These datasets are derived from diverse sources, including real chat logs, customer service interactions, and fictional narratives from less mainstream media. Such characters might include a typical office worker dealing with daily stressors or a mother showing love and responsibility in a family setting. The dialogues here involve specific, real-life scenarios, such as office interactions or typical family conversations, which lack the broad pre-existing knowledge base associated with public figures.

Our methodology diverges from conventional prompt engineering White et al. (2023) and fine-tuning Howard and Ruder (2018) We specialize in creating diverse, intricate character interactions that enhance role-playing scenarios’ depth and authenticity. Our unique approach combines emotion-driven character profiles with contextually coherent Q&A generation, fostering realistic and engaging interactions. Additionally, the integration of semantic-enhanced retrieval methods ensures both accuracy and relevance, setting our approach apart in its complexity and user engagement potential.

In the evaluation of LLMs Chen et al. (2021); Chang et al. (2023), the role of datasets is paramount, particularly in language processing and character portrayal. Traditional role playing datasets predominantly highlight eminent figures, such as the legendary Sun Wukong, whose familiar attributes and stories are widely acknowledged, facilitating model development Sabadoš (2021). However, these datasets often neglect the finer details and emotional complexity of lesser-known or everyday characters, leading to a representation gap Rolf et al. (2021). Our unique dataset bridges this gap by focusing on the rich, nuanced depiction of ordinary individuals. It involves an in-depth exploration and portrayal of each character’s distinct personality traits and emotional depths, delving into aspects usually overshadowed in dominant narratives.

In constructing our dataset, we design 28 unique and personalized Chinese characters to mirror a wide spectrum of real-world dialogues. These characters ranged from everyday individuals to specialized roles such as intelligent customer service agents and research assistants. Table 3.3 provides basic statistics for RoleInstruct. Our diverse data sources included social media interactions, film and television scripts, and customer service dialogues. We emphasized personal privacy and copyright law compliance, ensuring all data was cleansed and anonymized.

Figures 5 and 5 visually demonstrate the diversity of RoleInstruct, both in terms of linguistic usage and emotional expression. The verb-noun diagram provides insights into the varied narrative contexts, while the word cloud delves into the breadth of emotional and personality traits present.

In addition, we annotate each dataset entry with emotion labels to capture characters’ distinct emotional traits, adding an emotional layer to model training. We use Ekman’s "Six Basic Emotions Theory" Ekman (1992) to label utterances and included additional emotions like neutral, excited, and depressed, totaling ten categories. The use of emotion labels in dialogue datasets has been proven to enhance natural language processing by improving response retrieval and emotional relevance  Zhou et al. (2017). These labels also enrich conversational analysis and aid in building natural dialogue systems  Bothe et al. (2019). See Appendix B for more details.

We assess the ChatGLM3 model, enhancing its performance on specific datasets using the Low-Rank Adaptation (LoRA) fine-tuning method Hu et al. (2021). LoRA’s precision in fine-tuning, essential for handling personalized and emotionally rich content, maintains the model’s core capabilities while adapting to new data features. Please refer to Appendix C for more details.

We benchmark our RoleCraft-GLM’s performance against industry standards such as GPT-3.5 and GPT-4, and leading Chinese dialogue generation technologies like ChatGLM2³³3https://github.com/THUDM/ChatGLM2-6B and ChatGLM3, along with Baichuan2 Yang et al. (2023), Qwen Bai et al. (2023), InternLM Team (2023), SparkDesk⁴⁴4https://xinghuo.xfyun.cn/ and Xverse⁵⁵5https://github.com/xverse-ai/XVERSE-13B. Additionally, we introduce another role-playing baseline, RoleLLM, which utilizes LoRA for fine-tuning on a specific dataset.

Results from Tables 2 and 4 clearly demonstrate our model’s exceptional performance across multiple key performance indicators, particularly in specific role knowledge memory (SPE). Our model significantly outperformed GPT-4 and other models in this dimension, highlighting its superior ability in understanding and generating complex dialogues involving specific roles. However, GPT-4 leads in general instruction response accuracy (RAW) with a score of 0.5661, reflecting its strong capability in interpreting and responding to general instructions. Our model still maintains a high score, proving its effectiveness in handling everyday dialogues. These findings underscore our method’s significant effectiveness in deepening role understanding and enhancing dialogue generation quality. In particular, in emotion-driven role portrayal, our model demonstrated precise capture of each role’s emotional traits and personality, surpassing traditional role-playing models in expressing role-specific knowledge. Moreover, our model also show remarkable ability in maintaining natural flow and contextual consistency.

As indicated in the figure 6, we observe that RoleCraft-GLM leads with the highest score on the RPCS metric, demonstrating its high degree of alignment with expected responses in terms of emotional and contextual accuracy. In contrast, lower scores from models such as InternLM and Qwen may reflect deficiencies in handling emotional and contextual understanding in role-playing dialogues. These results emphasize the importance of specialized training and fine-tuning for role-playing applications in enhancing model performance.

The results across three dimensions are clearly illustrated in Table 4. Our model outperforms other mainstream models in overall performance. Particularly in terms of Communication Effectiveness and Expressive Diversity, our model demonstrates superior capabilities, reflecting its strengths in generating smooth, logically consistent, and emotionally rich dialogues.Please refer to Appendix F for a detailed manual assessment analysis.

The results of our ablation experiments (see Table 6) show that the RoleCraft-GLM model with emotional annotations scored higher in SPE than the version without them. This indicates that emotional annotations not only provide the model with key information for a deep understanding of the characters’ emotional states and personality traits but also ensure that the generated dialogues are more closely aligned with the characters’ true feelings and personalized expressions.

Table 6 demonstrates the hybrid instruction strategy’s superiority in overall performance (Avg), effectively balancing diverse aspects of dialogue. This superior performance originates from merging both general and specific instruction strengths. The general instruction strategy excels in handling broad dialogues and maintaining consistent character styles, as reflected in higher RAW and CUS scores. However, it is less effective in capturing detailed, character-specific knowledge, shown by a lower SPE score. In contrast, the specific instruction strategy shows proficiency in detailing character-specific traits, evidenced by a higher SPE score, but does not perform as well in wider conversational contexts. The effectiveness of the hybrid strategy is highlighted in its ability to combine the broad applicability of general instructions with the detailed focus of specific instructions, offering a more versatile and nuanced approach to dialogue generation.

The output of our model excellently combines the character traits of Zhou Xiaobei, colloquial expressions, emotional authenticity, and a close connection with the plot background as shown in Figure 7. Firstly, it aptly reflects Zhou Xiaobei’s mischievousness and intelligence by using phrases like ‘little troublemaker’ and ‘super fun’, which showcase his sense of humor and wit. Secondly, this natural expression, such as ‘you know’ and ‘haha’, adds a sense of closeness and authenticity, making the language sound more like a child’s natural way of speaking rather than too formal or mature. Moreover, the emotional expression about ‘not wanting to disappoint mom’ genuinely reflects Zhou Xiaobei’s love for his family and concern for his mother, showing the complexity and depth of his emotions. Lastly, this statement is closely connected to Zhou Xiaobei’s life background, displaying his life scenario and psychological state as a child from an ordinary family. See Appendix E for more generated examples.