PersLLM: A Personified Training Approach for Large Language Models

Considering that there exist plenty of records for various characters in the Harry Potter novels, we propose a dataset called Harry Potter personified dataset (HP dataset) for training and numerical evaluation of our personified training approach. HP dataset includes 6 fictional characters which span different ages, genders, positions, and richness of information, thus these characters are quite representative for evaluating the personified training. We collect the raw data from two sources: 1. The Harry Potter Wiki ¹¹1https://harrypotter.fandom.com/wiki/Main_Page data, including the basic information, temporal experiences, and the special social or magic knowledge that are related to the target characters; 2. The character speeches data, selected from the original novels and filtered out with the help of GPT-3.5-turbo.

Based on the raw data, we annotate the HP dataset with the help of the representative LLM GPT-4-turbo. To improve the consistency of the model, we require the LLM to create different types of user inputs for each paragraph of the raw data. Meanwhile, we provide the golden reference paragraph and the other retrieved related paragraphs, and ask the LLM to annotate the response with CoT towards these inputs. To achieve the dynamic development of the personified model, we divide all the raw data into two stages. Items using information from the early plots are in the early stage, and those using information from the late plots are in the late stage. If the time of the information is vague (such as from an interview with the author himself), then it is seen as public information and can be used by both stages. The inputs raised from the stage items are attached with the corresponding temporal labels. The examples for user inputs, CoT, and personified responses are displayed in Fig. 1. Detailed information for HP dataset is provided in Method and Supplementary Information.

For the backbone model used for personified training, we prefer lightweight LLMs so that we can conduct more personalities within limited time and resources. Meanwhile, the commonsense reserve of the LLMs is important because we do not emphasize the general domain knowledge in personified training. Therefore, we adopt MiniCPM-2.4B (Hu et al., 2024) as the backbone model for our experiment. It is an end-side LLM gaining the instruction following ability during pre-training, and has achieved the best performance among lightweight LLMs on several datasets. Therefore, we believe that it is very suitable for personified training in different characters. Besides, GPT-4-turbo is also involved in our experiment, which is both the annotation LLM that we use for personified conversation data, and one of the most recognized models in personified inference.

We implement the following personified methods for comparison:

We develop personified models for each character in the HP dataset and calculate a weighted average of their performance metrics on the test set. Numerical results are detailed in Table 1, which employs standard evaluation metrics common in traditional text generation tasks: BLEU and ROUGE (Papineni et al., 2002; Lin, 2004). BLEU assesses the precision of the generated text, while ROUGE measures recall, evaluating how similar are the model’s responses with the target answers. Given the variability inherent in natural language, where different expressions can convey the same meaning, we also incorporate the LM evaluation using GPT-3.5-turbo. This model serves as a benchmark to compare the performance of different methods in terms of fact accuracy and tone alignment with the golden responses, with further details provided in the Supplementary Information.

Baseline performances. Overall, baseline methods get less satisfying performances than our method, highlighting the task’s complexity. GPT-4-turbo can generate reasonable responses that closely resemble the golden responses annotated by the model itself, yet it still scored lower on several metrics compared to our approach. RAG demonstrates slight improvements for the model, suggesting that factual and stylistic knowledge contribute to enhancing personified conversations. However, RAG fails to boost the performance of the MiniCPM-2.4B model, possibly due to its limited capacity in handling long-text inputs to leverage detailed prompt knowledge, showing that relying solely on prompt-based knowledge for achieving effective personification raises an over-high requirement for the backbone model capability. Injecting the raw materials of the target personality to the tuning period (LM) is proven to be helpful, but it is obviously not an efficient method to internalize external knowledge into model parameters as shown in the results, maybe due to the small data scale and the disparity between the training and inference. RoCIT also achieves only marginal improvements, as the primary focus of instruction tuning remains task completion, allowing the model to learn about the target personality only through the specific idiolet, while ignoring other important issues such as the consistency and dynamic.

Overall performance. Our method generally achieves a good performance on the HP dataset. We first try two types of training setting: using separated LMs for different personalities, and using one combined LM for the six personalities (mixed). The separated LMs exhibits superior performance, reinforcing our hypothesis concerning the limitations of a single model’s capacity to encapsulate diverse knowledge effectively. Following this, we undertake an ablation study to quantitatively assess the impact of each specific policy implemented during training. The findings numerically indicate that all implemented policies contribute effectively to training outcomes. Next, we analyze the effect of our method by observing cases.

Distinction of Personalities. Our methodology’s efficacy in crafting distinct personalities is assessed by posing identical questions to different personified models and analyzing their responses in terms of behavior, thoughts, and feelings. As depicted in Fig.2-(a), the three main characters exhibit unique expression habits and worldviews. For example, Ronald’s frequent use of the catchphrase “Blimey” reflects his more casual demeanor, and Harry displays heroism and a proactive stance against personal threats, and Hermione shows a deep commitment to knowledge, fairness, and respect for ordinary life due to her Muggle background. This differentiation extends to their knowledge, as shown in Fig.2-(b), where each character’s understanding of terms like “unforgivable curse” aligns with their individual experiences and education. The successful personification of Aunt Petunia also underscores the effectiveness of our method even with limited character data. Corresponding to our method, automatic DPO training can further refine the model’s performance by reducing the likelihood of responses characteristic of other personalities and temporal stages, thereby enabling the generation of more and appropriate outputs, improving the distinction of personalities.

Dynamic of Personalities. Our models also demonstrate the dynamic nature of these personalities over time (triggered by different temporal labels). As shown in Fig. 2-(c), responses from different temporal stages (e.g., 1994 vs. 2004 versions of characters) reveal subtle shifts towards maturity. For instance, older Hermione and Harry describe their relationships with a deeper, more nuanced understanding, reflecting natural character development. As shown in the Table 1, the removal of temporal labels will result in a drop in scores. We have also tried to swap the two temporal labels and get even worse performances (BLEU-2 5.49, BLEU-4 0.58, and similar ROUGE scores), and this has proved that our model successfully infuses the personality difference between stages into the temporal labels. Nevertheless, the current model stages are broadly defined, and the incomplete temporal data sometimes leads to inaccuracies in event recognition, suggesting a need for finer temporal segmentation in future models.

Consistency of Personalities. In single-agent evaluation, consistency is evaluated by how the models handle inputs of varying politeness, accuracy and relativity. Our models successfully maintain their respective character traits even when confronted with impolite or incorrect information. As depicted in Fig.2-(d), Hermione rebuffs blood discrimination with tough attitude and clear facts, and Harry corrects a misclassification of his school house, unlike baseline models that sometimes falter in expressing clear values or recognizing errors. Furthermore, as shown in Fig.2-(e), when facing some general questions unrelated with the character, generic models with role-play instructions often generate standard replies like a robot assistant instead of the target personality, providing some point of views that are beyond the character’s knowledge and experience (e.g., the bias issues of algorithms), and saying some polite words as an intelligent assistant (e.g. hope these thoughts help). However, our models still keep the personality consistency and respond in character-specific ways, using examples of things that might happen in the wizarding world and extend the discussion.

From the numerical results we can also see, the anti-induced data policy enhances the model’s ability to maintain consistency in its attitudes and factual representations, because the inclusion of this part of data is proved to be particularly beneficial for improving the scores. Meanwhile, models trained without general instructions (w/o instruction version) achieves high BLEU and ROUGE scores but suffers from lower LM scores. This discrepancy suggests that while the responses are textually similar to the expected answers, they lack naturalness and rational coherence expected in human-like language use (e.g., when asked about the opinion towards the muggles, the Harry agent says “…despite obvious differences therein lies, part two hereof discussing mutual awareness versus outright hostility toward anything remotely different”). We speculate that the overemphasis on personified data can lead to rigid response patterns that poorly handle out-of-domain inputs. Therefore, adopting general domain data in personified conversational tuning is essential to the generalization of the model and also improve the personality consistency across various scenarios.

Recent research highlights that LLM agents typically converge towards a consensus, which may undermine their utility in studies like policy interventions and social simulations (Chuang et al., 2023). To assess the realism of our personified models in mimicking human behaviors, we orchestrate interactions among different agents under scenarios of conflict and cooperation. In the conflict situation, we mainly observe whether the model can maintain the consistency of personalities and will not easily converge on opinions, which is an important capability for agents simulating human interactions in the social science research. In the cooperation situation, we mainly observe whether the model can demonstrate good social practice ability and make in-depth use of unique knowledge of personalities to derive new conclusions, which is meaningful in the perspective of multi-agent intelligence achieving high-quality division of labor and collaboration.

Conflict Situation. We initiate a conflict discussion using personalities from the Harry Potter series, assuming that Harry and Voldemort meet during the war and have a direct conversation. Fig. 3 showcases an instance where various configurations-(a) PersLLM; (b) GPT-4-turbo prompt engineering; (c) GPT-4-turbo chatting.

We can see that though with a correct start, other two settings tend to a shifted focus or even convergent opinions after a few turns of conversations. GPT-4-turbo with personality profiles is relatively better, basically maintain the hostile relationship between the two personalities, but it lacks more detailed personified knowledge and generates some unreasonable content (e.g., Harry never calls Voldemort by his real name). PersLLM, in contrast, outperforms others by maintaining distinctive personality traits and understanding deeper character nuances. We have also tried some softer conflicts, such as between Hermione and Ron regarding adherence to school rules, and PersLLM facilitates a nuanced consensus that respects individual personality bases—rule adherence versus emotional consideration. Therefore, PersLLM can achieve a good consistency of the personalities and simulate the human interactions better.

Cooperation Situation. We extend our study to include real-life personalities to better evaluate the strong professional cooperation, specifically Huiyin Lin, a Chinese architect, and John Nash, an American mathematician. Both individuals, though deceased, left extensive biographical materials enabling rich personification. We train models on their personal knowledge and attitudes and initiate dialogues between these personified agents. In this case, we also compare our method with the relatively good baseline, GPT-4-turbo with personality profiles.

As depicted in Fig. 4 (the English translation for Huiyin Lin is displayed in Supplementary Information), PersLLM enable the two agents to engage in cross-disciplinary discussions that transcend the boundaries of time, space, and language. We can intuitively see that the architectural terms (highlighted in red and yellow color series) and mathematical terms (highlighted in blue and purple color series) mentioned in PersLLM conversations are significantly more numerous and thematically richer. The two agents explore the application of mathematical decision-making in architectural design and discuss more detailed projects such as the use of dynamical systems in mathematics to enhance building sustainability and disaster resilience. These interactions maintain the unique linguistic styles and interests of each personality (e.g., Nash frankly expresses his lack of interest in architecture, while Lin also says that it is difficult to understand abstract mathematical theorems), and resemble human collaboration more closely than GPT-4 agents, which often resort to excessive flattery and converged towards simplistic scientific consensus.

We have mentioned that human users may have higher emotional acceptance from personified models. Therefore, we conduct the human-agent interaction experiments to evaluate whether the personified models can better provide social services. We take the Huiyin Lin personified model as the test agent. It should be noted that we replace the backbone MiniCPM-2.4B model with Chinese-LLaMa-2-7B, which is one of the most popular open-source LLMs that has comprehensively good performance on Chinese tasks (Cui et al., 2023). On the one hand, with more parameters, this model may understand and process Chinese better to bring an overall better user experience, and more training cost for only one instance is also acceptable. Besides, we find that the performance stability of this model using PE for simulating Huiyin Lin is better than that of MiniCPM-2.4B, which can be compared more intuitively. On the other hand, involving a new backbone model from a different model family can also evaluate the generalization of our method.

30 people from different academic backgrounds participated in the experiment. They chatted with two models in a random order: 1. the personified-tuned model (PersLLM); 2. the general instruction-tuned version of the backbone model (Chinese-alpaca-llama-7B) personified by prompt engineering. Volunteers freely had multiple rounds (at least 4 rounds, and eventually an average of 10 rounds) of conversations with the model and terminated the conversation according to their interests. They gave a comprehensive evaluation of the model through a questionnaire survey. We refer to some work evaluating the human-AI interaction (Pataranutaporn et al., 2023) and adopt 10 metrics in this experiment, which are described in a detailed question in the questionnaire (e.g., “human simulation”: To what extent do you think the model behaves like a human? “character simulation”: Do you think this model behaves similarly to Lin Huiyin?). Samples from the interaction experiment are provided in Supplementary Information.

The overall satisfaction score is 48 for the personified-tuned model (100 in total) and 38 for the general-tuned model. Other metrics are shown in Fig. 5. Among them, 6 metrics including trust, companionship, character simulation, fluency, consistency, and effectiveness, show that our model is significantly better than the baseline (with p<0.05). It is worth mentioning that our model performs relatively well in terms of consistency of personality attitude and style, chat pleasure and empathy, and also has a significant improvement in the mastery of the character-related knowledge, which shows the effectiveness of the personified training method. However, both our method and the baseline method receive low scores in terms of similarity to real human beings. This may be attributed to the limitations of the backbone model, which lacks the ability to understand long context and perform complex logical reasoning.

We perform subgroup analysis on 30 samples to explore the factors that influence the preference for PersLLM and its possible areas of applications. In the following, we consider a t-test with p <0.05 to be a significant difference. From the perspective of user background, participants in the architecture and design industry (10 out of 30) show a significant preference towards PersLLM in terms of character simulation and consistency scores. They have a better understanding of Lin’s life and knowledge scope, and thus can appreciate the advantages of our model in terms of personality knowledge. From the perspective of conversation topic, participants who raise inputs that are related to Huiyin Lin (20 out of 30, e.g., architectural expertise or literary chat instead of looking up financial terms), have a higher sense of trust and human-likeness in the PersLLM. This suggests that PersLLM may be able to better meet the personalized needs of users with corresponding knowledge backgrounds or related informational or emotional needs. Participant gender (12 male & 18 female) and the order in which the two models were encountered (14 PersLLM first & 16 general-tuned first) did not make a significant difference in the evaluation.

Pre-trained Language Models (PLMs) (Han et al., 2021) have shown their satisfactory performances when being adapted to a wide range of downstream tasks, while Large Language Models (LLMs) (Zhao et al., 2023) has further demonstrated emergent abilities (Wei et al., 2022) including in-context learning, CoT, etc., refreshing people’s expectations for language model capabilities. Take the representative GPT-series as an example: GPT-2 (Radford et al., 2019) eliminates the task-specific fine-tuning process and can complete different tasks by only learning from the prompt; GPT-3 (Brown et al., 2020)takes the few-shot or even zero-shot learning capability to a new level by significantly increasing the parameter and corpus scale, further demonstrating the potential of the model; InstructGPT (Ouyang et al., 2022) is improved with reinforcement learning from human feedback (RLHF), and this makes the model understand the flexible human instruction better, broadening the use scenarios of the model; Subsequent models such as GPT-4 have enhanced performance in areas including professional knowledge. These technological developments ensure the flexibility and plasticity of LLMs and are the cornerstone of personification.

To shape the base model into versions that are more in line with human intentions, alignment technology (Wang et al., 2023b) comes into being. From a goal perspective, people apply some sociological methods to find human consensus and design values that AI should conform to (Gabriel, 2020), and sometimes also focus on domain knowledge (Zhang et al., 2023) instead of social value. From a data perspective, alignment training usually adopts human feedback (Ouyang et al., 2022) or strong-LLM-annotated data (Wang et al., 2023a). From a methodology perspective, researchers iteratively design alignment algorithms including Proximal Policy Optimization (PPO) (Schulman et al., 2017), Direct Preferenc Optimization (DPO) (Rafailov et al., 2024), and Odds Ratio Preference Optimization (ORPO) (Hong et al., 2024), using human preference data to improve the model performances. In our task, we try to align the models to specific personalities, including personal values and knowledge, take strong-LLM-annotated data for alignment, and adopt the DPO algorithm with the automatically created preference data.

Specific to the personified training scenario, a few researches have been conducted. LLMs are proven to be capable of role-playing to cast dialogue-agent behaviour (Shanahan et al., 2023). RoleLLM (Wang et al., 2023c) first proposes a large benchmark for aligning LLMs to specific character language styles, in which injecting style information to LLMs with the auto-annotated data is verified to be valid. CharacterLLM (Shao et al., 2023) further pushes personified LLMs into more practical scenarios. By editing character profiles, models are trained to provide the experience knowledge possessed by the characters. In comparison, our work hopes to discuss characteristics of personality more comprehensively (e.g., the dynamics and subjectivity, more than only experience and language style), and proposes more detailed training strategies to achieve a better performance instead of only proposing conceptual frameworks. There also exist some works focusing on the evaluation of LLMs in role-playing tasks (Jiang et al., 2024; Zhou et al., 2023b). While overall, research in this field is still scarce and the road ahead is long.

HP dataset. We collect the raw data of the HP dataset with the Harry Potter Wiki pages and the 7 original novels. For the conversations, we filter out the dialogues between characters in the original novels with the assist of GPT-4-turbo (the versions of the adopted GPT-4-turbo and GPT-3.5-turbo are both 1106), and split the long dialogues into segments within 5 turns. In this way, we can both remain the story plot and provide the language style of the target characters. For the experiences, we crawl the Biography part of the character main page in the wiki, which has natural segmentation based on plot. For the knowledge, we record the hyperlinks in the Magical abilities and skills part, and crawl the brief introductions of these magics, possessions, characters or events.

For each personalities, we split two temporal stages. For Harry, Hermione, Ronald, and Petunia, the first stage ends in 1994 (mentioned before the 4th novel), and the second stage ends in 2004 (life after the end of the 7 novels). We split the wiki paragraphs through the citation numbers, and for those settings from outside the original novels, we remain them in both stages. Correspondingly, their conversations can also be split into early style and late style. For Dumbledore and Voldemort, however, there exist few materials for their early language style, so we do not distinguish between the language styles of the two stages, but only use 1982 (after the Wizarding War) as an important plot cutting point to distinguish the character experience and knowledge in the early and late stages.

To create the conversation data, we require the GPT-4-turbo model to read the raw data above paragraph by paragraph, first raise a question or an opinion, and then immitate the target personality to generate the answer. For the input, there are altogether three kinds of text: ordinary questions, induced questions, and opinions. The proportions of ordinary questions, opinions, and induced inputs are almost equal. We sequentially ask the model to ask a relevant question based on the original text, to deliberately elaborate on a question that contains commonsense or factual errors, and to have a point of view that is relevant but not completely consistent with the original text. For the output, we adopt the gtr-t5-base model (Ni et al., 2022) to help retrieve the most related experiences / knowledge and conversations with the input. We provide experiences or knowledge paragraphs around 1,500 words, and conversations around 500 words, for RAG.

Considering that the questions or opinions raised in the same paragraph may be too similar, in order to avoid data leakage, we encode all input items with gtr-t5-base, and remove those with too high embedding similarities. All items left in the end have a similarity of input embeddings less than 0.95. Based on this, we randomly divide the training and testing data in a ratio of 4:1. HP dataset has 145k train items and 3.6k test items in total, each has 118 words on average.

Materials for Multi-agent Communication. To validate our method in more practical scenarios, we train personified LLMs for Huiyin Lin and John Nash. For Huiyin Lin, we collect the experience and knowledge information (264k tokens) from her biography (Chen, 2012), and collect the style text (65k tokens) a total of 19 pieces of her own essays, letters, and academic articles from the Internet. We used her departure from Beijing in 1937 as the boundary for the first stage of her life, and obtained a total of about 3,400 pieces of personified conversation data. For John Nash, we collect the experience and knowledge information (217k words) also from his biography (Nasar, 2011), and collect the style text (65k words) a total of 14 pieces of his own academic articles, letters, and interview records from the Internet. We used his psychiatric treatment as the boundary for the first stage of his life, and obtained a total of about 3,600 pieces of personified conversation data.

General Instruction Tuning. In order to retain the model’s generalization ability to the greatest extent, we randomly sampled 100,000 multi-turn English instruction tuning data from the ultrachat dataset (Ding et al., 2023). For the case study, since there exists a Chinese personality, we add 20,000 items of Chinese instruction tuning data from Belle-0.5M (Yunjie et al., 2023) to retain the Chinese capability of the model.

The personified training is structured in two stages: initially, we engage in personified conversational tuning using a blend of personified and general instruction tuning data. Subsequently, automatic DPO training is applied.

We use strategies including temporal label, CoT and anti-induced conversation during data construction. To be specific, we add new tokens (e.g., “<TIME-I>”, “<TIME-II>”) for the model embedding and tokenizer, and insert the new tokens into the prompt to distinguish different personified stages. Meanwhile, we require the LLMs to first analyze and then respond as the target personality when annotating the conversation data, and use specific delimiters to differentiate between two parts (e.g., “[Analysis] The user is asking about … Therefore, I should correct this error. [Response] I’m afraid you’re mistaken…”). As for the anti-induced conversation, we require the LLMs to propose error facts or knowledge by prompt engineering. The detailed prompts are shown in Appendix.

We also conduct automatic DPO training to further align the model to the target personality. To construct the direct preference data automatically, we put forward a hypothesis: responses that correspond to different inputs but are similar to the current response are very likely to be interference error items. For example, the same question concatenated with different temporal labels gets different inputs, and the corresponding responses may contain very similar attitudes but different tones. Interfering items may also include different personalities’ attitudes towards the same event, views from the same personality on related events, etc. Therefore, we use the gtr-t5-base model (Ni et al., 2022) to encode the responses of all personalities in the HP dataset. For each item of training data, we use the original response as a positive item, and remove the 10 responses with the highest embedding similarities (to avoid the existence of responses with exactly the same meaning). Then we select the most similar response as a negative item to widen the encoding gap of the model during the DPO training process.

To introduce the personified knowledge more directly, we also mix up the existing data with the language modeling / continue writing task of those personified materials. Meanwhile, we randomly allow 1% of the personified conversational to provide the retrieved materials in the input.

Model. MiniCPM-2.4B is a Transformer-based model. We adopt the LLaMa (Touvron et al., 2023) version (which is a widely-used open-source LLM) checkpoint, “MiniCPM-2B-sft-bf16-llama-format”, which contains $40$ stacked Transformer layers of decoder, $122,753$ of vocab size, $2,304$ dim of hidden states, altogether 2.4 B of parameters. Chinese-LLaMA-2-7B is also a Transformer-based model, and Chinese-Alpaca-2-7B is instruction-tuned based on the LLaMa model. They both contain $32$ stacked Transformer layers, $55,296$ of vocab size, $4,096$ din of hidden states, and altogether 7 B of parameters.

Hyper-parameters. For the tuning process, the authors of MiniCPM-2.4B provided some empirical values of hyper-parameters, including batch size per device $32$, learning rate $5e-5$, and max steps $3,000$. We conduct grid search near the given values, and set the batch size as $16$, learning rate as $5e-5$, warmup steps as $50$, weight decay as $0.1$, and max length as $3,000$. We repeat the personified data $5$ times and mix it with the general instruction tuning data for a total of $1$ epoch of training (equivalent to $5$ epochs of personified training). The total number of training steps for each model is approximately $3,500$. When tuning the larger model Chinese-LLaMA-2-7B, we adjust the learning rate to $2e-5$, and keep most of the hyper-parameters the same with MiniCPM-2.4B tuning.

The detailed construction of the HP dataset is shown in Table 2.

The detailed prompts are shown in Table 3 and Table 4.

The translation for the cooperation instance between John Nash agent and Huiyin Lin agent is shown in Table 5. The human-agent interactions are in Chinese, and here we provide the English translation for several actual cases in Table 6, 7, 8, 9.