The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agents

The earliest usage of educational computer software includes military applications such as flight simulations [14]. Increasing widespread use of computers opened the way for many interactive multimedia applications focusing on education [15]. One special form of computer-assisted learning includes life-like pedagogical agents that help with learning and motivation in multimedia environments [16]. Such agents can assume the role of instructors, coaches, tutors, or learning companions [17] and converse with the user using natural language with text or speech input [18]. Pedagogical agents can simulate instructional roles such as expert, motivator, and mentor with high accuracy [19]. Human-like agents have been shown to influence learner achievement, attitude, and retention of learning [20] and deliver a more interesting overall learning experience than learning without an agent [21].

Research has explored the use of 2D and 3D agents in educational settings, revealing varying effects on learning outcomes. 2D agents can outperform their more visually complex 3D counterparts in certain scenarios [22], as they reduce extraneous cognitive load and enhance coherence in multimedia learning [23]. High levels of anthropomorphism in agents can detract from social co-presence [24] and consequently limit their pedagogical benefits [25]. 3D agents, in contrast, offer a greater variety of expressions and gestures, which can boost learning and engagement [26]. Although high behavioral realism enhances social presence in virtual reality environments, where 3D agents are particularly effective, it can negatively impact factual learning outcomes [10]. However, the presence of a real instructor is associated with increased learning and satisfaction, leading to better information recall [27]. The mixed effects of high behavioral realism may be explained by the uncanny valley effect [28], where realistic yet not quite human-like agents induce discomfort and unease. In robotics, incorporating personality into humanoid robots has been shown to reduce these uncanny feelings and enhance the overall user experience [29], which can also apply to virtual humans.

Pedagogical agents can be embedded into different applications such as text-based mobile conversational systems [30], collaborative serious games [31], multiple-agent intelligent tutoring systems [32], collaborative augmented reality environments [33], and artificial intelligence-enabled remote learning[34]. In robotic educational agents, verbal cues are more effective than nonverbal cues in improving engagement [35]. Expressing various emotions through facial expressions and body motion can appeal to different learner types in animated pedagogical agents [36]. Pedagogical agents have been utilized in teaching a wide range of subjects, including STEM [37], foreign language [38], history [39], as well as work training [40]. Learning, engagement, human likeness, credibility, and personality can be used as measures of the psychometric structure of pedagogical agents [41].

Multi-modal communication elements are essential for expressing desired personality traits in digital characters. Nonverbal behavior elements are commonly used in affective virtual agents to convey these traits [42, 43, 44, 45]. Studies leverage high-level meanings of motion to express the target personality type [46]. For instance, PERFORM establishes a link between Laban Movement Analysis (LMA) parameters and the perceived personality of virtual human characters [47]. In addition to body motion, personality-specific voice, dialogue, and facial expressions help distinguish personality traits in expressive conversational agents [11]. Recent research indicates that appearance and movement significantly influence the expression of certain traits such as agreeableness and neuroticism [48].

The character’s rendering style also affects personality perception [49]. For example, cartoon-like rendered characters are perceived as more agreeable, whereas characters with unappealing, ill-looking rendering styles are found quarrelsome and less sympathetic [50]. Additionally, animation realism [51], face model [52], body shape [53, 54], clothing, environment, and facial expression [55], as well as skin texture and viewing angle [56] of virtual characters are all influential on perceived personality. In multi-agent scenarios, the interaction between agents and proximity is a successful indicator of personality [57, 58, 59], as well as emotional group dynamics [59]. Action choices in procedural story generation can also express different personality types [60, 61]. An agent’s perceived personality highly influences how users interact with the system; for example, users are more willing to trust and listen to serious-looking, assertive agents [62].

Human hands are highly expressive in communication [63]; specific hand movements can convey different personalities [64]. Gesture performance in combination with language highly influences perceived personality [65]. Linguistic elements such as the ratio of phrases, words of emotion and cognition, and exclamations correlate with apparent personality traits [66, 67]. Automatic personality assessment is possible using text input from social media messages [68, 69], using speech [70, 71], facial expressions [72], gait features [73] and body motion features [74]. Overall, different communication elements contribute to the perceived personality of digital characters where multi-modal approaches often yield better results [75, 76, 77].

LLMs such as GPT exhibit consistent personality cues [78] and can be customized for different personalities through prompting [79], as we do in this work. Analysis suggests that word choices and the length of the generated text are influential on this [80]. Data-driven personality estimation systems can predict different personality types when the generated text uses certain prompts [81, 82], supporting the success of LLMs in capturing personality cues from language.

LLMs span many applications, including natural language-based human-computer interaction [78]. The rise of LLMs has opened up new avenues for creating and populating digital worlds. For instance, language models can be used to generate and animate 3D avatars [83], schedule motions [84], control facial expressions and body motion styles [85], drive non-player character behavior in games [86, 85], and automate and refine digital storytelling [87]. 3D scenes can be injected into LLMs for captioning, 3D question answering, and navigation tasks [88]; models can describe or compare objects in 3D scenes [89].

LLMs have also started playing critical roles in innovative educational technologies [90]. For instance, LLMs are used with vocalized agents in Augmented Reality (AR) environments to aid students in foreign language learning [91]. Similarly, in healthcare, LLMs enhance patient experiences during consultation, diagnosis, and management [92]. Although dialog systems utilizing LLMs can generate highly sophisticated responses, they lack access to dynamic real-time data, such as the current date [93]. Consequently, LLM-based agent systems often focus on isolated tasks, such as answering questions based on pre-existing knowledge. To enable more context-aware conversations, additional mechanisms to enhance LLM memory can be implemented [94]. In this work, we use an LLM to generate educational responses for specific computing subjects in an educational setting.

Conversational agents aim to interpret and respond to user statements in ordinary natural language through integrating computational linguistics techniques [95]. Understanding and exhibiting emotions and personality are essential for successful natural language conversations [96]. For example, the same query may require different interpretations based on the user’s current mood, and similarly, the same response can be perceived differently based on the agent’s current body language and facial expression. Conversational systems that give relevant answers to user questions are perceived as more human-like and engaging [97]. The realism of animation and behavior is critical for agents with a visual representation [98]. Thus, studies synthesize gesture animation to accompany speech [99, 100]. Recent deep learning-based co-speech animation generation systems can produce highly realistic results [101, 102, 103, 104], and the generated animations can be authored to express the desired pose and style at specific frames [105]. Facial expressions are also crucial to create realistic experiences. For instance, agents that mimic user facial expressions deliver believable and empathetic conversations [106]. Conversational agents can take as input multiple stimuli, including user’s gaze [107], speech and facial expression [106], structured or natural language text [108]. In this study, we input natural language text from participants to keep the system requirements minimal and leave other modalities for future work.

To run our study, we updated the personality-driven conversational agent platform by Sonlu et al. [11], an open-source, multi-modal system for animating 3D conversational virtual agents. The platform runs on Unity [109] and controls multiple modalities, such as dialogue, facial expressions, and body movements, based on an input personality. Body movement control involves modifying a given animation clip via joint rotation and animation speed adjustments, noise addition, and inverse kinematics-based gesture changes following the LMA mappings defined in PERFORM [47]. Facial animation involves mouth movements during speech, frequent blinks associated with neuroticism, and updating blend shapes to express emotions associated with specific personality factors. Facial expressions are designed according to Facial Animation Control System (FACS) [110], and mouth movements are handled by Oculus LipSync [111] with a customized mapping to the facial blend shapes of the 3D model. The input personality determines the agent’s default facial expression. For example, an agreeable agent tends to smile by default with each turn of its dialogue, which diminishes with time. We designed 3D human models for the current study using Reallusion Character Creator [112]. To introduce a measure of diversity, we created four characters: two female and two male, each with light and dark skin tones, as depicted in Figure 1.

Our updated system differs from the existing personality-driven platform in terms of how it handles dialogue. The previous work used IBM Watson Assistant [113] to extract the intent from user queries mapped into domain-specific handcrafted dialogue lines. This work replaces the dialogue logic with an LLM-based text generation model, GPT-3.5 Turbo [114], facilitated by OpenAI’s Chat Completions Application Programming Interface (API), eliminating the need for manual dialogue crafting. When users type their prompts (e.g., ask a question), the system returns an answer coherent with the input personality description. We limit the token number to 750 for the output text to keep the conversation concise. Unlike the previous platform, which used Watson Text-to-Speech API for speech generation, our current system employs Microsoft® text-to-speech functionality, producing an almost immediate response to vocalize the agent’s answer. This local solution also lets us determine the currently spoken word we use to display partial subtitles. Since the generated responses could be fairly long, we followed a dynamic approach where five words centering the currently spoken word were displayed on top of the agent in models with visual representation. The subtitles were on by default, but the users could disable them in models with 3D agents if they found them distracting.

We used a temperature of 0.9 to promote diverse outputs from GPT while maintaining the information’s reliability. Temperatures greater than 1 introduce creativity; however, they lead to hallucinations, conflicting with the aim of the educational system. Chat Completions API takes as input a “messages” parameter consisting of message objects, where each object has a role of “system”, “user”, or “assistant” and content. For the role of “system”, we give the following messages as input for different agent personalities and teaching topics:

• •

  {“role”: “system”, “content”: Act as an extroverted teacher teaching about $<$topic$>$, give friendly and polite answers.}

• •

  {“role”: “system”, “content”: Act as an introverted teacher teaching about $<$topic$>$, give short and unfriendly answers.}

To produce a response that the agent speaks, the system sends the role prompt together with the last five dialogue messages, alternating between the user and assistant: {“role”: “user / assistant”, “content”: $<$ message $>$} We limited the number of messages to five to reduce costs, eliminate context drift, and prevent manipulation. This restriction helps prevent altering the language model’s perception through extended interactions [115].

We designed a $3\times 2$ independent subjects study for a comparative analysis of three models—D, A, and E—each tested with high and low values of agreeableness-extroversion combination. Model D is the dialogue-only setting, where the system’s answers were shown on screen sentence-by-sentence concurrently with audio playback. Model A included an animated 3D model of the agent, randomly chosen among four alternatives (two male, two female, each with dark and light skin tones). Model A involved the virtual human animated without any personality-based alterations. In models D and A, personality was conveyed only through synthesized dialogue. Model E was similar to Model A but additionally incorporated the expression of personality through face and body movements.

In model E, motions that display a combination of high extroversion and agreeableness involve the LMA parameters of Indirect Space, Light Weight, and Free Flow [47]. These correspond to multi-focal spatial attention, delicate, lifted-up movements, and uncontrolled and fluid motion. Because high extroversion and agreeableness are associated with opposite Time Efforts (Sudden, urgent vs. Sustained, lingering), we left the Time component of the animations unaltered. The animations expressing low extroversion and agreeableness involve Direct Space, Strong Weight, Bound Flow, and neutral Time, corresponding to single-focused, heavy, and controlled movements. The default facial expression of a highly extroverted and agreeable agent is relaxed and happy, with occasional smiles and direct eye contact. In contrast, an agent characterized by lower levels of these traits displays a more tense expression by default. Low-trait agents also avoid eye contact with the viewer.

Screenshots of different models are displayed in Figure 2. We name each variation of the system with its model name and whether they express high or low traits. For example, E-High refers to the variation where we express high trait personality using the model that uses both text and animation-based cues. Note that we display a single image for models A and D as they are visually similar in high and low variations. For the animation in Figure 2(b), the E-Low variation has the hands close to the body with a slightly more slanted posture (Figure 2(c)), and the E-High variation has the hands further from the body with more upright posture (Figure 2(d)).

The study involved an educational application with a conversational agent that teaches users complex subjects. We presented participants with six options and asked them to select the least familiar topic. The topics were quantum computing, blockchain technologies, transformer architectures, quantum mechanics, string theory, and general relativity. The selected topic was provided to the GPT model as part of the system role prompt to guide a focused conversation. The application required that participants pose the agent at least five questions to learn about the topic. There was no upper limit on the number of questions a participant could ask. Upon completing their queries, participants could proceed to answer survey questions. They could review the survey questions or the chat history at any point.

The survey questions appeared in two groups. The first group included 27 questions on a 5-point Likert scale, where 15 questions measured the perceived personality of the agent using the extra-short form of the Big Five Inventory–2 (BFI-2-XS) [116], and 12 questions measured self-assessment of learning, quality, and engagement using the Learning Object Evaluation Scale for Students (LOES-S) [117]. In LOES-S, learning-related questions are about the self-assessment of learning and how much the learning object, i.e., the tool in question, helped teach the subjects a new concept. Quality assesses the instructional design, ease of use, organization, and help features. Engagement evaluates how much the subjects liked the tool and whether they found it motivating.

The second group of questions required open-ended input to receive detailed participant feedback. There was no character limit for the answers to the open-ended questions. Completing both groups of questions directed the participants to the user study completion page, where they received a link for task approval.

We used the crowd-sourcing service Prolific to recruit participants. Before running the study, each participant was directed to a website to test whether they had installed the correct text-to-speech package. Only those with the supported system configurations could continue with the study. Two hundred ten unique participants (99 female, 95 male, 16 not specified) rated our system, with each alternative evaluated by 35 individuals, which provides a medium effect size (Cohen’s $f=0.26$) for both main effects and their interaction and power of $0.80$ at a significance level of $0.05$ for independent-subjects Analysis of Variance (ANOVA).

Each participant interacted with only one version of the system, where the average interaction time was $19.74\pm 9.25$ minutes. This time excludes the introduction, during which participants read about the task and downloaded the application, but includes the time spent answering the survey questions. The average participant age was $28.80\pm 8.57$. Upon entering their Prolific IDs, participants were shown an introduction message with information about the study details. We informed the participants about the data collected and the study’s aim to measure the system’s performance; we emphasized that the study did not aim to measure their knowledge in any manner.

BFI-2-XS includes three questions for each personality factor, half of which are inversely proportional to the measured dimensions. Responses were assigned integer values on a 5-point Likert scale, ranging from -2 to 2. We calculated the signed sum of these values to derive personality scores between -6 and 6, which were then re-scaled back to the range $[-2,2]$. Similarly, LOES-S has five questions measuring learning, four questions measuring quality, and three questions measuring the engagement of the learning object. We calculated the sum per measurement type and mapped the corresponding ranges into $[-2,2]$ to report the corresponding means.

For exploratory analysis, we display box plot diagrams of each model regarding perceived personality and LOES-S scores for learning, quality, and engagement (see Figure 3). The diagrams indicate positive mean scores for openness, conscientiousness, extroversion, and agreeableness and negative mean scores for neuroticism across all models. The models received particularly high ratings for conscientiousness. The plots also show high positive ratings for learning, quality, and engagement, with mean engagement scores slightly higher for high personality variations than for low personality ones. We can also observe that the model E-High, followed by A-High, represents high conscientiousness, high agreeableness, and low neuroticism better than the other models. In the next section, we perform descriptive analysis to identify potential statistically significant effects of the models and personality styles on the output variables.

To investigate the impact of model type (D, A, and E) and personality style (high or low) on learning outcomes and perceived personality factors, we ran seven two-way analysis of variance (ANOVA) models and one non-parametric alternative model (Welch’s ANOVA). Welch’s ANOVA was utilized to assess the quality scores of LOES-S across model type and personality style, given that the assumption of equal variances was violated, as indicated by a Bartlett test.

Apart from the non-parametric model, which combined model type and personality style as a single factor, all other models examined the influence of model type and personality style on outcome mean individually and any potential interaction between these factors. With balanced and sufficiently large sample sizes ($n{=}35$) across factor combinations and evidence for equal variances across factor levels (as measured by Bartlett’s test), all outcomes except for quality were appropriate for ANOVA modeling. To control the familywise error rate at $0.05$, we employed the Hommel method to adjust for multiple testing across all model terms, including one post-hoc analysis. Unlike the conservative Bonferroni correction, the Hommel method offers increased statistical power. Table I presents significant terms for all ANOVA runs before and after the correction for multiple testing. Adjusted-for ANOVA tests returned significant main effects of personality style on the outcomes of engagement ($F=9.502,p=0.0421$), openness ($F=4.474,p=0.00178$), extroversion ($F=10.148,p=0.03006$), agreeableness ($F=25.541,p=0.0000211$), and neuroticism ($F=9.708,p=0.0378$). Although conscientiousness initially carried a significant finding for personality style ($F=5.68,p=0.0181$), this term’s statistical significance dropped after multiple testing corrections. The main effect of the model type was initially significant for neuroticism, but the effect did not remain significant after the Hommel procedure ($p=0.417$). The box plots of the statistically significant effects are depicted in Figure 4.

The effects of agent gender and skin color were not among the hypotheses, so we randomly selected one 3D agent model among four different appearances, which also determined the agent’s voice, to support variety. We do not observe a significant effect due to the agent’s gender or skin color, which confirms previous work [22].

We report the Pearson Correlation between LOES-S and personality factors in Table II. In general, correlation coefficients higher than $0.4$ are considered moderate. Perceived openness, conscientiousness, extroversion, and agreeableness all positively correlate to the LOES-S scores, albeit some weakly. The highest correlations are for conscientiousness, particularly for Model D. Quality and engagement scores are strongly correlated ($>0.6$), and learning is moderately correlated, nearing the threshold for a strong correlation. For Model D, openness and agreeableness also have moderate correlations with all the learning parameters. In general, engagement is moderately correlated with all factors except neuroticism. The expressed neuroticism is weakly inversely proportional to each parameter, but its only statistically significant correlations are for learning in Model D and quality in Model A. The correlations are the strongest for Model D and weakest for Model E.

We posed the following open-ended questions to the participants for in-depth feedback:

1. 1.

   What is $<$topic$>$ based on your conversation with the system?

2. 2.

   Why is $<$topic$>$important, please discuss briefly?

3. 3.

   Did you learn anything new about $<$topic$>$ after your conversation with the system? If so, what did you learn?

4. 4.

   Do you think the system’s behavior influenced your learning experience?

5. 5.

   Please briefly describe your conversation with the system. Was it interesting? Did you encounter any problems?

Two independent researchers tagged the answers with at least 95% agreement for each theme. The themes were determined based on the following criteria:

• 1

  Benefit/No Benefit to Learning Experience: Regarding the third and fourth questions, this theme reflects the effect of the system behavior on the participant’s learning experience.

• 2

  Learning / No Learning: Regarding the third question, this theme captures whether the participant learned anything new interacting with the system.

• 3

  Interesting / Not Interesting: Regarding the fifth question, this theme considers whether the participant finds the system interesting or not.

• 4

  Realistic / Robotic / Poor Voice: This theme captures the overall human likeness of the system based on the answers to the fourth and fifth questions. In certain cases, the participants only commented about the system’s voice, which utilized less developed local solutions to support immediate response; otherwise, the participants’ feedback is on the overall image of the agent.

• 5

  Detailed / Moderate / Short Answer: Regarding the answers to the first and second questions, this theme captures how much detail the response includes. Although this may strongly depend on the participants’ characteristics, the system’s behavior may have also caused them to adjust the level of detail of their answers. These answers may also depend on the participant’s learning; the less they know, the shorter the answers will be. We tag the responses as detailed or short; any remaining response is considered moderate.

Table III shows the number of themes for each variation. The following subsections focus on each theme with quotes from the participants. At the end of each quote, we indicate the participant’s ID as a reference and the variation that the participant interacted with. We only fixed the typos in some of the responses to improve readability.

A primary objective of agent-based educational systems is to facilitate self-directed learning by providing a useful learning experience. Actively involving individuals in learning helps them develop a deeper understanding of the subject matter than passive learning through only reading or watching the material [118]. This level of interaction is naturally present with a human teacher who can promptly address a learner’s questions. We required participants to actively engage by asking questions to replicate this dynamic and prevent passive learning.

Most participants indicated that the system behavior improved their learning experience; they emphasized increased engagement due to having a human-like agent: “It was more engaging to have a human avatar instead of a blank screen or other representation.” – P3 (A-High). Even for the dialogue-only variations, the experience was more enjoyable due to interaction: “I think I enjoyed learning using the system more than I would have if I were reading on my own in a book or Google.” – P21 (D-High). Being able to ask questions using natural language and receive to-the-point answers was also found to be beneficial by certain participants: “It saved me some time from having to Google specific terms and read long texts on them, giving me the key points to get a basic understanding.” – P17 (E-High). A human-like interaction with a non-human system can help people with anxiety to experience interactive learning: “It can be useful a lot, and I’d like to use it because it is quite calming down the person who has anxiety.” – P27 (E-High).

However, a human teacher would typically simplify the material to facilitate easier learning, which can be challenging for a language model to emulate with limited user feedback: “I believe the system was very informational, which influenced my learning. However, the information was too much to handle at once, which clouded my mind.” – P30 (A-High). Participants who reported no benefit from the system often cited the difficulty of the answers: “I could have learned more on Wikipedia or Google. The system’s answers were too difficult to understand for me.” – P169 (D-Low). Although the high-trait models were generally found to be more beneficial, some participants noted that their responses were too lengthy: “The responses were quite long, sometimes too long. Some of the things it said could’ve been left out as they didn’t provide any useful information; it was just ’flavor text.’ I also read very fast, so waiting for it to stop talking was a bit boring. Other than that, it was a very positive experience. ” – P42 (D-High).

In addition to length, a major difference between the high and low variations was the LLM’s word choices. The high-trait variations use motivational language, supporting the learner instead of just giving the answer: “Yes, the system had a motivational tone that lifted my energy towards learning about something I had zero knowledge about.” – P37 (D-High). Such language can help create a more interactive and thus motivational experience: “Yes, I liked the easy-to-learn explanations and also the motivational part ’that is indeed a fantastic question’.” – P106 (D-High). Participants also referred to the system as friendly (P50 and P150, A-High) and polite (P209, D-High). On the other hand, the low-trait variations responded with shorter sentences that some users preferred: “It was not interesting per se; however, it was very informative and straight to the point.” – P2 (E-Low). Especially, the introverted language [66] of the low variations could influence the relatively short and simple responses: “Conversation was simple and quick. The agent gave me short and simple answers to my question that are easy to understand” – P170 (E-Low).

Participants found variations with embodied agents slightly more beneficial and interesting. Especially, they reported a positive influence of having an embodied human agent for the high-trait variations: “Somewhat, seeing a human-like face made it easier to memorize the information” – P163 (A-Low). “Interacting with a humanoid entity is more engaging than reading a book.” – P143 (A-High). “Yes! I really enjoyed it; it seemed very human. It was like talking to an expert; it can answer any question you have instantly.” – P12 (A-High). “It was really exciting to see a person/character in front of the screen. Of course, it has an influence and really affected my learning process positively.” – P134 (E-High). “I think the animations and the looks of the character were motivating, and this could help with the learning in general.” – P207 (E-High). “Body movement and text to speech allowed to be more engaged in the conversation.” – P184 (E-High). A human-like agent can help learners better focus on the conversation, positively affecting learning: “Having someone explaining a subject to you in human form generates a curiosity that is similar to listening to an enthusiastic teacher. As someone with a low attention span, the agent kept me engaged in the conversation and sparked further interest.” – P1 (E-High). “It was really exciting to see a person/character in front of the screen. Of course, it has an influence and really affected my learning process positively.” – P134 (E-High). While most participants focused on the body movements, a few reflected on the facial expressions and their positive effect in the E-High variant: “Slight facial “expressions” was noted and kind of felt like it made an impact, to be fair.” – P11 (E-High).

For the low-trait agents, participants noted that agent movements were monotonic and the visual representation brought no advantage: “The person itself is extremely dull, there is no life if that makes sense, and the movements and gestures are extremely weird, the hand and arm movements are strange, I would rather have that taken away, but being able to ask any questions to a topic and a response provided immediately is amazing, really like that aspect.” – P196 (E-Low). “Maybe, I think if the system were ”nicer” and less monotonic, the learning would be easier.” – P147 (E-Low). “I did not find the “graphics” to help. A chatbot would have basically had the same effect on me.” – P191 (A-Low).

The results suggest that high-trait personality variations lead to improved learning outcomes. However, the influence of expressivity appears limited, as Model E shows only a slight improvement in learning performance compared to Model A. Only two responses in each A-High and E-High were associated with no learning, pointing out that human-like appearance combined with high traits can improve learning.

The system sometimes inspired participants to learn more about the topic: “I did not know anything about this theory at all. After my conversation, I can proudly say that I am really into string theory. I learned the basic concept and the creators of the theory. I also asked how I could learn more, and the conversational agent suggested four different possible sources.” – P8 (A-High). “It really almost felt like talking to someone who knows quantum computing well. I especially appreciated the way it understood my questions, even though I felt a question or two were a bit vague. The system actually made me want to know more about the subject so I can ask better questions.” – P11 (E-High). Since we asked participants to choose the subject they had the least information about, most of them reported having almost no prior knowledge of the conversation topic: “It was a completely new topic to me, and I feel generally satisfied with the answers of your system” – P15 (E-Low). Therefore, the themes associated with learning correspond to the participants who are newly introduced to the subject and acquired new information. A few participants indicated they already had sufficient knowledge of the subject: “I didn’t learn anything new; I already knew the basic information about Blockchain.” – P29 (E-Low). Still, such participants found the system useful and reported a desire to use it again: “I am an enthusiast of general relativity. I honestly didn’t learn much, but I would love to use the AI to learn more in the future!” – P101 (E-Low).

Participants categorized under the “no learning” theme generally indicated the difficulty of the subject: “Previously, I had no idea what transformer architecture was. But unfortunately, I still believe that I did not learn a great deal about this type of technology. In my opinion, these new technologies that use AI are very difficult to understand if a person has no background knowledge.” – P20 (E-High). One detriment to learning could be the agent’s short answers in the low-trait variations: “Not much (learning) as the replies were brief, but I got a basic idea” – P101 (E-Low). Conversely, the lengthy answers of the high-trait variations may have been distracting: “The topics answered were on point, maybe a bit too long, and different questions had similar answers in common as part of it.” – P41 (E-High).

Promoting interest in a subject is essential for success in education [119]. For conversational agents, behavioral and graphical realism is an important factor for engagement [120], which also applies to pedagogical agents [121].

Participants found the D-Low variation to be the least interesting, followed by A-Low. All the high-trait variations and E-low were perceived as similarly interesting. The expressive gesturing in the E-Low variation may have mitigated the decrease in interest. A few participants found the agent realistic in all variations, with the E-High variation receiving the most responses associated with the “realistic” theme.

Participants who found the system interesting usually reported a positive influence on learning in the E-High variation: “I found it fascinating, really interesting, and quickly increased my knowledge on the subject. I would have loved to do more and carry on asking questions to discover more about blockchain.” – P199 (E-High). “It was really exciting to see a person/character in front of the screen. Of course, it has an influence and really affected my learning process positively.” – P134 (E-High). “I think that the system behaved as naturally as possible, so it was a good influence.” – P174 (E-High). Participants mentioned a positive effect on concentration due to the existence of the human avatar: “…Avatar of the agent helps concentrate on the conversation.” – P170 ( E-Low). The experience’s novelty could have resulted in some participants finding the study interesting: “It was quite interesting. I did not know what to expect when entering the task, but I was pleasantly surprised and engaged in the entire experience. It would definitely be something I would use again if I could.” – P175 (A-High). Participants found Model D less interesting but still beneficial in teaching: “I think it wasn’t interesting, but it taught me a topic I didn’t know about.” – P2 (D-Low). Since our focus was on a short conversation with the agent, the users could have endured the negative aspects of Models D and A more easily. The positive outcomes of expressive gestures may become more prominent in the long term. For example, keeping the agent’s behavior more interesting can be important to keep the user’s attention longer.

The system’s voice was highly criticized due to using a local solution for immediate responses at the cost of realism: “I didn’t like the voice because it seemed robotic and emotionless.” – P10 (D-Low). Some participants found the speech inhuman and slow: “I prefer the more traditional ’read what the bot says’ instead of it talking with its inhuman voice that speaks way slower and more monotonously than I would prefer.” – P41 (E-High). There was an increased focus on the negative aspects of speech synthesis when there was no visual representation, with 14 participants mentioning the poor voice of the agent in Model D. In contrast, negative comments on the agent’s voice were as low as 5 in Model E. Since Model D does not include an embodied agent, only a few participants regarded the agent as a robotic individual, with most of the negative comments focusing on the voice. Among the variations with a visual representation, E-High had the fewest responses associated with the “robotic” theme, which can be interpreted as the high-trait expressive motion effectively enhancing the agent’s realism.

Participants who found Model D human-like mostly reflected on the naturalness of the generated responses: “It was interesting. I liked the way it started to answer. It always added a kind of human comment and then gave you the answer.” – P59 (D-High). Participants assigned to the embodied representations tended to describe the agent as a real human:“It felt like a conversation with a real human, not like a robot. I felt like he was my teacher; I could ask any question.” – P6 (A-Low). “I really enjoyed it, it seemed very human.” – P12 (A-High). “I think it gave a more human experience and helped me focus on the subject.” – P179 (E-Low). The lack of expressive facial expressions in Model A was noticed by participants, which could explain why some found it robotic: “Some facial expressions might make the bot more relatable.” – P163 (A-Low). One participant found the overall appearance of the agent uncanny: “At first, I was surprised that this ’agent’ has a human-like avatar. I don’t think it helped me through the learning process because it felt kinda uncanny. Also, the TTS quality was poor, so it would be hard to understand some words if not for the subtitles. But I think that agent’s responses were very good and comprehensive.” – P57 (E-High).

The depth of participants’ answers to the first and second questions can indirectly measure their learning and attention. Therefore, we grouped the responses into three categories based on how much detail they captured. Short answers usually only summarize the conversation topic with a single sentence: “1. Quantum physics is a fundamental branch of physics that deals with the behavior of very small particles. 2. It provides information about the little particles, which are everywhere.” – P56 (E-Low). Moderate-length answers use examples from the conversation and include more detail on the subject: “1. Blockchain is a decentralized technology that records transactions. It consists of blocks of data linked together in a chain. 2. It is important because of its transparency that allows one to view the transaction history, it is also highly secure.” – P103 (A-High). Detailed answers contain more than one sentence for each question and include different aspects of the subject: “1. Blockchain is a decentralized system to store information about transactions. A single block contains data about the transaction and the hash of the previous block, so it can’t be removed without breaking the chain. Data are stored as copies on many computers called nodes. 2. Blockchain is important because it is decentralized, and there is no authority keeping watch on it, so it is hard to manipulate it. It is also very secure, as every transaction must be validated by miners before it becomes a part of the chain.” – P57 (E-High).

More participants provided detailed answers in the high-trait variations for Models D and A, possibly because the agent’s longer responses encouraged them to elaborate. However, more participants gave detailed responses in the low-trait variation for Model E compared to its high-trait variation. Notably, the E-High variation had the highest number of moderate-length responses, twice as many as the other models. Although the amount of detail in participant responses may depend on various factors, including their personality and attitude toward the experiment, we note a general trend toward lengthier responses for high-trait agents.