Collaborative Quest Completion with
LLM-driven Non-Player Characters in Minecraft

We design a minigame within Minecraft. In this minigame, there are two NPCs: Elena and Alaric. Elena informs the player about her friend Alaric being stuck on an spider-infested island far above and asks the player to build a path towards the island and rescue Alaric. On meeting Alaric, he further tasks the player with finding a special sword that is in a chest inside the island and is being guarded by zombies. We define the completion of the quest as the completion of the following sequence of steps:

1. (a)

   talk to Elena who will tell you about Alaric stuck on an island

2. (b)

   collect materials to build a path to the island

3. (c)

   build the path to reach the island

4. (d)

   fight spiders on the island

5. (e)

   talk to Alaric who will ask you to find his special sword

6. (f)

   find the special sword

7. (g)

   give sword to Alaric

Both the NPCs are powered by GPT-4. They have conversational abilities so the player can interact with them. They also have some limited in-game capabilities such as moving around, locating resources, mining resources, and attacking and defending from mobs. The player can take help of these NPCs to complete the quest. The player is given no prior knowledge about the steps of the quest. They have to figure the steps by interacting with the NPCs.

In most games today, conversation with an NPC is entirely pre-scripted and players are given options to choose from to navigate through a pre-written dialogue tree. Volum et al. (2022) proposed a method to prompt large language model to generate NPC responses in real-time enabling a freeform conversation between players and NPCs. We build on this work where we additionally prompt GPT-4 with following specific traits that we hypothesize would make for a better collaborative experience for human players in their quest completion.¹¹1Full prompt for both NPCs is included in the appendix.

We create a custom world within Minecraft for our minigame. The world consists of a village with several houses and a floating island (high up in the air above the village) infested with spiders. Elena is standing outside one of the houses in the village, whereas Alaric is in the floating island being attacked by spiders. The village has several resources that the player can mine to build a path to the island. The player initially spawns near Elena.

The player can interact with the NPCs either using text or using their voice. The NPC’s response is also conveyed to the player both using text on the screen and using voice. We use Azure’s speech-to-text and text-to-speech services.²²2https://azure.microsoft.com/en-us/products/ai-services/ai-speech

We design a user study where we recruit 28 gamers with prior experience playing both Minecraft and role-playing games. Each user session lasted for an hour where 15 mins was used to convey instructions to the players and setup the game, 30 mins was used for the game play and 15 mins was used to fill out the post-study survey.

One of the main goals of this study was to understand how human players collaborate with NPCs to complete the quest. To this end, we analyzed the game logs and the video recordings of the player’s gameplay and found many instances of NPC helping the player and the player helping the NPCs.

Our prompt design includes a mechanism for a sub-goal to be generated every few turns. We find that the GPT-4 generated sub-goals helps facilitate collaboration in different ways. The most notable cases were the ones where the NPC used the generated sub-goals to guide the player back to the main task of finishing the quest when they deviated from it (example 1 in Table 3). In other cases, the generated sub-goals helped the NPC reinforce their point (example 2 in Table 3), helped them negotiate resource exchange, or even helped them avoid a possible dispute with the player, all of which improves the NPC’s collaborative skills.

One of the issues with enabling code generation using LLMs is that it can be prone to errors. In our case, this could mean generation of function calls that do not exist or generation of incorrect code syntax. We find that since we include a mechanism to return the function call status (success or failure) back to GPT-4, it is able to communicate the errors in a more contextual way back to the player using natural language. Players say in the post-study survey that this behavior helps them accept these in-game errors better and encourages them to keep interacting with the NPCs. The examples in Table 4 show how the NPC makes a wrong function call but still conveys the mistake in a contextual way, creating a more collaborative experience.

We seek to understand how the collaborative role of the NPCs affects the player’s success in accomplishing the various steps that constitute the quest. Figure 2 depicts the number of players that completed the different steps of the quest. All of the 28 players spoke with Elena and most of them were willing to assist Alaric, but some of them needed to hear more persuasive arguments or even a reward offer (see appendix for an example), demonstrating the benefit of having an NPC driven by an LLM that can produce such strong reasoning in real-time when necessary.

24 players went on to collect the materials required to potentially build a path to the island, but only 18 went on to actually build the path to the island. On inspecting the logs and recordings, we found that this step of collecting materials and building a path can be a long one, taking an average of 10 mins even for skilled players. Hence, understandably, most of the dropout cases were due to lack of enthusiasm or essential building/survival skills or simply lack of time. However, we noted some cases where the players did not get Elena to help them out as much as they had hoped and so they dropped out. This suggests that with a more skilled NPC, the success rate of such a tedious step could be higher.

Only 13 players found Alaric. Others took too long to reach Alaric, and the monsters of the game had already attacked Alaric by then. Alaric spoke to those 13 players and asked them to find his special sword hidden in a chest guarded by zombies of the island. Only 7 were able to successfully find the sword and return it to Alaric, indicating an overall success rate of 25%. Others had difficulty either finding the sword or attacking the zombies or simply ran out of time. Although the NPC, Alaric, had attacking and defending skills, players could not always get him to help them in their fight with the zombies. So, a number of players reported in their survey answers that having a more skilled NPC could have helped them in this step, providing further evidence for how a more advanced game agent could enhance the collaborative experience.

The NPCs in our minigame are driven by GPT-4 which is a language-only model i.e. it does not have any visual grounding, except for those that are provided using limited game APIs (e.g. location of resources, path to a location, etc). Through our analysis we found this led to several issues in the game experience. Table 5 highlights some of the key issues. In example 1, the monsters of the game had moved Alaric away from where he initially was but since he does not have visual capabilities nor real-time game state information to track his location, he kept insisting that he is next to the player when he clearly was not. Similarly, in example 2, Elena was being attacked by a spider in the village but did not have this awareness. In example 3, player wanted Alaric to guide them to his location. But since Alaric did not have this game-state information, it hallucinated the directions.

In future, we think a number of enhancements could help improve the collaborative experience. Firstly, using a generative model with visual capabilities could help elevate issues around lack of visual context. Further, adding a capability to query game-state information in real-time (e.g. being aware of the NPC’s location in relation to the player and other NPCs, being aware of the damage caused by the mobs in the game, etc) would help the NPCs to be more contextually aware.

Players filled out a questionnaire³³3Full list of questions included in the appendix. post their gameplay and were asked about their overall experience, including things that they liked and disliked. 44% rated the overall experience above average, 53% found it better than their usual Minecraft gameplay experience and 65% said they would play this minigame again. These numbers suggest that although the preliminary study was useful in identifying strengths and weaknesses, there needs to be a lot of improvement before we can begin to use such AI-agents in 3D games. 88% of the players, however, reported that they had fun, suggesting value in continuing this line of research.

When asked what the players liked the most, a major theme was around AI in-game assistants being very helpful to new players. Others noted how talking to an AI character made the experience more enjoyable and immersive. Below are some relevant quotes from the players:

“The ability to inquire in-game entities in open-ended terms helps someone like me - a newbie - to feel more confident trying out things."
“The freedom while still adding structure, goals, and narrative to the game."
“I liked the moment where I entered the castle, the AI called out to me asking for help, and then we talked a bit as we worked together to take out the mass of enemies that had built up. It felt more immersive that way than the usual Minecraft adventure map."
“The dialog was far better than I expected and I liked that I could command the AI to do a couple things."

When asked what the players disliked the most, the main feedback was on the NPCs lacking essential 3D visual context and how this made it difficult for the players to get the NPCs to do certain tasks. Some also noted the time-lag in the NPC responses being an issue. Below are some relevant quotes from players:
“Sometimes, NPCs would struggle to adapt to environmental changes. For example, if Alaric got stuck in a hole completely outside of the island, but even after talking with him, struggled to understand that he was not on the island with spiders. "
“The NPCs did not have good contextual knowledge of the world, which resulted in them being adamantly incorrect at times (wrong about contents of chest, wrong about location of other NPC)."