Memory Sharing for Large Language Model based Agents

Equipping agents with memory mechanisms to enhance their abilities has attracted the attention of researchers. Memory can play an important role in helping agents remember conversation information, maintain behavioral consistency, and accumulate experience. Generative agents enhanced with memory features can store vast experience records, facilitating deeper self understanding (Park et al., 2023), while VOYAGER has developed a skill library that evolves by incorporating successful action programs, optimizing task resolution (Wang et al., 2023). In the case of Ghost in the Minecraft, a text-based memory system supports LLMs in maintaining reference plans for efficient plan formulation when similar objectives arise (Zhu et al., 2023). Later, based on the concept of “memorization-retrieval-response”, Memochat was proposed for maintaining consistent long-range open-domain conversation (Lu et al., 2023). With the emergence of MemGPT (Packer et al., 2023), a new memory hierarchy was developed to process long texts and maintain the long-term memory. And the TiM make LLMs to maintain an evolved memory for storing historical thoughts along the conversation stream to a reality (Liu et al., 2023a). Also, through maintain agents’ own reflective text in an episodic memory buffer and implementing the exemplar memory, the Reflexion (Shinn et al., 2023) and SYNAPSE (Zheng et al., 2023) successfully induce better decision-making and generalize successful trajectories to new task respectively. Our MS framework is primarily designed to enable multiple agents to share memories, facilitating collective self-enhancement through inter-agent interactions. In previously mentioned agents equipped with memory mechanisms, the memory mainly ensures conversational consistency and stores past experiences to achieve individual enhancement. However, our MS framework achieves collective enhancement through shared memories, providing a pathway for the evolution from individual intelligence to collective intelligence.

ICL enhances the problem-solving capabilities of LLMs by incorporating few-shot examples into prompts (Brown et al., 2020; Levine et al., 2021; Zhou et al., 2022; Liu et al., 2023b; White et al., 2023). Research has demonstrated that ICL can foster creative learning in LLMs to a certain extent (Swanson et al., 2021). By redesigning inputs, LLMs become more adept at handling logical challenges (Wiegreffe et al., 2022; Wu et al., 2022). Crowdsourced instructions also contribute to improved performance in LLMs (Mishra et al., 2022). Additionally, elucidating the relationship between examples and tasks has been shown to be highly beneficial for LLMs (Lampinen et al., 2022), while the CoT (Wei et al., 2022) and PAL (Gao et al., 2023), enhance LLMs’ performance in complex reasoning tasks by introducing intermediate reasoning steps. However, when dealing with open-ended questions, agents still face two primary challenges: insufficient problem descriptions, which impair the agents’ comprehension, and the lack of external knowledge bases and available reference materials. Our MS framework addresses these challenges by converting high-quality content generated by various LLM-based agents into shared memories, providing agents with useful reference examples, thereby improving their performance in open-ended questions.

RAG (Lewis et al., 2020; Ram et al., 2023; Shi et al., 2023) is a method that enhances LLMs’ ability to generate accurate and timely content by integrating retrieval techniques such as BM25(Luo et al., 2023; Liu et al., 2022) or SBERT(Reimers and Gurevych, 2019). Using dense retrievers combined with contrastive learning for feedback can effectively enhance the performance (Rubin et al., 2022). Furthermore, iteratively training the retriever with contrastive learning can further improve its performance (Wang et al., 2024). However, the retrievers in the aforementioned studies are typically trained only once before deployment, making it challenging to adapt to newly generated data. In contrast, the retriever in our MS framework undergoes continuous training. Whenever new memories are added to the memory pool, they are used for further training of the retriever. This continuous updating and evolution process ensures that the quality of the retrieved memories gradually improves over time. And most importantly, with the help of MS framework, the dependence on external databases has been greatly reduced.

A memory is essentially a Prompt-Answer (PA) pair. In some special cases, it is permissible for a PA pair to lack a prompt, typically applicable in initial scenarios. These PA pairs are stored in natural language, which serves as the shared memories. On one hand, these shared memories can be used to improve the response quality of different agents; on the other hand, they are generated by various agents and stored in the shared memory pool. The dynamic expansion of the shared memory pool ensures a continuous influx of new memories, thereby enriching the knowledge base of all agents. In addressing open-ended questions, these shared memories provide agents with a broader perspective and deeper understanding, which is crucial for generating high quality answers. Fig.3 shows how the Agent-Puzzle use the memory to make a prompt and get the answer through one-shot learning.

After each interaction, the PA pair is scored. If the score of PA pair exceeds a preset threshold, the answer and its corresponding prompt are packaged as a useful memory and stored in the memory. When scoring the answers, we established different scoring criteria for various domains and topics, delegating the scoring tasks to the LLM itself. To facilitate the LLM’s understanding of these criteria, they were autonomously generated by the LLM, based on the reasonable assumption that the LLM can better grasp criteria it designed. Therefore, these scoring criteria are established prior to the deployment of the framework to ensure consistency in the LLM’s scoring process, thereby guaranteeing fair evaluation of memories from different agents. Before implementation, these criteria undergo a manual review phase, assessing the relevance of potential memories to the current focal task and their relevance within the domain to ensure their utility. Manual review, rather than delegation to the LLM, is employed to provide additional precision and special consideration to align with the agents’ specific needs, particularly in potential or special application scenarios. The manual review of these scoring criteria requires minimal workload.

Prior to the operational deployment of the MS, a small subset of instances was manually archived within the memory pool. These instances fulfill a dual purpose: firstly, they provide a diversified array of memories upon which each agent may experiment with novel prompts in the face of new queries; secondly, they constitute the preliminary training corpus for our retriever. This foundational training regimen mirrors the methodology by which subsequently archived memories will be assimilated into our model in real time, thereby facilitating the model’s ongoing adaptive learning and optimization. During the answering phase, an agent retrieves memories from the shared memory pool based on the question with the help of a dense retriever, which are more similar to the target question in terms of cosine similarity. These retrieved memories, combined with question, form a prompt that is submitted to the agent, which then generates an answer. The memories extracted from the shared memory are used as context to enhance the quality of the agent’s response, a typical ICL method that usually improves the answer quality.

Memory Train. Whenever a new memory, denoted as $(X,Y)$, is added into the memory pool, it will also be used to train our retriever, which help the retriever to continuously update itself and continuously adapt to new memory. Based on the new generated memory $(X,Y)$, the classical method BM25 ascertain the most pertinent top-$n$ candidate pairs $\{(x_{i},y_{i})\}_{i=1}^{n}$, sourced from the diverse and extensive memory pool, labeled as $C$. Each candidate within $C$ will undergo a evaluation process utilizing the comprehensive scoring capabilities of LLMs.The scoring mechanism employed is defined by the following equation:

This equation seeks to determine, given a input-output pair $(x_{i},y_{i})$ in $C$ as a condition, the probability that the response generated for the input in the new memory contradicts the output in the new memory. This grading part serves as a preparatory step for the subsequent labeling of each candidate example. It is noteworthy that making $\neg Y$ as the result part is trying to make sure that the memory that the retriever gets from other agents is of reference value, but it does not have to be the most relevant to the current question, which means that it can help the current agent to learn from new examples. This approach diverges from a simplistic reliance on $Y$ as the outcome, which tends to restrict the retrieval process to memory previously stored by the current agent.

Within the defined set $C=\{(x_{i},y_{i})\}_{i=1}^{n}$, each candidate now is ascribed a score. We sort them from the lowest to the highest score and we select $v$ memory in total to label. The top $\frac{v}{2}$ candidates (lowest score) in $C$ are identified as being the pair with the reference value to $(X,Y)$ and accordingly, their labels are set to positive. Conversely, the bottom $\frac{v}{2}$ candidates are deemed as the least reference value to $(X,Y)$, and their labels are thus designated as negative. Those labeled data will be used to minimize the following function:

It enhances the predictive accuracy, which is especially critical in handling imbalanced memory pool. This strategic choice underscores our model’s preparation to extract meaningful insights from various memory, advancing our overarching goal of developing a robust and adaptable MS framework.

As described in sections 3.1 and 3.2, the memories stored within the memory pool effectively help agents improve their response quality. Moreover, the number of memories in memory pool is dynamically expanding. Over time, the increasing number of memories in memory pool enhances the assistance provided to the agents. However, the initial state of a newly created multi-agent system and its corresponding memory pool lacks any memories. How can it assist the agents from the outset? And how can we validate the effectiveness of this mechanism? To address this, we propose a rapid interactive learning method that enables multiple agents to engage in interactive prompt and answer. High-quality PA pair is stored as memory in the memory pool, facilitating self-learning and self-enhancement within the multi-agent system. Initially, a small set of answers (e.g., 100 records, theoretically even one record can initiate the process) is placed in the memory as the initial set. Agents then engage in prompt and answer based on this initial memory set, rapidly expanding the memory pool. Specially, give a standard answer, we ask the agent to give a corresponding question based on the standard answer. Then, we give this question back to agents again and ask them to answer this question. This is also the way how we construct our dataset and our initial memory pool. We measure the effectiveness of the MS framework by calculating the average quality of answers generated by the agents at different stages.

We aim to assess the efficacy of the MS framework in processing open-ended questions across three domains: Literary Creation, Unconventional Logic Problem-solving, and Plan Generation. Within the Literary Creation domain, we have appointed three specialized agents responsible for generating Wuyanlvshi (a form of classical Chinese poetry, Chinese form), Limericks, and Sonnets, respectively. In the Logic Problem-solving domain, dedicated agents are tasked with addressing Puzzles, Riddles, and Puns. Meanwhile, for Plan Generation, we have developed agents to create Study Plans, Travel Plans, and Fitness Plans. We use a total of nines datasets representing nine agents to evaluate. There are 1000 question-answer pairs in total, details in Appendix A.3. For each agent, a consistent, small subset of pre-provided, complete PA pairs were selected and incorporated into the memory pool for the initial phase of retriever training and prompt refinement. Subsequently, for each agent, an identical number of queries will be introduced to increment the volume of real-time memory within the memory pool. For our scoring LLM, we use gpt-3.5-turbo. As the backbones of our agents, we consider three LLMs: two close-source LLMs (gpt-3.5-turbo and gpt-4o) and one open-source LLM (open-mistral-7b). We use the BERTScore (Zhang et al., 2019) as our metric to measure the performance of each agent.

The evaluation of memory impact begins with the implementation of different retrieval strategies, including zero-shot, one-shot, two-shot, and three-shot learning. Subsequently, quantitative and qualitative analysis are conducted. In qualitative analysis, one question is that the memories of agents in the same domain can make up for the lack of diversity, so if the memories of agents in all domains are placed in the same memory pool, from the perspective of diversity, it will definitely increase further, but will the memory pool still provide positive help to the agents’ answers? Aiming to solve this, our study uses two distinct types of memory pools, Domain-pool and Single-pool. The Domain-pool means a dedicated memory pool is allocated for each domain and is shared for all agents with this domain, aiming at enhancing the integration of domain-specific memories. Conversely, the second pool integrates agents from all domains into a unified memory pool, facilitating the analysis of cross-domain memory utilization. Although not all the memories are very related to one single query, the memory comes from other unrelated agents in other domains may help the agents better understand the query from different angle.

In terms of the quantitative analysis, we measure the performance of each agent whenever the same proportion of new memory is added to the memory pool. There are five phase in total-20%, 40%, 60%, 80%, 100%. At each phase, an evaluation of agent performance was conducted to ascertain improvements or regressions. This dualfaceted approach enabled a thorough exploration of the naunced impacts and applicability of authentic memories across diverse domains.

Before the experiment, none of the agents have a suitable database for reference. While after the interactive learning stage, a continuously expanding memory pool with high quality memories is successfully be a database for agent to refer. The MS framework help the agents get rid of the dependence on external databases, and multiple agents can interactively expand the memory pool. The principal outcomes of our experiments are presented in Table.1, which shows the performance of each agent under various learning strategies within the MS framework. Compared to Zero-shot learning, other three learning strategies all help agents achieve better performance, which means that the memory from other agents can help current agent get desired answers, rather than interfering with the agent’s learning ability. Also, given the performance of all agents get improved after utilizing the shared memories, our previous hypothesis that the MS framework could enhance collective intelligence through multi-agent interactions, thereby advancing from individual to collective intelligence, has been confirmed. And for most agents, they achieve the best performance when they retrieve and utilize three memories from the memory pool.

It worth noting that, for all the agents in the domains-Literary Creation and Plan Generation, the performance of them under the three-shot learning with a open-source LLM successfully surpass themselves with close-source LLM under the zero-shot learning, which proves the potential power of the shared memories. Besides, when under the same learning strategy, the superior performance of the closed-source LLM over the open-source LLM can be attributed to the former’s enhanced understanding and reasoning capabilities. Specifically, in the Literary Creation domain, the improvement of those three agents’ performance are not as significant as other agents, which may resulted in the language used is different when storing memory. This deserves further study in future work.

Table.2 compares the scenario where all agents use the Domain-pool or Single-pool under the three-shot learning strategy, since the prior experiments showed us that most agents achieve the best performance under the three-shot learning. Excluding Agent-Study, all other agents exhibited diminished performance with the Single-pool. Although the Single-pool can further enrich the diversity of shared memories, for the agents, homologous memories can better help them get more reliable answers, no matter what the agent’s backbone is.

One follow up question is, even memory types are homogeneous, whether an excessive accumulation of memories will impede the agent’s output quality. Figure.4 shows the variations in performance across individual agents consequent to the integration of different ratios of newly generated memories into the pool. It is clear that as more and more high-quality memories are added to the memory pool, the performance of most agents is getting better and better, especially for the Agent-Limerick. For several agents, there is no further change of their performance as the shared memories increases further in the later stages. We assume this is since those newly added shared memories are not more suitable than the previous ones. If the memory pool can be further expanded, this stagnation may be broken.