InvAgent: A Large Language Model based Multi-Agent System for Inventory Management in Supply Chains

A multi-period, multi-echelon production-inventory system for a single non-perishable product is designed for illustrating and simulating a typical multi-stage supply chain. As shown in Figure 2, each stage in this supply chain consists of an inventory holding area and a production area. The inventory holding area stores the materials necessary for production at that stage. One unit of inventory produces one unit of product at each stage. There are lead times for transferring products between stages. The outgoing material from stage $i$ serves as the feed material for production at stage $i-1$. Stages are numbered in ascending order: $0,1,...,M-1$, with stage 0 being the retailer. Production at each stage is limited by the stage’s production capacity and available inventory. Figure 2 depicts the flow of raw materials through various stages of production and inventory management, ultimately culminating in the fulfillment of customer demand at the retail level.

There are $T$ periods in each simulation, starting from 1, with $t=0$ used for the initial condition of the supply chain. At the beginning of each time period, the following sequence of events occurs:

1. 1.

   Check deliveries: Each stage receives incoming inventory replenishment shipments that have arrived after the stage’s respective lead time.

2. 2.

   Check orders and demands: Each stage places replenishment orders to their respective suppliers. Replenishment orders are filled according to the available production capacity and inventory at the suppliers. Customer demand occurs at the retailer and is filled based on the available inventory at the retailer.

3. 3.

   Deliver orders and demands: Each stage delivers as many products as possible to satisfy downstream demand or replenishment orders. Unfulfilled sales and replenishment orders are backlogged, with backlogged sales taking priority in the following period.

4. 4.

   Compute profits: Each stage computes the profit and cost for product sales, material orders, backlog penalties, and surplus inventory holding costs.

With the notations defined in Table 1, the entire inventory management problem (IMP), inspired by Hubbs et al. (2020), can be expressed using following equations:

	$\displaystyle I_{m,t}=I_{m,t-1}+R_{m,t-L_{m}}-S_{m,t},$		(1)
	$\displaystyle\qquad\forall m\in\mathcal{M},$
	$\displaystyle R_{m,t}=\min(B_{m+1,t-1}+O_{m,t},c_{m+1},$
	$\displaystyle{\hskip 37.0pt}I_{m+1,t-1}+R_{m+1,t-L_{m+1}}),$		(2a)
	$\displaystyle\qquad\forall m=0,...,M-2,$
	$\displaystyle R_{M-1,t}=O_{M-1,t},$		(2b)
	$\displaystyle S_{m,t}=R_{m-1,t},\quad\forall m=1,...,M-1,$		(3a)
	$\displaystyle S_{0,t}=\min(B_{0,t-1}+D_{t},c_{0},$
	$\displaystyle{\hskip 30.0pt}I_{0,t-1}+R_{0,t-L_{0}}),$		(3b)
	$\displaystyle B_{m,t}=B_{m,t-1}+O_{m-1,t}-S_{m,t},$		(4a)
	$\displaystyle\qquad\forall m=1,...,M-1,$
	$\displaystyle B_{0,t}=B_{0,t-1}+D_{t}-S_{0,t},$		(4b)
	$\displaystyle P_{m,t}=p_{m}S_{m,t}-r_{m}R_{m,t}-k_{m}B_{m,t}$
	$\displaystyle{\hskip 30.0pt}-h_{m}I_{m,t},\qquad\forall m\in\mathcal{M}.$		(5)

In Equation 1, the current inventory at stage $m$ at the end of the current period $t$ is equal to the final inventory in the previous period, plus fulfilled order placed $L_{m}$ periods ago, minus the sales during the current period. In Equation 2a, fulfilled order at stage $m$ placed during period $t$ is decided by (1) previous backlog at the upstream stage plus newly requested orders, (2) upstream stage production capacity, and (3) total available inventory at the upstream stage $m+1$ at the start of period $t$, including leftover stock from the previous period and newly arrived orders after accounting for lead time. The final fulfilled order is the minimum of these three conditions, ensuring that the order does not exceed any of these constraints. This ensures that the supply chain operates within its capacity and inventory limits, preventing overcommitment and stockouts. Equation 2b tells us requested orders at the upmost stage are always fulfilled, because we assume an unlimited supply of raw materials. Sales are always equal to fulfilled orders except at stage 0 (retailer), as shown in Equation 3a. In Equation 3b, sales at stage 0 (retailer) during period $t$ are determined by the minimum of three conditions: (1) the previous backlog at stage 0 plus the current customer demand, (2) the production capacity of stage 0, and (3) the total available inventory at stage 0 at the start of period $t$, which includes leftover stock from the previous period and newly fulfilled orders after accounting for the lead time. This ensures that the sales at the retailer do not exceed the total demand, production capacity, or available stock. In Equation 4a, the backlog at stage $m$ during period $t$ is calculated as the sum of the previous period’s backlog at stage and the orders requested from the previous stage, minus the sales at stage $m$, for all stages except the retailer. In Equation 4b, the backlog at stage 0 (retailer) during period $t$ is calculated similarly to Equation 4a, but the requested order is replaced by customer demand because the retailer is directly in contact with customers. In Equation 5, the profit at each stage $m$ during period $t$ is calculated as the sales revenue minus the procurement costs, unfulfilled demand costs, and inventory holding costs.

In this work, we propose InvAgent, a LLM based multi-agent inventory management system for supply chain optimization. InvAgent includes several key agents: one user proxy and one agent for each stage. The user proxy serves as an intermediary between the environment and all supply chain agents, facilitating communication and managing the exchange of data. The framework of InvAgent method is illustrated in Figure 1, which follows these steps:

1. 1.

   The user proxy resets the environment at the beginning of the first round.

2. 2.

   The user proxy requests the state of the current round for each stage from the environment.

3. 3.

   The user proxy provides the state to each stage and requests the action from it.

4. 4.

   The user proxy sends the agent actions to the environment and obtains the next state and the reward for this step.

5. 5.

   The user proxy determines whether the simulation is terminated; if not, the simulation moves to step 2.

At the beginning of the simulation, we create system messages for agents in Figure 3, which provide essential information, such as definitions, roles, and goals in the supply chain. The state $s_{m,t}$ and action $a_{m,t}$ of an agent are defined as: $s_{m,t}=[c_{m},p_{m},r_{m},k_{m},h_{m},L_{m},I_{m,t-1},B_{m,t-1},\\ B_{m+1,t-1},S_{m,t-L_{\max}},\dots,S_{m,t-1},0,\dots,0,\\ R_{m,t-L_{m}},\dots,R_{m,t-1}]$ and $a_{m,t}=O_{m,t}$, where the state includes the current stage features, inventory, backlog, upstream backlog, recent sales, and arriving deliveries with left zero padding.

The prompt, as designed in Figure 4, aims to provide the state and request actions from each agent, ensuring effective decision-making and clear communication within the supply chain. It includes contextual information such as the current period, stage²²2The stage is counted from 1 instead of 0 in the prompt to prevent confusion for LLMs., and number of stages to position the model within the supply chain. The state description (Figure 5) provides a comprehensive snapshot of inventory levels, backlogs, previous sales, and incoming deliveries, enabling informed decisions. Demand (Figure 6) and downstream order (Figure 7) details help match supply with immediate needs, allowing upstream suppliers to quickly respond to downstream orders or demands. The strategy description (Figure 8) outlines guidelines like considering lead times and avoiding overordering to maintain inventory balance. By requesting reasoning before specifying the action, the prompt promotes transparency and interpretability in decision-making. This design leverages LLMs’ capabilities to enhance inventory management, ensuring decisions are well-informed, transparent, and aligned with the supply chain strategy. One example of the prompt and the response from GPT-4 is shown in Appendix A.

The features of the prompt design are as follows:

• •

  Zero-Shot Learning. Our designed prompt operates on a zero-shot basis without providing any specific examples to the LLM. This means that the model must generate responses based solely on its pre-existing knowledge and the information presented in the prompt.

• •

  Demand Description. Since we don’t have any prior training process, unlike reinforcement learning, which involves a training process to enhance understanding of the environment and demand, providing a clear and detailed description of the demand is crucial to ensure accurate understanding and effective responses.

• •

  Downstream Order. The prompt considers downstream order, where information can be delivered swiftly and shared efficiently between different stages.

• •

  Human-Crafted Strategy. The inherent strategy of the LLM is generally sufficient for simple scenarios, such as constant demands. However, for more complex scenarios like seasonal demands, it is assumed that additional human-crafted strategies can be helpful for the LLM’s decision-making.

• •

  Chain-of-Thought (CoT). The CoT approach can enhance the explainability of the results. By guiding the LLM through a structured reasoning process, CoT helps the model to better understand the scenario and improve its reasoning capabilities, ultimately leading to more accurate and reliable outcomes.

We describe the various experiment scenarios designed to evaluate the performance of our inventory management system in a multi-echelon supply chain. Each scenario introduces specific conditions to rigorously test the robustness and adaptability of the proposed model. Parameter settings for these scenarios are summarized in Table 2.

In the first scenario, a four-stage supply chain is tested with a constant demand of 4 units per period over 12 periods, starting with 12 units of inventory per stage and a lead time of 2 periods. This scenario aims to test the basic functionality of the model under stable conditions. The second scenario introduces variable demand, uniformly ranging between 0 and 4 units per period, adding randomness to evaluate the system’s ability to manage fluctuating demand while maintaining efficient inventory. The third scenario further increases demand variability, with uniform distribution between 0 and 8 units per period, and incorporates sales and order costs set at 5 units per period, testing the model’s capability to handle high variability and financial impacts. The fourth scenario simulates seasonal demand with a leaping pattern ranging from 4 to 8 units per period, maintaining the same financial parameters as the third scenario, to evaluate the system’s performance under predictable but varying demand patterns. Finally, the fifth scenario features normally distributed demand with a mean of 4 units and a standard deviation of 2 units per period, varying lead times, initial inventories, sales prices and order costs across the stages. This scenario tests the system’s performance under more realistic demand fluctuations and varying operational constraints. These scenarios collectively provide a comprehensive test bed for evaluating the efficacy and adaptability of our multi-agent system in managing dynamic inventory across a multi-echelon supply chain.

We have four baselines: two heuristic policies, namely the base-stock policy and the tracking demand policy, and two reinforcement learning (RL) policies, specifically independent proximal policy optimization (IPPO) and multi-agent proximal policy optimization (MAPPO).

Two heuristic baselines are designed based on the desired inventory levels, where each stage aims to maintain sufficient inventory to fulfill customer demands or downstream orders. The stage order (action) is computed by

$$O_{m,t}=\min(\max(0,\hat{O}_{m,t}),c_{m}),$$

(6)

where the $\hat{O}_{m,t}$ is determined by the desired inventory $\hat{I}_{m,t}$, current inventory $I_{m,t-1}$, upstream backlog order $B_{m+1,t-1}$, and cumulative sum of arriving deliveries as

$$\hat{O}_{m,t}=\hat{I}_{m,t}-I_{m,t-1}-B_{m+1,t-1}\\ -\sum_{\Delta t=1}^{L_{m}}R_{m,t-\Delta t}.$$

(7)

When an order is placed, the stage replenishes its inventory to this target level by ordering the difference between the base-stock level and the current inventory position with upstream backlog orders and arriving deliveries. Based on different choices of the desired inventory, we introduce two heuristic polices as follows.

Base-Stock Policy. The base-stock policy Lee et al. (1997); Oroojlooyjadid et al. (2022) is an inventory management strategy where each stage in the supply chain maintains a constant inventory level, or base-stock level. Here, the desired inventory level is set equal to the production capacity:

$$\hat{I}_{m,t}=c_{m}.$$

(8)

Tracking Demand Policy. The tracking demand policy is an inventory management strategy that adjusts orders based on observed demand (or sale) patterns rather than maintaining a constant base-stock level. By dynamically aligning supply with actual consumption, this policy ensures a responsive and efficient inventory system. In this policy, the desired inventory is set as:

$$\hat{I}_{m,t}=\bar{S}_{m,t-1}L_{m}+B_{m,t-1},$$

(9)

where the average sale for recent rounds is

$$\bar{S}_{m,t-1}=\frac{1}{L_{\max}}\sum_{\Delta t=1}^{L_{\max}}S_{m,t-\Delta t}.$$

(10)

More heuristic baselines are discussed in Appendix B.

Independent Proximal Policy Optimization (IPPO) with Parameter Sharing. Proximal policy optimization (PPO) Schulman et al. (2017) updates policies by iteratively sampling data through interaction with the environment and optimizing a clipped surrogate objective function using multiple epochs of stochastic gradient ascent. Independent PPO (IPPO) De Witt et al. (2020) is a RL approach where each agent is independently trained with the PPO algorithm. Here we employ the parameter sharing by using the same policy parameters for all agents to improve learning efficiency and coordination.

Multi-Agent Proximal Policy Optimization (MAPPO). Multi-agent PPO (MAPPO) Yu et al. (2022) is an extension of PPO algorithm designed for multi-agent environments. It utilizes a centralized value function that takes into account global information from all agents, improving variance reduction and stability during training.

The performance of our model InvAgent is evaluated using the total reward from all stages all periods during one simulation (episode), and the reported numbers are averaged over 5 episodes for each experiment to reduce the uncertainty. We utilize the Python packages such as AutoGen Wu et al. (2023), Gymnasium Towers et al. (2023), and RLlib Liang et al. (2018), and also LLMs including GPT-4, GPT-4O, and GPT-4-Turbo Achiam et al. (2023). For the constant demand scenario, we change the last part of the prompt in Figure 4 to "([0], [4], or [8] only)" to boost the InvAgent performance.

The performance of baseline models is evaluated based on the episode reward averaged over 100 episodes. For reinforcement learning (RL), we explore various hyper-parameter settings, including the numbers of hidden units ([128, 128] and [256, 256]), activation function (ReLU), learning rate (1e-4, 5e-4, 1e-3), training batch size (500, 1000, 2000), stochastic gradient descent (SGD) minibatch size (32, 64, 128), number of SGD iterations (5, 10, 20), number of training iterations (500, 800, 1000, 1500), and discount factor (1.0). We randomly select 20 combinations of these hyper-parameters during one experiment and keep the best one of them. The final hyper-parameters used for all scenarios are presented in Appendix C. The RL experiments are conducted on a NVIDIA A10 GPU.

Our experiment results, as shown in Table 3, highlight the performance of various models across different demand scenarios. The InvAgent model demonstrates competitive performance, particularly in the variable demand scenario, where InvAgent (without the hand-crafted strategy) achieves the highest mean episode rewards. While the MAPPO model exhibits the best performance in the other demand scenarios, InvAgent’s zero-shot capability and adaptability offer significant benefits. This adaptability allows InvAgent to make reasonable decisions and understand concepts without specific examples, showcasing a level of generalization and adaptability akin to human intuition.

When compared to heuristic baselines, InvAgent shows notable advantages. Unlike the base-stock policy and tracking-demand policy, which rely on fixed or historical data and often struggle with fluctuating demands, InvAgent dynamically adapts to real-time conditions, minimizing inventory costs and avoiding stockouts. InvAgent performs particularly well in the variable demand scenario, indicating its effectiveness in managing unpredictable demand patterns. These results demonstrate that InvAgent can achieve lower costs and better adaptability than the heuristic baselines.

In comparison to RL models, InvAgent offers distinct strengths despite certain performance limitations. RL models like MAPPO and IPPO achieve higher metrics in several scenarios due to extensive training but come with increased complexity and potential instability. RL training often requires significant computational resources and time, with risks of overfitting. In contrast, InvAgent provides key advantages in explainability and ease of implementation, making reasonable decisions without prior training. Although InvAgent does not always outperform RL models, its stability, simplicity, and interpretability make it a valuable alternative for dynamic inventory management.

The comparison of the InvAgent model with and without strategy shows that the inherent strategy of the LLM is generally sufficient for many simpler scenarios, such as constant and variable demands. However, in more complex scenarios like seasonal demands, the addition of a human-crafted strategy enhances the LLM’s decision-making. This demonstrates that while the model’s built-in capabilities are adequate for straightforward situations, incorporating explicit strategies proves beneficial for managing more complex and patterned demand scenarios, thereby improving performance and adaptability.

We conduct ablation studies to evaluate the impact of various prompt components in InvAgent under the variable demand scenario, as detailed in Table 4. This scenario introduces randomness with demand varying uniformly between 0 and 4 units per period over 12 periods. The results demonstrate that the contributions of different components to the model’s performance vary significantly. The prompt without the strategy component is the best one and other models are compared with it.

Overall, the ablation study confirms the robustness and adaptability of our model in dynamic supply chain environments. Components such as the demand description and downstream order are particularly essential for optimizing performance under variable demands. By keeping agent histories, all previous messages in the entire episode (simulation) are retained, allowing stage agents to use the entire chat history for context in their decision-making. Additionally, structured reasoning through Chain-of-Thought (CoT) also play crucial roles. These findings emphasize the importance of each component in achieving effective and efficient inventory management, guiding future improvements and applications in more complex scenarios.