Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning

The multi-agent decision-making problem is typically described as a Markov game (MG), which can be defined by a tuple $\Gamma\triangleq\langle\mathcal{I},\mathcal{S},\{\mathcal{A}^{i}\}_{i\in\mathcal{I}},P,\{r^{i}\}_{i\in\mathcal{I}},\gamma\rangle$. $\mathcal{I}=\{1,2,...,n\}$ denotes the set of agents and $s\in S$ is the global state set of the environment. $\mathcal{A}^{i}$ denotes the action space of agent $i$ and the joint action space $\mathcal{A}=\prod_{i=1}^{n}\mathcal{A}^{i}$ is the product of action spaces of all agents. $P:\mathcal{S\times\mathcal{A}}\to\Omega(\mathcal{S})$ denotes state transition function, where $\Omega(X)$ is the set of probability distributions in $X$ space. $r^{i}:\mathcal{S\times\mathcal{A}\to\mathbb{R}}$ is the reward function of agent $i$ and $\gamma$ is the discount factor. At time step $t$, each agent chooses an action $a^{i}_{t}\in\mathcal{A}^{i}$ at state $s_{t}\in\mathcal{S}$ based on its own policy $\pi^{i}:\mathcal{S}\to\Omega(\mathcal{A}^{i})$ and receives feedback in the form of $r^{i}(s_{t},\boldsymbol{a_{t}})$, where $\boldsymbol{a_{t}}=(a_{t}^{1},...,a_{t}^{n})\in\mathcal{A}$. The environment moves to a new state $s_{t+1}\sim P(s_{t+1}\mid s_{t},a_{t})$ as a result of joint action $\boldsymbol{a_{t}}$. The joint policy of all agents is expressed as $\boldsymbol{\pi}\left(s_{t}\right)=\prod_{i=1}^{n}\pi^{i}\left(s_{t}\right)$. Under the framework of MG, the state value function of agent $i$ is defined as :

$\displaystyle V^{i}_{\boldsymbol{\pi}}(s)=\mathbb{E}_{s\sim P,a^{-i}\sim\boldsymbol{\pi}^{-i}}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}(s_{t},\boldsymbol{a_{t}})\mid\mathrm{s}_{0}=s,a_{t}^{i}\sim\pi^{i}(s_{t})\right],$

(1)

where $-i$ represents all agents except $i$. According to Bellman equation, the action-state value function is denoted as:

$\displaystyle Q^{i}_{\boldsymbol{\pi}}(s,\mathbf{a})=r^{i}(s,\mathbf{a})+\gamma\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|s,\mathbf{a})\cdot V^{i}_{\boldsymbol{\pi}}(s^{\prime}).$

(2)

In a multi-agent system, each decision-maker strives to maximize their own expected utility, denoted as $\mathcal{J}^{i}(\theta)=\mathbb{E}_{s\sim P,a\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}\right]$, where $\theta=\{\theta^{i}\}_{i\in{\mathcal{I}}}$ represents the policy parameters of agents. However, using gradient ascent to directly update the strategy, i.e., $\theta^{i}\leftarrow\theta^{i}+\alpha\nabla_{\theta^{i}}\mathcal{J}^{i}(\theta)$, usually fails because $\mathcal{J}^{i}(\theta)$ is affected by all agents and the gradient update directions among agents may conflict. Consequently, it is vital to employ the concept of game equilibrium solutions in order to develop effective coordination strategies. One common solution objective is NE. Nash-Q learning, for example, computes agent $i$’s value function $V_{\mathcal{N}ash}^{i}(s)$ when all agents follow the NE strategy in each stage of the game, and updates the action-state value function $Q_{\mathcal{N}ash}^{i}(s,\mathbf{a})$ through Eq. (2). The algorithm can converge to NE strategies under the strong assumption that equilibrium exists at every game stage. All agents adopt equilibrium policies as their own convergence objective rather than selfishly maximizing their own individual utilities. With the emergence of deep learning, current popular MARL approaches frequently employ the CTDE paradigm to enable coordination via credit assignment Rashid et al. (2018) or centralized critics Lowe et al. (2017).

The Stackelberg leadership model dictates the order of actions among agents. Consider a two-player game, the leader can enforce his own strategy $\pi^{1}$ to the follower, who will react after observing the leader’s behavior $\pi^{2}:\mathcal{S\times A}^{1}\to\Omega(\mathcal{A}^{2})$, where $\pi^{i}\in\Pi^{i}$ and $\Pi^{i}$ represents the policy space. In this model, leader optimizes its objective on the premise that follower will provide the best response $BR(\pi^{1})$, while the follower tends to maximize its expected utility based on leader’s preconditions. It can be formalized as:

	$\displaystyle\max_{\pi^{1}\in\Pi^{1}}\{\mathcal{J}^{1}(\pi^{1},\pi^{2})|\pi^{2}$	$\displaystyle=BR(\pi^{1})\},$		(3)
	$\displaystyle\max_{\pi^{2}\in\Pi^{2}}\{\mathcal{J}^{2}(a^{1},\pi^{2})|$	$\displaystyle a^{1}\sim\pi^{1}\}.$

Equilibrium signifies that in a multi-player game, all players have adopted the optimal strategy and none can improve their performance by altering their own strategy. Correspondingly, Stackelberg equilibrium signifies that both agents adhere to the optimal solution strategies $({\pi^{1}_{SE}},\pi^{2}_{SE})$ of the the above optimization issue. It satisfies:

	$\displaystyle V^{1}_{\pi^{1}_{SE},BR(\pi^{1}_{SE})}(s)$	$\displaystyle\geq V^{1}_{\pi^{1},BR(\pi^{1})}(s),$		(4)
	$\displaystyle\pi^{2}_{SE}$	$\displaystyle=BR(\pi^{1}_{SE}).$

As a mandatory equilibrium, SE offers more benefits than NE in terms of the stability, certainty of equilibrium points and Pareto superiority Zhang et al. (2020). Consequently, it is a more suitable learning objective.

Current CTDE approaches, especially policy-based methods, tend to center around enhancing the richness of input information rather than boosting the effectiveness of collaboration. Actually, centralized training liberates us from restrictive circumstances. With centralized training, we can definitely create an interaction mechanism among agents for completing their tasks. Furthermore, the concept of the SE is more in line with the goals of RL as it requires merely additional constraints, which can be easily achieved by setting a specific interaction mechanism. So naturally, we utilize the form of multi-player Stackelberg leadership model and construct the spatio-temporal sequential Markov game (STMG) in the training phase to direct agents coordination and facilitate the implementation of SE policies.

Figure 1 shows the state transition process in STMG. For the sake of simplicity, we assume that the agent ID $i$ is assigned by priority $o^{i}$. Compared with MG, STMG assumes the form of a sequence decision in both temporal and spatial domains. Agents with higher priorities have greater initiative, whereas agents with lower priorities must respond to the actions of those with higher priorities. Correspondingly, the policy of each agent changes to $\pi^{i}:\mathcal{S}\times\mathcal{A}^{1}\cdot\cdot\cdot\mathcal{A}^{i-1}\to\Omega(\mathcal{A}^{i})$. The action-state value function of agent $i$ can be written as:

$\displaystyle\begin{aligned} Q^{i}_{\boldsymbol{\pi}}&(s,a^{1:i-1},a^{i})=\mathbb{E}_{s\sim P,a^{i+1:n}\sim\boldsymbol{\pi}^{i+1:n}}\bigg{[}\sum_{t=0}^{\infty}\gamma^{t}\cdot\\ &r^{i}_{t}(s_{t},\boldsymbol{a_{t}})\mid\mathrm{s}_{0}=s,\mathbf{a}_{0}=\boldsymbol{a},a_{t}^{i}\sim\pi^{i}(s_{t},a_{t}^{1},...,a_{t}^{i-1})\bigg{]}.\end{aligned}$

(5)

The state value function is denoted as:

$$V^{i}_{\boldsymbol{\pi}}(s,a^{1:i-1})=\sum_{a^{i}\in\mathcal{A}^{i}}\pi^{i}(a^{i}|s,a^{1:i-1})Q^{i}_{\boldsymbol{\pi}}(s,a^{1:i-1},a^{i}).$$

(6)

And then we have the advantage function:

$\displaystyle\begin{aligned} A^{i}_{\pi}(s,a^{1:i-1},a^{i})=Q^{i}_{\boldsymbol{\pi}}(s,a^{1:i-1},a^{i})-V^{i}_{\boldsymbol{\pi}}(s,a^{1:i-1}).\end{aligned}$

(7)

Within the STMG framework, agent with priority $i$ can consider the actions of the preceding $i-1$ superior agents as prerequisites, maximize its own expected return $\mathcal{J}^{i}(\theta)=\mathbb{E}_{s\sim P,a^{i+1:n}\sim\boldsymbol{\pi}^{i+1:n}_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}r^{i}_{t}\right]$ according to the current subgame state $s^{i-1}=(s,a^{1},...,a^{i-1})$, which is equivalent to learning the best response to the strategies of superior agents. In summary, during the training phase, agents’ execution mechanism can be divided into the Markov game process in the time domain and the Leader-Follower model in the space domain. The specific execution process is depicted in Figure 2.

Within the STMG framework, agents are able to acquire the actions of superior agents directly. This asymmetric approach arises naturally in the centralized training phase without communication restrictions, but it is obviously illegal in the decentralized execution phase. To this end, several approaches can be taken. We can train a policy network that is shared by all agents, allowing them to calculate the actions of other agents using the shared parameters. Designing a communication module that enables superior agents to broadcast their decisions to inferior agents is also an alternative. Moreover, agents can keep copies of other players’ policy or value networks to independently calculate their own policies. Nonetheless, parameter sharing can result in suboptimal solutions, and communication may not be feasible in fully decentralized execution settings or with communication bandwidth constraints. Additionally, the scalability of the algorithm may be limited as the number of agents increases and the volume of saved model copies grows.

In contrast to the aforementioned methods, we develop an N-level policy model using a conditional hypernetwork. It consists of a parameter generator, a state embedding, and a target policy module. Figure 3 illustrates its structure in detail. MARL is essentially a multi-task learning (multi-objective regression) problem Iqbal and Sha (2019). Instead of explicitly training and storing individual policy modules for each agent, we maintain a set of policy parameters $\{\theta_{tar}^{i}\}_{i\in\mathcal{I}}$ by training a meta-model (conditional hypernetwork) $\mathcal{H}(e^{i},\theta_{h})=\theta^{i}_{tar}$ with the weights $\theta_{h}$ shared by all agents. It maps the embedding of each agent’s priority ID $e^{i}$ to the parameter configuration of its policy network. It has been demonstrated that hypernetworks can handle continuous learning tasks von Oswald et al. (2020), where a model is trained on a series of tasks in sequence. This is highly compatible with our expectations. In addition, the input $s^{i-1}$ to the target policy module is encoded by the state embedding network $\mathcal{E}(s,a^{1},...,a^{i-1};\mathcal{\theta}_{s})$, which receives as input the current state of the environment and the actions of superior agents. Specifically, each component of the model is described as follows:

	State Embedding:	$\displaystyle s^{i-1}$	$\displaystyle=\mathcal{E}(s,a^{1},...,a^{i-1};\mathcal{\theta}_{s}),$		(8)
	Parameter Generator:	$\displaystyle\theta^{i}_{tar}$	$\displaystyle=\mathcal{H}(e^{i};\theta_{h}),$		(9)
	Target Policy:	$\displaystyle a^{i}$	$\displaystyle\sim\pi^{i}(s^{i-1};\theta^{i}_{tar}).$		(10)

It is important to note that the parameters of the policy model consist of $\theta^{i}=(\theta_{s},\theta_{h},\theta^{i}_{tar})$, where $\theta_{s},\theta_{h}$ are learnable parameters shared by all agents and $\theta^{i}_{tar}$ can be accessed by all agents through the parameter generator. By using the N-level policy model, we are able to share policy parameters while avoiding the suboptimal solutions and storage issues previously mentioned, resulting in efficient decentralized coordination.

The N-level policy model allows for the implementation of the asymmetric training with symmetric/decentralized execution (ATSE) paradigm in STEP. Meanwhile, we select Proximal Policy Optimization (PPO) Schulman et al. (2017) as the underlying algorithm due to its superior performance. To ensure the balance between exploration and exploitation during the training phase, agents sample actions from the categorized distribution or multivariate Gaussian distribution generated by the policy model. However, due to the unpredictability of sampling, agents are unable to calculate the actions of superior agents. As a result, we directly transmit the behaviors of superior agents to inferior agents during the training process. During the execution process, each agent selects the action with the highest probability, and inferior agents can easily compute actions of superior agents through the shared policy model to establish corresponding coordination. Additionally, throughout each epoch of the training process, each agent trains the identical policy module using its own data, which can result in catastrophic forgetting. To address this issue, we incorporate a regularization item to ensure that the policy network can train the current agent’s policy parameters while maintaining the capacity to fit the previously updated policies of other agents. The objective function of actor $i$ is expressed as:

		$\displaystyle\mathcal{L}^{i}(\theta)=\mathbb{E}_{s\sim P,\mathbf{a}\sim\boldsymbol{\pi}}[\mathcal{L}_{clip}-\mathcal{L}_{h}(\theta_{h},\theta_{h_{old}},e^{1:i-1})$		(11)
		$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad+\eta S(\pi_{\theta}^{i}(s,a^{1:i-1}))],$
		$\displaystyle\mathcal{L}_{clip}=\min(r^{i}_{\theta}A^{i}_{\pi},clip(r^{i}_{\theta},1\pm\epsilon)A^{i}_{\pi}),$
		$\displaystyle\mathcal{L}_{h}(\cdot)=\frac{\beta}{i-1}\sum_{j=1}^{i-1}||\mathcal{H}(e^{j};\theta_{h})-\mathcal{H}(e^{j};\theta_{h_{old}})||^{2}_{2},$

where $S(\cdot)$ is Shannon entropy used for strengthening exploration,$\quad r^{i}_{\theta}=\frac{\pi^{i}_{\theta}(a^{i}|s,a^{1:i-1})}{\pi^{i}_{\theta_{old}}(a^{i}|s,a^{1:i-1})}$ is the likelihood ratio between the current and previous policies, $\epsilon$ is the clipping ratio, $\eta$ and $\beta$ are coefficients of entropy and regularization term respectively. Critic network is used to fit the value function, and its loss function is expressed as:

$\displaystyle\begin{aligned} \mathcal{L}&(\phi^{i})=\max\big{[}\left(V_{\phi}\left(s,a^{1:i-1}\right)-R^{i}\right)^{2},\\ &\left(\operatorname{clip}\left(V_{\phi}\left(s,a^{1:i-1}\right),V_{\phi_{old}}\left(s,a^{1:i-1}\right)\pm\varepsilon\right)-R^{i}\right)^{2}\big{]},\end{aligned}$

(12)

where $R^{i}$ is the cumulative return and $\varepsilon$ is the clipping ratio.

The pseudo-code of STEP can be found in Appendix. Notably, our method provides comprehensive advantages over currently popular MARL techniques. Unlike prior work based on SE Könönen (2004); Zhang et al. (2020), STEP is easily extensible to include additional players; Unlike prior work based on parameter sharing Rashid et al. (2018), STEP is able to learn heterogeneous policies; In contrast to previous work on learning heterogeneous policies Kuba et al. (2022), STEP does not increase the learning cost as the number of agents increases; In contrast to previous CTDE work Lowe et al. (2017); Yu et al. (2021), we no longer focus on merely expanding the information space, but on specific forms of coordination.

We test STEP using the cooperative matrix game Penalty proposed by Claus and Boutilier Claus and Boutilier (1998) and the non-cooperative matrix game Mixing shown in Figure 4. Agents receive constant observations at each time step in both games, and the length of repeated matrix games is set to 25. To cultivate formidable critics, the popular CTDE paradigm utilises combined observation and action to enlarge the information space. However, there is no noticeable performance difference between independent learning (IL) and CTDE for empty state matrix games. Therefore, only MAPPO Yu et al. (2021) and MADDPG Lowe et al. (2017) are considered as baselines for our research. We present the experimental results over 100 trials of 10000 time steps in Figure 5 and Table 1, which show that STEP outperforms the baselines in both types of matrix games. More detailed results can be found in Appendix.

In Penalty, there are three NE points, and the only SE point corresponds to the optimal NE point. The challenge lies in the fact that during the exploration phase, any deviation from the optimal NE policy by any of the players will result in severe punishment for all players. The likelihood that agents choose the best NE action decreases as the penalty term increases. Consequently, MAPPO always falls into the trap of penalty and converges to the sub-optimal NE point $(a^{1}_{2},a^{2}_{2})$ at a rate of $100\%$, unless $k=0$. Similarly, when $k$ declines, MADDPG has a greater probability of converging to the sub-optimal NE point. Compared to the baselines, SE is the convergence objective for STEP. By computing the leader’s action, the follower is always able to select the best response action, allowing it to efficiently arrive at the optimal solution regardless of the configuration. Even under a high penalty of $k=-100$, there is a $93\%$ probability of convergence to the optimal value. Due to the use of neural network approximation, modest probability errors are permitted.

Table 1 displays the results of different methods in Mixing game. We evaluate the agents’ coordination based on the average return proposed by Zhang Zhang et al. (2018), as each agent has its own particular reward. It is clear that there are two optimal solutions in this scenario, with $(a^{1}_{1},a^{2}_{1})$ corresponding to the SE. In addition, the SE is Pareto superior to the only NE. As expected, STEP is able to rapidly converge to the SE with a probability of nearly $100\%$. In contrast, MAPPO and MADDPG are utterly ineffective and cannot even converge. It also demonstrates that STEP can effectively solve the problem of selecting multiple optimal solutions to some extent due to the certainty of the SE.

The Multi-Agent MuJoCo environment, which divides each part of the robot into separate agents that operate each part independently to achieve optimal motion, is used to assess the performance of the algorithm in a fully cooperative setting, as well as the impact of the regularization term coefficient on STEP’s performance. Specifically, to measure the collaboration quality of the agents, we choose representative cooperation scenarios such as Ant and HalfCheetah. And the results in Many-Agent Ant scenario can be found in Appendix due to the space limit.

Figure 6 demonstrates that STEP outperforms heterogeneous and homogeneous methods in all cases, which is certainly a result of the hierarchical structure among agents. Notably, the performance gap between STEP and HAPPO narrows as the number of agents increases. We think it’s because STEP achieves the effect of heterogeneous policies through parameter sharing. The number of STEP parameters remains constant regardless of the number of agents, whereas HAPPO with heterogeneous policies builds an individual policy network for each agent. In this sense, STEP shows a higher capacity to capture and represent diverse and intricate policies.

Figure 7 depicts the influence of various regularization coefficients on the experimental results. When the number of agents is small, the performance of various coefficients is similar. When there are more agents, however, greater coefficients lead to better results. The absence of regularization coefficients, on the other hand, has a negative impact on the experimental outcomes, demonstrating the efficiency of the regularization term. This outcome is expected and intuitive.

Lane merging in high-density, high-speed traffic presents a significant challenge for both autonomous vehicles and human drivers. To guarantee safe and efficient merging, vehicles must apply coordination mechanisms that allow them to merge at precise speeds and times. In this scenario, each agent has its own goal, and all of them seek to pass through the main road quickly. Due to the fact that HAPPO is built for a fully cooperative environment, we only employ MAPPO, IPPO, and MADDPG to compare with STEP in this mixed task scenario. The road conditions vary at different levels.

Results in Figure 8 show that STEP outperforms the other approaches in the mixed task scenario, with the performance gap between STEP and the other algorithms growing as traffic density and complexity rise. Additionally, the average speed of STEP, as illustrated in Figure 9, is also faster than that of the other algorithms after convergence occurs.