Cooperative Multi-Agent Deep Reinforcement Learning Methods for UAV-aided Mobile Edge Computing Networks

In practical UAV-aided MEC networks, the distributed operation is desirable so that observing states of environments and inferring actions for individual UAVs and IDs can be split across multiple UAVs and IDs. Existing centralized DRL methods [22, 23, 24, 25] combine all UAVs and IDs as a single agent to handle states and actions. Such a single-agent deep reinforcement learning (SADRL) approach brings state collection and decision-making processes in the central manner, which are infeasible for supporting a massive number of IDs. Also, a sole actor-critic NN can only be trained to a certain MEC network with a fixed number of UAVs and IDs, and thus it cannot be straightforwardly applied to larger MEC systems.

One viable approach for distributed UAV-aided MEC systems is multi-agent DRL (MADRL) [26, 27, 28, 29, 30, 31, 32, 33, 34, 35], which adopts multi-agent partially observable Markov decision processes (POMDP). Computing nodes at UAV servers and ground IDs are interpreted as agents which identify their actions based on partial observations of the overall MEC systems, e.g., channel state information, task volume, and locations. Various MADRL techniques such as the MAPPO [26, 27, 28], MADDPG [29, 30, 31, 32, 33] and MATD3 [34, 35] have shown their effectiveness for the decentralized management of UAVs and IDs.

As the multi-agent POMDP formalism lacks full knowledge of the entire MEC system, a coordination strategy among agents should be employed so that they can estimate the overall state from partial observations. Two popular methods include implicit coordination through reward shaping [26, 27, 28, 29, 30, 31, 32, 33] and exploit coordination through observation shaping [26, 27, 28, 29, 30, 31, 32, 33, 34, 35]. The former trains all the agents jointly using carefully designed reward functions such that individual agents can implicitly infer the knowledge of other agents in the training. Since agents can cooperate in the training step via long-term statistics, trained actor NNs are not able to accommodate highly fluctuating environments. In contrast, the latter allows agents to share messages, which are normally given as subsets or manipulations of partial observations. These messages can be conveyed through reliable control channels. Such an explicit message exchange mechanism can adapt to immediate environment changes at the expense of increased communication overhead.

Both approaches have proven to succeed in various configurations of UAV-based MEC networks. However, these approaches generally require time-consuming trial-and-error validation processes to check the feasibility of all possible combinations of partial observations and rewards. This challenge may become prohibitive as the dimension of observation statistics increases. Moreover, man-made agent coordination policies are not guaranteed to achieve good performance since input features of actor NNs normally resort to manual optimization. For these reasons, it is necessary to develop a new cooperative MADRL framework that can autonomously determine interaction strategies among agents, in particular, communication messages by leveraging NNs.

This paper investigates a cooperative MADRL for UAV-aided MEC networks where NNs at multiple agents determine their decision-making and coordination policies autonomously. We aim at minimizing the total energy consumption of ground IDs by offloading their computational tasks to a mobile edge server mounted on a UAV. This poses joint optimization of the trajectory and computing resource allocation of the UAV along with offloading decisions at IDs. These optimization variables need to be computed by the UAV and IDs individually. The considered problem is classified as a multi-agent POMDP where UAV and ID agents collaboratively identify their action variables only with partial observations. Compared to existing MADRL frameworks which require a handcraft design of UAV-ID coordination, the proposed scheme can handle highly fluctuating and heterogeneous network dynamics.

In this paper, we propose a novel cooperative multi-agent DDPG (C-MADDPG) framework where NNs at UAV and ID agents determine their policies by cooperating with other agents. Our system is first formulated as a cooperative multi-agent POMDP task which constructs two different actions for individual agents, namely, solution actions and message actions. The solution actions include optimization variables at agents, e.g., trajectory and resource allocation variables at the UAV agent and offloading variables at the ID agents. In addition, the message actions indicate communication messages to be exchanged among the UAV and ID agents. For effective information exchange, the proposed design establishes interactions in uplink (ID-to-UAV) and downlink (UAV-to-ID). Thus, along with the solution actions, the uplink message actions at IDs and the downlink message actions at the UAV are regarded as actions taken by actor NNs. This is a distinct feature of the proposed framework compared to conventional MADDPG approaches [36] where communication messages should be designed synthetically in advance and are fixed in all episodes of training steps.

In the proposed C-MADDPG, we build a solution actor NN and a message actor NN. In the inference stage, the UAV and ID agents calculate their solution actions by using the solution actor NN. Cooperative inference among these actor NNs establishes uplink-downlink coordination among the UAV and ID agents. To send encoded statistics of partial observations to the UAV agent, the ID agents utilize their message actor NNs that generate uplink message actions. These uplink message actions become an input to the message actor NN at the UAV agent, which creates the downlink message actions intended for individual ID agents. Once the coordination is completed, the UAV and ID agents calculate their solution actions by using the solution actor NNs. In this decision-making process, the message actions are leveraged as side inputs to the solution actor NNs so that the UAV and ID agents can collaboratively decide their optimization variables. Since the proposed cooperative inference does not need any centralized operations, we can construct decentralized optimization solutions for practical UAV-aided MEC networks.

In the proposed cooperative actor NN architecture, the UAV agent aggregates the uplink message actions sent by all ID agents. Therefore, the input dimension of actor NNs at the UAV agent, in particular, the message actor NN, scales with the number of IDs. For this reason, a naive NN structure whose input and output dimensions are fixed leads to poor generalization ability with respect to the ID population. To achieve the scalability, we exploit the concept of the graph attention network (GAT) [37], and adopt the parameter sharing technique where all ID agents utilize the identical actor NN. By doing so, the entire inference becomes independent of the number of IDs and can achieve the scalability.

The training process of the proposed C-MADDPG requests joint optimization of solution actor NNs, message actor NNs, and critic NNs. To this end, we adopt the centralized training decentralized execution (CTDE) strategy where all actor NNs are trained in an end-to-end manner under the supervision of the critic NN. The trained actor NNs are then deployed to the UAV and IDs for real-time decentralized inference. Consequently, the proposed parameter sharing policy can be implemented without additional communication overheads in the inference step. To further improve the scalablity of actor NNs, we develop a joint training process which leverages several episodes with the arbitrary number of ID agents. Also, we employ a random masking strategy that stochastically prunes input features of the critic NN. Numerical results validate the generalization ability of the proposed scheme and demonstrate the effectiveness of the proposed C-MADDPG framework over existing approaches.

The contributions of this paper are summarized as follows:

• •

  We propose a novel C-MADDPG framework which establishes self-organizing coordination strategies among UAV and ID agents. Compared to existing MADRL methods [26, 27, 28, 29, 30, 31, 32, 33, 34, 35] which design agent coordination messages manually, the proposed approach exploits message actor NNs to allow autonomous RL operations. This framework generates task-oriented agent interaction protocols that are optimized to enhance the expected reward function. Consequently, the proposed method does not require a handcrafted design of observations and rewards.

• •

  In practical UAV-aided MEC networks, the number of IDs may vary from time to time. This requests actor NNs whose inference calculations can be performed independent of the ID populations. To this end, we develop scalable actor NNs based on the parameter sharing strategy where all IDs leverage the identical NNs. However, a straightforward extension of the parameter sharing entails indistinguishable messages at all IDs. To address this difficulty, the GAT mechanism is employed which evaluates the importance of individual IDs and the resulting actor NNs successfully achieve the scalability to the number of IDs.

• •

  To further enhance the scalability, the training mechanism of the proposed C-MADDPG should be carefully designed such that shared actor NNs can observe various MEC configurations with arbitrary ID populations. Such randomized samples can be generated by using masking operations. We randomly prune IDs in the training step so that the actor NNs can be optimized over different MEC networks. This strategy helps the NNs to learn an efficient decentralized decision-making policy for arbitrary given ID populations.

The rest of the paper is organized as follows: Section II offers an overview of recent works on MADRL-based UAV MEC systems. In Section III, we describe a system model and formulate an optimization problem. Section IV provides a cooperative POMDP formulation. The proposed actor structure and its cooperative inference are presented in Section V. In Section VI, a joint training policy is introduced, and the performance evaluations are shown in Section VII. Finally, the paper is terminated with concluding remarks in Section VIII.

Let $\mathbf{q}_{j}^{(t)}=(q_{x,j}^{(t)},q_{y,j}^{(t)},0)\in\mathbb{R}^{3}$ be the 3D Cartesian coordinate of ID $j$ at time slot $t$ ($t=1,\cdots,T$). Mobile IDs change their positions time to time according to predefined missions. The randomness in ID positions can be modeled by the Gauss-Markov process [38]. At time slot $t$, the location of ID $j$ is written by

where $\tau$ represents the duration of a time slot and the speed $v_{j}^{(t)}$ and $o_{j}^{(t)}$ indicate speed and moving direction, respectively. Here, $v_{j}^{(t)}$ and $o_{j}^{(t)}$ are updated as

where $\kappa_{v}\in[0,1]$ and $\kappa_{o}\in[0,1]$ stand for the memory factors and $\bar{v}_{j}$ and $\bar{o}_{j}$ are the average speed and direction of ID $j$, respectively. The independent Gaussian random variables $\Phi_{v}\sim N(0,\varsigma_{v}^{2})$ and $\Phi_{o}\sim N(0,\varsigma_{o}^{2})$ characterize the randomness of the ID mobility. In the meantime, the UAV trajectory is optimized to enhance the MEC performance. Let us define $\beta^{(t)}\in[0,2\pi]$ and $\eta^{(t)}\in[0,\pi]$ respectively as the azimuth angle and the elevation angle of the UAV at time slot $t$. Then, the moving direction of the UAV $\mathbf{\Delta}^{(t)}\in\mathbb{R}^{3}$ can be expressed as

As a result, the 3D location vector of the UAV is obtained as

where $v^{(t)}\in[0,v_{\max}]$ equals the UAV velocity with $v_{\max}$ being the maximum speed constraint.

We define $P_{j}^{(t)}$ as the line of sight (LoS) probability given by [39]

where $K_{1}$ and $K_{2}$ are constants on the propagation environment and $\nu_{j}^{(t)}$ equals the elevation angle between the UAV and ID $j$. According to the air-to-ground propagation model [40, 41], the large-scale channel gain between the UAV and ID $j$ can be written by

where $\mathbf{q}_{j}^{(t)}$ is the 3D location vector of ID $j$, $\alpha$ represents the path loss exponent, $\rho_{0}$ indicates the reference path loss, and $\chi_{\text{LoS}}$ and $\chi_{\text{NLoS}}$ ($\chi_{\text{NLoS}}>\chi_{\text{LoS}}>1$) respectively account for the path loss of the LoS and non-LoS cases.

We employ the time division duplexing protocol where uplink and downlink communication are realized over the reciprocal channel. The uplink and downlink rates are respectively expressed by

where $p_{u}$ and $p_{d}$ equal the uplink and downlink transmit power at the IDs and the UAV, respectively, $B$ denotes the total bandwidth and $N_{0}$ stands for the noise power.

ID $j$ splits its task into $T$ subtasks each having $I_{j}/T$ bits. Each subtask is subject to be completed within one time slot of duration $\tau$. At the beginning of each time slot, the IDs determine their task offloading policies based on the partial offloading protocol. ID $j$ offloads $\lambda_{j}^{(t)}\in[0,1]$ portion of $I_{j}/T$ bits to the UAV server, whereas the remaining part $1-\lambda_{j}^{(t)}$ is processed locally. The energy consumption $E_{l_{j}}^{(t)}$ required for the local processing of $(1-\lambda_{j}^{(t)})I_{j}/T$ bits is written as [42]

where the constants $\vartheta$ and $C$ account for the hardware efficiency and the computational complexity, respectively.

To offload $\lambda_{j}^{(t)}\frac{{I}_{j}}{T}$ bits to the UAV, the communication energy $E_{o_{j}}^{(t)}$ of ID $j$ is given by

The computation capacity of the UAV is limited by the maximum CPU frequency $f_{\max}$. For parallel computations, a virtual machine (VM) [43] with the CPU frequency $f_{j}^{(t)}$ is dedicated to processing the task offloaded from ID $j$. This incurs the sum CPU frequency constraint as

The latency of the offloading procedure at ID $j$ comprises delays in the uplink task offloading from ID $j$ to the UAV, task computation at the UAV, and downlink transmission from the UAV to ID $j$ as

where the first term represents the uplink transmission delay of the offloaded task of size $\lambda_{j}^{(t)}I_{j}$ with $R_{u,j}^{(t)}$ bits/sec, and the second term indicates the computation latency with the CPU frequency $f_{j}^{(t)}$ cycles/sec, and the third term quantifies the downlink transmission delay for broadcasting the task of size $\delta\lambda_{j}^{(t)}I_{j}$ with $R_{d,j}^{(t)}$ bits/sec with the constant $\delta$ being the ratio of output to input task sizes. Finally, the latency constraint is imposed as

We aim at minimizing the total energy consumption of all IDs through the joint optimization of the UAV trajectory $\mathbf{U}=\{v^{(t)},\eta^{(t)},\beta^{(t)},\forall t\}$, computing resource allocation $\mathbf{F}=\{f_{j}^{(t)},\forall j,t\}$, and offloading ratio $\mathbf{\Lambda}=\{\lambda_{j}^{(t)},\forall j,t\}$. The total energy minimization problem can be formulated as

The above problem is a nonconvex problem due to the latency constraint and the objective function. Existing MADRL methods [26, 27, 28, 29, 30, 32, 31, 33, 34, 35] resort to computationally demanding exhaustive search processes for designing agent interaction mechanisms. To overcome this difficulty, we propose a novel cooperative MADRL framework that identifies optimization variables and coordination strategies autonomously by using NNs.

ID agent $j$ ($\forall j\in\mathcal{N}$) first obtains its message action $m^{(t)}_{j}$ by the message actor NN $\mu_{I}(\cdot;\varphi_{I})$ with trainable parameter $\varphi_{I}$ based on its partial observation $o_{j}^{(t)}$. The message action $m^{(t)}_{j}$ of ID agent $j$ ($\forall j\in\mathcal{N}$) is then expressed as

where the identical message actor $\mu_{I}(\cdot;\varphi_{I})$ is employed for all ID agents. Such a parameter sharing policy leads to a scalable structure so that a sole message actor can be universally applied to an arbitrary ID population.

A set of ID messages $\{m_{j}^{(t)}:\forall j\in\mathcal{N}\}$ are conveyed to the UAV agent through orthogonal uplink control channels. Without loss of the generality, each ID is assumed to be assigned by $M$ frequency resource blocks (RBs) to transmit its message action. To accommodate such a resource constraint, we design $m_{j}^{(t)}$ as an $M$-dimensional vector where the transmission of each element occupies one RB. As a result, total $NM$ RBs are needed for the uplink coordination.

After receiving the messages $\mathbf{m}_{I}^{(t)}\triangleq\{m_{j}^{(t)}:\forall j\in\mathcal{N}\}$, the UAV agent computes its message action $m_{0}^{(t)}$ to be multicast to all IDs through the downlink control channels. This UAV message action encodes the knowledge required for the decision processes of the solution action $x_{j}^{(t)}$ at ID agent $j\in\mathcal{N}$. To this end, the UAV agent combines its observation $o_{0}^{(t)}$ and the group of ID message actions $\mathbf{m}_{I}^{(t)}$. By doing so, the UAV successfully propagates partitioned information of individual IDs to the entire MEC network. Similar to the ID message actions, the UAV message actor $\mu_{U}(\cdot;\varphi_{U})$ with trainable parameter $\varphi_{U}$ is adopted to produce the UAV message action $m_{0}^{(t)}$ as

The dimension of the ID message actions $\mathbf{m}_{I}^{(t)}\in\mathbb{R}^{NM}$ scales with the number of IDs $N$. For this reason, a naive NN architecture, in particular, fully-connected layers, fails to achieve the scalability with respect to $N$.

To overcome this issue, we develop a scalable UAV message actor $\mu_{U}(\cdot;\varphi_{U})$ based on the GAT [37]. This framework modifies node interaction policies of the graph neural network (GNN) [44] such that each node can measure the importance of its neighbors, which is referred to as an attention score. As a result, the generalization capability can be fairly improved without sacrificing the scalability to the node population. Thus, the design goal of the message actor (23) is to aggregate the observation of the UAV agent $o_{0}^{(t)}$ and $\mathbf{m}_{I}^{(t)}$ based on the importance of individual IDs.

In general, the GAT facilitates multiple layers to extract useful features of input data. To realize such a layered GAT architecture, several communication rounds among the UAV and IDs are necessary to share the results of each GAT iteration. To avoid this issue, we modify conventional multi-iteration GAT architectures to a single-iteration GAT to leverage sole uplink-downlink coordination. Also, since the DRL involves temporal connections of actor NNs in consecutive time slots, our single-iteration GAT design becomes more powerful in the UAV-aided MEC systems.

A key enabler of the proposed GAT approach is to allow the UAV agent to have NN modules for computing the attention scores of all ID agents. By doing so, we can straightforwardly implement the GAT mechanism without propagating latent vectors of hidden layers multiple times. The UAV agent first extracts a hidden feature of agent $j\in\tilde{\mathcal{N}}$, denoted by $\mathbf{e}_{j}^{(t)}$ of length $E$, as

where $\epsilon_{U}(\cdot;\delta_{U})$ and $\epsilon_{I}(\cdot;\delta_{I})$ indicate the feature extractor NNs of the UAV agent and ID agents, respectively, which are responsible for generating $\mathbf{e}_{j}^{(t)}$ utilized for the computation of the attention scores. A group of feature vectors $\{\mathbf{e}_{j}^{(t)}:\forall j\in\tilde{\mathcal{N}}\}$ is adopted as an input to the GAT operation. A scalar attention score $z_{j,k}^{(t)}\in[0,1]$ about agent $k$ measured by agent $j$ is calculated as

where $\epsilon_{A}(\cdot;\delta_{A})$ stands for the attention NN that evaluates the affinity of two different agents. The attention score $z_{j,k}^{(t)}$ interprets the importance of agent $k$ for the decision-making process at agent $j$. Finally, the output of the GAT for agent $j$ becomes the weighted average of the feature vectors $\{\mathbf{e}_{j}^{(t)}:\forall j\in\tilde{\mathcal{N}}\}$ with coefficients $\{z_{j,k}^{(t)}:\forall j\in\tilde{\mathcal{N}}\}$ as

Notice that in conventional GAT, each agent has its dedicated feature extractor NN. Thus, to obtain the attention score $z_{j,k}^{(t)}$ in (25), ID agent $j$ should know all the feature vectors $\mathbf{e}_{k}^{(t)}$ ($\forall k\in\tilde{\mathcal{N}}$) which is not viable with sole uplink cooperation from IDs to the UAV. This can be addressed by allowing the UAV agent to reuse the feature extractor NNs $\epsilon_{I}(\cdot;\delta_{I})$ for all IDs.

Thanks to the GAT mechanism, $\mathbf{w}_{j}^{(t)}$ encodes sufficient statistics required at agent $j$ to take its solution actions. Thus, they can be utilized as an input of solution actor NNs. As will be discussed, $\mathbf{w}_{0}^{(t)}$ is leveraged internally to find the solution action of the UAV agent $x_{0}^{(t)}$. In contrast, the remaining vectors $\mathbf{w}_{j}^{(t)}$ ($\forall j\in\mathcal{N}$) need to be sent to the associated ID agents. To this end, the UAV message action $m_{0}^{(t)}$ is designed as

The UAV multicasts $m_{0}^{(t)}\in\mathbb{R}^{NE}$ to the IDs, which occupies $NE$ RBs for the downlink coordination.

The UAV message $m_{0}^{(t)}$ encapsulates the network state $s^{(t)}$, i.e., the set of all observations of the UAV and ID agents. Therefore, it is sufficient for all agents to determine their solution actions by leveraging $m_{0}^{(t)}$ only. Let $\pi_{U}(\cdot;\theta_{U})$ be the solution actor NN at the UAV agent with the learnable parameter $\theta_{U}$. The UAV solution action $x_{0}^{(t)}$ is obtained as

As shown in (18), the solution action of the UAV agent $x_{0}^{(t)}$ contains two types of optimization variables, i.e., the trajectory $\mathbf{v}^{(t)}$ and computing resource allocation $\mathbf{f}^{(t)}$. To yield such heterogeneous actions, the solution actor $\pi_{U}(\cdot;\theta_{U})$ comprises two component NNs $\gamma_{V}(\cdot;\zeta_{V})$ and $\gamma_{F}(\cdot;\zeta_{F})$ for calculating $\mathbf{v}^{(t)}$ and $\mathbf{f}^{(t)}$, respectively. Then, the trainable parameter set of the solution actor NN of the UAV becomes $\theta_{U}=\{\zeta_{V},\zeta_{F}\}$.

The trajectory variable is computed as

where an input is a concatenation of the UAV information vector $\mathbf{w}_{0}^{(t)}\in\mathbb{R}^{E}$ from the GAT (25) and the sum of the ID information vectors $\sum_{j\in\mathcal{N}}\mathbf{w}_{j}^{(t)}\in\mathbb{R}^{E}$. Since the dimension of these input vectors is independent of $N$, (29) preserves the scalability with respect to the ID population. The distinct input $\mathbf{w}_{0}^{(t)}$ helps the NN $\gamma_{V}(\cdot;\zeta_{V})$ distinguish the UAV information vector with those of the IDs $\sum_{j\in\mathcal{N}}\mathbf{w}_{j}$. Consequently, we can successfully produce the UAV-specific trajectory action $\mathbf{v}^{(t)}$ based on the aggregated ID information $\sum_{j\in\mathcal{N}}\mathbf{w}_{j}$.

Next, to determine the computing resource allocation $\mathbf{f}^{(t)}$, a sole NN produces each CPU frequency variable $f_{j}^{(t)}$ based on the associated ID agent information vector $\mathbf{w}_{j}^{(t)}$. To this end, we first calculate an intermediate value $\tilde{f}_{j}^{(t)}$ as