Navigation in a simplified Urban Flow through Deep Reinforcement Learning

The main features of an MDP are a state space $S$, and initial state space $S_{0}$ with an initial state distribution $p(s_{0})$, an action state space $A$, a state transition probability distribution $p(s_{t+1}|s_{t},a_{t})$ which satisfies the Markov property, and a reward function $r(s_{t},a_{t}):S\times A\rightarrow R$. The reward function provides the feedback of the environments when executing the action $a_{t}$ in a state $s_{t}$. State and action spaces can be continuous or discrete. These give a first indication about how to build the architecture and the network associated with the DRL algorithm. RL is generally used to solve a MDP. An optimal policy is learned through a trial-and-error process, and the policy could be stochastic or deterministic. A stochastic policy $a\sim\pi(\cdot|s):S\rightarrow P(A)$ returns the probability density of available state and action pairs ($s,a$), where $P(A)$ is the set of probability measures on $A$. Note that a deterministic policy $\mu(S):S\rightarrow A$ only projects states into actions.
A POMDP is characterized by the fact that the agent cannot directly observe the state $s_{t}$, but receives a set of observations $o_{t}$ with a distribution $p(o_{t}|s_{t})$. The sequence of observations do not satisfy the Markov property, since $p(o_{t+1}|a_{t},o_{t},a_{t-1},o_{t-1},...,o_{0})\neq p(o_{t+1}|o_{t},a_{t})$. Consequently, the agent has to infer the current state $s_{t}$ based on the history of trajectories.
In this work, we exploit in particular the ability of LSTMs to capture temporal dependencies in a POMDP, in combination with PPO, in an environment characterized by a partial observability, in order to design an algorithm powerful enough to be prone to extension to a 3D problem and to even more uncertain environments. The detailed description of policy-gradient methods, PPO, LSTMs and their combination is described in detail in the Appendix.

UAV navigation is typically described with a set of non-linear differential equations in a three-dimensional space. Since in the present work the flow field is two dimensional, the UAV is modelled as a mass point and the set of non-linear equations is reduced to a 2D space. The variables describing the UAV motion here are: position vector ($\boldsymbol{x}$), velocity vector($\boldsymbol{v}_{\textrm{global}}$), orientation (heading angle, $\theta$) and angular velocity ($\omega$). The dynamics and state of the UAV are significantly influenced by the underlying flow field. The equations of motion take into account the presence of the surrounding flow field with velocity components $u_{\text{flow}}$ and $v_{\text{flow}}$, and are given by:

$$\begin{cases}\frac{\mathrm{d}\boldsymbol{x}}{\mathrm{dt}}&=\boldsymbol{v}_{\textrm{global}}+\boldsymbol{v}_{\textrm{flow}},\\ \frac{\boldsymbol{v}_{\text{global}}}{\mathrm{dt}}&=a\begin{pmatrix}\cos(\theta)\\ \sin(\theta)\end{pmatrix},\\ \frac{\mathrm{d}\theta}{\mathrm{dt}}&=\omega,\\ \frac{\mathrm{d}\omega}{\mathrm{dt}}&=\dot{\omega},\end{cases}\$$

(1)

where $\vec{v}_{\text{flow}}=\begin{pmatrix}u_{\text{flow}}\\ v_{\text{flow}}\end{pmatrix}$ represents the velocity of the flow field at the UAV’s position, while $a$ and $\dot{\omega}$ represent the linear and angular accelerations, respectively. $a$ and $\dot{\omega}$ are also the controls which represent the actions taken bu the UAV when interacting with the two dimensional flow-field.
To solve the equations of motion, the fourth-order Runge–Kutta method is used, yielding the state of the UAV at the next time step.

Both observation and action spaces are continuous. The observation space is desgined as follows:

$$o=\left\{\theta,\phi,d_{0}\right\}+\left\{\beta_{i}\right\},$$

(2)

where $\theta$ is the heading angle of the UAV, $\phi$ is the relative angle of the UAV with respect to the target, $d_{0}$ is the distance between of the UAV and the target and $\left\{\beta_{i}\right\}$ are the are the angles associated with the sensors for obstacle detection, with $i\in[0,8]$ spanning the angles between $-\pi$ and $\pi$. Figure 2 shows graphically the definition of the observation space.

The action space is designed to include the linear and angular accelerations of the UAV, where $a\in[-3.0,3.0]$ and $\dot{\omega}\in[-\pi/4,\pi/4]$, while setting constraints to the velocity, which is bounded in magnitude, $v_{\textrm{UAV}}\in[-1.4v_{\textrm{max}},1.4v_{\textrm{max}}]$, where $v_{\textrm{max}}$ is the magnitude of the maximum velocity of the flow field across all the frames used by the algorithm.
From the formulation of the observation space, it can be observed that obstacle detection is achieved by providing to the agent a set of directions, since in UAVs the relation with the surroundings is typically given by images from cameras, radar signals or range finders. In this work, we take as inputs for the observation space the angles which represent the orientation of rays sent by range finders mounted on the UAV. Obstacle detection is achieved by implementing a ray-tracing technique ([47]). First of all, the UAV has to check for free space in its perspective. The input is the position of the UAV and the output is a boolean variable which indicates whether the path is free from obstacles or not. Then, if the obstacle is present, the intersection with the traced rays is computed. First, the direction of the ray is calculated, based on the ray origin and final point, as well as the coordinates of the obstacles. Then, it is verified whether parallel directions to the obstacles are present. If the detected directions are not parallel to the obstacles, the intersection point between the ray and the obstacle is calculated and the distance to the intersection is returned. Figure 3 sketches the process.

The starting and target areas are chosen randomly before the first obstacle and after the second obstacle, respectively. The agent is allowed to take a maximum of 80 steps in the environment for each episode. The starting frame of the simulation is random, meaning that the navigation task does not always start from the same simulation data, but changes randomly for every episode, so that the initial conditions of the flow field themselves exhibit uncertainties.
The UAV state is described as follows:

$$s=\left\{\boldsymbol{x},\theta,\boldsymbol{v},\omega\right\},$$

(3)

where $\boldsymbol{x}$ is the position vector, $\theta$ is the heading angle, $\boldsymbol{v}$ is the velocity vector and $\omega$ is the angular velocity.
The state of the UAV is inferred from the observations, which are given as input to the NN and described in Equation (2). The environment has been implemented using the format of OpenAI Gymnasium ([49]).

As mentioned in Section 1, DRL is a process that encourages learning by trial and error and this process is triggered by a reward which is given to the agent when it takes the right actions to complete the assigned task. The structure of the reward is crucial because this guides the agent towards a more effective learning, so this component of the algorithm has to be carefully designed and tuned for a specific task. The reward structure of this work is inspired by other obstacle-avoidance and navigation problems ([66], [54], [19]).
The reward structure is designed to guide the UAV towards the target while minimizing collisions with obstacles, reducing energy consumption and preventing leaving the designated operational bounds. The reward function is constructed from several components, each one addressing a different aspect of the UAV’s performance. The final reward is a sum of the following components:

• •

  Transition reward $r_{trans}$: it describes the progression of the UAV towards the target, and is defined as:

  $$r_{\mathrm{trans}}=\sigma d_{\mathrm{dist}},$$

  (4)

  where $\sigma\in\mathbb{R}$ is a scaling factor to weight the influence of this term in the total reward, and $d_{\textrm{dist}}$ is the reduced distance from the starting point to the target:

  $$d_{\mathrm{dist}}=\left\|\ x_{t-1}-x_{\mathrm{target}}\right\|-\left\|\ x_{t}-x_{\mathrm{target}}\right\|,$$

  (5)

  where $x_{t-1}$ is the position of the UAV at the previous time step, $x_{\textrm{target}}$ is the position of the target point and $x_{t}$ is the current position of the UAV;

• •

  Obstacle penalty $r_{\textrm{obs}}$: it detects how close the UAV is to the obstacle. The closest the UAV is to the obstacle, the larger the value of the penalty, described here as:

  $$r_{\textrm{obs}}=-\alpha e^{-\psi d_{\textrm{min}}},$$

  (6)

  where $\alpha$, $\psi$ $\in\mathbb{R}$ are constants, and $d_{\textrm{min}}$ = $\textrm{min}\left\{d_{1},..,d_{n}\right\}$, with $n\in$(0,8) being the minimum of the distances between the UAV and the obstacles provided by the sensor measurement.

• •

  Free-space reward $r_{\textrm{free}}$: it is the reward assigned for navigating in a direction free of obstacles:

  $$r_{\textrm{free}}=\begin{cases}r_{\textrm{free}}&\text{if free space ahead is detected}\\ 0&\text{otherwise},\end{cases}$$

  (7)

  where $r_{\textrm{free}}\in\mathbb{R}$ is a constant.

• •

  Best-direction free-space reward $r_{\textrm{best}}$: if the path ahead is not free of obstacles, the agent chooses an alternative direction:

  $$r_{\textrm{best}}=\begin{cases}\zeta\beta_{\textrm{best}}&\text{if no free space ahead}\\ 0&\text{otherwise},\end{cases}$$

  (8)

  where $\zeta\in\mathbb{R}$ is a constant and $\beta_{\textrm{best}}$ is the angle which returns a direction representing the maximum distance from the obstacle in the set detected by the sensors;

• •

  Step penalty $r_{\textrm{step}}$: it is the penalty given for each step taken by the agent, which encourages the UAV to find the shortest path to the target, $r_{\textrm{step}}\in\mathbb{R}$;

• •

  Energy penalty $r_{\textrm{energy}}$: it accounts for the UAV’s propulsion energy consumption. It is calculated as:

  $$r_{\textrm{energy}}=-\kappa||\mathbf{v_{\textrm{prop}}}||,$$

  (9)

  where $\mathbf{v_{\textrm{prop}}}$ is the propulsion velocity, defined as the difference between the UAV’s current velocity and the flow velocity. This penalty discourages excessive energy use by reducing the reward proportionally to the effort required to move relative to the surrounding flow.

The total reward is then given by:

$$R_{\textrm{tot}}=\sum_{i=0}^{m}r_{\textrm{trans}_{i}}+r_{\textrm{obs}_{i}}+r_{\textrm{free}_{i}}+r_{\textrm{best}_{i}}+r_{\textrm{step}_{i}}+r_{\textrm{energy}_{i}},$$

(10)

where $m\in\mathbb{N}$ is the number of steps taken by the agent in the environment. At the end of each episode of $m$ steps, additional terms are considered. If the agent reaches the target it gets an extra reward, whereas if it hits an obstacle it receives an extra penalty, as well as if it hits the bounds of the domain.

The algorithm used in this work has been completely developed in the context of this work, and is written in PyTorch ([29]). Figure 4 shows a sketch of the network structure.

The first layer takes as input the vector of observations ($\boldsymbol{o_{\textrm{k}}}$) described in Section 2.4 and feeds the raw observations into the first hidden layer of the neural network. The first hidden layer is a fully connected layer with 64 neurons and a hyperbolic tangent (tanh) activation function, which allows to capture the non-linear relationships in the data. The second hidden layer is a again a fully connected layer, with a reduced size of 32 neurons, again with a tanh activation function. The LSTM layer follows the two dense layers, with 16 neurons. It processes the temporal features and models the temporal dynamics of the problem. The LSTM layer takes as additional inputs also the last reward ($R_{k-1}$) and last actions ($\boldsymbol{a_{\textrm{k-1}}}$), thus benefiting from the memory of past experiences. Finally, the network outputs are divided into two distinct streams. The first stream produces parameters for each action in the action space, specifically the mean ($\mu$) and standard deviation ($\sigma$), which are used for sampling actions in a stochastic policy. The second stream outputs a single scalar value representing the value function, which estimates the expected return from the current state. The output layer uses a linear activation function, appropriate for both continuous action distribution parameters and value estimation. Note that $\boldsymbol{c}$ and $\boldsymbol{h}$ represent the parameters of the hidden state which are fed and then given as output of the LSTM layer.
The hyperparameters of the algorithm were tuned by a trial-and-error procedure and are summarized in Table 1. The Adam optimizer ([20]) is chosen because of its robustness and adaptive property.

The results obtained with a PPO+LSTM approach are compared with the results obtained applying a TD3 and a traditional PPO algorithm, also custom implemented, which are widely used for similar tasks in less complex environments. The structures of these networks are described in Figure 5.

For Figure 5(a), the same nomenclature as for Figure 4 applies. Figure 5(b) describes the classical structure of a TD3 architecture. In our implementation, the actor network comprises 5 fully connected layers with 256, 128, 64, 32, and 2 nodes, respectively. Rectified-linear-unit (ReLU) activations are applied after each of the first four layers, with the final layer using a tanh activation scaled by the maximum action value. The critic network also has 5 fully connected layers with 256, 128, 64, 32, and 1 node. It takes both the state and action as input, concatenates them, and processes the combined input through ReLU activations after each layer to output a single Q value. The learning rates for the actor and critic networks are set to $10^{-4}$ and $10^{-3}$, respectively, for both PPO and TD3.

Policy-gradient methods parameterise and optimize policies by maximizing the expected returns explicitly, unlike value-based methods that derive policies indirectly by a value function.
In policy-gradient methods, the expected cumulative reward has to be maximized as:

$$J(\theta)=\mathbb{E}_{\pi_{\theta}}\Bigg{[}\sum_{t=0}^{T}\gamma^{t}r(s_{t},a_{t})\Bigg{]},$$

(12)

where $\gamma^{t}$ is the discount factor raised to the power of $t$, which discounts the reward $r(s_{t},a_{t})$ received at time $t$.
The policy-gradient theorem yields the gradient of the objective function calculated with respect to the policy characterized by the set of weights of the NN $\theta$ ([42]):

$$\nabla_{\theta}J_{\theta}=\mathbb{E}_{\pi_{\theta}}[\nabla_{\theta}\textrm{log}\pi_{\theta}(a|s)Q^{\pi_{\theta}}(s,a)],$$

(13)

with $Q^{\pi_{\theta}}(s,a)$ being the action-value function under the policy $\pi_{\theta}$.
In this work we use a stochastic policy in the training steps. The output of the policy will be the covariance ${\sigma}_{\theta}$ and the mean value ${\mu}_{\theta}$ of the Gaussian distribution.

The proximal-policy optimization (PPO) is an advanced policy-gradient method which uses a first-order trust region to limit the policy update step size and prevent too large variations of the policy, balancing at the same time exploration and exploitation ([38]). This is achieved by clipping to zero the probability that the new policy $\pi_{\theta}$ deviates more than a given $\epsilon$ from the previous policy $\pi_{\theta_{\textrm{old}}}$ as follows:

$$\begin{split}J^{\textrm{clipped}}(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\Bigg{[}\frac{1}{K}\sum_{k=0}^{K-1}\textrm{min}(\tilde{r}_{k}(\theta)A^{\pi_{\theta}}(\boldsymbol{y}_{k},\boldsymbol{u}_{k}),\\ \textrm{clip}(\tilde{r}_{k}(\theta),1-\epsilon,1+\epsilon)A^{\pi_{\theta}}(\boldsymbol{y}_{k},\boldsymbol{u}_{k}))\Bigg{]},\end{split}$$

(14)

with:

$$\tilde{r}_{k}(\theta)=\frac{\pi_{\theta}(\boldsymbol{u}_{k}|\boldsymbol{y}_{k})}{\pi_{\textrm{old}}(\boldsymbol{u}_{k}|\boldsymbol{y}_{k})},$$

(15)

being the ratio between the new and the old policy and:

$$A^{\pi_{\theta}}(\boldsymbol{y}_{k},\boldsymbol{u}_{k})=Q^{\pi_{\theta}}(\boldsymbol{y}_{k},\boldsymbol{u}_{k})-V^{\pi_{\theta}}(\boldsymbol{y}_{k}),$$

(16)

the advantage function. $\epsilon$ is the clip range, which prevents dramatic changes in the policy.
The advantage function represents the average improvement in the cumulative reward by selecting a specific action. It is usually computed using an estimate of the value function $V_{\phi}$ by the generalized advantage estimator (GAE) ([37]):

$$A_{\phi,k}=\sum_{k^{\prime}=k}^{K-1}(\gamma\lambda)^{k^{\prime}-k}\delta_{\phi,k^{\prime}},$$

(17)

where

$$\delta_{\phi,k}=R_{k}+\gamma V_{\phi,k+1}-V_{\phi,k}$$

(18)

is the temporal difference residual of $V_{\phi,k}$ with a discount $\gamma$ and $\lambda\in[0,1]$ is the GAE factor.
The mean-squared error (MSE) $L^{\textrm{value}}(\phi)$ between the value function estimate $V_{\phi}$ and the cumulative reward calculated from step $k$ it is the loss function used to update the critic network:

$$L^{\textrm{value}}(\theta,\phi)=\mathbb{E}_{\tau\sim\pi_{\theta}}\Bigg{[}\frac{1}{K}\sum_{k=0}^{K-1}\Bigg{(}V_{\phi,k}-\sum_{k^{\prime}=k}^{K}R_{k^{\prime}}\Bigg{)}^{2}\Bigg{]},$$

(19)

If the value function estimate has both the actor and critic roles, being then an additional output of the policy network, $\phi=\theta$ and the MSE is added to the clipped objective function to get the final PPO:

$$J^{\textrm{tot}}(\theta)=J^{\textrm{clipped}}(\theta)-\lambda_{v}L^{\textrm{value}}(\theta),$$

(20)

with $\lambda_{v}$ being the value-function coefficient, which measures the weight of the last term in the total objective function.
The steps of the algorithm are the following: first an initial policy $\pi_{\theta}$ and a value function $V_{\phi}$ are initialized. Then the interaction with the environment starts to gather a set of trajectories $\left\{s_{t},a_{t},r_{t},s_{t+1}\right\}$ and the advantage function is estimated using GAE to enhance stability and reduce the variance of the policy throughout the training. The clipped objective function is then used to update the policy parameters and the value function parameters are adjusted to minimize the MSE. This process is then repeated until convergence of the policy is achieved, meaning that the reward does not oscillate anymore, so that the calculated standard deviation stays withing acceptable bounds and the variance reduces over time.

The effectivenss of PPO can be enhanced by implementing LSTM cells, especially when dealing with environments in which sequential data and temporal dependencies are crucial, as it is in trajectory-optimization problems ([14]). LSTMs are recurrent neural networks (RNNs) that handle sequences and capture long-term dependencies, making them suitable for problems which require memory of previous states.
The drawbacks of traditional RNNs are addressed by LSTM cells which can keep information over long periods. LSTM cells have three main components: a forget gate, an input gate and an output gate. The forget gate decides which information has to be discarded from the cell state, and it is defined as:

$$f_{t}=\sigma(W_{f}\cdot[h_{t-1},x_{t}]+b_{f}),$$

(21)

with $\sigma$ is the sigmoid function, $W_{f}$ are the weights, $h_{t-1}$ the previous hidden state, $x_{t}$ the current input and $b_{f}$ the bias, respectively.
The input gate provides the new information which has to be added to the cell state. It is defined as:

$$i_{t}=\sigma(W_{i}\cdot[h_{t-1},x_{t}]+b_{i}),$$

(22)

and the candidate cell state $\tilde{C}_{t}$ can be expressed as:

$$\tilde{C}_{t}=\textrm{tanh}(W_{C}\cdot[h_{t-1},x_{t}]+b_{C})$$

(23)

The new cell state is then updated as:

$$C_{t}=f_{t}\odot C_{t-1}+i_{t}\odot\tilde{C}_{t},$$

(24)

where $\odot$ indicates an element-wise multiplication.
The output gate controls the output based on the cell state:

$$o_{t}=\sigma(W_{o}\cdot[h_{t-1},x_{t}]+b_{o}),$$

(25)

$$h_{t}=o_{t}\odot\textrm{tanh}(C_{t}),$$

(26)

where $o_{t}$ denotes the output state and $h_{t}$ the new hidden state respectively.
The combination of these components enables LSTMs to effectively capture temporal dependencies and mitigate the vanishing-gradient problem common in standard RNNs.
When applying LSTMs to PPO, these cells have to be integrated both in the policy and value networks. This allows the agent to learn also from past experiences, which is crucial when dealing with POMDPs. In fact, many real-world scenarios are not represented by a fully observable state of the environment. LSTMs enhance the agents’ behavior by keeping information from past observations and the agents can then infer missing information and take actions based on more informed decisions.
Combining LSTM networks with PPO gives several advantages:

• •

  The agent can learn from temporal sequences, leading to a better-informed decision-making process;

• •

  LSTMs make the whole process more suitable for complex environments with partial observability, leveraging the long-term dependencies they are able to learn;

• •

  The agent can store and reuse past experiences and this a key feature for problems which are characterized by temporal dependencies.

On the other hand, they exhibit some drawbacks. In particular, the computational cost increases, they require a careful hyperparameter tuning and stabilizing the training can be more challenging due to the interaction between PPO and LSTM networks.