\modelname: Hyperbolic Planning and Curiosity for Crowd Navigation

Early studies in crowd navigation [5, 13, 14] propose models describing agents’ behavior within crowds. These studies introduce fundamental concepts such as Social Forces [5] and provide models that guarantee collision avoidance in case of perfect information even for multiple agents simultaneously [14]. The approaches only employ past information about the crowd motion and do not explicitly model the future trajectories of the obstacles. A second line of work [15, 16, 17] focuses on predicting the trajectories of the dynamic obstacles to devise a safe path for the robot. While effective in terms of predictions, these techniques require a lot of computation, especially when dealing with large crowds, causing the decision-making model to be too slow for real-time applications. Chen et al. [18] describe the decision-making problem as an MDP and proposes to adopt DRL to solve the task. Following this approach, Chen et al. [4] extend the agent’s attention from human-robot interaction to a more comprehensive crowd-robot interaction, explicitly modeling the human-robot and human-human interactions. SG-D3QN [7] introduces a social attention mechanism to retrieve a graph representation of the crowd-robot state, which can be further improved [10, 6] by relying on intrinsic rewards [12, 19] and spatio-temporal maps for environment representations. Our work takes inspiration from Martinez et al. [10] and introduces hyperbolic latent representations to encode the MDP states, enabling us to achieve high performance with low-dimensional embeddings.

Exploration in Crowd Navigation is paramount for the autonomous agent to learn nuanced decision-making, balancing the need to navigate close to obstacles for efficiency with ensuring safety in populated environments. ICM [12] and RE3 [19] are recent methods that implement this strategy providing the policy network with “bonus rewards” when the agent visits unknown spaces. In ICM, a small network is fed with the current state and the action the agent is performing, and it is tasked to predict the representation of the subsequent state. The more the error (meaning that the state is unknown), the more the intrinsic reward. This work extends ICM, adopting hyperbolic state representations to increase further the reward for exploring novel states.

Hyperbolic Neural Networks (HNNs) are emerging as a powerful tool for capturing hierarchical representations, yielding compact representations, dealing with uncertainty, and much more [20, 9]. Recent works in computer vision for example demonstrate the potential of HNNs, outperforming traditional Euclidean models in several tasks [21, 22, 23]. Recently, Cetin et al. [24], acknowledge that MDPs can be represented as tree graphs in the states space, thus they advocate for using hyperbolic latent representations in the domain of DRL, as they natively extract hierarchical features from the data. They demonstrate the advantages of coupling hyperbolic learning with DRL methods on Procgen [25] and Atari-100k [26], reporting performance improvements on a wide range of test environments. They define a hybrid network with an Euclidean backbone and hyperbolic projective and classification layers. This work, for the first time applies the insights of Cetin et al. [24] to the crowd navigation problem. We furthermore introduce new value and curiosity modules that operate entirely in hyperbolic space. Moreover, we investigate the efficacy of hyperbolic intrinsic rewards and interpretability by analyzing hyperbolic features.

The crowd navigation problem consists of finding the optimal policy that drives an autonomous agent to a goal position in a crowd. For our simulation, we leverage the widely adopted CrowdNav simulator [4], see e.g.  [4, 7, 6, 10, 18, 27]. In this environment, the dynamic obstacles are informed of each other agent’s state (apart from the robot’s state) to avoid reciprocal collisions. Those states encompass their current position ($p\in\mathbb{R}^{2}$), their velocity ($v\in\mathbb{R}^{2}$), and the radius $r$ used as a 2D proxy of their volume. Thus, we describe the $i$-th obstacle visible state at time $t$ as:

$$w_{t}^{i}=[p_{x},p_{y},v_{x},v_{y},r]$$

(1)

and refer to the state of all obstacles as $w_{t}^{h}=[w_{t}^{1},\cdots,w_{t}^{n}]$ The robot is designed as a holonomic robot, and its state encompasses the current position ($p\in\mathbb{R}^{2}$) and velocity ($v\in\mathbb{R}^{2}$), its radius $r$, the maximum scalar velocity $v_{M}$, the current steering angle $\theta$ and the goal position ($g\in\mathbb{R}^{2}$), such that:

$$w_{t}^{r}=[p^{r}_{x},p^{r}_{y},v_{x},v_{y},r,v_{M},\theta]$$

(2)

At each timestep, the robot senses the environment’s visible state, i.e., $w_{t}=[w_{t}^{r},w_{t}^{h}]$. The autonomous agent moves according to the policy provided by our RL algorithm, which allows the models to steer along 16 equally spaced angles in $[0,2\pi)$ and move at 5 different velocities or stay still, resulting in 81 possible actions in each state. In the following, we define $\mathcal{A}$ as the set of allowed actions for the robot.

Following leading approaches [4, 6, 7, 10], we cast the problem of crowd navigation as an MDP, where at each timestep $t$ the robot has to perform the optimal action ($a_{t}^{*}\in\mathcal{A}$) according to the current visible state. The problem is set as finding the optimal deterministic policy ${\pi^{*}:w_{t}\mapsto a^{*}_{t}}$ that associates the optimal action to the state $w_{t}$ for every timestep $t$. For retrieving the optimal policy, we define $Q^{*}(w,a)$ as the optimal state-action function describing the expected value of taking a particular action $a$ being in a certain state $w$. $Q^{*}(w,a)$ satisfies the Bellman equation:

$$Q^{*}(w,a)=\sum_{w^{\prime},r}\mathbf{P}(w^{\prime},r|w,a)[r+\gamma^{t}\max_{a^{\prime}}Q^{*}(w^{\prime},a^{\prime})]$$

(3)

where $w^{\prime}$ is the state following $w$ after performing the action $a$, $r$ is the extrinsic reward provided by the environment, and $\gamma\in(0,1)$ is the discount factor that adjusts the interset for future rewards. The optimal policy $\pi^{*}$ is then defined as:

$$\pi^{*}(w_{t})=\arg\max_{a}Q^{*}(w_{t},a)$$

(4)

From Eq. 3, it follows that accurately defining the reward is a key factor for RL algorithms. The extrinsic reward ($r^{e}_{t}=r^{e}(w_{t})$) should be shaped to encourage the agent to reach its goal fast while avoiding collisions and creating discomfort to the human dynamic obstacles. As done in [10], we formalize it as follows:

$$r^{e}_{t}=\begin{cases}0.25&\text{if goal reached}\\ -0.25&\text{if collision}\\ -0.2\ d_{t}^{g}+\sum_{i=0}^{N}f(d^{i}_{t})&\text{otherwise}\end{cases}$$

(5)

where $d_{t}^{g}$ and $d^{i}_{t}$ represent the current distance from the goal and from the $i$-th obstacle, respectively. The summation term promotes a safety constraint, increasing the reward if the robot maintains a minimal distance from each obstacle. Thus, we devise $f$ as:

$$f(d^{i}_{t})=\begin{cases}d^{i}_{t}-0.2&d^{i}_{t}<0.2\\ 0&\text{otherwise}\end{cases}$$

(6)

This paper advocates for hyperbolic learning to perform crowd navigation. A hyperbolic metric space is a Riemannian manifold with constant negative curvature $-c$ (in this paper, we consider $c=1$ unless explicitly stated) [11]. Its definition encompasses several isometric models, among which we select the Poincaré ball $\mathbb{D}^{n}$, following recent literature [24, 21, 23, 28, 29]. The Poincaré ball is an open ball of radius $1$, $\mathbb{D}^{n}=\{x\in\mathbb{R}^{n+1}:\ ||x||<1\}$, endowed with the Riemannian metric:

$$g_{x}^{\mathbb{D}}=\frac{2I_{n}}{1-||x||^{2}}$$

(7)

with $I^{n}$ an $n\times n$ Identity matrix. Important in this paper is the ability to map points between hyperbolic space and its tangent space (i.e. Euclidean space), which are given by the exponential and logarithmic mapping functions. Given a point $v\in\mathbb{D}^{n}$, the exponential map with basepoint $v$ $\exp_{v}$ projects any point from the tangent space of $\mathbb{D}^{n}$ in $u$ $u\in T_{v}(\mathbb{D}^{n})$ into $\mathbb{D}^{n}$:

$$\exp_{v}(u)=v\oplus(\tanh(\frac{||u||}{1-||v||^{2}})\frac{u}{||u||})$$

(8)

where $\oplus$ represent the Möbius addition [11]. As a common practice, we consider the origin $O$ of the Poincaré ball to be the basepoint for projections. This simplifies Eq. 8 to:

$$\exp_{O}(u)=\tanh(||u||)\frac{u}{||u||}$$

(9)

Inversely, the logarithmic map with basepoint $O$, $\log_{O}$, is defined as:

$$\log_{O}(v)=\tanh^{-1}(||v||)\frac{v}{||v||}$$

(10)

The distance between points on the Poincaré ball is:

$$d_{\mathbb{D}}(x,y)=\mathrm{arcosh}\ {(1+\frac{2||x-y||}{(1-||x||^{2})(1-||y||^{2})})}$$

(11)

Eq. 11, highlights the growth of the volume in hyperbolic space: when a point approaches the boundary of $\mathbb{D}^{n}$, its norm grows exponentially. Leveraging the base operations described in Ganea et al. [11], namely Möbius addition and Möbius matrix multiplication, we can extend the main modules commonly used for deep learning. In this work, we adopt hyperbolic multi-layer perceptrons (h-MLPs), which rely on Möbius matrix multiplication and addition.

We introduce a model to navigate autonomous agents in dynamic environments. For this, we employ graph encoding for the current state, followed by a value network adapted to fully operate in hyperbolic space that is tailored for graph representations and allows for extreme dimensionality reduction. Inspired by the Dueling-DQN [30], the HyperPlanner focuses on accurately determining the value of potential actions in each state, facilitating informed decision-making by the robot.

Environment graph. We consider the environment’s visible state $w_{t}$ as a graph whose nodes are represented by the agents in the scene, and the features of each node coincide with its observable state. This work relies on two subsequent graph attention (GAT) modules to encode this graph effectively. The first GAT layer receives as input an embedding $\Phi(w^{*}_{t})$ of each agent’s current state, obtained via an MLP $\Phi$ to have the same dimension for both $w_{t}^{r}$ ($\in\mathbb{R}^{9}$) and $w_{t}^{i}$ ($\in\mathbb{R}^{5}$), while the second layer outputs high-level representations along with softmax-normalized attention coefficient that describe the strength of the interaction of each agent with each other. We employ two GNN layers to account for second-order interactions, as the reciprocal interactions of two obstacles can affect the robot planning.

Hyperbolic Policy Network. Next, we map the weighted representations to hyperbolic space via exponential mapping (cf. Eq. 9). A hyperbolic-MLP (h-MLP) module is in charge of extracting the compact hierarchical features and refining the coefficients given by the GATs. In the remainder of this section and in Fig. 2, with a slight abuse of notation, we omit to represent the action of this module to keep the notations easy to read. Starting from these representations, two separate modules. are tasked to (1) estimate the value of the current state (the V-Net, $V$) and (2) quantify the relative advantage of each possible action on the current state (the A-Net, $A$). The two modules are modeled as h-MLPs with ReLU activations:

$$\begin{split}V(w_{t})&=\text{h-MLP}(\text{ReLU}(\text{h-MLP}(w_{t}))),\\ A(w_{t})&=[A(w_{t},a^{0}),...,A(w_{t},a^{|\mathcal{A}|})]\\ &=\text{h-MLP}(\text{ReLU}(\text{h-MLP}(w_{t})))\end{split}$$

(12)

Notably, aiming to reduce the computational overhead severely, we constraint these modules to operate in 2-dimensional latent spaces. Indeed, by leveraging the capacity of hyperbolic modules to encode data with a hierarchical structure, we rely on the compact yet expressive power of this reduced dimensional setting, allowing for efficient computation without sacrificing the depth and quality of the agent’s environmental understanding. Finally, we aggregate the outputs to retrieve a single value for each action in the current state as:

$$Q(w,a)=V(w)+(A(w,a)-\frac{1}{\mathcal{A}}\sum_{a^{\prime}\in\mathcal{A}}A(w,a^{\prime}))$$

(13)

and we choose the next action as

$$a_{t}=\pi(w_{t})=\arg\max_{a\in\mathcal{A}}Q(w,a)$$

(14)

Within our DRL framework, we propose a module, dubbed HyperCuriosity, that refines the exploration-exploitation trade-off by leveraging the distinctive metric properties of hyperbolic space. We build upon the concept of ICM [12], which, akin to human curiosity, encourages exploration by generating intrinsic rewards based on the prediction error of the consequences (next state) of the agent’s action. Pathak et al. [12] show that the intrinsic reward mechanism benefits from state representations invariants to factors that do not affect the agent’s decisions. HyperCuriosity further extends this study, adopting hyperbolic latent space to compactly encode the hierarchical features of the environment.

HyperCuriosity consists of three interconnected components settled in hyperbolic space: a feature extractor $\phi$, a forward module $f$, and an inverse module $g$; its input corresponds to the triplet $(w_{t},a_{t},w_{t+1})$ representing the interaction between the agent and the environment produced by the current policy $a_{t}=\pi(w_{t})$. The feature extractor $\phi$ consists of an exponential mapping to embed the states into hyperbolic space and an h-MLP layer:

$$\phi(w)=\text{h-MLP}(\exp_{O}(w))$$

(15)

We apply $\phi$ to two subsequent states ($w_{t}\xrightarrow{a_{t}}w_{t+1}$) recovering their high-level representations $\phi(w_{t}),\phi(w_{t+1})$. Next, the forward module $f$, devised as an h-MLP, uses the current state’s representation and the transition action $a_{t}$ (encoded as a one-hot vector) to predict the next state’s representation provided by $\phi$. Since the concatenation involves two terms defined in different spaces, we first apply the $\log_{O}$ map (Eq. 10) to $\phi(w_{t})$ to project it in the Euclidean space. We formalize this layer as:

$$\begin{split}f(w_{t},a_{t})&=\text{h-MLP}(\exp_{O}([\log_{O}{\phi(w_{t})},a_{t}]))\\ &=\hat{\phi}(w_{t+1})\end{split}$$

(16)

where $[\cdot,\cdot]$ is the concatenation operation. The third component, $g$, acts as a regularizing module, using the representations from the feature extractor to infer the action that transitions between the two, indirectly modeling the information conveyed by $\phi$’s representations. Besides using hyperbolic latent spaces to compactly represent the relevant features of the environment, our proposal lies in the intrinsic reward computation, which is derived from the discrepancy between $\phi(w_{t+1})$ and the predictive module’s anticipated next-state representation $\hat{\phi}(w_{t+1})$:

$$r^{i}_{t}=d_{\mathbb{D}}(\phi(w_{t+1}),\hat{\phi}(w_{t+1}))$$

(17)

Fully-Hyperbolic Architecture. The entire model architecture is hyperbolic, and it inherits its metric properties. The advantages of this choice are twofold. First, when representing hidden states of an MDP, the hyperbolic latent space is more appropriate as this space is naturally suited to embed tree-like structures, adhering to the hierarchical nature of consecutive decisions and state space decision-depending evolution [24]. Second, the Poincaré distance sets a penalty that grows exponentially with the hyperbolic radius. This is reflected in an increased contribution of the intrinsic reward, effectively motivating the agent to explore novel states during training. Moreover, this property yields, as shown in Sec. VI-B, the opportunity for the model to increase exponentially the penalty for mistaken decisions when the situation is simple and to reduce the penalty when the situation is complex. Interestingly, when guided so, the hyperbolic radius of the embeddings is unsupervisedly learned, and its magnitude correlates with the situation’s complexity.

All the components of HyperCuriosity are randomly initialized and trained end-to-end with the policy network. This scheme entails that the intrinsic rewards are more influential in the first part of the training and smoothly decrease throughout the training due to the optimization of the predictive network. This represents an opportunity for the model to tune the exploration importance directly from data, which represents an intriguing alternative to the systematic decrease of exploration adopted in classical exploration strategies [10].

Implementation details. We train \modelnamefor 10k episodes and then select the best checkpoint. We adopt RiemmanianAdam as optimizer with a learning rate of ${10^{-3}}$. Fig. 3 shows the training curves.

Simulations. To evaluate the proposed \modelname, we follow Zhou et al. [7] and use two scenarios from the CrowdNav [4] simulation environment: simple and complex. The simple scenario consists of 5 humans in the scene, positioned in a circle, who must reach a predefined goal by crossing the circle’s center. The complex scenario involves 10 humans, each with a predetermined objective, where 5 are positioned in a circle or a square, and the other 5 are randomly set. The agent obtains a new random destination goal upon reaching its current one to avoid still obstacles. The ORCA [14] algorithm determines the human agents’ paths, allowing them to navigate without collisions. Following literature, we define the interval between two consecutive timesteps as 0.25 seconds. We train \modelnamefor 10000 episodes, evaluating its performance every 500 episodes. Finally, we consider three conditions for ending an episode, namely collision(the robot collides with an obstacle), out-of-time (the robot fails to reach its destination within 30 seconds, but no collision occurs), and success (the robot safely reaches the goal within 30 seconds).

Baselines. We benchmark our proposed \modelnameagainst several state-of-the-art methods which we adopt as comparative baselines. We consider ORCA [14], which employs a reactive policy, setting pedestrian radii to $d_{s}=0.2$ for preserving safety distances. Additionally, we evaluate against two versions of Intrinsic-SGD3QN [10], ICM and RE3, which adapts the exploration modules from Pathak et al. [12] to social navigation tasks. Furthermore, we assess SafeCrowdNav [6], the former state-of-the-art method, which utilizes intrinsic exploration rewards as in Martinez et al. [10] and provides quantitative safety score assessments. We report the results of our solution for both the low (2) and high-dimensional (128) embedding versions.

Metrics. We test each method on 1000 randomly generated episodes and evaluate them using three metrics: “Average Return”, the average cumulative return over steps, “Navigation Time”, the average time required for the robot to reach the goal, and “Success Rate”, the rate of the robot reaching the goal without collisions.

We analyze the quantitative performance of \modelnamewith state-of-the-art approaches on complex and simple scenarios, reporting the results in Tables I and II, respectively.

Complex Scenario. Table I presents the evaluation on the complex scenario where the robot navigates through 10 humans to reach the final goal. Our proposal demonstrates higher success rates than other methods across the 2- and 128-dimensional versions. Notably, the 2-dimensional variant achieves a success rate of 99.3%, surpassing the 128-dimensional version by 1.4% and outperforming the best baseline by 2.9% in the circle scenario, and by 1.2% and 2.1% in the square one. Moreover, our method achieves higher average returns than state-of-the-art methods, particularly with the low-dimensional version in both cases. Interestingly, although the 128-dimensional version reaches its destination $\sim$1 second faster on average than the 2-dimensional counterpart in both scenarios, \modelnameconsistently yields the best success rate, provoking fewer collisions. The advantage of navigation in hyperbolic space is that this performance comes at a fraction of the parameters. Compared to the current state-of-the-art, our approach is 2.7$\times$ to 6.5$\times$ more parameter-efficient.

Simple Scenario. Table II shows the evaluation results for the simple scenario. Similar to the complex case, our method demonstrates superior success rates and average returns for the 2- and 128-dimensional variants, showing improvements of 1.4% and 5.6%, respectively. Our solution achieves a perfect success rate in all 1000 test cases, reporting no collisions. Interestingly, in contrast to the complex scenario, the high-dimensional version outperforms the low-dimensional one in this case. In the simple scenario, the Euclidean models appear more aggressive, as confirmed by the lower navigation time of [8, 10]. However, this also results in decreased safety, as the Euclidean-based models report substantial drops in success rate, more likely to expose potential humans to collisions.

Fig. 4 illustrates the performance comparison between two methods, Intrinsic-SGD3QN [10] and our proposed approach \modelname, across different embedding dimensions (2, 8, and 128). In all the considered dimensions, our hyperbolic learner consistently outperforms its Euclidean counterpart, illustrating two critical advantages: the effectiveness of hyperbolic modules even with minimal embedding sizes and the overall convenience of adopting a hyperbolic approach for state representation in the crowd navigation task. Indeed, the Euclidean competitor fails to converge when employing only 2 dimensions, reporting a clear drop in both the average navigation time and success rate metrics.

In Fig. 3 we compare the training trends of \modelnameand Intrinsic-SGD3QN [10]. The plot clearly shows that the Euclidean model fails to converge when the latent space is constrained to have dimensionality 2 (orange curve), reporting low success rate, and high average time, indicating that most episodes conclude matching an out-of-time condition.

While training its policy, \modelnamelearns to encode the different states composing an episode in different areas of the Poincaré Disk. We reach this conclusion upon investigation of \modelname’s representations by analyzing their hyperbolic radius, which is the distance from the center of the Poincaré Disk to the embeddings. In Fig. 5, we plot the hyperbolic radius of the current state Vs the attention the system is paying to obstacles other than self for a single timestep. The plot reveals a high negative correlation between these measures (0.55). This confirms that more complex situations correspond to a smaller radius, indicating that the agent is more attentive to possible collisions. Consequently, the smaller radius reflects increased uncertainty.

On the sides of Fig. 5, we associate to some points the visualization of the current dynamic running in the simulation. In particular, we select two points with low radius ($\leq 0.5$, red and purple points) and two with high radius ($\geq 0.5$, yellow and green points). As can be seen from the Figure, we found that the points with lower radius values correspond to situations in which the obstacles are likely to impede the robot from easily moving forward. In the bottom-left frame, the robot cannot move toward the goal as a person is approaching on its left and cannot devise a safe root moving to the right, given the presence of a second obstacle on its right, demonstrating awareness of the situation’s complexity. Similarly, in the purple frame, the robot is surrounded by several obstacles moving toward the line between the agent and its goal. Still, the robot successfully avoids collision, but the safer way it has to take is opposite to the direction of the goal. On the other hand, the other two examples show the robot being sure of the next action, as no obstacle seems to approach the path to the goal. Surprisingly, the robot is near an obstacle in the bottom-right frame, but it correctly predicts its intention to move away. In the top-right one, the robot is mainly focused on obstacles that are far away from it. Still, they point to the optimal direction for the robot and risk colliding in a future step.

We perform a study on the generalization capabilities of our model to further investigate the quality of \modelname’s internal representations. For this experiment, we increase the scenario’s complexity; Table III reports the results of the best performer models from Table I when doubling the number of dynamic obstacles in the scene. In particular, all the baselines represented in Table III have been trained in the complex scenario (10 obstacles) but are assessed with 20 humans. Even in this challenging scenario, \modelnameshows the best performance for success rate and average return, outperforming the most recent model from [6] by 4 p.p. points and surpassing the Intrinsic-SGD3QN (128-embedding dimensions). Although there is a slight increase ($\leq 1$ sec.) in navigation time compared to the best performer baseline, the trade-off is balanced by the gains in navigational safety and efficiency. These findings highlight the robustness and adaptability of \modelname, which is key in real-world crowd navigation when the number of entities composing the crowd is unknown and can vary with time.