Informed Spectral Normalized Gaussian Processes for Trajectory Prediction

The PIL approach of Schlauch [?] integrates prior expert and world knowledge in a supervised learning setup. The basic idea is to define a sequence of knowledge tasks $i=1,\ldots,M-1$ on datasets $D_{i}=\{(x^{(i)}_{j},y^{(t)}_{j})\}^{n_{i}}_{j=1}$ with $n_{i}$ samples each. These datasets can be synthetically generated, for example, by leveraging semantic annotations to map the prior knowledge to the prediction target. Semantic annotations are readily available in domains like autonomous driving, but are often underutilized in state-of-the-art models that learn from observations in the conventional task $i=M$ alone [?].

Given a probabilistic DL model parameterized by $\theta$ and an initial uninformative prior $\pi_{0}(\theta)$, the goal is to recursively learn from the sequence of tasks by applying Bayes’ rule

$$\begin{split}p(\theta|D_{1:i})\propto\pi_{0}(\theta)\prod^{i}_{j=1}p_{\theta}(y_{j}|x_{j}),\end{split}$$

(1)

where $p_{\theta}(y_{j}|x_{j})$ are the likelihood functions at task $j$, which are assumed to be conditionally independent given $\theta$. This computationally intractable recursion is approximated by repurposing regularization-based continual learning methods.

The PIL approach can generally be applied, as long as first, the prior knowledge is strongly related to the observational task, second, the prior knowledge can be mapped to the prediction target and third, the posterior parameter distribution can be estimated. The informative priors make information explicit and shape the loss surface in the downstream task, improving the training outcome; even without using probabilistic inference in the end [?].

SNGPs [?] employ a composition $f_{\theta}=g_{\theta_{\text{GP}}}\circ h_{\theta_{\text{NN}}}:\mathcal{X}\to\mathcal{Y}$, $\theta=\{\theta_{\text{NN}},\theta_{\text{GP}}\}$. Its first component is a deterministic, spectral normalized feature extractor $h_{\theta_{\text{NN}}}:\mathcal{X}\to\mathcal{H}$ with trainable parameters $\theta_{\text{NN}}$ mapping the high dimensional input space $\mathcal{X}$ into a low dimensional hidden space $\mathcal{H}$. The second component is a GP output layer $g_{\theta_{\text{GP}}}:\mathcal{H}\to\mathcal{Y}$ with a radial basis function (RBF) kernel mapping into the output space $\mathcal{Y}$. The RBF kernel can be approximated by (random) Fourier features using Bochner’s Theorem [?]. This effectively reduces the GP to a Bayesian linear model, that can be written as a neural network layer with fixed hidden weights and trainable output weight parameters $\theta_{\text{GP}}$ and enables end-to-end training with the feature extractor. The distance-sensitive of the composition prevents a “feature-collapse” [?], improving the calibration against adversarial and outlier samples. In total, SNGP introduces five additional hyperparameters, namely an upper bound $s$ and number of power iterations $N_{p}$ for the spectral normalization for the feature extractor and the number of Fourier features $N_{f}$, the kernel’s length scale $l_{s}$ and Gaussian prior choice for the last layer.

There are two problems prohibiting the direct application of the PIL approach to composite last layer kernel approximations like the SNGP. First, there is no existing continual learning method for kernels that does not require rehearsal memories or parameter expansions (see Sec. 2). Second, estimating the posterior parameter distribution of the feature extractor (e.g. via a Laplace approximation or variational inference) contradicts the motivation for the last layer kernel approximation regarding compute-efficiency.

We tackle the first problem by leveraging the Fourier feature approximation of the RBF kernel of the GP. The posterior distributions of the parameters of the last layer at task $i$ can be made tractable through Laplace approximation [?], that is, we assume

$\displaystyle p(\theta_{\text{GP}}|D_{1:i})\approx\mathcal{N}(\theta_{\text{GP}};\theta_{\text{GP},i}^{*},\Sigma^{-1}_{\text{GP},i}),$

given a maximum a posteriori (MAP) estimate $\theta^{*}_{\text{GP},i}$ at task $i$. Similar to online EWC [?], $\theta^{*}_{\text{GP},i}$ can be obtained by minimizing

$$\displaystyle-\log{p_{\theta_{\text{GP}}}(y_{i}|x_{i})}-\frac{\lambda_{\text{GP}}}{2}(\theta_{\text{GP}}-\theta^{*}_{\text{GP},i-1})^{\top}\Sigma_{\text{GP},i-1}^{-1}(\theta_{\text{GP}}-\theta^{*}_{\text{GP},i-1})$$

(2)

with respect to $\theta_{\text{GP}}$, where the precision $\Sigma_{\text{GP},i}^{-1}$ is approximated by the sum of the Hessian at the MAP estimate and a scaled precision at task $i-1$, that is,

$$\Sigma_{\text{GP},i}^{-1}\approx H_{\text{GP},i}(\theta_{\text{GP},i}^{*})+\gamma_{\text{GP}}\Sigma_{\text{GP},i-1}^{-1}.$$

Above, $\lambda_{\text{GP}}>0$ is a temperature parameter, that scales the importance of the previous task [?], and $0<\gamma_{\text{GP}}\leq 1$ is a decay parameter, that allows for more plasticity over very long task sequences [?]. In contrast to online EWC, we can cheaply compute the Hessian using moving averages [?] instead of using a Fisher matrix approximation. In the first task $i=1$, we use an uninformative zero-mean, unit-variance prior $\pi_{0}$, which amounts to a simple $\mathcal{L}$2-regularization.

To tackle the second problem and regularize the feature extractor, we approximate the precision $\Sigma_{\text{NN},i-1}^{-1}$ with the identity matrix $\mathbb{I}$. This implies a simple $\mathcal{L}$2-regularization for the MAP estimates $\theta^{*}_{\text{NN},i}$ obtained by minimizing

$\displaystyle-\log{p_{\theta_{\text{NN}}}(y_{i}|x_{i})}-\frac{\lambda_{\text{NN}}}{2}(\theta_{\text{NN}}-\theta^{*}_{\text{NN},i-1})^{2}$

with respect to $\theta_{\text{NN}}$, where $\lambda_{\text{NN}}$ is the extractor specific temperature parameter. This idea is conceptually simple, but should be sufficient, since the learned representation in knowledge tasks should be suitable downstream due to the close relation between tasks.

In result, the complete model $f_{\theta}:\mathcal{X}\to\mathcal{Y}$, parameterized by $\theta=\{\theta_{\text{NN}},\theta_{\text{GP}}\}$, can be effectively regularized and used in the PIL approach, as visualized in Fig. 1. Our method introduces three hyperparameters $\{\lambda_{\text{GP}},\gamma_{\text{GP}},\lambda_{\text{NN}}\}$. It only requires the parameters of the previous task in memory and has little computational overhead like online EWC [?].

We limit ourselves to the single-agent trajectory prediction problem [?]. An autonomous driving system is assumed to observe the states in the state space $\mathcal{Y}$ of all agents $\mathcal{A}$ present in a scene on the road. Let $y^{(t)}\in\mathcal{Y}$ denote the state of target agent $a\in\mathcal{A}$ at time $t$ and let $y^{(t-T_{o}\,:\,t)}=\big{(}y^{(t-T_{o})},y^{(t-T_{o}+\delta t)},\ldots,y^{(t)}\big{)}$ be its observed trajectory over an observation history $T_{o}$ with sampling period $\delta t$. Additionally, we assume access to agent-centered maps $\mathcal{M}$, which include semantic annotations such as the drivable area. Map and states make up the scene context of agent $a$, denoted as ${x=(\{y_{j}^{(t-T_{o}\,:\,t)}\}^{|\mathcal{A}|}_{j=1},\mathcal{M})}$. Given $x$, the goal is to predict the distribution of $a$’s future trajectories $p(y^{(t+\delta t\,:\,t+T_{h})}|x)$ over the prediction horizon $T_{h}$, where $y^{(t-\delta t\,:\,t+T_{h})}=\big{(}y^{(t+\delta t)},y^{(t+2\delta t)},\ldots,y^{(t+T_{h})}\big{)}$.

CoverNet [?] approaches the single-agent trajectory problem by considering a birds-eye-view RGB rendering of the scene context $x$ and the current state $y^{(t)}$ of the target agent $a$ as inputs. The RGB rendering is processed by a computer-vision backbone, before concatenated with the target’s current state and processed by another dense layer. The output is represented as a set $\mathcal{K}$ of $K$ candidate trajectories $y_{k}^{(t+\delta t\,:\,t+T_{h})}$. Doing so reduces the prediction problem to a classification problem, where each trajectory in the set $\mathcal{K}$ is treated as a sample of the predictive distribution $p(y^{(t+\delta t\,:\,t+T_{h})}|x)$ and only the conditional probability of each sample is required. In principle, any space-filling heuristic may be used to define $\mathcal{K}$, for example, by using a dynamical model that integrates physical limitations [?], which could be applied in combination with the PIL approach. Here, we follow Phan-Minh’s [?] definition of a fixed set $\mathcal{K}$ by solving a set-covering problem over a subsample of observed trajectories in the training split, using a greedy-algorithm²²2Further details in our supplemental. Also see Chapter 35.3 of Cormen [?] regarding set-covering problems in general. given a coverage-bound $\epsilon$, which determines the number of total candidates $K$.

The modification of CoverNet with SNGP is straightforward if a convolutional neural network (CNN) is used as backbone. In that case, a spectral normalization can be directly applied to the architecture while the last layer is replaced with a Gaussian process, approximated by Fourier features as described in Sec. 3.2.

The PIL approach is applied sequentially on two consecutive tasks as follows. In task $i$, we integrate the prior drivability knowledge, that trajectories are likely to stay on-road. To this end, we derive new training labels (see Sec. 3.1), where all candidate trajectories in $\mathcal{K}$ with way-points inside the drivable area for a given training scene $x$ are labeled as positive [?]. We then train in a multi-label classification with a binary cross-entropy loss on these labels. In task $i+1$, the closest candidate trajectory in $\mathcal{K}$ to the observed ground truth is labeled as positive. We train in a multi-class classification with a sparse categorical cross-entropy loss (using softmax normalized logit transformations) on these labels [?]. In effect, the consecutive tasks are only differing in the labels and loss functions used. Applying our method described in Sec. 3.3, we first train our CoverNet-SNGP model on task $i$ and then regularize its training on task $i+1$, as exemplified in Fig. 1. We denote the resulting informed CoverNet-SNGP as CoverNet-SNGP${}_{\textbf{I}}$, opposed to the non-informed version CoverNet-SNGP${}_{\textbf{U}}$ trained on task $i+1$ only without integration of prior knowledge from task $i$.

We use the public NuScenes [?] and Argoverse2 [?] datasets. We replicate the NuScenes data split by Phan-Minh [?] on Argoverse2, only considering vehicle targets (exlcuding pedestrians and cyclists not driving on-road), as summarized in Tab. 1. For the RGB rendering, we consider each scene with a one-second history ($T_{o}=1\text{s}$). For the candidate trajectories in $\mathcal{K}$, we consider a six-second prediction horizon ($T_{h}=6\text{s}$), sampled at $2\text{Hz}$ in NuScenes and $10\text{Hz}$ in Argoverse2. Both datasets include drivable areas in the semantic map data, allowing us to define the first task as described in Sec. 4.3.

We consider the unmodified CoverNet as baseline, once as non-informed Base-CoverNet [?] and once as Transfer-CoverNet. The Transfer-CoverNet baseline, pretrained on task $i$ and then trained on the current task $i+1$, has previously been proposed by Boulton [?]. We can also understand it as an ablation-type baseline to the PIL approach without regularization. In addition, we compare to GVCL-Det-CoverNet proposed by Schlauch [?], since it only needs a single-inference pass too. However, GVCl-Det-CoverNet also requires computationally extremely expensive training of a GVCL-CoverNet model. For example, in our setting, training until convergence on a single Nvidia RTX A5000 GPUs with $10\%$ of NuScenes data needs around 120 hours for GVCL-CoverNet, in contrast to 8 hours for CoverNet-SNGP${}_{\textbf{I}}$ and 6 hours for Base-CoverNet.

We measure the average displacement error minADE${}_{1}$ and final displacement error minFDE${}_{1}$ , evaluating the quality of the most likely trajectory, and the minADE${}_{5}$ , which considers the five most likely trajectories [?]. The minADE${}_{5}$ depends on the probability-based ordering and, thus, indirectly on the calibration. We also consider the drivable area compliance (DAC) to evaluate the extent to which predictions align with our prior drivability knowledge.

Since observed ground truth trajectories may not be part of the trajectory set $y_{\text{true}}^{(t+\delta t\,:\,t+T_{h})}\notin\mathcal{K}$, the CoverNet model exhibits an irreducible approximation error. To more clearly assess the impact of our method, we also consider the classification-based negative log likelihood (NLL) and the rank of the positively labeled trajectory (RNK), both directly depending on the calibration, and the Top1-accuracy (ACC).

We use the output representation described in Sec. 4 with a coverage bound $\epsilon=4\text{m}$, for NuScenes with $K_{\text{Nusc}}=415$ and for Argoverse2 with $K_{\text{Argo}}=518$ candidates. We employ a ResNet-50 as backbone and SGD as optimizer. For the CoverNet-SNGPs, we fix power iterations $N_{p}$ to one and the number of Fourier features $N_{f}$ to 1024, following Liu [?]. The spectral normalization’s upper bound $s$ and the kernel length scale $l_{s}$ are treated as additional hyperparameters. We tune the hyperparameters of each model on the respective tasks with 100% of the data using the validation NLL³³3Configurations are available in our supplemental and on Github.. The exception is CoverNet-SNGP${}_{\textbf{I}}$, which uses the same settings as CoverNet-SNGP${}_{\textbf{U}}$ on task $i+1$. We also fix both temperature parameters $\lambda_{\text{NN}}$ and $\lambda_{\text{GP}}$ ad-hoc to the inverse of the effective dataset size to keep tuning costs low. The decay parameter $\gamma_{\text{GP}}$ is mostly relevant for very long task sequences (see Sec. 3), such that we set $\gamma_{\text{GP}}=1$.

Tab. 2 and Tab. 3 show the performance of our CoverNet-SNGP${}_{\text{I}}$ in comparison to the baselines on NuScenes and Argoverse2, respectively. We observe, that the prior drivability knowledge leads to notable performance benefits in our CoverNet-SNGP${}_{\text{I}}$ and informed baselines (Transfer-CoverNet, GVCl-Det-CoverNet) across most metrics. The benefits from the prior drivability knowledge are most substantial in the calibration-sensitive metrics (RNK and notably NLL, e.g., as seen in Fig 2) that directly benefit from the optimization in the knowledge tasks. The drivability knowledge is less helpful in discerning the best candidate between the remaining drivable candidate trajectories, leading to lower benefits in the respective metrics (minADE${}_{1}$ , minFDE${}_{1}$ , ACC).

We also observe, that Transfer-CoverNet’s benefits are limited to higher data regimes. In low data regimes, Transfer-CoverNet can even perform substantially worse than Base-CoverNet across all metrics (except DAC). In these low data regimes, Transfer-CoverNet may converge to less adequate minima, due to its weight initialization being overly biased towards drivability (illustrated by the rising DAC). In contrast, GVCL-Det-CoverNet and our CoverNet-SNGP${}_{\text{I}}$ never decrease performance, with consistent benefits especially in low data regimes. This highlights a principal advantage of the PIL approach, where the informative prior helps to shape the complete loss landscape during training.

In comparison to GVCL-Det-CoverNet, our CoverNet-SNGP${}_{\text{I}}$ shows benefits across most metrics, especially in low data regimes, even though both are trained using the PIL approach. The advantage is most visible in the metrics concerning the most-likely trajectory (minADE${}_{1}$ , ACC). CoverNet-SNGP${}_{\text{I}}$ also shows more stable results with a lower standard deviations. Here, our CoverNet-SNGP${}_{\text{I}}$ profits from using the full information of the posterior distribution at inference.