Learn to Memorize and to Forget: A Continual Learning Perspective of Dynamic SLAM

The objective is to guarantee a photorealistic and multi-view consistent map representation by minimizing the photometric and geometric errors in static areas between the rendered images and observed images as:

$$L_{pho}=\frac{1}{|H|}\sum_{(u,v)\in H}\left|I\left[u,v\right]-\hat{I}(\xi,\theta_{M})\left[u,v\right]\right|,$$

(1)

$$L_{geo}=\frac{1}{|H|}\sum_{(u,v)\in H}\left|D\left[u,v\right]-\hat{D}(\xi,\theta_{M})\left[u,v\right]\right|,$$

(2)

where $H\subset\mathbbm{1}(R^{t})$ is the set of static samples given motion segmentation mask $R^{t}$ indicated by the classifier. $\hat{I}(\xi^{t},\theta^{t}_{M})$ and $\hat{D}(\xi^{t},\theta^{t}_{M})$ are predicted color and depth images through volume rendering.

In this paper, we follow Co-SLAM [43] to model the scene as a truncated signed distance field $s$ with color values $\mathbf{c}$ as $f(\mathbf{x};\theta)\rightarrow(\mathbf{c},s)$. Nevertheless, the proposed method can also adopt the conventional NeRF representation with density-based volume rendering as [20]. In this work, the learnable map parameters $\theta=\{\alpha,\phi,\tau\}$ denote the encoder $\mathcal{V}(\mathbf{x};\alpha)$ for grid features and the decoders for color and geometry. The volume rendering for color and depth prediction is a weighted sum of all samples $\{\mathbf{p}_{i}\}_{i=1}^{N}$ along a ray determined by the pixel coordinate and the camera origin as:

$$\hat{I}(\xi,\theta_{M})[u,v]=\frac{1}{\sum_{i=1}^{N}w_{i}}\sum_{i=1}^{N}w_{i}\mathbf{c}_{i},$$

(3)

$$\hat{D}(\xi,\theta_{M})[u,v]=\frac{1}{\sum_{i=1}^{N}w_{i}}\sum_{i=1}^{N}w_{i}d_{i},$$

(4)

where $w_{i}$ is the computed weight; $d_{i}$ is the distance between the sampled point $\mathbf{p}_{i}$ and the corresponding camera origin. It should be noted that in Co-SLAM [43], the weight $w$ is computed by multiplying two Sigmoid functions as:

$$w=\sigma\left(\frac{s}{\lambda_{tr}}\right)\sigma\left(-\frac{s}{\lambda_{tr}}\right).$$

(5)

where $\lambda_{tr}$ is the truncation distance.

Besides the photometric and geometric constraints in Eq. 1 and 2, we also adopt the SDF losses $L_{sdf}$ in near-surface and free space $L_{free}$ along with the feature smoothness loss $L_{smooth}$ in Co-SLAM [43] for further constraints.

The SDF loss is employed to enhance the quality of map reconstruction, where the depth values of pixels are utilized as SDF approximations. For points within the truncated region $(|D[u,v]-\hat{D}[u,v]|\leq tr)$:

$$L_{sdf}=\frac{1}{|R_{H}|}\sum_{r\in R_{H}}\frac{1}{|S_{r}^{tr}|}\sum_{p\in S_{r}^{tr}}\big{(}s_{p}-(D[u,v]-\hat{D}(\xi,\theta_{M})[u,v])\big{)}^{2}$$

(6)

where $R_{H}$ represents the set of rays corresponding to the static pixel set $H$.

For points outside of the truncation region $(|D[u,v]-\hat{D}[u,v]|>tr)$, we calculate the free space loss:

$$L_{free}=\frac{1}{|R_{H}|}\sum_{r\in R_{H}}\frac{1}{|S_{r}\setminus S_{r}^{tr}|}\sum_{p\in S_{r}\setminus S_{r}^{tr}}(s_{p}-tr)^{2},$$

(7)

To mitigate the noisy reconstruction of unvisited areas induced by hash collision, a smoothness loss for the interpolated features is applied:

$$L_{smooth}=\frac{1}{|\mathcal{G}|}\sum_{p\in\mathcal{G}}\Delta_{x}^{2}+\Delta_{y}^{2}+\Delta_{z}^{2},$$

(8)

where $\Delta x,y,z=V_{\alpha}(p+\epsilon_{x,y,z})-V_{\alpha}(p)$ refers to the change in feature metrics between adjacent vertices sampled on the hash-grid.

We apply ADAM optimizer on the weighted sum of these five loss terms:

$$L=L_{pho}+\lambda_{1}L_{geo}+\lambda_{2}L_{sdf}+\lambda_{3}L_{free}+\lambda_{4}L_{smooth}.$$

(9)

Note that in the loss function, only errors in the static areas will be back-propagated for optimizing the pose and map parameters, where the motion segmentation mask $R^{t}$ is inferred using the instance-wise classifier $g(\mathbf{z};\theta^{t}_{C})$ in Sec. 4.3. During the tracking process, we randomly select $|H_{t}|$ pixels within the static area of the current frame and optimize the estimated camera pose by minimizing the objective function with fixed map and classifier parameters.

Similarly, the bundle adjustment for jointly optimizing camera poses and map parameters is carried out by randomly selecting $|H_{b}|$ pixels within the static area of the past keyframes and the current frame. Stored keyframes serve as past experience to avoid catastrophic forgetting [45, 34]. As the error of pixels in the dynamic areas is inhibited from back-propagation, only knowledge in the static areas will be reserved in the neural map. As illustrated in Fig. 4, the moving object will be gradually forgotten in the neural map given constantly propagated errors from recent observations.

As illustrated in Fig. 2, the motion status is inferred at the instance level. An image $I^{t}$ is decomposed into $K^{t}$ segments $\{S_{k}^{t}\}_{k=1}^{K^{t}}$ using FastSAM [47], where each segment is then fed to a visual context encoder [28, 17, 22]. Such decomposition and encoding process turns the image into $K^{t}$ vectors $\{\mathbf{z}_{k}^{t}\}_{k=1}^{K^{t}}$ to represent $K^{t}$ different instances. Intuitively, we aim to maintain the motion status of each instance across time. Explicitly aggregating information of each instance across views is non-trivial as the dynamic objects lead to inconsistency. We turn to a simple but effective solution that implicitly records the motion status of each instance using a two-layer MLP as $g(\mathbf{z};\theta_{C}^{t})$. The network serves as a classifier that determines the motion status of the corresponding instance feature, where an instance will be treated as a moving object if $g(\mathbf{z}^{t}_{k};\theta_{C}^{t})>0.5$.

The object movement leads to inconsistency between the observation and the stored knowledge. As emitted radiance is view-dependent [20], we merely examine the geometry inconsistency between the rendered depth map and the observed one. The moving status $o_{k}^{t}$ of the instance $k$ will be treated as True if a certain portion of pixels within the segment $S^{t}_{k}$ exhibit large discrepancies as:

$$\frac{\sum_{(u,v)\in S_{k}^{t}}\mathbbm{1}\left(\hat{D}(\xi,\theta_{M})[u,v]-D[u,v]>t_{d}\right)}{|S_{k}^{t}|}>t_{p},$$

(10)

where $t_{d}$ and $t_{p}$ controls the sensitivity of motion segmentation.

The classifier will be updated if the current moving status of any instance $o_{k}^{t}$ contradicts the classification result $\mathbbm{1}(g(\mathbf{z}^{t}_{k};\theta_{C}^{t-1})>0.5)$. The network is optimized using the binary cross-entropy loss as:

$$-\sum_{k=1}^{K^{t}}\big{[}o_{k}^{t}\log(g(\mathbf{z}^{t}_{k};\theta_{C}^{*}))+(1-o_{k}^{t})\log(1-g(\mathbf{z}^{t}_{k};\theta_{C}^{*}))\big{]}.$$

(11)

Once the classifier is updated, the motion segmentation mask of all keyframes will be updated. The observations regarding the moving object will no longer contribute to the pose and map optimization. As the neural map is continually trained under constant distribution shift, the knowledge regarding the moving object will be gradually forgotten without manual operations.

The experiments are conducted on a desktop PC with an Intel i9-12900K CPU, an NVIDIA RTX 3090 GPU, and 64GB memory. The keyframe is automatically stored every 5 frames.

Specific to implementation details, the number of sampling points along each ray $N=128$, sampling pixels during tracking and bundle adjustment $H_{t}=1024,H_{b}=2048$, truncation distance $tr=10cm$, threshold used for determining moving status $t_{d}=0.3,t_{p}=0.05$. For the weights of each loss in Eq. 9, we follow the settings of Co-SLAM as: $\lambda_{1}=0.1,\lambda_{2}=5000,\lambda_{3}=10,\lambda_{4}=1e-8$.

As illustrated in Fig. 3, the photometric and geometric errors in dynamic areas lead to inherent ambiguities as they will be back-propagated to modify the pose estimates and mapping results. As tracking and mapping are coupled during the optimization, the erroneous tracking and the fusion of dynamic objects into the map would eventually lead to system collapse. The proposed method, on the other hand, achieves robust camera tracking results even in the challenging ’crowd’ sequences. As presented in Tab. 1, the proposed method achieves better results compared against the feature-based DynaSLAM [2] and the dense SLAM systems of ReFusion [25] and StaticFusion [32]. The effects arising from the dynamic objects are well alleviated by the proposed motion status classifier.

As illustrated in Fig. 6, the continual learning of the classifier leads to dynamic updating of the object motion status. The successive changes in motion status can be well reflected by the geometric inconsistency, and the classifier can quickly identify the motion changes and learn the instant motion status. The encoded feature of masked objects can well distinguish between two people even if they fall into the same "person" category.

Segmentation with FastSAM [47] leads to varying granularities. A person may be separated into different parts as arms, legs, body, and head. In such a case, we wish to retain the instance-level segmentation instead of the part-level decomposition, thereby reducing the number of classifier updates.

Specifically, as illustrated in Fig. 10, for any two segments $R_{1}$ and $R_{2}$, we consider $R_{1}$ to be a part of $R_{2}$ and delete it if the portion of overlapped areas between $R_{1}$ and $R_{2}$ is larger than $T_{R}$ as:

$$S_{R_{1}\cap R_{2}}>T_{R}\times S_{R_{1}}$$

(13)

where $T_{R}$ is set to 0.9 in our experiments.

We follow the dense dynamic SLAM system of ReFusion [25] to evaluate the mesh results quantitatively. As shown in Fig. 11 and  12, our method achieves comparable results with ReFusion [25] and outperforms StaticFusion [32]. Note that NeRF-based methods mainly focus on realistic rendering (as presented in our main paper and supp. video) instead of surface reconstruction, the quantitative results sufficiently verify our map quality in dynamic environments.

We compare the number of updates for the classifier based on whether replay training is performed. The experiment is conducted with different image encoders across two high-dynamic sequences: ’person_tracking’ and ’balloon’. The results are illustrated in Fig. 13. It is clearly observable that the classifier undergoing replay training requires fewer updates compared to the one without replay, regardless of the image encoder used. Detailed analysis can be found in the "continual learning of the classifier" parts of Sec. 4.3 and Sec. 5.3.

We also tested the average run-time of each component of the system in the two sequences mentioned above. In high-dynamic environments, we effectively balanced the system’s speed and accuracy. Our method can run at a frame rate of around 1 fps. In contrast, the frame rates of Co-SLAM [43], iMap [34], and NICE-SLAM [49] are approximately 3 fps, 2 fps, and less than 0.1 fps, respectively.

In Table 1 of the main paper, we present the quantitative results of camera tracking, showing that our method achieves higher accuracy compared to other dynamic SLAM approaches. Here, we also provide a qualitative demonstration of these results. Fig. 14 shows a comparison of our trajectories with the ground truth on three sequences.

We tested the impact of different experimental settings on the accuracy of camera tracking. Table 4 and Table 4 present the ablation study results on Bonn RGB-D dataset. As shown in Table 4, using different visual encoders causes slight variations in the tracking results, with BLIP-2 [17] performing the best and DINOv2 [22] performing the worst. Table 4 reflects that the presence of prior knowledge has minimal impact on tracking accuracy. In most cases, our method achieves precise camera tracking without prior knowledge. Only in handling extremely challenging scenarios (such as Fig. 9 in the main paper), do we need to incorporate prior knowledge.