MotionGS : Compact Gaussian Splatting SLAM by Motion Filter

Xinli Guo, Weidong Zhang^∗, Ruonan Liu, Peng Han, and Hongtian Chen All authors are with Shanghai Jiao Tong University. ^∗Corresponding Author: Weidong Zhang ([email protected]). This paper is partly supported by the National Key R&D Program of China (2022ZD0119900), the National Natural Science Foundation of China (U2141234), Shanghai Science and Technology program (22015810300), Hainan Province Science and Technology Special Fund (ZDYF2024GXJS003), National Natural Science Foundation of China (62303308), Shanghai Pujiang Program (23PJ1404700), Joint Research Fund of Shanghai Academy of Spaceflight Technology (USCAST2023-22).

Abstract

With their high-fidelity scene representation capability, the attention of SLAM field is deeply attracted by the Neural Radiation Field (NeRF) and 3D Gaussian Splatting (3DGS). Recently, there has been a surge in NeRF-based SLAM, while 3DGS-based SLAM is sparse. A novel 3DGS-based SLAM approach with a fusion of deep visual feature, dual keyframe selection and 3DGS is presented in this paper. Compared with the existing methods, the proposed tracking is achieved by feature extraction and motion filter on each frame. The joint optimization of poses and 3D Gaussians runs through the entire mapping process. Additionally, the coarse-to-fine pose estimation and compact Gaussian scene representation are implemented by dual keyframe selection and novel loss functions. Experimental results demonstrate that the proposed algorithm not only outperforms the existing methods in tracking and mapping, but also has less memory usage.

Index Terms:

SLAM, 3D Gaussian splatting, Keyframe selection, Feature extraction, 3D reconstruction.

Refer to caption — Figure 1: Overview of MotionGS. The input to MotionGS at each timestep is the current RGB-D/RGB image. After the motion filter, the directly pose optimization of the motion keyframe is done based on the photometric error between the GT and render result. After information filter, the joint optimization of keyframe poses and 3D scene geometry on sliding windows and random historical frames is carried out in the mapping thread. Finally, the scene is refined.

I Introduction

The main purpose of simultaneous localization and mapping (SLAM) is to enable real-time positioning and simultaneous mapping in unknown environments. This technology plays a crucial role in various fields such as autonomous driving [1], unmanned systems [2], virtual reality [3], and augmented reality [4]. Over the past years, the SLAM field has made remarkable strides in both localization accuracy and speed [5]. However, recent attention has been attracted towards scene representation and comprehension. This shift has given rise to numerous map-centric SLAM. The scene representation of traditional map-centric SLAM includes point clouds or surfels [6, 7, 8, 9] , meshs [10, 11], voxels [12, 13, 14, 15]. As a pivotal component of SLAM systems, the selection of the mapping approach not only shapes the overall system design but also defines its capabilities.

In term of dense visual SLAM, these classic map-centric methods can not achieve the high-fidelity representation nor fail in reconstructing the fine textures and repetitive scenes. Recently, Neural Radiance Fields (NeRF) [16] have garnered significant attention in the SLAM field. NeRF represents a novel view synthesis approach with an implicit scene representation, which has the advantages of photorealism and minimal memory usage. A multitude of NeRF-based dense SLAM methods has sprung up with respective to this evolving NeRF [17, 18, 19, 20, 21, 22, 23, 24]. However, NeRF-based dense SLAM methods all rely on ray-based volume rendering by dense sampling, which is highly time-consuming and unreliable [25].

As a new scene representation, 3D Gaussian Splatting (3DGS) [26] explicitly represents scenes using 3D Gaussian as primitives, achieving rendering effects comparable to NeRF while offering faster speed of optimization and rendering. 3DGS is exceptionally well-suited as the mapping method in SLAM systems, as demonstrated by the recently proposed approaches [25, 27, 28].

In this paper, a novel dense 3DGS-based SLAM approach, named MotionGS, is proposed. Deep feature extraction, dual keyframe selection, and 3DGS are integrated in this method. In the tracking thread, we extract deep features from each image, obtain motion keyframes through motion filter, and employ direct pose optimization tailored for 3DGS to reduce the number of processed frames and improve tracking performance. Finally, the tracking thread continuously updates and maintains a sliding window of information keyframes by information filter. In the mapping thread, a new loss function is designed to simultaneously optimize keyframe pose and 3D Gaussian in the diff-gaussian-rasterization framework. The dual optimizations in the tracking and mapping threads achieve coarse-to-fine pose optimization and reduce storage needs. Extensive evaluations on indoor RGB-D datasets have demonstrated the state-of-the-art performance of our approach in tracking, rendering, and mapping. In summary, the main contributions are given as follows:

1.

A novel dense visual SLAM method based on 3DGS (that integrates deep feature extraction, dual keyframe selection and 3DGS) is proposed. This approach not only achieves accurate real-time tracking and high-fidelity reconstruction, but also supports both RGB and RGB-D inputs.
2.

A novel dual keyframe strategy and a new loss function are designed to enhance the tracking accuracy and maintain rendering quality while reducing memory consumption.
3.

Our approach achieves state-of-the-art performance on Replica and TUM-RGBD datasets in terms of tracking, mapping and runs at 2.5 fps with less memory usage.

II Related Work

Dense visual SLAM: Traditional dense visual SLAM mainly consists of two types: stereo vision and RGB-D dense mapping. For example, DTAM [29] achieved real-time camera pose tracking and point cloud based mapping by estimating optical flow and depth between consecutive frames, while ORB-SLAM2 pro achieved real-time pose tracking and point cloud based mapping in RGB-D mode using conventional feature point methods. ElasticFusion [9] achieved elastic registration and nonlinear optimization for real-time pose estimation and map reconstruction. Kintinuous [30] combined dense visual SLAM with RGB-D sensors and motion estimation, enabling long-time high-quality map reconstruction. Traditional dense mapping methods include point clouds, triangulated meshes, surfels, and voxels. Recently, the fusion of deep learning with traditional SLAM frameworks has achieved more precise localization and reconstruction effects. For example, DROID-SLAM [31] extracted dense feature maps and iteratively updated camera poses and pixel-level depth by the depth update operators. It achieved excellent real-time positioning and dense map reconstruction.
NeRF-based SLAM: According to sensor types, this category can be divided into two forms: RGB and RGB-D. iMAP [17] was the first RGB-D SLAM method to adopt implicit neural representation. The idea of using MLP for scene representation had been widely recognized and promoted. NICE-SLAM [19] employed a hierarchical strategy by using multiple MLPs to encode scene geometry into (coarse, medium, and fine) three voxel grids. The centralized three-layer voxel grid significantly improved the precision and efficiency of the optimization process and effectively replied catastrophic forgetting issues. Furthermore, ESLAM [20] adopted large-scale axis-aligned representations of three coarse and three fine feature planes with TSDF as the scene geometry representation. This greatly reduced memory usage and significantly improved frame rates. For the same purpose, Co-SLAM [22] and Point-SLAM [23] respectively utilized encoding of joint coordinate with parameter and neural point cloud representation to enhance mapping performance. As for RGB forms, based on the DROID-SLAM algorithm, Hi-SLAM [32] supplemented with depth priors and joint depth and scale adjustment to compensate for the lack of depth information.
3DGS-based SLAM: Research in this field is rapidly developing, but there are relatively few related algorithms. In addition to the differentiable optimization and fast re-rendering of 3DGS, GS-SLAM [25] used adaptive expansion strategy and coarse-to-fine camera tracking policy to achieve high fidelity and fast high-resolution rendering performance. MonoGS [27] adopted a direct optimization approach against 3D Gaussian ellipsoid for camera tracking, coupled with geometric verification and regularization techniques, achieving continuous high-fidelity 3D scene reconstruction of even small and transparent objects. SplaTAM [28] employed a customized online tracking and mapping system, combined with silhouette masks to achieve fast rendering, dense optimization, efficient map reconstruction, and structured map expansion. Photo-SLAM [33] integrated explicit geometric features and implicit texture representations into a super-primitive map based on the classic ORB-SLAM3 method. It also introduced a training method based on Gaussian pyramid to gradually learn multi-level features, improving the performance of realistic mapping. Gaussian-SLAM [34] introduced several new techniques such as differential depth rendering, explicit Gaussian gradients, and frame-to-model alignment, significantly enhancing its performance. However, recent methods still exhibited slightly inferior performance in frame processing rates and memory usage. GS-ICP-SLAM [35] took a different approach by utilizing point cloud matching to greatly increase frame processing rates.

III The Proposed Approach

III-A Compact 3DGS Scene Representation

3DGS explicitly represents the scene with a set of anisotropic Gaussian. Each Gaussian $g$ encapsulates properties: $c_{g}$ , $o_{g}$ , $sh_{g}$ , $p_{g}$ , and $\Sigma_{g}$ , representing color, opacity, spherical harmonics, position in the world coordinate system, and Gaussian covariance matrix, respectively. The explicit voxel-based rendering of 3DGS representation can render the color $c_{p}$ and depth $d_{p}$ of specified pixels by frustum culling and blending $N$ Gaussian:

	$\displaystyle c_{p}=\sum_{i}^{N}c_{i}o_{i}\prod_{j}^{i-1}(1-o_{j}),$		(1)
	$\displaystyle d_{p}=\sum_{i}^{N}d_{i}o_{i}\prod_{j}^{i-1}(1-o_{j}),$		(2)

where $c_{i}$ represents the color of the $i$ th Gaussian along with the camera ray; $d_{i}$ represents the distance from the camera along with the ray to the $i$ th Gaussian. In the process of splatting in 3DGS, the differentiable transformation function from 3D-Gaussian $(p_{w},\Sigma_{w})$ to 2D-Gaussian $(p_{c},\Sigma_{c})$ is expressed as follows:

p_{c}=T_{cw}\ast p_{w},\Sigma_{c}=JW\Sigma_{w}W^{T}J^{T},

(3)

where $c$ and $w$ respectively represent camera and world Coordinate System; $T_{cw}\in SE^{3}$ represents camera poses; $J$ is the linear approximation of the Jacobian matrix for the projection transformation; $W$ is the rotational component of $T_{cw}$ . Given that the Gaussian geometry in the scene is similar to an ellipsoid, its covariance matrix $\Sigma$ can be represented in an ellipsoidal form:

\Sigma=RSS^{T}R^{T},

(4)

where $R$ and $S$ represent the rotation and scaling matrix of the Gaussian ellipsoid derived from quaternion $q$ and scale $s$ , respectively. Based on the above formula, the open-source diff-gaussian-rasterization [26] achieves differentiable 3D Gaussian rasterization. This library is founded on differentiable rendering gradients, streamlining computations in void spaces and thereby expediting both training and rendering processes. In each iteration, the GS scene is optimized end to end by the weighted sum of $L_{1}$ and structural similarity index ( $L_{ssim}$ ) between the groundtruth and rendered image.

Inspired by [36], a new penalty term is set based on the scale and opacity of Gaussian in the optimization process to compactly represent the scene while preserving original performance. The binary masks $B\in[0,1]$ are set on the opacity $o_{i}$ and scale $s_{i}$ of each Gaussian and participate in the optimization process. To make them differentiable[37], an auxiliary parameters $b\in{R}$ is introduced. The masked scale $\hat{s}_{i}$ and masked opacity $\hat{o}_{i}$ are given as follows:

	$\displaystyle B_{i}=Stop(1[\sigma(b_{i})<\epsilon]-\sigma(b_{i}))+\sigma(b_{i}),$		(5)
	$\displaystyle\hat{s}_{i}=B_{i}s_{i},\hat{o}_{i}=B_{i}o_{i},$		(6)

where $Stop$ and $\sigma$ respectively represent the stop gradient and sigmoid function. This mask eliminates the occlusion of Mini-Gaussian, which has little effect during rasterization rendering. In each densification, besides splitting, cloning, and pruning, Based on this mask, some scene geometries are also removed to reduce the number of Gaussian involved in optimization. Moreover, to balance high-fidelity rendering with reducing scene geometries, a new mask loss is designed:

L_{M}=\frac{1}{N}\sum_{i=1}^{N}\sigma(b_{i}).

(7)

Thus, the complete loss function is constructed as follows:

L=(1-\lambda_{1})L_{1}+\lambda_{1}L_{ssim}+\lambda_{2}L_{M},

(8)

where $\lambda_{1}$ and $\lambda_{2}$ represent the weights. After optimization based on this loss function (8) is finished, the masks will be discarded as there are no longer useful.

III-B Dual Keyframe Strategy

The dual keyframe strategy consists of a motion filter and an information filter, respectively corresponding to motion keyframe and information keyframe.

Motion Filter It performs feature extraction on each frame and only retains frames that exceed the correlation threshold. Similar to [31], three different networks are designed as: the feature network, context network, and update network.

Each input image is processed by the feature network consisting of six residual modules and three downsampling modules, producing dense feature maps at 1/8 resolution of the original image size. Given that the input image $img\in R^{H\times W\times 3}$ , the output dense feature map.The ouput dense feature map $dmap\in R^{H/8\times W/8\times D}$ where we set $D=256$ . These dense feature maps are used to construct correlation pyramids and obtain sets of correlated feature vectors, while finer texture features extracted by the context network are added to the update network during each update process. Inter-frame motion vector is obtained after the update process.

Correlation Pyramid For any pair of frames $(I_{i},I_{j})$ in the frame set, the 4D correlation vectors $Cor\in R$ are computed by taking the dot product of all pairs of feature vectors from the dense feature maps $dmap_{i}$ and $dmap_{j}$ .

Cor(I_{i},I_{j})_{u_{1}v_{1}u_{2}v_{2}}=\left\langle dmap(I_{i})_{u_{1}v_{1}},% dmap(I_{j})_{u_{2}v_{2}}\right\rangle.

(9)

Then, average pooling is performed on the last two dimensions of the relevant vectors to construct a 4-level correlation pyramid. We also employ a query operation, which takes an $H\times W$ coordinate grid as the input and retrieves values from the correlation volume using bilinear interpolation. This operator is applied to each correlation volume in the pyramid, and the final feature vector is computed by concatenating the results from each level.

During the motion filter process, the motion threshold and maximum frame interval are preset. When the norm of its inter-frame motion vectors exceeds the motion threshold or its inter-frame interval with respect to the previous motion keyframe exceeds the maximum frame interval, the current frame is added as a new motion keyframe. The initial pose of the new motion keyframe is set based on the motion model, where the motion vector is defined as the pose transformation between the previous motion keyframe and the one before it.

Information Filter Considered the high real-time demand of SLAM algorithms, it is impractical to use all motion keyframes for optimization. Therefore, the tracking thread performs dual keyframe selection to update and maintain a sliding window of information keyframes. The information threshold and minimum mapping frame distance are preset. A motion keyframe is added to the sliding window if its relative complement to the previous information keyframe exceeds the threshold or its distance to the previous informative keyframe exceeds the minimum mapping frame distance. To maintain a constant window size, deletion operation is performed as well. An information keyframe is removed if its overlap coefficient with the new informative keyframe falls below a threshold or its inter-frame displacement from the previous informative keyframe is minimal.

For the motion keyframe $i$ , its relative complement ( $RC$ ) and overlap coefficient ( $OC$ ) are defined as follows:

RC(i,j)=\frac{G_{i}-G_{i}\cap G_{j}}{G_{i}\cup G_{j}},

(10)

OC_{i,j}=\frac{G_{i}\cap G_{j}}{min(G_{i},G_{j})},

(11)

where $G_{i}$ represents the scene geometry in the $i$ th frame; $j$ denotes the preceding information keyframe for motion keyframe $i$ , while $k$ denotes the preceding information keyframe for information keyframe $j$ .

III-C 3DGS-based Direct pose optimization

Firstly, it is crucial to acknowledge that the rendering results of 3DGS inherently exhibit some degree of blur. It is impractical to optimize the partial pixels between real photos and rendered images. Although the pose optimization using the entire photo may slightly lag in speed, it aligns with the differentiable rendering framework of 3DGS. Therefore, all pixels are directly used to the pose optimization framework based on the photometric error between real photos and rendered images. The photometric error relative to the camera pose and depth (if available) for pose optimization is defined as follows:

	$\displaystyle E_{rgb}=opacity*\left\\|I_{render}-I_{gt}\right\\|_{1},$		(12)
	$\displaystyle E_{d}=opacity*\left\\|D_{render}-D_{gt}\right\\|_{1}.$		(13)

Inspired by [27], the Lie algebra helps find the smallest Jacobian matrix with the right number of degrees of freedom, so no extra calculations are needed.The chain expansion of the transformation function for rasterization rendering in 3DGS is expressed as follows:

\frac{\partial p_{c}}{\partial T_{cw}}=\frac{\partial p_{c}}{\partial p_{w}}% \frac{\partial p_{w}}{\partial T_{cw}},

(14)

\frac{\partial\Sigma_{c}}{\partial T_{cw}}=\frac{\partial\Sigma_{c}}{\partial J% }\frac{\partial J}{\partial p_{c}}\frac{\partial p_{c}}{\partial T_{cw}}+\frac% {\partial\Sigma_{c}}{\partial W}\frac{\partial W}{\partial T_{cw}}.

(15)

The Jacobian matrix of a function $f$ acting on a manifold was proposed in [38]:

\frac{{}^{\chi}\partial f(\chi)}{\partial\tau}\overset{\bigtriangleup}{=}\lim_% {\tau\to 0}\frac{\partial Log(f(\chi)^{-1}\circ f(\chi\circ Exp(\chi)))}{% \partial\tau},

(16)

where $\chi\in SE^{3}$ ; $\tau\in\mathfrak{se}^{3}$ ; $\circ$ denotes the group operator; $Exp$ and $Log$ respectively represent the exponential and logarithmic functions mapping from the Lie algebra to the Lie group. Therefore, we can get:

\frac{\partial p_{c}}{\partial T_{cw}}=[1,-p_{c}^{\times}],

(17)

\frac{\partial W}{\partial T_{cw}}=\begin{bmatrix}0&W_{:,1}^{\times}\\ 0&W_{:,2}^{\times}\\ 0&W_{:,3}^{\times}\end{bmatrix},

(18)

where $\times$ represents the skew-symmetric matrix of a three-dimensional vector.

III-D SLAM System

Figure 1 illustrates the overall framework of the proposed MotionGS. Integrating the aforementioned modules, MotionGS mainly consists of two parts: tracking and Gaussian dense mapping. Each part runs as an independent thread and maintains communication with each other, collectively constructing the high-fidelity dense map reconstruction .

III-D1 Tracking

The tracking thread is responsible for the dual keyframe strategy: motion and information keyframes. Motion keyframes are primarily utilized for tracking purposes, whereas information keyframes are crucial for mapping.

The tracking thread performs feature extraction and motion filter on each frame image. It sets the initial pose of the current motion keyframe based on the previous motion keyframe and then minimizes the photometric error between the groundtruth image and the rendered image to obtain a coarse pose of the motion keyframe. Finally, the tracking thread continuously updates and maintains a sliding window of information keyframes.

III-D2 Mapping

The mapping thread is responsible for the joint optimization of 3D scene geometries and keyframe poses, as well as real-time rendering. As the main content of interaction between tracking and mapping, the information keyframe sliding window is directly used for the joint optimization. The Adam optimizer in pytorch is utilized and the mapping loss $L_{map}$ between rendered images and groundtruth images is showed below.

\displaystyle L_{map}=\sum_{i}^{m+n}(\left\|I_{i}^{*}-I^{gt}_{i}\right\|_{1})+% \beta\sum_{j}^{G}\left\|S_{j}-\bar{S}\right\|_{1}),

(19)

where $m$ represents the size of the sliding window, which is the number of frames used to compute the photometric loss within the window; $n$ denotes the number of randomly selected historical frames; $I^{*}$ denotes the rendered results; $G_{i}$ represents the set of all Gaussians; $S$ denotes the scale of Gaussian and the $\bar{S}$ denotes the mean scale of all Gaussians.

In addition to general splitting and pruning, the color refinement of the scene will also be regularly performed based on the new loss function (8). After the tracking is completed, the entire scene refinement is done in the mapping thread like SfM. In scene refinement, redundant Gaussian are removed, and the size of the scene model is reduced accordingly.

IV Experiment

IV-A Experimental Setting

The experiments have been conducted on two well-known datasets, TUM RGB-D [39] (3 sequences) and Replica [40] (8 sequences), to compare the performance of the MotionGS with other methods. The algorithm results are obtained on the server computer equipped with 64 cores and RTX 3090 GPU.

IV-A1 Metrics

As the two key components of SLAM, tracking and mapping are both crucial. For the tracking part, the root mean square error (RMSE) of the absolute trajectory error (ATE) of keyframes is calculated based on the tracking result of each algorithm. For the mapping part, the standard photometric rendering quality metrics are used, including: the peak signal to noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). Additionally, the storage usage for map representation is also analyzed and compared .

IV-A2 Baseline Methods

We compare and analyze MotionGS against classic traditional SLAM approach (ORB-SLAM2 [6]), deep learning based method (DROID-SLAM [31]), NeRF-based SLAM methods (iMAP [17], NICE-SLAM [19], Vox-Fusion [18], ESLAM [20], Co-SLAM [22], Point-SLAM [23]), and 3DGS-based SLAM methods (MonoGS [27], SplaTAM [28], GS-SLAM [25]). For non-dense SLAM methods, only tracking accuracy is compared. The metrics of the baselines are took from [27, 28, 25].

IV-B Evaluation

IV-B1 Track

Table I displays the ATE metrics of all above methods on the TUM dataset. In the RGB-D setting, MotionGS has surpassed the state-of-the-art methods of the 3DGS-based SLAM and NeRF-based SLAM in the fr1 and fr3 scenes, only slightly inferior to the baseline methods in the fr2 scene. In the Mono setting, MotionGS outperforms MonoGS, which is also applicable to monocular setups. However, MotionGS still exhibits higher errors compared with DROID-SLAM and ORB-SLAM2 due to the absence of loop closure detection and global BA. This demonstrates the pivotal role of loop closure detection and global BA in enhancing the localization accuracy of SLAM methods and their potential in 3DGS scene.

Table II presents the ATE metrics of all above methods on the Replica dataset. While the baseline methods have achieved remarkable millimeter-level accuracy, MotionGS remains competitive, surpassing them in five out of eight scenes. In the o0 and r0 scenarios, the best method only outperforms MotionGS by 0.14 cm in ATE. The excellent performance shown in Tables I and II is mainly attributed to the coarse to fine pose estimation. Tracking thread makes a coarse pose optimizition of the motion keyframe to reduce the accumulated tracking error caused by pose estimation for consecutive frames. Then, the joint optimization of poses and 3D Gaussians in Mapping thread further improves the tracking accuracy.

Class	Method	fr1	fr2	fr3	Arg
Mono	ORB-SLAM2	2.00	0.60	2.30	1.60
	DROID-SLAM	1.80	0.50	2.80	1.70
	MonoGS	4.15	4.79	4.39	4.44
	Ours	3.53	3.93	2.43	3.30
RGB-D(NeRF)	iMAP	4.90	2.00	5.80	4.23
	NICE-SLAM	4.26	6.19	6.87	5.77
	ESLAM	2.47	1.11	2.42	2.00
	Vox-Fusion	3.52	1.49	26.01	10.34
	Co-SLAM	2.40	1.70	2.40	2.17
	Point-SLAM	4.34	1.31	3.48	3.04
RGB-D(3DGS)	MonoGS	1.52	1.58	1.65	1.58
	SplaTAM	3.35	1.24	5.16	3.25
	GS-SLAM	3.30	1.32	6.60	3.70
	Ours	1.47	1.38	1.41	1.46

TABLE I: Comparison of tracking results ATE (cm) on TUM. The red, blue, and green in the above table represent the first, second, and third, respectively.

Method	r0	r1	r2	o0	o1	o2	o3	o4	Arg
iMAP	3.12	2.54	2.31	1.69	1.03	3.99	4.05	1.93	2.58
NICE-SLAM	0.97	1.31	1.07	0.88	1.00	1.06	1.10	1.13	1.07
Vox-Fusion	1.37	4.70	1.47	8.48	2.04	2.58	1.11	2.94	3.09
ESLAM	0.71	0.70	0.52	0.57	0.55	0.58	0.72	0.63	0.63
Point-SLAM	0.61	0.41	0.37	0.38	0.48	0.54	0.69	0.72	0.53
MonoGS	0.47	0.43	0.31	0.70	0.57	0.31	0.31	3.2	0.79
SplaTAM	0.36	0.31	0.40	0.29	0.47	0.27	0.29	0.32	0.55
GS-SLAM	0.48	0.53	0.33	0.52	0.41	0.59	0.46	0.7	0.50
Our	0.46	0.28	0.31	0.43	0.27	0.19	0.14	1.85	0.49

TABLE II: Comparison of tracking results ATE (cm) on Replica. The red, blue, and green in the above table represent the first, second, and third, respectively.

IV-B2 Render

Table III shows the rendering performance of MotionGS on the Replica dataset, compared with NICE-SLAM, Point-SLAM, MonoGS, SplaTAM and GS-SLAM. The result shows that 3DGS-based SLAM has competitive rendering performance compared with the state-of-the-art NeRF-based SLAM. Moreover, MotionGS attains the best PSNR and LPIPS on each scenes, outperforming the current state-of-the-art 3DGS-based SLAM. Although the SSIM metrics of MotionGS are not the optimal performance on each scenes, the mean of all SSIM metrics ranks first. We attribute the success to the mask on the scale and opacity of Gaussian, which protects MotionGS from the influence of the almost transparent or negligible Gaussians in rendering. The scene visualizations in these datasets are showed in the Figs 3 and 2. Compared with the baselines, higher fidelity and more scene details are achieved in the MotionGS. Our rendering effect showcases enhanced detail representation, as exemplified by the clocks, flower pots, murals, and books within the scene.

Method	Metric	r0	r1	r2	o0	o1	o2	o3	o4	Arg
NICE-SLAM	PSNR $\left[\mathrm{dB}\right]\uparrow$	22.12	22.47	24.52	29.07	30.34	19.66	22.23	24.94	24.42
	$\mathrm{SSIM}\uparrow$	0.689	0.757	0.814	0.874	0.886	0.797	0.801	0.856	0.809
	$\mathrm{LPIPS}\downarrow$	0.33	0.271	0.208	0.229	0.181	0.235	0.209	0.198	0.233
Point-SLAM	$\mathrm{PSNR}\left[\mathrm{dB}\right]\uparrow$	32.40	34.08	35.50	38.26	39.16	33.99	33.48	33.49	35.17
	$\mathrm{SSIM}\uparrow$	0.974	0.977	0.982	0.983	0.986	0.960	0.960	0.979	0.975
	$\mathrm{LPIPS}\downarrow$	0.113	0.116	0.111	0.100	0.118	0.156	0.132	0.142	0.124
MonoGS	$\mathrm{PSNR}\left[\mathrm{dB}\right]\uparrow$	34.83	36.43	37.49	39.95	42.09	36.24	36.7	36.07	37.50
	$\mathrm{SSIM}\uparrow$	0.954	0.959	0.965	0.971	0.977	0.964	0.963	0.957	0.960
	$\mathrm{LPIPS}\downarrow$	0.068	0.076	0.075	0.072	0.055	0.078	0.065	0.099	0.070
SplaTAM	$\mathrm{PSNR}\left[\mathrm{dB}\right]\uparrow$	32.86	33.89	35.25	38.26	39.17	31.97	29.70	31.81	34.11
	$\mathrm{SSIM}\uparrow$	0.98	0.97	0.98	0.98	0.98	0.97	0.95	0.95	0.97
	$\mathrm{LPIPS}\downarrow$	0.07	0.10	0.08	0.09	0.09	0.10	0.12	0.15	0.10
GS-SLAM	$\mathrm{PSNR}\left[\mathrm{dB}\right]\uparrow$	31.56	32.86	32.59	38.70	41.17	32.36	32.03	32.92	34.27
	$\mathrm{SSIM}\uparrow$	0.968	0.973	0.971	0.986	0.993	0.978	0.97	0.968	0.975
	$\mathrm{LPIPS}\downarrow$	0.094	0.075	0.093	0.050	0.033	0.094	0.110	0.112	0.082
Ours	$\mathrm{PSNR}\left[\mathrm{dB}\right]\uparrow$	36.58	38.82	39.71	43.47	43.85	38.87	37.66	37.86	39.60
	$\mathrm{SSIM}\uparrow$	0.967	0.973	0.977	0.985	0.985	0.976	0.973	0.97	0.976
	$\mathrm{LPIPS}\downarrow$	0.05	0.05	0.05	0.05	0.03	0.042	0.041	0.031	0.043

TABLE III: Comparison of rendering results on Replica. The red, blue, and green in the above table represent the first, second, and third, respectively.

IV-B3 Ablation Analysis

The ablation experiments are showed in Tables IV and V. It is clearly evident that keyframe selection exerts a profound influence on the tracking accuracy of MotionGS, thereby highlighting the advantage of the dual keyframe selection strategy. Furthermore, the SplaTAM has nearly 0.5GB of storage memory as shown in Table V, while storage memory of MotionGS is less than 50MB. The mask effectively reduces the number of Gaussians utilized in the scene representation, making it a more efficient solution in comparison with the SplaTAM.

Method (RGB-D)	Metric	fr1	fr2	fr3	Arg
Out of kf strategy	ATE $\downarrow$	1.87	1.27	3.48	2.21
Our	ATE $\downarrow$	1.47	1.38	1.41	1.46

TABLE IV: Ablation Analysis of the KF Strategy

Method	Metric	r0	r1	r2	o0	o1	o2	o3	o4
SplaTAM	Storage(Mb)	329	436	371	404	365	295	341	346
SplaTAM	Num of Gaussian( $10^{4}$ )	509	673	573	624	564	455	527	534
Ours^∗	Storage(Mb)	20.5	18.8	18.3	18.4	13.8	26.0	22.2	20.5
Ours^∗	Num of Gaussian( $10^{4}$ )	37	29	28	28	21	39	36	32
Ours	Storage(Mb)	16.6	14.2	15.9	15.6	11.8	23.6	20.8	17.6
Ours	Num of Gaussian( $10^{4}$ )	31	24	26	25	18	35	33	27

TABLE V: Ablation Analysis of the Mask loss. The Ours^∗ refers to the MotionGS without mask loss.

V Conclusion

This study proposed a 3DGS-based SLAM named MotionGS that integrates deep visual features, dual keyframe selection, and 3DGS. With its exquisite design, the state-of-the-art performance of the MonoGS has been fully demonstrated in extensive experiments. The proposed approach further emphasizes the broad potential of 3DGS in the SLAM field. Based on this work, the multi-sensor 3DGS-based SLAM tailored for large-scale outdoor scenes would be the next research.

References

[1] G. Bresson, Z. Alsayed, L. Yu, and S. Glaser, “Simultaneous localization and mapping: A survey of current trends in autonomous driving,” IEEE Transactions on Intelligent Vehicles, vol. 2, pp. 194–220, 2017.
[2] J. Drupt, A. I. Comport, C. Dune, and V. Hugel, “Mam3slam: Towards underwater-robust multi-agent visual slam,” Ocean Engineering, vol. 302, p. 117643, 2024.
[3] X. Jiang, L. Zhu, J. Liu, and A. Song, “A slam-based 6dof controller with smooth auto-calibration for virtual reality,” The Visual Computer, vol. 39, pp. 1–14, 06 2022.
[4] C. Theodorou, V. Velisavljevic, V. Dyo, and F. Nonyelu, “Visual slam algorithms and their application for ar, mapping, localization and wayfinding,” Array, vol. 15, pp. 100–222, 2022.
[5] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021.
[6] R. Mur-Artal and J. D. Tardós, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.
[7] M. Keller, D. Lefloch, M. Lambers, S. Izadi, T. Weyrich, and A. Kolb, “Real-time 3d reconstruction in dynamic scenes using point-based fusion,” in 2013 International Conference on 3D Vision - 3DV 2013, 2013, pp. 1–8.
[8] K. Wang, F. Gao, and S. Shen, “Real-time scalable dense surfel mapping,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 6919–6925.
[9] T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger, “Elasticfusion: Real-time dense slam and light source estimation,” The International Journal of Robotics Research, vol. 35, no. 14, pp. 1697–1716, 2016.
[10] F. Ruetz, E. Hernández, M. Pfeiffer, H. Oleynikova, M. Cox, T. Lowe, and P. Borges, “Ovpc mesh: 3d free-space representation for local ground vehicle navigation,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8648–8654.
[11] T. Schöps, T. Sattler, and M. Pollefeys, “Surfelmeshing: Online surfel-based mesh reconstruction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 10, pp. 2494–2507, 2020.
[12] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, “Real-time 3d reconstruction at scale using voxel hashing,” ACM Trans. Graph., vol. 32, no. 6, nov 2013.
[13] C. Sun, M. Sun, and H.-T. Chen, “Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 5459–5469.
[14] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 5501–5510.
[15] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in 2011 10th IEEE International Symposium on Mixed and Augmented Reality, 2011, pp. 127–136.
[16] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: representing scenes as neural radiance fields for view synthesis,” Commun. ACM, vol. 65, no. 1, p. 99–106, dec 2021.
[17] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 6229–6238.
[18] X. Yang, H. Li, H. Zhai, Y. Ming, Y. Liu, and G. Zhang, “Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation,” in 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2022, pp. 499–507.
[19] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “Nice-slam: Neural implicit scalable encoding for slam,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 12 786–12 796.
[20] M. M. Johari, C. Carta, and F. Fleuret, “Eslam: Efficient dense slam system based on hybrid representation of signed distance fields,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 17 408–17 419.
[21] Y. Zhang, F. Tosi, S. Mattoccia, and M. Poggi, “Go-slam: Global optimization for consistent 3d instant reconstruction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 3727–3737.
[22] H. Wang, J. Wang, and L. Agapito, “Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 13 293–13 302.
[23] E. Sandström, Y. Li, L. Van Gool, and M. R. Oswald, “Point-slam: Dense neural point cloud-based slam,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 18 433–18 444.
[24] A. Rosinol, J. J. Leonard, and L. Carlone, “Nerf-slam: Real-time dense monocular slam with neural radiance fields,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 3437–3444.
[25] C. Yan, D. Qu, D. Xu, B. Zhao, Z. Wang, D. Wang, and X. Li, “Gs-slam: Dense visual slam with 3d gaussian splatting,” 2024.
[26] B. Kerbl, G. Kopanas, T. Leimkuehler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Trans. Graph., vol. 42, no. 4, jul 2023.
[27] H. Matsuki, R. Murai, P. H. J. Kelly, and A. J. Davison, “Gaussian splatting slam,” 2024.
[28] N. Keetha, J. Karhade, K. M. Jatavallabhula, G. Yang, S. Scherer, D. Ramanan, and J. Luiten, “Splatam: Splat, track & map 3d gaussians for dense rgb-d slam,” 2024.
[29] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, “Dtam: Dense tracking and mapping in real-time,” in 2011 International Conference on Computer Vision, 2011, pp. 2320–2327.
[30] T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, and J. Mcdonald, “Kintinuous : Spatially extended kinectfusion,” 01 2012.
[31] Z. Teed and J. Deng, “Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 16 558–16 569.
[32] W. Zhang, T. Sun, S. Wang, Q. Cheng, and N. Haala, “Hi-slam: Monocular real-time dense mapping with hybrid implicit fields,” IEEE Robotics and Automation Letters, vol. 9, no. 2, pp. 1548–1555, 2024.
[33] H. Huang, L. Li, H. Cheng, and S.-K. Yeung, “Photo-slam: Real-time simultaneous localization and photorealistic mapping for monocular, stereo, and rgb-d cameras,” 2024.
[34] V. Yugay, Y. Li, T. Gevers, and M. R. Oswald, “Gaussian-slam: Photo-realistic dense slam with gaussian splatting,” 2024.
[35] S. Ha, J. Yeon, and H. Yu, “Rgbd gs-icp slam,” 2024.
[36] J. C. Lee, D. Rho, X. Sun, J. H. Ko, and E. Park, “Compact 3d gaussian representation for radiance field,” 2024.
[37] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” 2013.
[38] J. Solà, J. Deray, and D. Atchuthan, “A micro lie theory for state estimation in robotics,” 2021.
[39] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of rgb-d slam systems,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, pp. 573–580.
[40] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe, “The replica dataset: A digital replica of indoor spaces,” 2019.