Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction

3DGS [9] introduces the 3D Gaussians as differential volumetric representations of radiance fields, allowing high-quality real-time novel view synthesis. The set of 3D Gaussians is initialized from the calibrated camera poses and sparse point clouds generated from SfM. Each Gaussian is defined by position $\boldsymbol{\mu}$, covariance matrix $\boldsymbol{\Sigma}$: $G(\mathbf{x})=e^{-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^{T}\boldsymbol{\Sigma}^{-1}(x-\boldsymbol{\mu})}.$ The covariance can be decomposed from a scaling matrix $\mathbf{S}$ and rotation matrix $\mathbf{R}$: $\boldsymbol{\Sigma}=\mathbf{R}\mathbf{S}\mathbf{S}^{T}\mathbf{R}^{T}$. To render a novel view, the 3D Gaussians are projected to 2D camera view $\mathbf{T}$: $\boldsymbol{\Sigma}^{\prime}=\mathbf{J}\mathbf{T}\boldsymbol{\Sigma}\mathbf{T}^{T}\mathbf{J}^{T}$, where $\mathbf{J}$ is the Jacobian of the affine approximation of the projective transformation. To render the color, 3DGS further optimizes opacity and SH coefficients, following the point-based differential rendering by rasterizing anisotropic splats with $\alpha$-blending. The color and depth are rasterized following:

where $\mathbb{c}_{i}$ and $\alpha_{i}$ denote the color and opacity of the Gaussian, $d_{i}$ is the z-axis of the points by projecting the center of 3D Gaussians $\boldsymbol{\mu}$ to the camera coordinate. In summary, the parameters to optimize for the Gaussians include: $\Theta=\{\boldsymbol{\mu},\boldsymbol{\Sigma},\alpha,\mathbf{c}\}$. To realize SfM-free scene reconstruction, we need to both recover the camera poses $\mathbf{T}$ and optimize the Gaussian parameters $\Theta$.

Given first frame $\mathbf{I}_{0}$ and the known intrinsic $\mathbf{K}$, we generate the pointcloud $\mathbf{P}$ by unprojecting the monocular depth $\mathbf{D}_{0}$ by the initial identity camera pose $\mathbf{T}_{0}$: $\mathbf{P}=\pi^{-1}(\mathbf{T}_{0},\mathbf{D}_{0},\mathbf{K})$, where $\pi^{-1}$ is the pixel-to-world projection. The center of Gaussians $\boldsymbol{\mu}$ is initialized by $\mathbf{P}$. The color of each point $\mathbf{c}$ is initialized with the SH coefficient from the first frame. Other parameters are initialized following the implementation in 3DGS [9]. After initialization, we optimize the 3D Gaussians $G_{0}$ by minimizing the losses introduced in Sec. 2.4.

In this step, we fix the parameters of 3D Gaussians (i.e. assume the current GS is pseudo-GT) and update the camera pose by matching the projection flow from 3D Gaussians with the robust correspondences from filtered optical flow.

To update the camera pose by gradient descent, we first transform the 3D Gaussians $G$ with the camera pose $\mathbf{T}$. We take the camera poses as the optimizable variables and represent the rotations in quaternion $\mathbf{q}$ and translation vector $\mathbf{t}$. At timestep $t+1$, its camera pose $\hat{\mathbf{T}}_{t+1}$ is initialized from the previous camera poses $\hat{\mathbf{T}}_{t}$ and $\hat{\mathbf{T}}_{t-1}$ based on the constant velocity assumption: $\hat{\mathbb{q}}_{t+1}=\hat{\mathbb{q}}_{t}+(\hat{\mathbb{q}}_{t}-\hat{\mathbb{q}}_{t-1})+\Delta\mathbb{q},\quad\hat{\mathbb{t}}_{t+1}=\hat{\mathbb{t}}_{t}+(\hat{\mathbb{t}}_{t}-\hat{\mathbb{t}}_{t-1})+\Delta\mathbb{t}.$

Previous methods [2, 11] mostly adopt the photometric loss $\mathcal{L}_{rgb}$ to match the rendered color $\hat{\mathbb{C}}_{t+1}$ and ground truth color $\mathbb{I}_{t+1}$ for pose estimation with gradient optimization:

However, the application of photometric loss in optimizing camera poses within surgical scenes encounters limitations. First, the homogeneity and sparse texturing of the surgical surfaces lead to ambiguities in feature matching. Second, photometric inconsistencies across different views are quite common due to varied lighting conditions, the existence of reflective surfaces of the surgical instruments and tissues, and the presence of non-Lambertian surfaces. Consequently, using only the photometric loss for pose estimation is prone to converge to some local minima, thus leading to inaccurate reconstruction in the following step.

Second, a consistency check is introduced to remove the outliers to maintain rigid and reliable points in the optical flow. In dynamic surgical environments characterized by transient objects and photometric inconsistencies, it is essential to identify and preserve correspondences that are both rigid and reliable for accurate matching. Utilizing the optical flow $\mathbf{O}_{t-1\rightarrow t}$, we assess the epipolar geometry informed by the estimated camera poses $\hat{\mathbf{T}}_{t-1}$ and $\hat{\mathbf{T}}_{t}$. This assessment ensures that correctly matched points align with their respective epipolar lines for robust matching. Therefore, we can find the rigid and reliable points that better satisfy the epipolar geometry in $t$ to further filter out outliers in $\mathbf{O}_{t\rightarrow t+1}$ based on the continuity of endoscopic video. As shown in Fig. 3(a), we compute the Sampson distance [5] to measure the geometric error between a point in one image and its corresponding epipolar line in the other image. We take a threshold $\beta$ to obtain a rigid mask $\mathbf{M}_{r}$ for time $t$, ensuring that only robust correspondences are utilized for subsequent pose estimation tasks from $t$ to $t+1$. Finally, we obtain the flow mask from the consistency check: $\mathbf{M}=\mathbf{M}_{v}\odot\mathbf{M}_{r}.$

After estimating the camera pose $\hat{\mathbf{T}}_{t+1}$, we optimize the parameters $\Theta$ of 3D Gaussians $G$. Here, we keep the camera pose fixed and optimize the scene representation by minimizing the photometric loss, depth loss, and flow loss:

where $\lambda_{3}$ is the weight for depth loss, $\mathcal{L}_{depth}$ denotes a scale-invariant loss [16] between rendered depth $\hat{\mathbf{D}}$ and monocular depth $\mathbf{D}$ generated from Depth-Anything [20]. Since the projection flow is derived from rendered depth for reprojection, the flow loss directly contributes to a more precise estimation of depth. By optimizing the 3D Gaussians for both photometric consistency and flow dynamics, the geometry of the 3D Gaussians is not only consistent with the observed image data but also adheres to the expected motion patterns across frames. Finally, we add or prune the 3D Gaussians with adaptive density control, resulting in a progressive growing process for reconstruction.

We compare our method with existing state-of-the-art SfM-free methods: Nope-NeRF [2], BARF [11], NeRFmm [19] and SC-NeRF [7]. Quantitative results in Tab. 1 demonstrate that our method outperforms all the baselines. Only based on photometric loss, BARF [11], NeRFmm [19], and SC-NeRF [7] fail to recover the correct camera pose, suffering from the challenging surgical scenes. With constraints from depth distortion, Nope-NeRF [2] improves the performance compared to other baselines but still fails to handle large endoscopic movement (See Fig. 4). Thanks to the flow matching and the consistency check, our Free-SurGS could estimate accurate camera poses for scene reconstruction and render photo-realistic images with 3DGS. The efficiency comparison in Tab. 1 also demonstrates our faster training, higher inference speed, and lower memory of parameters, satisfying real-world surgical applications.

We conduct ablation studies to validate the effectiveness of the proposed modules in Tab. 2. The flow loss $\mathcal{L}_{flow}$ compensates for the limitation of photometric loss and improves the accuracy of pose estimation. The consistency check could further enhance the robustness of large movement and semi-static scenes. With more accurate poses as input, the performance of 3DGS is further improved to reconstruct the surgical scene.