CoFiI2P: Coarse-to-Fine Correspondences-Based Image to Point Cloud Registration

Before the age of deep-learning, hand-crafted detectors and descriptors (i.e., SIFT [13] and ORB [14]) are widely used to extract correspondences for matching. Compared to traditional methods, learning-based methods improve the robustness and accuracy of image matching with large viewpoint differences and illumination changes. SuperPoint [15] proposed a self-learning training method through homography adaptation. SuperGlue [16] proposed an attention-based graph neural network (GNN) for feature matching. Patch2Pix [17] is the first work to obtain patch matches and regress pixel-wise matches in a coarse-to-fine manner.

Point cloud registration aims to estimate the optimal rigid transformation of two point clouds. Correspondence-based methods [18, 19] estimate the correspondences first and recover the transformation with robust estimation methods [20]. RoReg [21] embeds orientation information of point cloud to estimate local orientation and refine coarse rotation through residual regression to achieve fine registration. In the transformer era, point-based transformers [22, 12] have emerged and shown great performance. CoFiNet [22] followed LoFTR’s [10] design and proposed the coarse-to-fine correspondences for registration, which computed coarse matches with descriptors strengthened by the transformer and refined coarse matches through density adaptive matching module. GeoTransformer [12] uses the Transformer to fully exploit the 3D properties of point clouds. However, I2I or P2P registration methods can not be directly used for the I2P registration, which extracts heterogeneous features from cross-modality data.

Fine registration methods have been thoroughly studied for several decades. Early-stage works [25, 26] utilize various artificial targets as calibration constraints. In recent years, some approaches have argued that structured features shared in images and point cloud could be used for target-less I2P registration, e.g., edge feature [23] and line feature [5]. [23] employs edge information and optimizes transformation according to the response value. [5] extracts lines from image and point clouds for registration. Recently, researches on 2D and 3D semantic segmentation motivate semantic feature-based I2P registration. Se-calib [24] further studies semantic edges in I2P registration to employ more common semantic information instead of a certain type of object. Overall, fine registration methods achieve high accuracy but depend on the quality of the initial values.

2D3D-Matchnet [7] is the pioneering method to regress the relative transform parameters with CNN. It extracts SIFT [13] and ISS [27] keypoints from image and point cloud and learns descriptors with a Siamese network. Experimental results show that hand-crafted detectors from different modalities match poorly. HAS-Net [28] proposed a novel network for learning cross-domain descriptors from 2D image patches and 3D point cloud volumes. However, both 2D3D-Matchnet and HAS-Net split the image and point cloud into patches and volumes, and then match with the descriptors of patches and volumes, resulting in the loss of long-range context and high-level information. DeepI2P [8] proposed a feature fusion module to merge image and point cloud information and classified points in/beyond the camera frustum. CorrI2P [9] predicted pixels and points in overlapping areas and matched with dense per-pixel/per-point features directly to get I2P correspondences. Although overlapping region detectors significantly reduce the number of false candidates, I2P registration only using the low-level feature without global guidance leads to serious mismatches. Inspired by the coarse-to-fine strategy in CoFiNet [22], we propose the CoFiI2P network, a coarse-to-fine I2P registration approach that integrates high-level correspondence information into low-level matching to effectively reject mismatches.

[32, 33] have shown that vision transformer (ViT) outperforms traditional CNN-based methods with a large margin in classification, detection, segmentation and other downstream tasks. Furthermore, recent approaches [11, 10, 34] have introduced transformer modules for I2I and P2P registration tasks. Therefore, we introduce the I2P transformer module customized for cross-modality registration task to enhance the representability and robustness of descriptors. Different from ViT used in the same-modality registration tasks, our I2P transformer contains both self-attention modules for space context capturing in homogeneous data and cross-attention modules for hybrid feature integration among heterogeneous data.

For the self-attention module, given a coarse-level feature map $\mathbf{F}\in\mathbb{R}^{N\times C}$ of image or point cloud, the query, key and value vectors $\mathbf{q},\mathbf{k},\mathbf{v}$ are generated as:

where $\mathbf{W}_{q},\mathbf{W}_{k}\in\mathbb{R}^{C_{qk}^{sa}\times C},\mathbf{W}_{v}\in\mathbb{R}^{C_{v}^{sa}\times C}$ are learnable weight matrixs. Then, the global attention enhanced feature map $\mathbf{F}^{sa}$ is calculated as:

Softmax operates row-wise on the attention matrix $\mathbf{q}\mathbf{k}^{\top}$ to obtain the weights on the values. Extracted global-aware features $\mathbf{F}^{sa}$ are fed into the feed-forward network (FFN) to fuse the spatial relation information in channel dimension. Given a feature map $\mathbf{F}$, the relative positions are encoded with multi-layer perception (MLP) [35].

Cross-attention is designed for fusing image and point cloud features in the I2P registration task. Given the feature maps $\mathbf{F}_{\tilde{P}},\mathbf{F}_{\tilde{I}}$ for super-points set $\tilde{P}$ and super-pixels set $\tilde{I}$, cross-attention enhanced feature maps $\mathbf{F}_{\tilde{P}}^{ca}$ of point cloud and $\mathbf{F}_{\tilde{I}}^{ca}$ of image are calculated as:

where $\mathbf{q}_{\tilde{P}},\mathbf{k}_{\tilde{P}},\mathbf{v}_{\tilde{P}}$ are query, key and value vectors of point cloud feature $\mathbf{F}_{\tilde{P}}$, and $\mathbf{q}_{\tilde{I}},\mathbf{k}_{\tilde{I}},\mathbf{v}_{\tilde{I}}$ are query, key and value vectors of image feature $\mathbf{F}_{\tilde{I}}$. The softmax operation is the same as in the self-attention module.

Remark 1. While the self-attention module encodes the spatial and geometric features for each super-pixel and super-point, the cross-attention module injects the geometric structure information and texture information across image and point cloud respectively. Outputs of the I2P transformer carry powerful cross-modality information for matching.

The field of view (FoV) of the monocular camera is obviously smaller than the laser scans of 3D Lidar (e.g., Velodyne-H64), which sweeps 360 degrees in the horizontal direction. Therefore, only a small number of super-points are in the camera frustum. To filter out beyond-frustum super-points, we add a simple binary classification head to predict super-points in or beyond the frustum of the camera. The coarse level correspondences are estimated between in-frustum super-points set $\tilde{P}$ and super-pixels set $\tilde{I}$ by finding the nearest super-pixel $\tilde{i}$ in feature space:

where $\mathbf{F}_{\tilde{P}}$ and $\mathbf{F}_{\tilde{I}}$ are corresponding feature maps.

We evaluate our method on two public datasets, KITTI Odometry [37] and Nuscenes [38].

• •

  KITTI Odometry [37]: It consists of 11 sequences of images and point cloud data in urban environments with ground-truth calibration files. The camera intrinsic function $\Gamma$ extracted from calibration files is thought unbiased during experiments. Sequences 0-8 are used for training and 9-10 for testing as previous approaches [8, 9]. The image resolution is resized to 160$\times$512 and the point cloud is downsampled and randomly selected to 20480.

• •

  Nuscenes [38]: The image and point cloud pairs are generated by the official SDK, where the point cloud is accumulated from the nearby frames and the image is from the current frame. We follow the official data split of nuScenes to utilize 850 scenes for training, and 150 scenes for testing. We downsample the image resolution to 160$\times$320 and the point cloud size to 20480 similarly.

We compare the proposed CoFiI2P with two open-sourced I2P methods as follows.

• •

  DeepI2P [8]: It uses the frustum classification and inverse camera projection to estimate the camera pose. We use officially released code for reimplementation and use the 2D and 3D inverse camera projection for optimization, namely DeepI2P (2D) and DeepI2P (3D).

• •

  CorrI2P [9]: It is the SOTA I2P method. CorrI2P predicts the overlapping area and establishes correspondences densely for pose estimation. We use officially released code for reimplementation.

We calculate relative rotation error (RRE), relative translation error (RTE) and registration recall (RR) to evaluate the registration accuracy. Inlier ratio (IR) used in previous approaches [22, 39] is introduced to evaluate the quality of correspondences. For efficiency analysis, we report the frame-per-second (FPS) on the KITTI Odometry dataset, evaluated with a batch size of 1 on an Intel i9-13900K CPU and an NVIDIA RTX 4090 GPU. RRE and RTE are defined as :

where $\mathbf{r}$ is the Euler angle vector of $\mathbf{R}_{gt}^{-1}\mathbf{R}_{e}$, $\mathbf{R}_{gt}$ and $\mathbf{t}_{gt}$ are the ground-truth rotation and translation matrix, $\mathbf{R}_{e}$ and $\mathbf{t}_{e}$ are the estimated rotation and translation matrix.

RR metric estimates the percentage of correctly matched I2P pairs, indicating the descriptor learning ability of the network. RR is defined as:

$(\delta_{r},\delta_{t})$ is the threshold to remove false registration results, i.e. $(10^{\circ},5m)$. IR denotes the inlier ratio of matching pairs, which measures the accuracy of correspondences. The IR for point/pixel correspondences set $\Omega_{\mathcal{M}}$ is defined as :

in which $\tau$ is used to control the reprojection error tolerance.

With correct transform parameters provided by the calibration files during the training process, the ground truth correspondences $\Omega_{\mathcal{M^{\star}}}$ are established to supervise the network. We trained the whole network 25 epochs for the KITTI Odometry dataset and 10 epochs for the Nuscenes dataset. We use the Adam [40] to optimize the network, and the initial learning rate is 0.001 and multiple by 0.25 after every 5 epochs. For our joint loss, we set $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$. The safe radius $r$, positive margin $\Delta_{pos}$ and negative margin $\Delta_{neg}$ in loss function are set to 1, 0.2 and 1.8. Scale factor $\gamma$ is set to 10. Model configurations and more implementation details are available in our open-source code.

I2P transformer [41] with self-attention module and cross-attention module is crucial to image-to-point cloud alignment at the global level. In this part, we conduct ablation studies to assess the effectiveness of the I2P transformer. We train the CoFiI2P without any attention module as baseline. Then, the self-attention modules and cross-attention modules are added on the coarse level respectively. As shown in Table IV, performance of the baseline method drops significantly without any attention module. Besides, the self-attention module reduces the RRE to $1.78^{\circ}$ and the RTE about $0.41m$, and the cross-attention module reduces the RRE to $1.74^{\circ}$ and the RTE to $0.43m$ respectively. With both the self-attention and cross-attention modules, the CoFiI2P achieves the smallest registration error. Moreover, with both self-attention and cross-attention modules, the variance reduces by a large margin, which indicates that the I2P transformer block enhances both the accuracy and robustness.

our CoFiI2P proposes to estimate coarse correspondences at super-point/-pixel level first and then generate fine correspondences at point-pixel level sequentially. We conduct ablation experiments on the coarse-to-fine matching scheme to demonstrate that the progressive two-stage registration operates better than the one-stage registration used in previous I2P approaches. This ablation study employs the backbone with full I2P transformer blocks as baseline and evaluates the registration accuracy with only coarse or fine matching schemes. For coarse matching only, matching pairs are established on the coarse level and remapped to the fine resolution for pose estimation. By contrast, for fine matching only, matching pairs are established on the fine level directly, without guidance of coarse level correspondences. Experiment results in Table IV show that removing either the coarse matching stage or the fine matching stage leads to higher registration error and variance. We indicate that coarse-level registration provides robust correspondences and fine-level registration provides accurate matching pairs. Combining the coarse-level and fine-level registration sequentially makes it easier to access the global optimal solution in I2P registration.

Given the significant impact of point cloud density on representative learning, we conducted ablation studies to examine this influence. The results in Table IV present registration accuracy and computational complexity for various point cloud densities. The findings reveal that as point cloud density decreases, the qualitative metrics deteriorate, while higher point cloud densities correspond to a sharp increase in computational complexity. It’s an intuitive observation that low-density point clouds lose local structured information, and high-density point clouds carry a heavy computational burden. As a result, we opt for a compromise and select 20480 points, striking a balance between efficiency and accuracy.