RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAM

Currently, multi-modal 3D object detection has received considerable attention on popular datasets Geiger et al. (2012); Caesar et al. (2020). BEVFusion Liu et al. (2023) fuse multi-modal representations in a unified 3D or BEV space. TransFusion Bai et al. (2022) builds a two-stage pipeline where proposals are generated based on LiDAR features and further refined using query image features. DeepInteraction Yang et al. (2022) and SparseFusion Xie et al. (2023) further optimize the camera branch on top of TransFusion. Previous methods are highly optimized to achieve the best performance on clean datasets. However, they ignore common factors in the real world (e.g., bad weather and sensor noise). In this work, we consider a real-world robustness perspective and design a robust multi-modal 3D perception framework, RoboFusion.

Motivated by the success of Large Language Models (LLMs) OpenAI (2023), VFMs start to be explored in CV community. SAM Kirillov et al. (2023) leverages ViT Dosovitskiy et al. (2020) to train on the huge SA-1B dataset, containing 11 million samples, which enables SAM to be generalized to many scenes. Currently, there have been a few research endeavors aiming at integrating 3D object detectors with SAM. For instance, SAM3D Zhang et al. (2023b), as a LiDAR-only method, solely transforms LiDAR’s 3D perspective into a BEV (Bird’s Eye View) 2D space to harness the generalization capabilities of SAM, yielding sub-optimal performance on ‘clean’ datasets. Another in progress work, 3D-Box-Segment-Anything ¹¹1https://github.com/dvlab-research/3D-Box-Segment-Anything, tries to utilize SAM for 3D object detection. This indicates the highly attention of SAM like foundation models in 3D scenes in the literature. Our RoboFusion, as a multi-modal method, gives clear strategies to leverage the generalization capabilities of VFMs to address the OOD noise challenges inherent in existing 3D multi-modal object detection methods.

Preliminaries. SAM Kirillov et al. (2023), a VFM, achieves generalization across diverse scenes due to its extensive training on the large-scale SA-1B dataset—with over 11 million samples and 1 billion high-quality masks. Currently, SAM family Kirillov et al. (2023); Zhao et al. (2023); Zhang et al. (2023a) primarily support 2D tasks. However, directly extending VFMs like SAM to 3D tasks presents a gap. To address this, we combine SAM with multi-modal 3D models, merging 2D robust feature representations with 3D point cloud features to achieve robust fused features.

SAM-AD. To further adapt SAM with AD (autonomous driving) scenarios, we perform pre-training on SAM to obtain SAM-AD. Specifically, we curate an extensive collection of image samples from well-established datasets (i.e., KITTI Geiger et al. (2012) and nuScenes Caesar et al. (2020)), forming the foundational AD dataset. Following DMAE Wu et al. (2023), we perform pre-training on SAM to obtain SAM-AD in AD scenarios, as shown in Fig. 3. We denote $x$ as a clean image from the AD dataset (i.e. KITTI Geiger et al. (2012) and nuScenes Caesar et al. (2020)) and $\eta$ as a set of noise images generated by Dong et al. (2023) based on $x$. And the noise type and the severity are randomly chosen from the four weather (i.e., rain, snow, fog, and strong sunlight) and the five severities from 1 to 5, respectively. We employ the image encoder of SAM Kirillov et al. (2023) , MobileSAM Zhang et al. (2023a) as our encoder while the decoder and the reconstruction loss are the same as DMAE Wu et al. (2023). For FastSAM Zhao et al. (2023), we adopt YOLOv8 ²²2https://github.com/ultralytics/ultralytics to pre-train FastSAM on the AD dataset. To avoid overfitting, we use random resizing and cropping as data augmentation. We also set the mask ratio as 0.75 and have trained 400 epochs on 8 NVIDIA A100 GPUs.

AD-FPN. As a promptable segmentation model, SAM has three components: image encoder, prompt encoder and mask decoder. Generally, the image encoder can provide high-quality and highly robust image embedding for downstream models, while the mask decoder is only designed to provide decoding services for semantic segmentation. Furthermore, what we require are robust image features rather than the processing of prompting information by the prompt encoder. Therefore, we employ SAM’s image encoder to extract robust image features. However, SAM utilizes the ViT series Dosovitskiy et al. (2020) as its image encoder, which excludes multi-scale features and provides only high-dimensional low-resolution features. To generate the multi-scale features required for object detection, inspired by Li et al. (2022a), we design an AD-FPN that offers ViT-based multi-scale features. Specifically, leveraging height-dimensional image embedding with stride 16 (scale=$1/16$) provided by SAM, we produce a series of multi-scale features $F_{ms}$ with stride of $\left\{32,16,8,4\right\}$. Sequentially, we acquire multi-scale feature $F_{i}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C_{i}}$ by integrate $F_{ms}$ in a bottom-up manner similar to FPN Lin et al. (2017).

Although SAM-AD or SAM has the capability to extract robust image features, the gap between 2D and 3D domains still persists and cameras lacking geometric information in a corrupted environment often amplify noise and give rise to negative migration issues. To mitigate this problem, we propose the Depth-Guided Wavelet Attention (DGWA) module, which can be split into two steps. 1) A depth-guided network is designed, that adds geometry prior to image features by combining image features and depth features from a point cloud. 2) The features of an image are decomposed into four wavelet subbands using the Haar wavelet transform Liu et al. (2020a), then attention mechanism allows to denoise informative features in the subbands.

Formally, given image features $F_{i}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C_{i}}$ and raw points $P\in\mathbb{R}^{N,C_{p}}$ as input. We project $P$ onto the image plane to acquire a sparse depth map $S\in\mathbb{R}^{H\times W\times 2}$. Next, we feed $S$ into the depth encoder $DE(\cdot)$, which consists of several convolution and max pooling blocks, to acquire depth features $F_{d}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C_{i}}$. Afterward, we leverage convolution encode $(F_{i},F_{d})$ to acquire depth-guided image features $\hat{F_{i}}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times{16}}$, given by

Subsequently, we employ discrete wavelet transform (DWT), a reversible operator, to partition the input $\hat{F_{i}}$ into four subbands. Specifically, we encode the rows and columns of the input separately into one low-frequency band $\widetilde{f}_{i}^{LL}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 4}$ and three high-frequency bands $(\widetilde{f}_{i}^{LH},\widetilde{f}_{i}^{HL},\widetilde{f}_{i}^{HH})\in\mathbb{R}^{\frac{H}{8},\frac{W}{8},4}$, with the low-filter $f_{L}=(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})$ and the high-filter $f_{H}=(\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$. In this state, the low-frequency band retains coarse-grained information while the high-frequency band retains fine-grained information. In other words, it is easier to capture the mutation signal, so as to filter the noise information. We concatenate the four-subband features along channel dimension to acquire wavelet features $\widetilde{F_{i}}=[\hat{f}_{i}^{LL},\hat{f}_{i}^{LH},\hat{f}_{i}^{HL},\hat{f}_{i}^{HH}]\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 16}$. Next, we perform wave-attention $Att_{\omega}$ to query informative features in the wavelet features. Concretely, we employ $\hat{F}_{i}$ as a Query and $\widetilde{F_{i}}$ as a Key/Value given by

Finally, we leverage the IDWT (inverse DWT) to convert $\widetilde{F_{i}}$ back to $\hat{F_{i}}$ and integrate this converted $\hat{F_{i}}$ and $F_{att}$ to obtain denoise features $F_{out}\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times 16}$ by

where $F_{out}$ preserves informative features and restrains redundant mutation noise in the frequency domain.

Following the incorporation of image depth features within the DGWA module, we propose the Adaptive Fusion technique to combine point cloud attributes with robust image features enriched with depth information. Specifically, different types of noise affect LiDAR and images to different degrees, which raises a corruption imbalance problem. Therefore, considering the distinct influences of various noises on LiDAR and camera, we employ self-attention to re-weight the fused features adaptively as shown in Fig. 4. The corruption degree of modality-specificity is dynamic, and self-attention mechanism allows adaptive re-weighting features to enhance informative features and suppress redundant noise.

We perform experiments on both the clean public benchmarks (KITTI Geiger et al. (2012) and nuScenes Caesar et al. (2020)) and the noisy public benchmarks (KITTI-CDong et al. (2023) and nuScenes-C Dong et al. (2023)).

We conduct evaluations on the clean datasets KITTI and nuScenes, as well as the noisy datasets KITTI-C and nuScenes-C. While SOTA methods are primarily focused on achieving high accuracy, we place greater emphasis on the robustness and generalization of the methods. These factors are crucial for the practical deployment of 3D object detection in AD scenarios, making the evaluation on the noisy datasets more important in our perspective.