MonoCD: Monocular 3D Object Detection with Complementary Depths

Many recent works [39, 7, 46, 23, 19, 44] are extended from the popular center-based paradigm CenterNet [45], which is an anchor-free method initially applied to 2D object detection. It makes the detection process simpler and more efficient due to converting all attributes of a 3D bounding box into a center to estimate. SMOKE [21] inherits the center-based framework and proposes that the estimation of the 2D bounding box can be omitted. MonoDLE [24] finds that the estimation of the 2D bounding box contributes to the prediction of 3D attributes and demonstrates that depth error is the main reason limiting the accuracy of monocular 3D object detection. MonoCon [20] finds that adding auxiliary learning tasks around the center can improve the generalization performance. Although there are many benefits in the center-based framework, it makes the prediction of all 3D attributes highly correlated with the local center. It ignores the exploitation of global information, leading to the coupling of predicted 3D attributes.

Benefiting from the non-local encoding of attention mechanism [35] and its development in object detection [4], multiple Transformer-based monocular 3D detectors have recently been proposed to enhance the global perception capability. MonoDTR [10] proposes to perform depth position encoding to inject global depth information into Transformer to guide the detection, which requires LIDAR for auxiliary supervision. Different from it, MonoDETR [42] uses foreground object labels to predict foreground Depth Maps to achieve depth guidance. In order to improve the inference efficiency, MonoATT [47] proposes an adaptive token Transformer and makes it possible for finer tokens to be assigned to more significant regions in images. Although the above methods perform well, the drawbacks of high computational complexity and slow inference of Transformer-based monocular 3D detectors are still apparent. Thus there is currently a lack of a method that has both the capability of synthesizing global information and low latency in real-world autonomous driving scenarios.

In addition to directly estimating object depth using deep neural networks, many recent works have broadened the depth estimation branch by mediately predicting geometric clues associated with depth. [32, 23] utilizes mathematical priors and uncertainty modeling to restore depth information through the ratio of 3D to 2D height. Based on them, MonoFlex [43] further extends the geometric depths to three sets by other supporting lines of the 3D bounding box and proposes to use uncertainties as weights to combine multiple depths into a final depth. MonoGround [28] introduces a local ground plane prior and enriches the depth supervision sources using randomly sampling dense points in the bottom plane of each object. MonoDDE [18] utilizes keypoint information to expand the number of depth prediction branches to 20, highlighting the importance of depth diversity. However, the complementarity between multiple depths is hardly explored. Errors in geometric clues (such as 2D/3D height) accumulate into the corresponding depth errors. Without effective complementarity, existing depth errors cannot be neutralized.

The task of monocular 3D object detection is to recognize objects of interest from a 2D image only and predict their corresponding 3D attributes including 3D location $(x,y,z)$, dimension $(h,w,l)$, and orientation $\theta$. The 3D location $(x,y,z)$ is usually transformed into 2.5D information $(u_{c},v_{c},z)$ for prediction. The recovery process of $x$ and $y$ can be formulated as:

$$x=\frac{(u_{c}-c_{u})z}{f_{x}},\quad y=\frac{(v_{c}-c_{v})z}{f_{y}}$$

(1)

where $(u_{c},v_{c})$ is the projected 3D center in the image and $(c_{u},c_{v})$ is the camera optical center. $f_{x}$ and $f_{y}$ denote the horizontal and vertical focal lengths respectively.

As described in Sec. 1, many methods [43, 28, 18] have realized that depth $z$ is the main reason limiting the performance of monocular 3D detector and utilize multi-depth to improve the accuracy of depth prediction via:

$$z_{soft}=\displaystyle\sum_{i=1}^{n}w_{i}z_{i}$$

(2)

where $\{z_{i}\}_{i=1}^{n}$ represents n predicted depths and $\{w_{i}\}_{i=1}^{n}$ represents their weights determined by the predicted uncertainty [11, 12]. $z_{soft}$ is used as the final depth of the output.

To demonstrate the effectiveness of complementary depths, we present its superiority from a mathematical perspective. Define two different depth prediction branches $\hat{z}_{1}$ and $\hat{z}_{2}$ as follows:

$$\hat{z}_{1}=z^{*}+e_{1},\quad\hat{z}_{2}=z^{*}+e_{2}$$

(3)

where $z^{*}$ represents the ground truth of depth. $e_{1}$ and $e_{2}$ are the errors of the two depth branches in a single prediction, respectively. Note that the positive and negative of $e_{1}$ and $e_{2}$ correspond to the sign of error. We define $e_{1}e_{2}>0$ to simulate the case of multiple depth coupling, as shown in Fig. 1(a). We term the final combination error of multiple coupling depths as coupling depth error. Hence, referring to Eq. 2, the coupling depth error $E_{1}$ of $\hat{z}_{1}$ and $\hat{z}_{2}$ can be formulated as:

	$\displaystyle E_{1}$	$\displaystyle=|w_{1}\hat{z}_{1}+w_{2}\hat{z}_{2}-z^{*}|$		(4)
		$\displaystyle=|w_{1}e_{1}+w_{2}e_{2}|$

where $w_{1}$ and $w_{2}$ satisfy $w_{1},w_{2}>0$ and $w_{1}+w_{2}=1$. We then flip $\hat{z}_{1}$ symmetrically along $z^{*}$ without changing the accuracy of the prediction through:

	$\displaystyle\hat{z}_{1}^{\prime}$	$\displaystyle=z^{*}-(\hat{z}_{1}-z^{*})$		(5)
		$\displaystyle=z^{*}-e_{1}$

After flipping, the error sign in $\hat{z}_{1}^{\prime}$ and $\hat{z}_{2}$ are opposite and higher complementarity between them is artificially achieved. We term the final combination error of multiple complementary depths as complementary depth error. Similarly, the complementary depth error $E_{2}$ of $\hat{z}_{1}^{\prime}$ and $\hat{z}_{2}$ can be formulated as:

	$\displaystyle E_{2}$	$\displaystyle=|w_{1}\hat{z}_{1}^{\prime}+w_{2}\hat{z}_{2}-z^{*}|$		(6)
		$\displaystyle=|w_{1}e_{1}-w_{2}e_{2}|$

By mathematical transformations we further express Eqs. 4 and 6 as:

	$\displaystyle E_{1}$	$\displaystyle=\sqrt{(w_{1}e_{1}+w_{2}e_{2})^{2}}$		(7)
		$\displaystyle=\sqrt{(w_{1}e_{1})^{2}+2w_{1}w_{2}e_{1}e_{2}+(w_{2}e_{2})^{2}}$

	$\displaystyle E_{2}$	$\displaystyle=\sqrt{(w_{1}e_{1}-w_{2}e_{2})^{2}}$		(8)
		$\displaystyle=\sqrt{(w_{1}e_{1})^{2}-2w_{1}w_{2}e_{1}e_{2}+(w_{2}e_{2})^{2}}$

It is obvious that the complementary depth error $E_{2}$ is consistently less than the coupling depth error $E_{1}$ due to the condition $e_{1}e_{2}>0$. Regardless of variations in weight or error magnitude, this relationship remains constant. Similarly, the conclusion is equivalent by maintaining $z_{1}$ unchanged during the flip of $z_{2}$. Therefore we can draw the conclusion: realizing the complementary relationship between two depth branches contributes to reducing the overall depth error, even without improving the accuracy of individual branches.

To demonstrate the effectiveness of complementary depths in practice, we select a classical multi-depth prediction baseline [43] for evaluation in KITTI val set. It contains 4 depth prediction branches (1 directly estimated depth and 3 geometric depths) and the coupling rate of any two branches is around 95% after testing. As shown on the left in Fig. 3, we flip the direct depth estimation branch among them symmetrically along the ground truth based on Eq. 5 across a 0% to 100% sample scale to achieve depths complementary at different levels. Additionally, considering the difficulty of obtaining depth predictions with opposite error signs while maintaining the same accuracy in practice, we conduct another experiment by flipping the depth branch while applying random disturbances of different magnitudes on top of it. The results are presented on the right of Fig. 3. Similar results are observed in other branches by performing the same operation as above. Based on this, we have the following three observations:

Observation 1: On the left of Fig. 3, the detection accuracy increases as the proportion of flipped samples rises. It demonstrates that increasing complementarity between multiple depth prediction branches can improve detection accuracy continuously.

Observation 2: For two independent depth prediction branches, ideally, the proportion of their predictions with opposite signs in all samples should be 50%. The situation is similar to the 50% flipped proportion on the left of Fig. 3 due to the coupling of multiple branches in the baseline. Therefore reducing the similarity of multiple depth prediction branches can also increase their complementarity.

Observation 3: In the case where the flipped proportion is fixed at 50%, as shown in the right of Fig. 3, it is not until the application of random disturbance with an amplitude of 2 meters (which is quite significant [24] for Car in KITTI) that the complementary effect disappeared. This indicates that complementary effect can still contribute to overall performance even if losing some depth estimation accuracy and ultimately whether the overall performance can be improved depends on both the proportion of opposite signs and the depth estimation accuracy.

Additionally, we select models with different total numbers of depth prediction branches to perform flipping and evaluation. We find that as the number of flipped branches approaches the number of unflipped branches, the overall performance improves accordingly. For more experiments and details, please refer to the supplementary materials.

To avoid falling into the coupling, we do not utilize the center-based approach discussed in Sec. 2.1 to predict $y$. We propose to first obtain the sloping trend of $y$ in the scene by the ground plane equation. The prediction of the ground plane equation is based on the Horizon Heatmap branch, similar to [38], but we omit the edge prediction and obtain prediction results as:

	$\displaystyle Ax+By+Cz+1.65=0$		(10)
	$\displaystyle s.t.\quad A^{2}+B^{2}+C^{2}=1$

where $A=F\frac{k_{h}f_{x}}{f_{y}}$, $B=-F$ and $C=F\frac{k_{h}c_{u}+b_{h}-c_{v}}{f_{y}}$. $k_{h}$ and $b_{h}$ represent the slope and intercept of the horizon fitted by Horizon Heatmap. After it, then considering Eq. 1 and the projected bottom center $(u_{b},v_{b})$ of the object, $y$ with global information can be derived as:

$${y}_{glo}=-\frac{1.65}{An+Cm+B}$$

(11)

where $n=\frac{f_{y}(u_{b}-c_{u})}{f_{x}(v_{b}-c_{v})}$, $m=\frac{f_{y}}{v_{b}-c_{v}}$.

Inserting Eq. 11 into Eq. 9, a new depth prediction branch with the global clue is obtained:

$$z_{glo}=\frac{{f}_{y}y_{glo}}{v_{b}-c_{v}}$$

(12)

In addition, to better utilize the global features as well as to expand the receptive field, we use dilated convolution [40] to predict the Horizon Heatmap.

Our experiments are conducted on the widely-adopted KITTI 3D Object [9] dataset, which contains 7481 training images and 7518 test images. Since the annotations of the test images are not publicly accessible, we follow [6] and further divide the 7481 training images into 3712 and 3769 as the training and validation sets, respectively. Each category is further refined into three difficulties: Easy, Moderate, and Hard based on 2D height, truncation, and occlusion.

As in previous methods, we use Average Precision ${AP}_{3D}$ and ${AP}_{BEV}$ as the overall evaluation metrics. Following [34], 40 recall positions are used for the above AP calculations. The IoU threshold is 0.7 for Car.

In the ablation study of Sec. 4.5, the mean absolute error (MAE) of $y$ is introduced as a metric to evaluate the accuracy of the different $y$ sources. In addition, to better measure the complementarity between different designs, we quantify the magnitude of complementarity as the Complementarity Score. As discussed in Sec. 3.2, both the error sign opposite proportion and depth estimation accuracy are crucial in achieving enhanced performance. Thus we formulate the Complementarity Score(CS) as:

$$CS=\frac{ESOP_{z}}{{MAE}_{z}}$$

(15)

where $ESOP_{z}$ represents depths Error Sign Opposite Proportion (ESOP) between global and local clue branches, and ${MAE}_{z}$ represents the Mean Absolute Error of $z_{comp}$. For a baseline without $z_{comp}$, ESOP counts the proportion between $z_{key}$ and $z_{dir}$.

In order to demonstrate the effectiveness of the proposed framework, we choose three recent center-based methods with excellent performance as the baseline model, MonoFlex [43], MonoDLE [24], and MonoCon [20]. All experiments are performed on a single RTX 2080Ti GPU. The aforementioned baseline models all employ DLA-34 [41] as the feature extraction network. In the Global Clues branch, the prediction head of Horizon Heatmap contains two 3×3 conv layers with BN and ReLU (where the dilation rate is set to 2) and an output conv layer. The horizon equation is obtained by taking out all the largest elements in each column of the Horizon heatmap and fitting them. The ground truth of Horizon Heatmap is generated by fitting the scene ground plane through the bottom coordinate annotation of each object and then projecting to the 2D image plane [38], so only RGB image data and camera annotations are used throughout the training process. The radius of the Gaussian kernel used for each pixel is 2 when mapping the horizon equation into Heatmap. The $z_{direct}$, $z_{key}$ and $z_{comp}$ loss weight proportions are set to $1:0.2:0.1$. The remaining settings such as optimizer, batch sizes, image padding size, etc. remain consistent with the baseline.

To demonstrate the effectiveness of the proposed method, we conduct quantitative experiments on test and val sets of KITTI [9].

As shown in Tab. 6, the proposed method is compared with the state-of-the-art methods in recent years on the widely used KITTI test set. Our method achieves the best performance in the majority of metrics without using any additional data. Compared with the previous multi-depth solving method MonoFlex [43], our performance for ${AP}_{3D}/{AP}_{BEV}$ improves by 19.44%/15.49%, respectively. The performance for ${AP}_{3D}/{AP}_{BEV}$ improves from 15.44/20.79 to 16.59/22.81 compared to the method GPENet [38], which also incorporated the ground plane equation solution. Even when compared to the latest Transformer-based detector MonoDETR [42], we outperform it in most metrics while ensuring real-time operation.

As shown in Tab. 7, we extend the complementary depth branch to three competitive center-based monocular 3d detectors. The results of the KITTI val set demonstrate that the proposed complementary depth is flexible and achieves stable increments across multiple frameworks and metrics. It is worth noting that the boost of our design performs better on ${AP}_{BEV}$ than ${AP}_{3D}$ in general. We attribute this to the focus of our method on improvements in depth estimation, since ${AP}_{BEV}$ is more emphasis on the accuracy of localization along the Z-axis compared to ${AP}_{3D}$ [9].

In this section, we select MonoFlex [43] as the baseline to discuss the impact of different designs.

By comparing the results of $z_{comp}$ MAE and ESOP pairs under different settings, it can be found that determining whether complementary depth can lead to overall performance enhancement often requires evaluation from two perspectives: depth estimation accuracy and ESOP. This trend can be effectively quantified by complementary scores.

The results in the 6th to 7th rows of Tab. 8 justify the removal of edge detection and the use of dilated convolution when predicting the ground plane equation.

To validate the effectiveness of achieving complementary form in enhancing detection accuracy, we present the results of different depth forms in Tab. 4. According to the results of the 2nd and 4th row in Tab. 4, the ESOP and CS of Eq. 14 are further enhanced after considering the complementary form compared to Eq. 12. Although a part of the depth estimation accuracy is sacrificed, the complementarity and overall performance are eventually improved, which is consistent with observation 3 in Sec. 3.2.

In addition to Eqs. 12 and 14 mentioned in Sec. 3.3, we also consider the following complementary form:

$$z=\frac{f_{y}(y_{glo}-H)}{v_{t}-c_{v}}$$

(16)

Although it appears that Eq. 16 is more symmetrical and complementary to $z_{key}$ in form, its depth estimation error is significantly higher than that of Eq. 14. This is due to the fact that $v_{t}$ and $c_{v}$ in the denominator are relatively close, as well as the $y_{glo}$ and $H$ in the numerator, which causes an unstable depth estimation. This is also why Eq. 16 has a higher ESOP because the instability of the estimate mitigates the prediction tendency, but it does not contribute to the overall performance. It demonstrates the importance of an appropriate form of complementary depth.

Based on the qualitative results shown in Fig. 5, it can be observed that $z_{comp}$ from the global clue branch is significantly different from $z_{dir}$ and $z_{key}$ from the local clue branch and has the opposite error sign. After combining $z_{comp}$, the predicted box is closer to the ground truth. This visualizes the process of error neutralization.