-
Uplifting Range-View-based 3D Semantic Segmentation in Real-Time with Multi-Sensor Fusion
Authors:
Shiqi Tan,
Hamidreza Fazlali,
Yixuan Xu,
Yuan Ren,
Bingbing Liu
Abstract:
Range-View(RV)-based 3D point cloud segmentation is widely adopted due to its compact data form. However, RV-based methods fall short in providing robust segmentation for the occluded points and suffer from distortion of projected RGB images due to the sparse nature of 3D point clouds. To alleviate these problems, we propose a new LiDAR and Camera Range-view-based 3D point cloud semantic segmentat…
▽ More
Range-View(RV)-based 3D point cloud segmentation is widely adopted due to its compact data form. However, RV-based methods fall short in providing robust segmentation for the occluded points and suffer from distortion of projected RGB images due to the sparse nature of 3D point clouds. To alleviate these problems, we propose a new LiDAR and Camera Range-view-based 3D point cloud semantic segmentation method (LaCRange). Specifically, a distortion-compensating knowledge distillation (DCKD) strategy is designed to remedy the adverse effect of RV projection of RGB images. Moreover, a context-based feature fusion module is introduced for robust and preservative sensor fusion. Finally, in order to address the limited resolution of RV and its insufficiency of 3D topology, a new point refinement scheme is devised for proper aggregation of features in 2D and augmentation of point features in 3D. We evaluated the proposed method on large-scale autonomous driving datasets \ie SemanticKITTI and nuScenes. In addition to being real-time, the proposed method achieves state-of-the-art results on nuScenes benchmark
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
AutoSplat: Constrained Gaussian Splatting for Autonomous Driving Scene Reconstruction
Authors:
Mustafa Khan,
Hamidreza Fazlali,
Dhruv Sharma,
Tongtong Cao,
Dongfeng Bai,
Yuan Ren,
Bingbing Liu
Abstract:
Realistic scene reconstruction and view synthesis are essential for advancing autonomous driving systems by simulating safety-critical scenarios. 3D Gaussian Splatting excels in real-time rendering and static scene reconstructions but struggles with modeling driving scenarios due to complex backgrounds, dynamic objects, and sparse views. We propose AutoSplat, a framework employing Gaussian splatti…
▽ More
Realistic scene reconstruction and view synthesis are essential for advancing autonomous driving systems by simulating safety-critical scenarios. 3D Gaussian Splatting excels in real-time rendering and static scene reconstructions but struggles with modeling driving scenarios due to complex backgrounds, dynamic objects, and sparse views. We propose AutoSplat, a framework employing Gaussian splatting to achieve highly realistic reconstructions of autonomous driving scenes. By imposing geometric constraints on Gaussians representing the road and sky regions, our method enables multi-view consistent simulation of challenging scenarios including lane changes. Leveraging 3D templates, we introduce a reflected Gaussian consistency constraint to supervise both the visible and unseen side of foreground objects. Moreover, to model the dynamic appearance of foreground objects, we estimate residual spherical harmonics for each foreground Gaussian. Extensive experiments on Pandaset and KITTI demonstrate that AutoSplat outperforms state-of-the-art methods in scene reconstruction and novel view synthesis across diverse driving scenarios. Visit our project page at https://autosplat.github.io/.
△ Less
Submitted 3 July, 2024; v1 submitted 2 July, 2024;
originally announced July 2024.
-
AOP-Net: All-in-One Perception Network for Joint LiDAR-based 3D Object Detection and Panoptic Segmentation
Authors:
Yixuan Xu,
Hamidreza Fazlali,
Yuan Ren,
Bingbing Liu
Abstract:
LiDAR-based 3D object detection and panoptic segmentation are two crucial tasks in the perception systems of autonomous vehicles and robots. In this paper, we propose All-in-One Perception Network (AOP-Net), a LiDAR-based multi-task framework that combines 3D object detection and panoptic segmentation. In this method, a dual-task 3D backbone is developed to extract both panoptic- and detection-lev…
▽ More
LiDAR-based 3D object detection and panoptic segmentation are two crucial tasks in the perception systems of autonomous vehicles and robots. In this paper, we propose All-in-One Perception Network (AOP-Net), a LiDAR-based multi-task framework that combines 3D object detection and panoptic segmentation. In this method, a dual-task 3D backbone is developed to extract both panoptic- and detection-level features from the input LiDAR point cloud. Also, a new 2D backbone that intertwines Multi-Layer Perceptron (MLP) and convolution layers is designed to further improve the detection task performance. Finally, a novel module is proposed to guide the detection head by recovering useful features discarded during down-sampling operations in the 3D backbone. This module leverages estimated instance segmentation masks to recover detailed information from each candidate object. The AOP-Net achieves state-of-the-art performance for published works on the nuScenes benchmark for both 3D object detection and panoptic segmentation tasks. Also, experiments show that our method easily adapts to and significantly improves the performance of any BEV-based 3D object detection method.
△ Less
Submitted 2 February, 2023;
originally announced February 2023.
-
A Versatile Multi-View Framework for LiDAR-based 3D Object Detection with Guidance from Panoptic Segmentation
Authors:
Hamidreza Fazlali,
Yixuan Xu,
Yuan Ren,
Bingbing Liu
Abstract:
3D object detection using LiDAR data is an indispensable component for autonomous driving systems. Yet, only a few LiDAR-based 3D object detection methods leverage segmentation information to further guide the detection process. In this paper, we propose a novel multi-task framework that jointly performs 3D object detection and panoptic segmentation. In our method, the 3D object detection backbone…
▽ More
3D object detection using LiDAR data is an indispensable component for autonomous driving systems. Yet, only a few LiDAR-based 3D object detection methods leverage segmentation information to further guide the detection process. In this paper, we propose a novel multi-task framework that jointly performs 3D object detection and panoptic segmentation. In our method, the 3D object detection backbone in Bird's-Eye-View (BEV) plane is augmented by the injection of Range-View (RV) feature maps from the 3D panoptic segmentation backbone. This enables the detection backbone to leverage multi-view information to address the shortcomings of each projection view. Furthermore, foreground semantic information is incorporated to ease the detection task by highlighting the locations of each object class in the feature maps. Finally, a new center density heatmap generated based on the instance-level information further guides the detection backbone by suggesting possible box center locations for objects. Our method works with any BEV-based 3D object detection method, and as shown by extensive experiments on the nuScenes dataset, it provides significant performance gains. Notably, the proposed method based on a single-stage CenterPoint 3D object detection network achieved state-of-the-art performance on nuScenes 3D Detection Benchmark with 67.3 NDS.
△ Less
Submitted 3 March, 2022;
originally announced March 2022.
-
Vessel Segmentation and Catheter Detection in X-Ray Angiograms Using Superpixels
Authors:
Hamid R. Fazlali,
Nader Karimi,
S. M. Reza Soroushmehr,
Shahram Shirani,
Brahmajee. K. Nallamothu,
Kevin R. Ward,
Shadrokh Samavi,
Kayvan Najarian
Abstract:
Coronary artery disease (CAD) is the leading causes of death around the world. One of the most common imaging methods for diagnosing this disease is X-ray angiography. Diagnosing using these images is usually challenging due to non-uniform illumination, low contrast, presence of other body tissues, presence of catheter etc. These challenges make the diagnoses task of cardiologists tougher and more…
▽ More
Coronary artery disease (CAD) is the leading causes of death around the world. One of the most common imaging methods for diagnosing this disease is X-ray angiography. Diagnosing using these images is usually challenging due to non-uniform illumination, low contrast, presence of other body tissues, presence of catheter etc. These challenges make the diagnoses task of cardiologists tougher and more prone to misdiagnosis. In this paper we propose a new automated framework for coronary arteries segmentation, catheter detection and center-line extraction in x-ray angiography images. Our proposed segmentation method is based on superpixels. In this method at first three different superpixel scales are exploited and a measure for vesselness probability of each superpixel is determined. A majority voting is used for obtaining an initial segmentation map from these three superpixel scales. This initial segmentation is refined by finding the orthogonal line on each ridge pixel of vessel region. In this framework we use our catheter detection and tracking method which detects the catheter by finding its ridge in the first frame and traces in other frames by fitting a second order polynomial on it. Also we use the image ridges for extracting the coronary arteries centerlines. We evaluated our method qualitatively and quantitatively on two different challenging datasets and compared it with one of the previous well-known coronary arteries segmentation methods. Our method could detect the catheter and reduced the false positive rate in addition to achieving better segmentation results. The evaluation results prove that our method performs better in a much shorter time.
△ Less
Submitted 8 September, 2017;
originally announced September 2017.
