-
Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap
Authors:
Carl Lindström,
Georg Hess,
Adam Lilja,
Maryam Fatemi,
Lars Hammarstrand,
Christoffer Petersson,
Lennart Svensson
Abstract:
Neural Radiance Fields (NeRFs) have emerged as promising tools for advancing autonomous driving (AD) research, offering scalable closed-loop simulation and data augmentation capabilities. However, to trust the results achieved in simulation, one needs to ensure that AD systems perceive real and rendered data in the same way. Although the performance of rendering methods is increasing, many scenari…
▽ More
Neural Radiance Fields (NeRFs) have emerged as promising tools for advancing autonomous driving (AD) research, offering scalable closed-loop simulation and data augmentation capabilities. However, to trust the results achieved in simulation, one needs to ensure that AD systems perceive real and rendered data in the same way. Although the performance of rendering methods is increasing, many scenarios will remain inherently challenging to reconstruct faithfully. To this end, we propose a novel perspective for addressing the real-to-simulated data gap. Rather than solely focusing on improving rendering fidelity, we explore simple yet effective methods to enhance perception model robustness to NeRF artifacts without compromising performance on real data. Moreover, we conduct the first large-scale investigation into the real-to-simulated data gap in an AD setting using a state-of-the-art neural rendering technique. Specifically, we evaluate object detectors and an online mapping model on real and simulated data, and study the effects of different fine-tuning strategies.Our results show notable improvements in model robustness to simulated data, even improving real-world performance in some cases. Last, we delve into the correlation between the real-to-simulated gap and image reconstruction metrics, identifying FID and LPIPS as strong indicators. See https://research.zenseact.com/publications/closing-real2sim-gap for our project page.
△ Less
Submitted 15 April, 2024; v1 submitted 24 March, 2024;
originally announced March 2024.
-
NeuRAD: Neural Rendering for Autonomous Driving
Authors:
Adam Tonderski,
Carl Lindström,
Georg Hess,
William Ljungbergh,
Lennart Svensson,
Christoffer Petersson
Abstract:
Neural radiance fields (NeRFs) have gained popularity in the autonomous driving (AD) community. Recent methods show NeRFs' potential for closed-loop simulation, enabling testing of AD systems, and as an advanced training data augmentation technique. However, existing methods often require long training times, dense semantic supervision, or lack generalizability. This, in turn, hinders the applicat…
▽ More
Neural radiance fields (NeRFs) have gained popularity in the autonomous driving (AD) community. Recent methods show NeRFs' potential for closed-loop simulation, enabling testing of AD systems, and as an advanced training data augmentation technique. However, existing methods often require long training times, dense semantic supervision, or lack generalizability. This, in turn, hinders the application of NeRFs for AD at scale. In this paper, we propose NeuRAD, a robust novel view synthesis method tailored to dynamic AD data. Our method features simple network design, extensive sensor modeling for both camera and lidar -- including rolling shutter, beam divergence and ray dropping -- and is applicable to multiple datasets out of the box. We verify its performance on five popular AD datasets, achieving state-of-the-art performance across the board. To encourage further development, we will openly release the NeuRAD source code. See https://github.com/georghess/NeuRAD .
△ Less
Submitted 18 April, 2024; v1 submitted 26 November, 2023;
originally announced November 2023.
-
Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous driving
Authors:
Mina Alibeigi,
William Ljungbergh,
Adam Tonderski,
Georg Hess,
Adam Lilja,
Carl Lindstrom,
Daria Motorniuk,
Junsheng Fu,
Jenny Widahl,
Christoffer Petersson
Abstract:
Existing datasets for autonomous driving (AD) often lack diversity and long-range capabilities, focusing instead on 360° perception and temporal reasoning. To address this gap, we introduce Zenseact Open Dataset (ZOD), a large-scale and diverse multimodal dataset collected over two years in various European countries, covering an area 9x that of existing datasets. ZOD boasts the highest range and…
▽ More
Existing datasets for autonomous driving (AD) often lack diversity and long-range capabilities, focusing instead on 360° perception and temporal reasoning. To address this gap, we introduce Zenseact Open Dataset (ZOD), a large-scale and diverse multimodal dataset collected over two years in various European countries, covering an area 9x that of existing datasets. ZOD boasts the highest range and resolution sensors among comparable datasets, coupled with detailed keyframe annotations for 2D and 3D objects (up to 245m), road instance/semantic segmentation, traffic sign recognition, and road classification. We believe that this unique combination will facilitate breakthroughs in long-range perception and multi-task learning. The dataset is composed of Frames, Sequences, and Drives, designed to encompass both data diversity and support for spatio-temporal learning, sensor fusion, localization, and mapping. Frames consist of 100k curated camera images with two seconds of other supporting sensor data, while the 1473 Sequences and 29 Drives include the entire sensor suite for 20 seconds and a few minutes, respectively. ZOD is the only large-scale AD dataset released under a permissive license, allowing for both research and commercial use. More information, and an extensive devkit, can be found at https://zod.zenseact.com
△ Less
Submitted 21 October, 2023; v1 submitted 3 May, 2023;
originally announced May 2023.
-
QUIC Throughput and Fairness over Dual Connectivity
Authors:
David Hasselquist,
Christoffer Lindström,
Nikita Korzhitskii,
Niklas Carlsson,
Andrei Gurtov
Abstract:
Dual Connectivity (DC) is an important lower-layer feature accelerating the transition from 4G to 5G that also is expected to play an important role in standalone 5G radio networks. However, even though the packet reordering introduced by DC can significantly impact the performance of upper-layer protocols, no prior work has studied the impact of DC on QUIC. In this paper, we present the first suc…
▽ More
Dual Connectivity (DC) is an important lower-layer feature accelerating the transition from 4G to 5G that also is expected to play an important role in standalone 5G radio networks. However, even though the packet reordering introduced by DC can significantly impact the performance of upper-layer protocols, no prior work has studied the impact of DC on QUIC. In this paper, we present the first such performance study. Using a series of throughput and fairness experiments, we show how QUIC is affected by different DC parameters, network conditions, and whether the DC implementation aims to improve throughput or reliability. Results for two QUIC implementations (aioquic, ngtcp2) and two congestion control algorithms (NewReno, CUBIC) are presented under both static and highly time-varying network conditions. Our findings provide network operators with insights and understanding into the impacts of splitting QUIC traffic in a DC environment. With reasonably selected DC parameters and increased UDP receive buffers, QUIC over DC performs similarly to TCP over DC and achieves optimal fairness under symmetric link conditions when DC is not used for packet duplication. The insights can help network operators provide modern users with better end-to-end service when deploying DC.
△ Less
Submitted 20 December, 2022; v1 submitted 28 December, 2021;
originally announced December 2021.
-
GPS-Denied Navigation Using SAR Images and Neural Networks
Authors:
Teresa White,
Jesse Wheeler,
Colton Lindstrom,
Randall Christensen,
Kevin R. Moon
Abstract:
Unmanned aerial vehicles (UAV) often rely on GPS for navigation. GPS signals, however, are very low in power and easily jammed or otherwise disrupted. This paper presents a method for determining the navigation errors present at the beginning of a GPS-denied period utilizing data from a synthetic aperture radar (SAR) system. This is accomplished by comparing an online-generated SAR image with a re…
▽ More
Unmanned aerial vehicles (UAV) often rely on GPS for navigation. GPS signals, however, are very low in power and easily jammed or otherwise disrupted. This paper presents a method for determining the navigation errors present at the beginning of a GPS-denied period utilizing data from a synthetic aperture radar (SAR) system. This is accomplished by comparing an online-generated SAR image with a reference image obtained a priori. The distortions relative to the reference image are learned and exploited with a convolutional neural network to recover the initial navigational errors, which can be used to recover the true flight trajectory throughout the synthetic aperture. The proposed neural network approach is able to learn to predict the initial errors on both simulated and real SAR image data.
△ Less
Submitted 22 October, 2020;
originally announced October 2020.
