-
Recurrent Video Restoration Transformer with Guided Deformable Attention
Authors:
Jingyun Liang,
Yuchen Fan,
Xiaoyu Xiang,
Rakesh Ranjan,
Eddy Ilg,
Simon Green,
Jiezhang Cao,
Kai Zhang,
Radu Timofte,
Luc Van Gool
Abstract:
Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusi…
▽ More
Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
△ Less
Submitted 12 November, 2022; v1 submitted 5 June, 2022;
originally announced June 2022.
-
Trajectory Planning with Deep Reinforcement Learning in High-Level Action Spaces
Authors:
Kyle R. Williams,
Rachel Schlossman,
Daniel Whitten,
Joe Ingram,
Srideep Musuvathy,
Anirudh Patel,
James Pagan,
Kyle A. Williams,
Sam Green,
Anirban Mazumdar,
Julie Parish
Abstract:
This paper presents a technique for trajectory planning based on continuously parameterized high-level actions (motion primitives) of variable duration. This technique leverages deep reinforcement learning (Deep RL) to formulate a policy which is suitable for real-time implementation. There is no separation of motion primitive generation and trajectory planning: each individual short-horizon motio…
▽ More
This paper presents a technique for trajectory planning based on continuously parameterized high-level actions (motion primitives) of variable duration. This technique leverages deep reinforcement learning (Deep RL) to formulate a policy which is suitable for real-time implementation. There is no separation of motion primitive generation and trajectory planning: each individual short-horizon motion is formed during the Deep RL training to achieve the full-horizon objective. Effectiveness of the technique is demonstrated numerically on a well-studied trajectory generation problem and a planning problem on a known obstacle-rich map. This paper also develops a new loss function term for policy-gradient-based Deep RL, which is analogous to an anti-windup mechanism in feedback control. We demonstrate the inclusion of this new term in the underlying optimization increases the average policy return in our numerical example.
△ Less
Submitted 12 August, 2022; v1 submitted 30 September, 2021;
originally announced October 2021.
-
The Replica Dataset: A Digital Replica of Indoor Spaces
Authors:
Julian Straub,
Thomas Whelan,
Lingni Ma,
Yufan Chen,
Erik Wijmans,
Simon Green,
Jakob J. Engel,
Raul Mur-Artal,
Carl Ren,
Shobhit Verma,
Anton Clarkson,
Mingfei Yan,
Brian Budge,
Yajie Yan,
Xiaqing Pan,
June Yon,
Yuyang Zou,
Kimberly Leon,
Nigel Carter,
Jesus Briales,
Tyler Gillingham,
Elias Mueggler,
Luis Pesqueira,
Manolis Savva,
Dhruv Batra
, et al. (5 additional authors not shown)
Abstract:
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors. The goal of Replica is to enable machine learning (ML) research that relies on visually, geometr…
▽ More
We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors. The goal of Replica is to enable machine learning (ML) research that relies on visually, geometrically, and semantically realistic generative models of the world - for instance, egocentric computer vision, semantic segmentation in 2D and 3D, geometric inference, and the development of embodied agents (virtual robots) performing navigation, instruction following, and question answering. Due to the high level of realism of the renderings from Replica, there is hope that ML systems trained on Replica may transfer directly to real world image and video data. Together with the data, we are releasing a minimal C++ SDK as a starting point for working with the Replica dataset. In addition, Replica is `Habitat-compatible', i.e. can be natively used with AI Habitat for training and testing embodied agents.
△ Less
Submitted 13 June, 2019;
originally announced June 2019.
