-
Towards High-Frequency Tracking and Fast Edge-Aware Optimization
Authors:
Akash Bapat
Abstract:
This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization.
AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with com…
▽ More
This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization.
AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with computing. Humans are sensitive to small misalignments between the real and the virtual world, and tracking at kilo-Hertz frequencies becomes essential. Current vision-based systems fall short, as their tracking frequency is implicitly limited by the frame-rate of the camera. This thesis presents a prototype system which can track at orders of magnitude higher than the state-of-the-art methods using multiple commodity cameras. The proposed system exploits characteristics of the camera traditionally considered as flaws, namely rolling shutter and radial distortion. The experimental evaluation shows the effectiveness of the method for various degrees of motion.
Furthermore, edge-aware optimization is an indispensable tool in the computer vision arsenal for accurate filtering of depth-data and image-based rendering, which is increasingly being used for content creation and geometry processing for AR/VR. As applications increasingly demand higher resolution and speed, there exists a need to develop methods that scale accordingly. This dissertation proposes such an edge-aware optimization framework which is efficient, accurate, and algorithmically scales well, all of which are much desirable traits not found jointly in the state of the art. The experiments show the effectiveness of the framework in a multitude of computer vision tasks such as computational photography and stereo.
△ Less
Submitted 1 September, 2023;
originally announced September 2023.
-
A Practical Stereo Depth System for Smart Glasses
Authors:
Jialiang Wang,
Daniel Scharstein,
Akash Bapat,
Kevin Blackburn-Matzen,
Matthew Yu,
Jonathan Lehman,
Suhib Alsisan,
Yanghan Wang,
Sam Tsai,
Jan-Michael Frahm,
Zijian He,
Peter Vajda,
Michael F. Cohen,
Matt Uyttendaele
Abstract:
We present the design of a productionized end-to-end stereo depth sensing system that does pre-processing, online stereo rectification, and stereo depth estimation with a fallback to monocular depth estimation when rectification is unreliable. The output of our depth sensing system is then used in a novel view generation pipeline to create 3D computational photography effects using point-of-view i…
▽ More
We present the design of a productionized end-to-end stereo depth sensing system that does pre-processing, online stereo rectification, and stereo depth estimation with a fallback to monocular depth estimation when rectification is unreliable. The output of our depth sensing system is then used in a novel view generation pipeline to create 3D computational photography effects using point-of-view images captured by smart glasses. All these steps are executed on-device on the stringent compute budget of a mobile phone, and because we expect the users can use a wide range of smartphones, our design needs to be general and cannot be dependent on a particular hardware or ML accelerator such as a smartphone GPU. Although each of these steps is well studied, a description of a practical system is still lacking. For such a system, all these steps need to work in tandem with one another and fallback gracefully on failures within the system or less than ideal input data. We show how we handle unforeseen changes to calibration, e.g., due to heat, robustly support depth estimation in the wild, and still abide by the memory and latency constraints required for a smooth user experience. We show that our trained models are fast, and run in less than 1s on a six-year-old Samsung Galaxy S8 phone's CPU. Our models generalize well to unseen data and achieve good results on Middlebury and in-the-wild images captured from the smart glasses.
△ Less
Submitted 31 March, 2023; v1 submitted 18 November, 2022;
originally announced November 2022.
-
Computing the Matching Distance of 2-Parameter Persistence Modules from Critical Values
Authors:
Asilata Bapat,
Robyn Brooks,
Celia Hacker,
Claudia Landi,
Barbara I. Mahler,
Elizabeth R. Stephenson
Abstract:
The exact computation of the matching distance for multi-parameter persistence modules is an active area of research in computational topology. Achieving an easily obtainable exact computation of this distance would allow multi-parameter persistent homology to be a viable option for data analysis. In this paper, we provide theoretical results for the computation of the matching distance in two dim…
▽ More
The exact computation of the matching distance for multi-parameter persistence modules is an active area of research in computational topology. Achieving an easily obtainable exact computation of this distance would allow multi-parameter persistent homology to be a viable option for data analysis. In this paper, we provide theoretical results for the computation of the matching distance in two dimensions along with a geometric interpretation of the lines through parameter space realizing this distance. The crucial point of the method we propose is that it can be easily implemented.
△ Less
Submitted 23 October, 2022;
originally announced October 2022.
-
Advantages and limitations of quantum routing
Authors:
Aniruddha Bapat,
Andrew M. Childs,
Alexey V. Gorshkov,
Eddie Schoute
Abstract:
The Swap gate is a ubiquitous tool for moving information on quantum hardware, yet it can be considered a classical operation because it does not entangle product states. Genuinely quantum operations could outperform Swap for the task of permuting qubits within an architecture, which we call routing. We consider quantum routing in two models: (1) allowing arbitrary two-qubit unitaries, or (2) allo…
▽ More
The Swap gate is a ubiquitous tool for moving information on quantum hardware, yet it can be considered a classical operation because it does not entangle product states. Genuinely quantum operations could outperform Swap for the task of permuting qubits within an architecture, which we call routing. We consider quantum routing in two models: (1) allowing arbitrary two-qubit unitaries, or (2) allowing Hamiltonians with norm-bounded interactions. We lower bound the circuit depth or time of quantum routing in terms of spectral properties of graphs representing the architecture interaction constraints, and give a generalized upper bound for all simple connected $n$-vertex graphs. In particular, we give conditions for a superpolynomial classical-quantum routing separation, which exclude graphs with a small spectral gap and graphs of bounded degree. Finally, we provide examples of a quadratic separation between gate-based and Hamiltonian routing models with a constant number of local ancillas per qubit and of an $Ω(n)$ speedup if we also allow fast local interactions.
△ Less
Submitted 3 June, 2022;
originally announced June 2022.
-
Quantum routing with fast reversals
Authors:
Aniruddha Bapat,
Andrew M. Childs,
Alexey V. Gorshkov,
Samuel King,
Eddie Schoute,
Hrishee Shastri
Abstract:
We present methods for implementing arbitrary permutations of qubits under interaction constraints. Our protocols make use of previous methods for rapidly reversing the order of qubits along a path. Given nearest-neighbor interactions on a path of length $n$, we show that there exists a constant $ε\approx 0.034$ such that the quantum routing time is at most $(1-ε)n$, whereas any swap-based protoco…
▽ More
We present methods for implementing arbitrary permutations of qubits under interaction constraints. Our protocols make use of previous methods for rapidly reversing the order of qubits along a path. Given nearest-neighbor interactions on a path of length $n$, we show that there exists a constant $ε\approx 0.034$ such that the quantum routing time is at most $(1-ε)n$, whereas any swap-based protocol needs at least time $n-1$. This represents the first known quantum advantage over swap-based routing methods and also gives improved quantum routing times for realistic architectures such as grids. Furthermore, we show that our algorithm approaches a quantum routing time of $2n/3$ in expectation for uniformly random permutations, whereas swap-based protocols require time $n$ asymptotically. Additionally, we consider sparse permutations that route $k \le n$ qubits and give algorithms with quantum routing time at most $n/3 + O(k^2)$ on paths and at most $2r/3 + O(k^2)$ on general graphs with radius $r$.
△ Less
Submitted 24 August, 2021; v1 submitted 4 March, 2021;
originally announced March 2021.
-
Morse-based Fibering of the Persistence Rank Invariant
Authors:
Asilata Bapat,
Robyn Brooks,
Celia Hacker,
Claudia Landi,
Barbara I. Mahler
Abstract:
Although there is no doubt that multi-parameter persistent homology is a useful tool to analyse multi-variate data, efficient ways to compute these modules are still lacking in the available topological data analysis toolboxes. Other issues such as interpretation and visualization of the output remain difficult to solve. Software visualizing multi-parameter persistence diagrams is currently only a…
▽ More
Although there is no doubt that multi-parameter persistent homology is a useful tool to analyse multi-variate data, efficient ways to compute these modules are still lacking in the available topological data analysis toolboxes. Other issues such as interpretation and visualization of the output remain difficult to solve. Software visualizing multi-parameter persistence diagrams is currently only available for 2-dimensional persistence modules. One of the simplest invariants for a multi-parameter persistence module is its rank invariant, defined as the function that counts the number of linearly independent homology classes that live in the filtration through a given pair of values of the multi-parameter. We propose a step towards interpretation and visualization of the rank invariant for persistence modules for any given number of parameters. We show how discrete Morse theory may be used to compute the rank invariant, proving that it is completely determined by its values at points whose coordinates are critical with respect to a discrete Morse gradient vector field. These critical points partition the set of all lines of positive slope in the parameter space into equivalence classes, such that the rank invariant along lines in the same class are also equivalent. We show that we can deduce all persistence diagrams of the restrictions to the lines in a given class from the persistence diagram of the restriction to a representative in that class.
△ Less
Submitted 13 April, 2021; v1 submitted 30 November, 2020;
originally announced November 2020.
-
Mapped Convolutions
Authors:
Marc Eder,
True Price,
Thanh Vu,
Akash Bapat,
Jan-Michael Frahm
Abstract:
We present a versatile formulation of the convolution operation that we term a "mapped convolution." The standard convolution operation implicitly samples the pixel grid and computes a weighted sum. Our mapped convolution decouples these two components, freeing the operation from the confines of the image grid and allowing the kernel to process any type of structured data. As a test case, we demon…
▽ More
We present a versatile formulation of the convolution operation that we term a "mapped convolution." The standard convolution operation implicitly samples the pixel grid and computes a weighted sum. Our mapped convolution decouples these two components, freeing the operation from the confines of the image grid and allowing the kernel to process any type of structured data. As a test case, we demonstrate its use by applying it to dense inference on spherical data. We perform an in-depth study of existing spherical image convolution methods and propose an improved sampling method for equirectangular images. Then, we discuss the impact of data discretization when deriving a sampling function, highlighting drawbacks of the cube map representation for spherical data. Finally, we illustrate how mapped convolutions enable us to convolve directly on a mesh by projecting the spherical image onto a geodesic grid and training on the textured mesh. This method exceeds the state of the art for spherical depth estimation by nearly 17%. Our findings suggest that mapped convolutions can be instrumental in expanding the application scope of convolutional neural networks.
△ Less
Submitted 26 June, 2019;
originally announced June 2019.
-
The Domain Transform Solver
Authors:
Akash Bapat,
Jan-Michael Frahm
Abstract:
We present a framework for edge-aware optimization that is an order of magnitude faster than the state of the art while having comparable performance. Our key insight is that the optimization can be formulated by leveraging properties of the domain transform, a method for edge-aware filtering that defines a distance-preserving 1D mapping of the input space. This enables our method to improve perfo…
▽ More
We present a framework for edge-aware optimization that is an order of magnitude faster than the state of the art while having comparable performance. Our key insight is that the optimization can be formulated by leveraging properties of the domain transform, a method for edge-aware filtering that defines a distance-preserving 1D mapping of the input space. This enables our method to improve performance for a variety of problems including stereo, depth super-resolution, and render from defocus, while keeping the computational complexity linear in the number of pixels. Our method is highly parallelizable and adaptable, and it has demonstrable scalability with respect to image resolution.
△ Less
Submitted 11 May, 2018;
originally announced May 2018.
