-
Autonomous Rendezvous with Non-cooperative Target Objects with Swarm Chasers and Observers
Authors:
Trupti Mahendrakar,
Steven Holmberg,
Andrew Ekblad,
Emma Conti,
Ryan T. White,
Markus Wilde,
Isaac Silver
Abstract:
Space debris is on the rise due to the increasing demand for spacecraft for com-munication, navigation, and other applications. The Space Surveillance Network (SSN) tracks over 27,000 large pieces of debris and estimates the number of small, un-trackable fragments at over 1,00,000. To control the growth of debris, the for-mation of further debris must be reduced. Some solutions include deorbiting…
▽ More
Space debris is on the rise due to the increasing demand for spacecraft for com-munication, navigation, and other applications. The Space Surveillance Network (SSN) tracks over 27,000 large pieces of debris and estimates the number of small, un-trackable fragments at over 1,00,000. To control the growth of debris, the for-mation of further debris must be reduced. Some solutions include deorbiting larger non-cooperative resident space objects (RSOs) or servicing satellites in or-bit. Both require rendezvous with RSOs, and the scale of the problem calls for autonomous missions. This paper introduces the Multipurpose Autonomous Ren-dezvous Vision-Integrated Navigation system (MARVIN) developed and tested at the ORION Facility at Florida Institution of Technology. MARVIN consists of two sub-systems: a machine vision-aided navigation system and an artificial po-tential field (APF) guidance algorithm which work together to command a swarm of chasers to safely rendezvous with the RSO. We present the MARVIN architec-ture and hardware-in-the-loop experiments demonstrating autonomous, collabo-rative swarm satellite operations successfully guiding three drones to rendezvous with a physical mockup of a non-cooperative satellite in motion.
△ Less
Submitted 22 January, 2023;
originally announced January 2023.
-
BEFANA: A Tool for Biodiversity-Ecosystem Functioning Assessment by Network Analysis
Authors:
Martin Marzidovšek,
Vid Podpečan,
Erminia Conti,
Marko Debeljak,
Christian Mulder
Abstract:
BEFANA is a free and open-source software tool for ecological network analysis and visualisation. It is adapted to ecologists' needs and allows them to study the topology and dynamics of ecological networks as well as apply selected machine learning algorithms. BEFANA is implemented in Python, and structured as an ordered collection of interactive computational notebooks. It relies on widely used…
▽ More
BEFANA is a free and open-source software tool for ecological network analysis and visualisation. It is adapted to ecologists' needs and allows them to study the topology and dynamics of ecological networks as well as apply selected machine learning algorithms. BEFANA is implemented in Python, and structured as an ordered collection of interactive computational notebooks. It relies on widely used open-source libraries, and aims to achieve simplicity, interactivity, and extensibility. BEFANA provides methods and implementations for data loading and preprocessing, network analysis and interactive visualisation, modelling with experimental data, and predictive modelling with machine learning. We showcase BEFANA through a concrete example of a detrital soil food web of agricultural grasslands, and demonstrate all of its main components and functionalities.
△ Less
Submitted 28 March, 2022; v1 submitted 21 March, 2022;
originally announced March 2022.
-
Benchmarking Batch Deep Reinforcement Learning Algorithms
Authors:
Scott Fujimoto,
Edoardo Conti,
Mohammad Ghavamzadeh,
Joelle Pineau
Abstract:
Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement lea…
▽ More
Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.
△ Less
Submitted 3 October, 2019;
originally announced October 2019.
-
Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
Authors:
Jason Gauci,
Edoardo Conti,
Yitao Liang,
Kittipat Virochsiri,
Yuchen He,
Zachary Kaden,
Vivek Narayanan,
Xiaohui Ye,
Zhengxing Chen,
Scott Fujimoto
Abstract:
In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, w…
▽ More
In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, optimized serving, and a model-based data understanding tool. We also showcase and describe real examples where reinforcement learning models trained with Horizon significantly outperformed and replaced supervised learning systems at Facebook.
△ Less
Submitted 4 September, 2019; v1 submitted 1 November, 2018;
originally announced November 2018.
-
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
Authors:
Felipe Petroski Such,
Vashisht Madhavan,
Edoardo Conti,
Joel Lehman,
Kenneth O. Stanley,
Jeff Clune
Abstract:
Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an ope…
▽ More
Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g.\ DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ${\raise.17ex\hbox{$\scriptstyle\sim$}}$4 hours on one desktop or ${\raise.17ex\hbox{$\scriptstyle\sim$}}$1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.
△ Less
Submitted 20 April, 2018; v1 submitted 18 December, 2017;
originally announced December 2017.
-
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
Authors:
Edoardo Conti,
Vashisht Madhavan,
Felipe Petroski Such,
Joel Lehman,
Kenneth O. Stanley,
Jeff Clune
Abstract:
Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are…
▽ More
Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.
△ Less
Submitted 29 October, 2018; v1 submitted 18 December, 2017;
originally announced December 2017.
-
Disruptions in the U.S. Airport Network
Authors:
Edoardo Conti,
Steve Cao,
A. J. Thomas
Abstract:
Our project analyzes the United States domestic airport network. We attempt to determine which airports are most vital in maintaining the underlying infrastructure for all domestic flights within the United States. To perform our analysis, we use data from the first quarter of 2010 and use several methods and algorithms that are frequently used in network science. Using these statistics, we identi…
▽ More
Our project analyzes the United States domestic airport network. We attempt to determine which airports are most vital in maintaining the underlying infrastructure for all domestic flights within the United States. To perform our analysis, we use data from the first quarter of 2010 and use several methods and algorithms that are frequently used in network science. Using these statistics, we identified the most important airports in the United States and investigate the role and significance that these airports play in maintaining the structure of the entire domestic airport network. Some of these airports include Denver International and Ted Stevens Anchorage International. We also identified any structural holes and suggested improvements that can be made to the network. Finally, through our analysis, we developed a disaster response algorithm that calculates flight path reroutes in emergency situations.
△ Less
Submitted 8 January, 2013;
originally announced January 2013.
