-
Adversarial Attacks on Reinforcement Learning Agents for Command and Control
Authors:
Ahaan Dabholkar,
James Z. Hare,
Mark Mittrick,
John Richardson,
Nicholas Waytowich,
Priya Narayanan,
Saurabh Bagchi
Abstract:
Given the recent impact of Deep Reinforcement Learning in training agents to win complex games like StarCraft and DoTA(Defense Of The Ancients) - there has been a surge in research for exploiting learning based techniques for professional wargaming, battlefield simulation and modeling. Real time strategy games and simulators have become a valuable resource for operational planning and military res…
▽ More
Given the recent impact of Deep Reinforcement Learning in training agents to win complex games like StarCraft and DoTA(Defense Of The Ancients) - there has been a surge in research for exploiting learning based techniques for professional wargaming, battlefield simulation and modeling. Real time strategy games and simulators have become a valuable resource for operational planning and military research. However, recent work has shown that such learning based approaches are highly susceptible to adversarial perturbations. In this paper, we investigate the robustness of an agent trained for a Command and Control task in an environment that is controlled by an active adversary. The C2 agent is trained on custom StarCraft II maps using the state of the art RL algorithms - A3C and PPO. We empirically show that an agent trained using these algorithms is highly susceptible to noise injected by the adversary and investigate the effects these perturbations have on the performance of the trained agent. Our work highlights the urgent need to develop more robust training algorithms especially for critical arenas like the battlefield.
△ Less
Submitted 1 July, 2024; v1 submitted 2 May, 2024;
originally announced May 2024.
-
StarCraftImage: A Dataset For Prototyping Spatial Reasoning Methods For Multi-Agent Environments
Authors:
Sean Kulinski,
Nicholas R. Waytowich,
James Z. Hare,
David I. Inouye
Abstract:
Spatial reasoning tasks in multi-agent environments such as event prediction, agent type identification, or missing data imputation are important for multiple applications (e.g., autonomous surveillance over sensor networks and subtasks for reinforcement learning (RL)). StarCraft II game replays encode intelligent (and adversarial) multi-agent behavior and could provide a testbed for these tasks;…
▽ More
Spatial reasoning tasks in multi-agent environments such as event prediction, agent type identification, or missing data imputation are important for multiple applications (e.g., autonomous surveillance over sensor networks and subtasks for reinforcement learning (RL)). StarCraft II game replays encode intelligent (and adversarial) multi-agent behavior and could provide a testbed for these tasks; however, extracting simple and standardized representations for prototyping these tasks is laborious and hinders reproducibility. In contrast, MNIST and CIFAR10, despite their extreme simplicity, have enabled rapid prototyping and reproducibility of ML methods. Following the simplicity of these datasets, we construct a benchmark spatial reasoning dataset based on StarCraft II replays that exhibit complex multi-agent behaviors, while still being as easy to use as MNIST and CIFAR10. Specifically, we carefully summarize a window of 255 consecutive game states to create 3.6 million summary images from 60,000 replays, including all relevant metadata such as game outcome and player races. We develop three formats of decreasing complexity: Hyperspectral images that include one channel for every unit type (similar to multispectral geospatial images), RGB images that mimic CIFAR10, and grayscale images that mimic MNIST. We show how this dataset can be used for prototyping spatial reasoning methods. All datasets, code for extraction, and code for dataset loading can be found at https://starcraftdata.davidinouye.com
△ Less
Submitted 8 January, 2024;
originally announced January 2024.
-
Enhancing Multi-Agent Coordination through Common Operating Picture Integration
Authors:
Peihong Yu,
Bhoram Lee,
Aswin Raghavan,
Supun Samarasekara,
Pratap Tokekar,
James Zachary Hare
Abstract:
In multi-agent systems, agents possess only local observations of the environment. Communication between teammates becomes crucial for enhancing coordination. Past research has primarily focused on encoding local information into embedding messages which are unintelligible to humans. We find that using these messages in agent's policy learning leads to brittle policies when tested on out-of-distri…
▽ More
In multi-agent systems, agents possess only local observations of the environment. Communication between teammates becomes crucial for enhancing coordination. Past research has primarily focused on encoding local information into embedding messages which are unintelligible to humans. We find that using these messages in agent's policy learning leads to brittle policies when tested on out-of-distribution initial states. We present an approach to multi-agent coordination, where each agent is equipped with the capability to integrate its (history of) observations, actions and messages received into a Common Operating Picture (COP) and disseminate the COP. This process takes into account the dynamic nature of the environment and the shared mission. We conducted experiments in the StarCraft2 environment to validate our approach. Our results demonstrate the efficacy of COP integration, and show that COP-based training leads to robust policies compared to state-of-the-art Multi-Agent Reinforcement Learning (MARL) methods when faced with out-of-distribution initial states.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
A General Framework for Distributed Inference with Uncertain Models
Authors:
James Z. Hare,
Cesar A. Uribe,
Lance Kaplan,
Ali Jadbabaie
Abstract:
This paper studies the problem of distributed classification with a network of heterogeneous agents. The agents seek to jointly identify the underlying target class that best describes a sequence of observations. The problem is first abstracted to a hypothesis-testing framework, where we assume that the agents seek to agree on the hypothesis (target class) that best matches the distribution of obs…
▽ More
This paper studies the problem of distributed classification with a network of heterogeneous agents. The agents seek to jointly identify the underlying target class that best describes a sequence of observations. The problem is first abstracted to a hypothesis-testing framework, where we assume that the agents seek to agree on the hypothesis (target class) that best matches the distribution of observations. Non-Bayesian social learning theory provides a framework that solves this problem in an efficient manner by allowing the agents to sequentially communicate and update their beliefs for each hypothesis over the network. Most existing approaches assume that agents have access to exact statistical models for each hypothesis. However, in many practical applications, agents learn the likelihood models based on limited data, which induces uncertainty in the likelihood function parameters. In this work, we build upon the concept of uncertain models to incorporate the agents' uncertainty in the likelihoods by identifying a broad set of parametric distribution that allows the agents' beliefs to converge to the same result as a centralized approach. Furthermore, we empirically explore extensions to non-parametric models to provide a generalized framework of uncertain models in non-Bayesian social learning.
△ Less
Submitted 20 November, 2020;
originally announced November 2020.
-
Non-Bayesian Social Learning with Gaussian Uncertain Models
Authors:
James Z. Hare,
Cesar Uribe,
Lance Kaplan,
Ali Jadbabaie
Abstract:
Non-Bayesian social learning theory provides a framework for distributed inference of a group of agents interacting over a social network by sequentially communicating and updating beliefs about the unknown state of the world through likelihood updates from their observations. Typically, likelihood models are assumed known precisely. However, in many situations the models are generated from sparse…
▽ More
Non-Bayesian social learning theory provides a framework for distributed inference of a group of agents interacting over a social network by sequentially communicating and updating beliefs about the unknown state of the world through likelihood updates from their observations. Typically, likelihood models are assumed known precisely. However, in many situations the models are generated from sparse training data due to lack of data availability, high cost of collection/calibration, limits within the communications network, and/or the high dynamics of the operational environment. Recently, social learning theory was extended to handle those model uncertainties for categorical models. In this paper, we introduce the theory of Gaussian uncertain models and study the properties of the beliefs generated by the network of agents. We show that even with finite amounts of training data, non-Bayesian social learning can be achieved and all agents in the network will converge to a consensus belief that provably identifies the best estimate for the state of the world given the set of prior information.
△ Less
Submitted 24 October, 2019;
originally announced October 2019.
-
POSE.R: Prediction-based Opportunistic Sensing for Resilient and Efficient Sensor Networks
Authors:
James Z. Hare,
Junnan Song,
Shalabh Gupta,
Thomas A. Wettergren
Abstract:
The paper presents a distributed algorithm, called Prediction-based Opportunistic Sensing for Resilient and Efficient Sensor Networks (POSE.R), where the sensor nodes utilize predictions of the targets positions to probabilistically control their multi-modal operating states to track the target. There are two desired features of the algorithm: energy-efficiency and resilience. If the target is tra…
▽ More
The paper presents a distributed algorithm, called Prediction-based Opportunistic Sensing for Resilient and Efficient Sensor Networks (POSE.R), where the sensor nodes utilize predictions of the targets positions to probabilistically control their multi-modal operating states to track the target. There are two desired features of the algorithm: energy-efficiency and resilience. If the target is traveling through a high node density area, then an optimal sensor selection approach is employed that maximizes a joint cost function of remaining energy and geometric diversity around the targets position. This provides energy-efficiency and increases the network lifetime while preventing redundant nodes from tracking the target. On the other hand, if the target is traveling through a low node density area or in a coverage gap (e.g., formed by node failures or non-uniform node deployment), then a potential game is played amongst the surrounding nodes to optimally expand their sensing ranges via minimizing energy consumption and maximizing target coverage. This provides resilience, that is the self-healing capability to track the target in the presence of low node densities and coverage gaps. The algorithm is comparatively evaluated against existing approaches through Monte Carlo simulations which demonstrate its superiority in terms of tracking performance, network-resilience and network-lifetime.
△ Less
Submitted 2 September, 2020; v1 submitted 23 October, 2019;
originally announced October 2019.
-
Non-Bayesian Social Learning with Uncertain Models
Authors:
James Z. Hare,
Cesar A. Uribe,
Lance Kaplan,
Ali Jadbabaie
Abstract:
Non-Bayesian social learning theory provides a framework that models distributed inference for a group of agents interacting over a social network. In this framework, each agent iteratively forms and communicates beliefs about an unknown state of the world with their neighbors using a learning rule. Existing approaches assume agents have access to precise statistical models (in the form of likelih…
▽ More
Non-Bayesian social learning theory provides a framework that models distributed inference for a group of agents interacting over a social network. In this framework, each agent iteratively forms and communicates beliefs about an unknown state of the world with their neighbors using a learning rule. Existing approaches assume agents have access to precise statistical models (in the form of likelihoods) for the state of the world. However in many situations, such models must be learned from finite data. We propose a social learning rule that takes into account uncertainty in the statistical models using second-order probabilities. Therefore, beliefs derived from uncertain models are sensitive to the amount of past evidence collected for each hypothesis. We characterize how well the hypotheses can be tested on a social network, as consistent or not with the state of the world. We explicitly show the dependency of the generated beliefs with respect to the amount of prior evidence. Moreover, as the amount of prior evidence goes to infinity, learning occurs and is consistent with traditional social learning theory.
△ Less
Submitted 27 September, 2019; v1 submitted 9 September, 2019;
originally announced September 2019.
-
Non-Bayesian Social Learning with Uncertain Models over Time-Varying Directed Graphs
Authors:
César A. Uribe,
James Z. Hare,
Lance Kaplan,
Ali Jadbabaie
Abstract:
We study the problem of non-Bayesian social learning with uncertain models, in which a network of agents seek to cooperatively identify the state of the world based on a sequence of observed signals. In contrast with the existing literature, we focus our attention on the scenario where the statistical models held by the agents about possible states of the world are built from finite observations.…
▽ More
We study the problem of non-Bayesian social learning with uncertain models, in which a network of agents seek to cooperatively identify the state of the world based on a sequence of observed signals. In contrast with the existing literature, we focus our attention on the scenario where the statistical models held by the agents about possible states of the world are built from finite observations. We show that existing non-Bayesian social learning approaches may select a wrong hypothesis with non-zero probability under these conditions. Therefore, we propose a new algorithm to iteratively construct a set of beliefs that indicate whether a certain hypothesis is supported by the empirical evidence. This new algorithm can be implemented over time-varying directed graphs, with non{-}doubly stochastic weights.
△ Less
Submitted 9 September, 2019;
originally announced September 2019.
