Showing 1–48 of 48 results for author: Zavlanos, M M

Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies

Authors: Pingcheng Jian, Easop Lee, Zachary Bell, Michael M. Zavlanos, Boyuan Chen

Abstract: Vision-based imitation learning has shown promising capabilities of endowing robots with various motion skills given visual observation. However, current visuomotor policies fail to adapt to drastic changes in their visual observations. We present Perception Stitching that enables strong zero-shot adaptation to large visual changes by directly stitching novel combinations of visual encoders. Our k… ▽ More Vision-based imitation learning has shown promising capabilities of endowing robots with various motion skills given visual observation. However, current visuomotor policies fail to adapt to drastic changes in their visual observations. We present Perception Stitching that enables strong zero-shot adaptation to large visual changes by directly stitching novel combinations of visual encoders. Our key idea is to enforce modularity of visual encoders by aligning the latent visual features among different visuomotor policies. Our method disentangles the perceptual knowledge with the downstream motion skills and allows the reuse of the visual encoders by directly stitching them to a policy network trained with partially different visual conditions. We evaluate our method in various simulated and real-world manipulation tasks. While baseline methods failed at all attempts, our method could achieve zero-shot success in real-world visuomotor tasks. Our quantitative and qualitative analysis of the learned features of the policy network provides more insights into the high performance of our proposed method. △ Less

Submitted 28 June, 2024; originally announced June 2024.

Risk-averse Learning with Non-Stationary Distributions

Authors: Siyi Wang, Zifan Wang, Xinlei Yi, Michael M. Zavlanos, Karl H. Johansson, Sandra Hirche

Abstract: Considering non-stationary environments in online optimization enables decision-maker to effectively adapt to changes and improve its performance over time. In such cases, it is favorable to adopt a strategy that minimizes the negative impact of change to avoid potentially risky situations. In this paper, we investigate risk-averse online optimization where the distribution of the random cost chan… ▽ More Considering non-stationary environments in online optimization enables decision-maker to effectively adapt to changes and improve its performance over time. In such cases, it is favorable to adopt a strategy that minimizes the negative impact of change to avoid potentially risky situations. In this paper, we investigate risk-averse online optimization where the distribution of the random cost changes over time. We minimize risk-averse objective function using the Conditional Value at Risk (CVaR) as risk measure. Due to the difficulty in obtaining the exact CVaR gradient, we employ a zeroth-order optimization approach that queries the cost function values multiple times at each iteration and estimates the CVaR gradient using the sampled values. To facilitate the regret analysis, we use a variation metric based on Wasserstein distance to capture time-varying distributions. Given that the distribution variation is sub-linear in the total number of episodes, we show that our designed learning algorithm achieves sub-linear dynamic regret with high probability for both convex and strongly convex functions. Moreover, theoretical results suggest that increasing the number of samples leads to a reduction in the dynamic regret bounds until the sampling number reaches a specific limit. Finally, we provide numerical experiments of dynamic pricing in a parking lot to illustrate the efficacy of the designed algorithm. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Path Signatures and Graph Neural Networks for Slow Earthquake Analysis: Better Together?

Authors: Hans Riess, Manolis Veveakis, Michael M. Zavlanos

Abstract: The path signature, having enjoyed recent success in the machine learning community, is a theoretically-driven method for engineering features from irregular paths. On the other hand, graph neural networks (GNN), neural architectures for processing data on graphs, excel on tasks with irregular domains, such as sensor networks. In this paper, we introduce a novel approach, Path Signature Graph Conv… ▽ More The path signature, having enjoyed recent success in the machine learning community, is a theoretically-driven method for engineering features from irregular paths. On the other hand, graph neural networks (GNN), neural architectures for processing data on graphs, excel on tasks with irregular domains, such as sensor networks. In this paper, we introduce a novel approach, Path Signature Graph Convolutional Neural Networks (PS-GCNN), integrating path signatures into graph convolutional neural networks (GCNN), and leveraging the strengths of both path signatures, for feature extraction, and GCNNs, for handling spatial interactions. We apply our method to analyze slow earthquake sequences, also called slow slip events (SSE), utilizing data from GPS timeseries, with a case study on a GPS sensor network on the east coast of New Zealand's north island. We also establish benchmarks for our method on simulated stochastic differential equations, which model similar reaction-diffusion phenomenon. Our methodology shows promise for future advancement in earthquake prediction and sensor network analysis. △ Less

Submitted 5 February, 2024; originally announced February 2024.

Network Preference Dynamics using Lattice Theory

Authors: Hans Riess, Gregory Henselman-Petrusek, Michael C. Munger, Robert Ghrist, Zachary I. Bell, Michael M. Zavlanos

Abstract: Preferences, fundamental in all forms of strategic behavior and collective decision-making, in their raw form, are an abstract ordering on a set of alternatives. Agents, we assume, revise their preferences as they gain more information about other agents. Exploiting the ordered algebraic structure of preferences, we introduce a message-passing algorithm for heterogeneous agents distributed over a… ▽ More Preferences, fundamental in all forms of strategic behavior and collective decision-making, in their raw form, are an abstract ordering on a set of alternatives. Agents, we assume, revise their preferences as they gain more information about other agents. Exploiting the ordered algebraic structure of preferences, we introduce a message-passing algorithm for heterogeneous agents distributed over a network to update their preferences based on aggregations of the preferences of their neighbors in a graph. We demonstrate the existence of equilibrium points of the resulting global dynamical system of local preference updates and provide a sufficient condition for trajectories to converge to equilibria: stable preferences. Finally, we present numerical simulations demonstrating our preliminary results. △ Less

Submitted 10 July, 2024; v1 submitted 29 September, 2023; originally announced October 2023.

Policy Stitching: Learning Transferable Robot Policies

Authors: Pingcheng Jian, Easop Lee, Zachary Bell, Michael M. Zavlanos, Boyuan Chen

Abstract: Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations… ▽ More Training robots with reinforcement learning (RL) typically involves heavy interactions with the environment, and the acquired skills are often sensitive to changes in task environments and robot kinematics. Transfer RL aims to leverage previous knowledge to accelerate learning of new tasks or new body configurations. However, existing methods struggle to generalize to novel robot-task combinations and scale to realistic tasks due to complex architecture design or strong regularization that limits the capacity of the learned policy. We propose Policy Stitching, a novel framework that facilitates robot transfer learning for novel combinations of robots and tasks. Our key idea is to apply modular policy design and align the latent representations between the modular interfaces. Our method allows direct stitching of the robot and task modules trained separately to form a new policy for fast adaptation. Our simulated and real-world experiments on various 3D manipulation tasks demonstrate the superior zero-shot and few-shot transfer learning performances of our method. Our project website is at: http://generalroboticslab.com/PolicyStitching/ . △ Less

Submitted 24 September, 2023; originally announced September 2023.

Comments: CoRL 2023

Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Bandits

Authors: Yi Shen, Pan Xu, Michael M. Zavlanos

Abstract: Off-policy evaluation and learning are concerned with assessing a given policy and learning an optimal policy from offline data without direct interaction with the environment. Often, the environment in which the data are collected differs from the environment in which the learned policy is applied. To account for the effect of different environments during learning and execution, distributionally… ▽ More Off-policy evaluation and learning are concerned with assessing a given policy and learning an optimal policy from offline data without direct interaction with the environment. Often, the environment in which the data are collected differs from the environment in which the learned policy is applied. To account for the effect of different environments during learning and execution, distributionally robust optimization (DRO) methods have been developed that compute worst-case bounds on the policy values assuming that the distribution of the new environment lies within an uncertainty set. Typically, this uncertainty set is defined based on the KL divergence around the empirical distribution computed from the logging dataset. However, the KL uncertainty set fails to encompass distributions with varying support and lacks awareness of the geometry of the distribution support. As a result, KL approaches fall short in addressing practical environment mismatches and lead to over-fitting to worst-case scenarios. To overcome these limitations, we propose a novel DRO approach that employs the Wasserstein distance instead. While Wasserstein DRO is generally computationally more expensive compared to KL DRO, we present a regularized method and a practical (biased) stochastic gradient descent method to optimize the policy efficiently. We also provide a theoretical analysis of the finite sample complexity and iteration complexity for our proposed method. We further validate our approach using a public dataset that was recorded in a randomized stoke trial. △ Less

Submitted 17 January, 2024; v1 submitted 15 September, 2023; originally announced September 2023.

Convergence Analysis of the Best Response Algorithm for Time-Varying Games

Authors: Zifan Wang, Yi Shen, Michael M. Zavlanos, Karl H. Johansson

Abstract: This paper studies a class of strongly monotone games involving non-cooperative agents that optimize their own time-varying cost functions. We assume that the agents can observe other agents' historical actions and choose actions that best respond to other agents' previous actions; we call this a best response scheme. We start by analyzing the convergence rate of this best response scheme for stan… ▽ More This paper studies a class of strongly monotone games involving non-cooperative agents that optimize their own time-varying cost functions. We assume that the agents can observe other agents' historical actions and choose actions that best respond to other agents' previous actions; we call this a best response scheme. We start by analyzing the convergence rate of this best response scheme for standard time-invariant games. Specifically, we provide a sufficient condition on the strong monotonicity parameter of the time-invariant games under which the proposed best response algorithm achieves exponential convergence to the static Nash equilibrium. We further illustrate that this best response algorithm may oscillate when the proposed sufficient condition fails to hold, which indicates that this condition is tight. Next, we analyze this best response algorithm for time-varying games where the cost functions of each agent change over time. Under similar conditions as for time-invariant games, we show that the proposed best response algorithm stays asymptotically close to the evolving equilibrium. We do so by analyzing both the equilibrium tracking error and the dynamic regret. Numerical experiments on economic market problems are presented to validate our analysis. △ Less

Submitted 1 September, 2023; originally announced September 2023.

Max-Plus Synchronization in Decentralized Trading Systems

Authors: Hans Riess, Michael Munger, Michael M. Zavlanos

Abstract: We introduce a decentralized mechanism for pricing and exchanging alternatives constrained by transaction costs. We characterize the time-invariant solutions of a heat equation involving a (weighted) Tarski Laplacian operator, defined for max-plus matrix-weighted graphs, as approximate equilibria of the trading system. We study algebraic properties of the solution sets as well as convergence behav… ▽ More We introduce a decentralized mechanism for pricing and exchanging alternatives constrained by transaction costs. We characterize the time-invariant solutions of a heat equation involving a (weighted) Tarski Laplacian operator, defined for max-plus matrix-weighted graphs, as approximate equilibria of the trading system. We study algebraic properties of the solution sets as well as convergence behavior of the dynamical system. We apply these tools to the "economic problem" of allocating scarce resources among competing uses. Our theory suggests differences in competitive equilibrium, bargaining, or cost-benefit analysis, depending on the context, are largely due to differences in the way that transaction costs are incorporated into the decision-making process. We present numerical simulations of the synchronization algorithm (RRAggU), demonstrating our theoretical findings. △ Less

Submitted 13 September, 2023; v1 submitted 31 March, 2023; originally announced April 2023.

MSC Class: 15A80; 90C24 ACM Class: F.2.2

Policy Evaluation in Distributional LQR

Authors: Zifan Wang, Yulong Gao, Siyi Wang, Michael M. Zavlanos, Alessandro Abate, Karl H. Johansson

Abstract: Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard RL. At the same time, a main challenge in DRL is that policy evaluation in DRL typically relies on the representation of the return distribution, which needs to be carefu… ▽ More Distributional reinforcement learning (DRL) enhances the understanding of the effects of the randomness in the environment by letting agents learn the distribution of a random return, rather than its expected value as in standard RL. At the same time, a main challenge in DRL is that policy evaluation in DRL typically relies on the representation of the return distribution, which needs to be carefully designed. In this paper, we address this challenge for a special class of DRL problems that rely on linear quadratic regulator (LQR) for control, advocating for a new distributional approach to LQR, which we call \emph{distributional LQR}. Specifically, we provide a closed-form expression of the distribution of the random return which, remarkably, is applicable to all exogenous disturbances on the dynamics, as long as they are independent and identically distributed (i.i.d.). While the proposed exact return distribution consists of infinitely many random variables, we show that this distribution can be approximated by a finite number of random variables, and the associated approximation error can be analytically bounded under mild assumptions. Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR using the Conditional Value at Risk (CVaR) as a measure of risk. Numerical experiments are provided to illustrate our theoretical results. △ Less

Submitted 23 March, 2023; originally announced March 2023.

Comments: 12pages

Risk-Averse Multi-Armed Bandits with Unobserved Confounders: A Case Study in Emotion Regulation in Mobile Health

Authors: Yi Shen, Jessilyn Dunn, Michael M. Zavlanos

Abstract: In this paper, we consider a risk-averse multi-armed bandit (MAB) problem where the goal is to learn a policy that minimizes the risk of low expected return, as opposed to maximizing the expected return itself, which is the objective in the usual approach to risk-neutral MAB. Specifically, we formulate this problem as a transfer learning problem between an expert and a learner agent in the presenc… ▽ More In this paper, we consider a risk-averse multi-armed bandit (MAB) problem where the goal is to learn a policy that minimizes the risk of low expected return, as opposed to maximizing the expected return itself, which is the objective in the usual approach to risk-neutral MAB. Specifically, we formulate this problem as a transfer learning problem between an expert and a learner agent in the presence of contexts that are only observable by the expert but not by the learner. Thus, such contexts are unobserved confounders (UCs) from the learner's perspective. Given a dataset generated by the expert that excludes the UCs, the goal for the learner is to identify the true minimum-risk arm with fewer online learning steps, while avoiding possible biased decisions due to the presence of UCs in the expert's data. △ Less

Submitted 9 September, 2022; originally announced September 2022.

A Zeroth-Order Momentum Method for Risk-Averse Online Convex Games

Authors: Zifan Wang, Yi Shen, Zachary I. Bell, Scott Nivison, Michael M. Zavlanos, Karl H. Johansson

Abstract: We consider risk-averse learning in repeated unknown games where the goal of the agents is to minimize their individual risk of incurring significantly high cost. Specifically, the agents use the conditional value at risk (CVaR) as a risk measure and rely on bandit feedback in the form of the cost values of the selected actions at every episode to estimate their CVaR values and update their action… ▽ More We consider risk-averse learning in repeated unknown games where the goal of the agents is to minimize their individual risk of incurring significantly high cost. Specifically, the agents use the conditional value at risk (CVaR) as a risk measure and rely on bandit feedback in the form of the cost values of the selected actions at every episode to estimate their CVaR values and update their actions. A major challenge in using bandit feedback to estimate CVaR is that the agents can only access their own cost values, which, however, depend on the actions of all agents. To address this challenge, we propose a new risk-averse learning algorithm with momentum that utilizes the full historical information on the cost values. We show that this algorithm achieves sub-linear regret and matches the best known algorithms in the literature. We provide numerical experiments for a Cournot game that show that our method outperforms existing methods. △ Less

Submitted 6 September, 2022; originally announced September 2022.

Risk-Averse No-Regret Learning in Online Convex Games

Authors: Zifan Wang, Yi Shen, Michael M. Zavlanos

Abstract: We consider an online stochastic game with risk-averse agents whose goal is to learn optimal decisions that minimize the risk of incurring significantly high costs. Specifically, we use the Conditional Value at Risk (CVaR) as a risk measure that the agents can estimate using bandit feedback in the form of the cost values of only their selected actions. Since the distributions of the cost functions… ▽ More We consider an online stochastic game with risk-averse agents whose goal is to learn optimal decisions that minimize the risk of incurring significantly high costs. Specifically, we use the Conditional Value at Risk (CVaR) as a risk measure that the agents can estimate using bandit feedback in the form of the cost values of only their selected actions. Since the distributions of the cost functions depend on the actions of all agents that are generally unobservable, they are themselves unknown and, therefore, the CVaR values of the costs are difficult to compute. To address this challenge, we propose a new online risk-averse learning algorithm that relies on one-point zeroth-order estimation of the CVaR gradients computed using CVaR values that are estimated by appropriately sampling the cost functions. We show that this algorithm achieves sub-linear regret with high probability. We also propose two variants of this algorithm that improve performance. The first variant relies on a new sampling strategy that uses samples from the previous iteration to improve the estimation accuracy of the CVaR values. The second variant employs residual feedback that uses CVaR values from the previous iteration to reduce the variance of the CVaR gradient estimates. We theoretically analyze the convergence properties of these variants and illustrate their performance on an online market problem that we model as a Cournot game. △ Less

Submitted 15 June, 2022; v1 submitted 16 March, 2022; originally announced March 2022.

Failing with Grace: Learning Neural Network Controllers that are Boundedly Unsafe

Authors: Panagiotis Vlantis, Leila J. Bridgeman, Michael M. Zavlanos

Abstract: In this work, we consider the problem of learning a feed-forward neural network controller to safely steer an arbitrarily shaped planar robot in a compact and obstacle-occluded workspace. Unlike existing methods that depend strongly on the density of data points close to the boundary of the safe state space to train neural network controllers with closed-loop safety guarantees, here we propose an… ▽ More In this work, we consider the problem of learning a feed-forward neural network controller to safely steer an arbitrarily shaped planar robot in a compact and obstacle-occluded workspace. Unlike existing methods that depend strongly on the density of data points close to the boundary of the safe state space to train neural network controllers with closed-loop safety guarantees, here we propose an alternative approach that lifts such strong assumptions on the data that are hard to satisfy in practice and instead allows for graceful safety violations, i.e., of a bounded magnitude that can be spatially controlled. To do so, we employ reachability analysis techniques to encapsulate safety constraints in the training process. Specifically, to obtain a computationally efficient over-approximation of the forward reachable set of the closed-loop system, we partition the robot's state space into cells and adaptively subdivide the cells that contain states which may escape the safe set under the trained control law. Then, using the overlap between each cell's forward reachable set and the set of infeasible robot configurations as a measure for safety violations, we introduce appropriate terms into the loss function that penalize this overlap in the training process. As a result, our method can learn a safe vector field for the closed-loop system and, at the same time, provide worst-case bounds on safety violation over the whole configuration space, defined by the overlap between the over-approximation of the forward reachable set of the closed-loop system and the set of unsafe states. Moreover, it can control the tradeoff between computational complexity and tightness of these bounds. Our proposed method is supported by both theoretical results and simulation studies. △ Less

Submitted 12 December, 2022; v1 submitted 22 June, 2021; originally announced June 2021.

Learning without Knowing: Unobserved Context in Continuous Transfer Reinforcement Learning

Authors: Chenyu Liu, Yan Zhang, Yi Shen, Michael M. Zavlanos

Abstract: In this paper, we consider a transfer Reinforcement Learning (RL) problem in continuous state and action spaces, under unobserved contextual information. For example, the context can represent the mental view of the world that an expert agent has formed through past interactions with this world. We assume that this context is not accessible to a learner agent who can only observe the expert data.… ▽ More In this paper, we consider a transfer Reinforcement Learning (RL) problem in continuous state and action spaces, under unobserved contextual information. For example, the context can represent the mental view of the world that an expert agent has formed through past interactions with this world. We assume that this context is not accessible to a learner agent who can only observe the expert data. Then, our goal is to use the context-aware expert data to learn an optimal context-unaware policy for the learner using only a few new data samples. Such problems are typically solved using imitation learning that assumes that both the expert and learner agents have access to the same information. However, if the learner does not know the expert context, using the expert data alone will result in a biased learner policy and will require many new data samples to improve. To address this challenge, in this paper, we formulate the learning problem as a causal bound-constrained Multi-Armed-Bandit (MAB) problem. The arms of this MAB correspond to a set of basis policy functions that can be initialized in an unsupervised way using the expert data and represent the different expert behaviors affected by the unobserved context. On the other hand, the MAB constraints correspond to causal bounds on the accumulated rewards of these basis policy functions that we also compute from the expert data. The solution to this MAB allows the learner agent to select the best basis policy and improve it online. And the use of causal bounds reduces the exploration variance and, therefore, improves the learning rate. We provide numerical experiments on an autonomous driving example that show that our proposed transfer RL method improves the learner's policy faster compared to existing imitation learning methods and enjoys much lower variance during training. △ Less

Submitted 7 June, 2021; originally announced June 2021.

Formal Verification of Stochastic Systems with ReLU Neural Network Controllers

Authors: Shiqi Sun, Yan Zhang, Xusheng Luo, Panagiotis Vlantis, Miroslav Pajic, Michael M. Zavlanos

Abstract: In this work, we address the problem of formal safety verification for stochastic cyber-physical systems (CPS) equipped with ReLU neural network (NN) controllers. Our goal is to find the set of initial states from where, with a predetermined confidence, the system will not reach an unsafe configuration within a specified time horizon. Specifically, we consider discrete-time LTI systems with Gaussi… ▽ More In this work, we address the problem of formal safety verification for stochastic cyber-physical systems (CPS) equipped with ReLU neural network (NN) controllers. Our goal is to find the set of initial states from where, with a predetermined confidence, the system will not reach an unsafe configuration within a specified time horizon. Specifically, we consider discrete-time LTI systems with Gaussian noise, which we abstract by a suitable graph. Then, we formulate a Satisfiability Modulo Convex (SMC) problem to estimate upper bounds on the transition probabilities between nodes in the graph. Using this abstraction, we propose a method to compute tight bounds on the safety probabilities of nodes in this graph, despite possible over-approximations of the transition probabilities between these nodes. Additionally, using the proposed SMC formula, we devise a heuristic method to refine the abstraction of the system in order to further improve the estimated safety bounds. Finally, we corroborate the efficacy of the proposed method with simulation results considering a robot navigation example and comparison against a state-of-the-art verification scheme. △ Less

Submitted 8 March, 2021; originally announced March 2021.

Learning Optimal Strategies for Temporal Tasks in Stochastic Games

Authors: Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, Miroslav Pajic

Abstract: Synthesis from linear temporal logic (LTL) specifications provides assured controllers for systems operating in stochastic and potentially adversarial environments. Automatic synthesis tools, however, require a model of the environment to construct controllers. In this work, we introduce a model-free reinforcement learning (RL) approach to derive controllers from given LTL specifications even when… ▽ More Synthesis from linear temporal logic (LTL) specifications provides assured controllers for systems operating in stochastic and potentially adversarial environments. Automatic synthesis tools, however, require a model of the environment to construct controllers. In this work, we introduce a model-free reinforcement learning (RL) approach to derive controllers from given LTL specifications even when the environment is completely unknown. We model the problem as a stochastic game (SG) between the controller and the adversarial environment; we then learn optimal control strategies that maximize the probability of satisfying the LTL specifications against the worst-case environment behavior. We first construct a product game using the deterministic parity automaton (DPA) translated from the given LTL specification. By deriving distinct rewards and discount factors from the acceptance condition of the DPA, we reduce the maximization of the worst-case probability of satisfying the LTL specification into the maximization of a discounted reward objective in the product game; this enables the use of model-free RL algorithms to learn an optimal controller strategy. To deal with the common scalability problems when the number of sets defining the acceptance condition of the DPA (usually referred as colors), is large, we propose a lazy color generation method where distinct rewards and discount factors are utilized only when needed, and an approximate method where the controller eventually focuses on only one color. In several case studies, we show that our approach is scalable to a wide range of LTL formulas, significantly outperforming existing methods for learning controllers from LTL specifications in SGs. △ Less

Submitted 30 August, 2023; v1 submitted 8 February, 2021; originally announced February 2021.

Temporal Logic Task Allocation in Heterogeneous Multi-Robot Systems

Authors: Xusheng Luo, Michael M. Zavlanos

Abstract: In this paper, we consider the problem of optimally allocating tasks, expressed as global Linear Temporal Logic (LTL) specifications, to teams of heterogeneous mobile robots. The robots are classified in different types that capture their different capabilities, and each task may require robots of multiple types. The specific robots assigned to each task are immaterial, as long as they are of the… ▽ More In this paper, we consider the problem of optimally allocating tasks, expressed as global Linear Temporal Logic (LTL) specifications, to teams of heterogeneous mobile robots. The robots are classified in different types that capture their different capabilities, and each task may require robots of multiple types. The specific robots assigned to each task are immaterial, as long as they are of the desired type. Given a discrete workspace, our goal is to design paths, i.e., sequences of discrete states, for the robots so that the LTL specification is satisfied. To obtain a scalable solution to this complex temporal logic task allocation problem, we propose a hierarchical approach that first allocates specific robots to tasks using the information about the tasks contained in the Nondeterministic Buchi Automaton (NBA) that captures the LTL specification, and then designs low-level executable plans for the robots that respect the high-level assignment. Specifically, we first prune and relax the NBA by removing all negative atomic propositions. This step is motivated by "lazy collision checking" methods in robotics and allows to simplify the planning problem by checking constraint satisfaction only when needed. Then, we extract sequences of subtasks from the relaxed NBA along with their temporal orders, and formulate a Mixed Integer Linear Program (MILP) to allocate these subtasks to the robots. Finally, we define generalized multi-robot path planning problems to obtain low-level executable robot plans that satisfy both the high-level task allocation and the temporal constraints captured by the negative atomic propositions in the original NBA. We show that our method is complete for a subclass of LTL that covers a broad range of tasks and present numerical simulations demonstrating that it can generate paths with lower cost, considerably faster than existing methods. △ Less

Submitted 13 February, 2022; v1 submitted 14 January, 2021; originally announced January 2021.

Comments: 47pages, 17 figures

Plane Wave Elastography: A Frequency-Domain Ultrasound Shear Wave Elastography Approach

Authors: Reza Khodayi-mehr, Matthew W. Urban, Michael M. Zavlanos, Wilkins Aquino

Abstract: In this paper, we propose Plane Wave Elastography (PWE), a novel ultrasound shear wave elastography (SWE) approach. Currently, commercial methods for SWE rely on directional filtering based on the prior knowledge of the wave propagation direction, to remove complicated wave patterns formed due to reflection and refraction. The result is a set of decomposed directional waves that are separately ana… ▽ More In this paper, we propose Plane Wave Elastography (PWE), a novel ultrasound shear wave elastography (SWE) approach. Currently, commercial methods for SWE rely on directional filtering based on the prior knowledge of the wave propagation direction, to remove complicated wave patterns formed due to reflection and refraction. The result is a set of decomposed directional waves that are separately analyzed to construct shear modulus fields that are then combined through compounding. Instead, PWE relies on a rigorous representation of the wave propagation using the frequency-domain scalar wave equation to automatically select appropriate propagation directions and simultaneously reconstruct shear modulus fields. Specifically, assuming a homogeneous, isotropic, incompressible, linear-elastic medium, we represent the solution of the wave equation using a linear combination of plane waves propagating in arbitrary directions. Given this closed-form solution, we formulate the SWE problem as a nonlinear least-squares optimization problem which can be solved very efficiently. Through numerous phantom studies, we show that PWE can handle complicated waveforms without prior filtering and is competitive with state-of-the-art that requires prior filtering based on the knowledge of propagation directions. △ Less

Submitted 7 December, 2020; originally announced December 2020.

Comments: 26 pages, 14 figures

Human-in-the-Loop Robot Planning with Non-Contextual Bandit Feedback

Authors: Yijie Zhou, Yan Zhang, Xusheng Luo, Michael M. Zavlanos

Abstract: In this paper, we consider a robot navigation problem in environments populated by humans. The goal is to determine collision-free and dynamically feasible trajectories that also maximize human satisfaction. This is because they may drive the robot close to humans that need help with their work or because they may keep the robot away from humans when it can interfere with human sight or work. In p… ▽ More In this paper, we consider a robot navigation problem in environments populated by humans. The goal is to determine collision-free and dynamically feasible trajectories that also maximize human satisfaction. This is because they may drive the robot close to humans that need help with their work or because they may keep the robot away from humans when it can interfere with human sight or work. In practice, human satisfaction is subjective and hard to describe mathematically. As a result, the planning problem we consider in this paper may lack important contextual information. To address this challenge, we propose a semi-supervised Bayesian Optimization (BO) method to design globally optimal robot trajectories using non-contextual bandit human feedback in the form of complaints or satisfaction ratings that express how satisfactory a trajectory is, without revealing the reason. Since trajectory planning is typically a high-dimensional optimization problem in the space of waypoints that define a trajectory, BO may require prohibitively many queries for human feedback to return a good solution. To this end, we use an autoencoder to reduce the high-dimensional problem space into a low dimensional latent space, which we update using human feedback. Moreover, we improve the exploration efficiency of BO by biasing the search for new trajectories towards dynamically feasible and collision-free trajectories obtained using off-the-shelf motion planners. We demonstrate the efficiency of our proposed trajectory planning method in a scenario with humans that have diversified and unknown demands. △ Less

Submitted 3 November, 2020; originally announced November 2020.

Boosting One-Point Derivative-Free Online Optimization via Residual Feedback

Authors: Yan Zhang, Yi Zhou, Kaiyi Ji, Michael M. Zavlanos

Abstract: Zeroth-order optimization (ZO) typically relies on two-point feedback to estimate the unknown gradient of the objective function. Nevertheless, two-point feedback can not be used for online optimization of time-varying objective functions, where only a single query of the function value is possible at each time step. In this work, we propose a new one-point feedback method for online optimization… ▽ More Zeroth-order optimization (ZO) typically relies on two-point feedback to estimate the unknown gradient of the objective function. Nevertheless, two-point feedback can not be used for online optimization of time-varying objective functions, where only a single query of the function value is possible at each time step. In this work, we propose a new one-point feedback method for online optimization that estimates the objective function gradient using the residual between two feedback points at consecutive time instants. Moreover, we develop regret bounds for ZO with residual feedback for both convex and nonconvex online optimization problems. Specifically, for both deterministic and stochastic problems and for both Lipschitz and smooth objective functions, we show that using residual feedback can produce gradient estimates with much smaller variance compared to conventional one-point feedback methods. As a result, our regret bounds are much tighter compared to existing regret bounds for ZO with conventional one-point feedback, which suggests that ZO with residual feedback can better track the optimizer of online optimization problems. Additionally, our regret bounds rely on weaker assumptions than those used in conventional one-point feedback methods. Numerical experiments show that ZO with residual feedback significantly outperforms existing one-point feedback methods also in practice. △ Less

Submitted 2 December, 2020; v1 submitted 14 October, 2020; originally announced October 2020.

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

Authors: Yan Zhang, Michael M. Zavlanos

Abstract: In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of all the other agents in the network. This can be impractical in large-scale problems, where sharing the state and action information with multi-hop neighbors may incur significant… ▽ More In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of all the other agents in the network. This can be impractical in large-scale problems, where sharing the state and action information with multi-hop neighbors may incur significant communication overhead. The advantage of the proposed zeroth-order policy optimization method is that it allows the agents to compute the local policy gradients needed to update their local policy functions using local estimates of the global accumulated rewards that depend on partial state and action information only and can be obtained using consensus. Specifically, to calculate the local policy gradients, we develop a new distributed zeroth-order policy gradient estimator that relies on one-point residual-feedback which, compared to existing zeroth-order estimators that also rely on one-point feedback, significantly reduces the variance of the policy gradient estimates improving, in this way, the learning performance. We show that the proposed distributed zeroth-order policy optimization method with constant stepsize converges to the neighborhood of a policy that is a stationary point of the global objective function. The size of this neighborhood depends on the agents' learning rates, the exploration parameters, and the number of consensus steps used to calculate the local estimates of the global accumulated rewards. Moreover, we provide numerical experiments that demonstrate that our new zeroth-order policy gradient estimator is more sample-efficient compared to other existing one-point estimators. △ Less

Submitted 16 June, 2023; v1 submitted 18 June, 2020; originally announced June 2020.

A New One-Point Residual-Feedback Oracle For Black-Box Learning and Control

Authors: Yan Zhang, Yi Zhou, Kaiyi Ji, Michael M. Zavlanos

Abstract: Zeroth-order optimization (ZO) algorithms have been recently used to solve black-box or simulation-based learning and control problems, where the gradient of the objective function cannot be easily computed but can be approximated using the objective function values. Many existing ZO algorithms adopt two-point feedback schemes due to their fast convergence rate compared to one-point feedback schem… ▽ More Zeroth-order optimization (ZO) algorithms have been recently used to solve black-box or simulation-based learning and control problems, where the gradient of the objective function cannot be easily computed but can be approximated using the objective function values. Many existing ZO algorithms adopt two-point feedback schemes due to their fast convergence rate compared to one-point feedback schemes. However, two-point schemes require two evaluations of the objective function at each iteration, which can be impractical in applications where the data are not all available a priori, e.g., in online optimization. In this paper, we propose a novel one-point feedback scheme that queries the function value once at each iteration and estimates the gradient using the residual between two consecutive points. When optimizing a deterministic Lipschitz function, we show that the query complexity of ZO with the proposed one-point residual feedback matches that of ZO with the existing two-point schemes. Moreover, the query complexity of the proposed algorithm can be improved when the objective function has Lipschitz gradient. Then, for stochastic bandit optimization problems where only noisy objective function values are given, we show that ZO with one-point residual feedback achieves the same convergence rate as that of two-point scheme with uncontrollable data samples. We demonstrate the effectiveness of the proposed one-point residual feedback via extensive numerical experiments. △ Less

Submitted 7 September, 2021; v1 submitted 18 June, 2020; originally announced June 2020.

Transfer Reinforcement Learning under Unobserved Contextual Information

Authors: Yan Zhang, Michael M. Zavlanos

Abstract: In this paper, we study a transfer reinforcement learning problem where the state transitions and rewards are affected by the environmental context. Specifically, we consider a demonstrator agent that has access to a context-aware policy and can generate transition and reward data based on that policy. These data constitute the experience of the demonstrator. Then, the goal is to transfer this exp… ▽ More In this paper, we study a transfer reinforcement learning problem where the state transitions and rewards are affected by the environmental context. Specifically, we consider a demonstrator agent that has access to a context-aware policy and can generate transition and reward data based on that policy. These data constitute the experience of the demonstrator. Then, the goal is to transfer this experience, excluding the underlying contextual information, to a learner agent that does not have access to the environmental context, so that they can learn a control policy using fewer samples. It is well known that, disregarding the causal effect of the contextual information, can introduce bias in the transition and reward models estimated by the learner, resulting in a learned suboptimal policy. To address this challenge, in this paper, we develop a method to obtain causal bounds on the transition and reward functions using the demonstrator's data, which we then use to obtain causal bounds on the value functions. Using these value function bounds, we propose new Q learning and UCB-Q learning algorithms that converge to the true value function without bias. We provide numerical experiments for robot motion planning problems that validate the proposed value function bounds and demonstrate that the proposed algorithms can effectively make use of the data from the demonstrator to accelerate the learning process of the learner. △ Less

Submitted 9 March, 2020; originally announced March 2020.

Socially-Aware Robot Planning via Bandit Human Feedback

Authors: Xusheng Luo, Yan Zhang, Michael M. Zavlanos

Abstract: In this paper, we consider the problem of designing collision-free, dynamically feasible, and socially-aware trajectories for robots operating in environments populated by humans. We define trajectories to be social-aware if they do not interfere with humans in any way that causes discomfort. In this paper, discomfort is defined broadly and, depending on specific individuals, it can result from th… ▽ More In this paper, we consider the problem of designing collision-free, dynamically feasible, and socially-aware trajectories for robots operating in environments populated by humans. We define trajectories to be social-aware if they do not interfere with humans in any way that causes discomfort. In this paper, discomfort is defined broadly and, depending on specific individuals, it can result from the robot being too close to a human or from interfering with human sight or tasks. Moreover, we assume that human feedback is a bandit feedback indicating a complaint or no complaint on the part of the robot trajectory that interferes with the humans, and it does not reveal any contextual information about the locations of the humans or the reason for a complaint. Finally, we assume that humans can move in the obstacle-free space and, as a result, human utility can change. We formulate this planning problem as an online optimization problem that minimizes the social value of the time-varying robot trajectory, defined by the total number of incurred human complaints. As the human utility is unknown, we employ zeroth order, or derivative-free, optimization methods to solve this problem, which we combine with off-the-shelf motion planners to satisfy the dynamic feasibility and collision-free specifications of the resulting trajectories. To the best of our knowledge, this is a new framework for socially-aware robot planning that is not restricted to avoiding collisions with humans but, instead, focuses on increasing the social value of the robot trajectories using only bandit human feedback. △ Less

Submitted 1 March, 2020; originally announced March 2020.

Comments: 10 pages, 3 figures

VarNet: Variational Neural Networks for the Solution of Partial Differential Equations

Authors: Reza Khodayi-Mehr, Michael M. Zavlanos

Abstract: In this paper we propose a new model-based unsupervised learning method, called VarNet, for the solution of partial differential equations (PDEs) using deep neural networks (NNs). Particularly, we propose a novel loss function that relies on the variational (integral) form of PDEs as apposed to their differential form which is commonly used in the literature. Our loss function is discretization-fr… ▽ More In this paper we propose a new model-based unsupervised learning method, called VarNet, for the solution of partial differential equations (PDEs) using deep neural networks (NNs). Particularly, we propose a novel loss function that relies on the variational (integral) form of PDEs as apposed to their differential form which is commonly used in the literature. Our loss function is discretization-free, highly parallelizable, and more effective in capturing the solution of PDEs since it employs lower-order derivatives and trains over measure non-zero regions of space-time. Given this loss function, we also propose an approach to optimally select the space-time samples, used to train the NN, that is based on the feedback provided from the PDE residual. The models obtained using VarNet are smooth and do not require interpolation. They are also easily differentiable and can directly be used for control and optimization of PDEs. Finally, VarNet can straight-forwardly incorporate parametric PDE models making it a natural tool for model order reduction (MOR) of PDEs. We demonstrate the performance of our method through extensive numerical experiments for the advection-diffusion PDE as an important case-study. △ Less

Submitted 16 December, 2019; originally announced December 2019.

Comments: 15 pages, 27 figures

A Distributed Online Convex Optimization Algorithm with Improved Dynamic Regret

Authors: Yan Zhang, Robert J. Ravier, Michael M. Zavlanos, Vahid Tarokh

Abstract: In this paper, we consider the problem of distributed online convex optimization, where a network of local agents aim to jointly optimize a convex function over a period of multiple time steps. The agents do not have any information about the future. Existing algorithms have established dynamic regret bounds that have explicit dependence on the number of time steps. In this work, we show that we c… ▽ More In this paper, we consider the problem of distributed online convex optimization, where a network of local agents aim to jointly optimize a convex function over a period of multiple time steps. The agents do not have any information about the future. Existing algorithms have established dynamic regret bounds that have explicit dependence on the number of time steps. In this work, we show that we can remove this dependence assuming that the local objective functions are strongly convex. More precisely, we propose a gradient tracking algorithm where agents jointly communicate and descend based on corrected gradient steps. We verify our theoretical results through numerical experiments. △ Less

Submitted 12 November, 2019; originally announced November 2019.

Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning

Authors: Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, Miroslav Pajic

Abstract: We present a reinforcement learning (RL) framework to synthesize a control policy from a given linear temporal logic (LTL) specification in an unknown stochastic environment that can be modeled as a Markov Decision Process (MDP). Specifically, we learn a policy that maximizes the probability of satisfying the LTL formula without learning the transition probabilities. We introduce a novel rewarding… ▽ More We present a reinforcement learning (RL) framework to synthesize a control policy from a given linear temporal logic (LTL) specification in an unknown stochastic environment that can be modeled as a Markov Decision Process (MDP). Specifically, we learn a policy that maximizes the probability of satisfying the LTL formula without learning the transition probabilities. We introduce a novel rewarding and path-dependent discounting mechanism based on the LTL formula such that (i) an optimal policy maximizing the total discounted reward effectively maximizes the probabilities of satisfying LTL objectives, and (ii) a model-free RL algorithm using these rewards and discount factors is guaranteed to converge to such policy. Finally, we illustrate the applicability of our RL-based synthesis approach on two motion planning case studies. △ Less

Submitted 5 March, 2020; v1 submitted 16 September, 2019; originally announced September 2019.

An Abstraction-Free Method for Multi-Robot Temporal Logic Optimal Control Synthesis

Authors: Xusheng Luo, Yiannis Kantaros, Michael M. Zavlanos

Abstract: The majority of existing Linear Temporal Logic (LTL) planning methods rely on the construction of a discrete product automaton, that combines a discrete abstraction of robot mobility and a B$\ddot{\text{u}}$chi automaton that captures the LTL specification. Representing this product automaton as a graph and using graph search techniques, optimal plans that satisfy the LTL task can be synthesized.… ▽ More The majority of existing Linear Temporal Logic (LTL) planning methods rely on the construction of a discrete product automaton, that combines a discrete abstraction of robot mobility and a B$\ddot{\text{u}}$chi automaton that captures the LTL specification. Representing this product automaton as a graph and using graph search techniques, optimal plans that satisfy the LTL task can be synthesized. However, constructing expressive discrete abstractions makes the synthesis problem computationally intractable. In this paper, we propose a new sampling-based LTL planning algorithm that does not require any discrete abstraction of robot mobility. Instead, it incrementally builds trees that explore the product state-space, until a maximum number of iterations is reached or a feasible plan is found. The use of trees makes data storage and graph search tractable, which significantly increases the scalability of our algorithm. To accelerate the construction of feasible plans, we introduce bias in the sampling process which is guided by transitions in the B$\ddot{\text{u}}$chi automaton that belong to the shortest path to the accepting states. We show that our planning algorithm, with and without bias, is probabilistically complete and asymptotically optimal. Finally, we present numerical experiments showing that our method outperforms relevant temporal logic planning methods. △ Less

Submitted 23 March, 2021; v1 submitted 1 September, 2019; originally announced September 2019.

Comments: 21 pages, 10 figures

An Optimal Graph-Search Method for Secure State Estimation

Authors: Xusheng Luo, Miroslav Pajic, Michael M. Zavlanos

Abstract: The growing complexity of modern Cyber-Physical Systems (CPS) and the frequent communication between their components make them vulnerable to malicious attacks. As a result, secure state estimation is a critical requirement for the control of these systems. Many existing secure state estimation methods suffer from combinatorial complexity which grows with the number of states and sensors in the sy… ▽ More The growing complexity of modern Cyber-Physical Systems (CPS) and the frequent communication between their components make them vulnerable to malicious attacks. As a result, secure state estimation is a critical requirement for the control of these systems. Many existing secure state estimation methods suffer from combinatorial complexity which grows with the number of states and sensors in the system. This complexity can be mitigated using optimization-based methods that relax the original state estimation problem, although at the cost of optimality as these methods often identify attack-free sensors as attacked. In this paper, we propose a new optimal graph-search algorithm to correctly identify malicious attacks and to securely estimate the states even in large-scale CPS modeled as linear time-invariant systems. The graph consists of layers, each one containing two nodes capturing a truth assignment of any given sensor, and directed edges connecting adjacent layers only. Then, our algorithm searches the layers of this graph incrementally, favoring directions at higher layers with more attack-free assignments, while actively managing a repository of nodes to be expanded at later iterations. The proposed search bias and the ability to revisit nodes in the repository and self-correct, allow our graph-search algorithm to reach the optimal assignment faster and tackle larger problems. We show that our algorithm is complete and optimal provided that process and measurement noises do not dominate the attack signal. Moreover, we provide numerical simulations that demonstrate the ability of our algorithm to correctly identify attacked sensors and securely reconstruct the state. Our simulations show that our method outperforms existing algorithms both in terms of optimality and execution time. △ Less

Submitted 7 October, 2020; v1 submitted 25 March, 2019; originally announced March 2019.

Comments: 16 pages, 10 figures

Distributed off-Policy Actor-Critic Reinforcement Learning with Policy Consensus

Authors: Yan Zhang, Michael M. Zavlanos

Abstract: In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional consensus step to let all the agents asymptotically achieve agreement on the gl… ▽ More In this paper, we propose a distributed off-policy actor critic method to solve multi-agent reinforcement learning problems. Specifically, we assume that all agents keep local estimates of the global optimal policy parameter and update their local value function estimates independently. Then, we introduce an additional consensus step to let all the agents asymptotically achieve agreement on the global optimal policy function. The convergence analysis of the proposed algorithm is provided and the effectiveness of the proposed algorithm is validated using a distributed resource allocation example. Compared to relevant distributed actor critic methods, here the agents do not share information about their local tasks, but instead they coordinate to estimate the global policy function. △ Less

Submitted 21 March, 2019; originally announced March 2019.

Deep Learning for Robotic Mass Transport Cloaking

Authors: Reza Khodayi-mehr, Michael M. Zavlanos

Abstract: We consider the problem of mass transport cloaking using mobile robots. The robots move along a predefined curve that encloses a safe zone and carry sources that collectively counteract a chemical agent released in the environment. The goal is to steer the mass flux around a desired region so that it remains unaffected by the external concentration. We formulate the problem of controlling the robo… ▽ More We consider the problem of mass transport cloaking using mobile robots. The robots move along a predefined curve that encloses a safe zone and carry sources that collectively counteract a chemical agent released in the environment. The goal is to steer the mass flux around a desired region so that it remains unaffected by the external concentration. We formulate the problem of controlling the robot positions and release rates as a PDE-constrained optimization, where the propagation of the chemical is modeled by the advection-diffusion (AD) PDE. We use a neural network (NN) to approximate the solution of the PDE. Particularly, we propose a novel loss function for the NN that utilizes the variational form of the AD-PDE and allows us to reformulate the planning problem as an unsupervised model-based learning problem. Our loss function is discretization-free and highly parallelizable. Unlike passive cloaking methods that use metamaterials to steer the mass flux, our method is the first to use mobile robots to actively control the concentration levels and create safe zones independent of environmental conditions. We demonstrate the performance of our method in simulations. △ Less

Submitted 9 March, 2020; v1 submitted 10 December, 2018; originally announced December 2018.

Comments: 8 pages, 11 figures, 1 table

Physics-Based Learning for Robotic Environmental Sensing

Authors: Reza Khodayi-mehr, Michael M. Zavlanos

Abstract: We propose a physics-based method to learn environmental fields (EFs) using a mobile robot. Common purely data-driven methods require prohibitively many measurements to accurately learn such complex EFs. Alternatively, physics-based models provide global knowledge of EFs but require experimental validation, depend on uncertain parameters, and are intractable for mobile robots. To address these cha… ▽ More We propose a physics-based method to learn environmental fields (EFs) using a mobile robot. Common purely data-driven methods require prohibitively many measurements to accurately learn such complex EFs. Alternatively, physics-based models provide global knowledge of EFs but require experimental validation, depend on uncertain parameters, and are intractable for mobile robots. To address these challenges, we propose a Bayesian framework to select the most likely physics-based models of EFs in real-time, from a pool of numerical solutions generated offline as a function of the uncertain parameters. Specifically, we focus on turbulent flow fields and utilize Gaussian processes (GPs) to construct statistical models for them, using the pool of numerical solutions to inform their prior mean. To incorporate flow measurements into these GPs, we control a custom-built mobile robot through a sequence of waypoints that maximize the information content of the measurements. We experimentally demonstrate that our proposed framework constructs a posterior distribution of the flow field that better approximates the real flow compared to the prior numerical solutions and purely data-driven methods. △ Less

Submitted 13 January, 2021; v1 submitted 10 December, 2018; originally announced December 2018.

Comments: 20 pages, 26 figures

STyLuS*: A Temporal Logic Optimal Control Synthesis Algorithm for Large-Scale Multi-Robot Systems

Authors: Yiannis Kantaros, Michael M. Zavlanos

Abstract: This paper proposes a new highly scalable and asymptotically optimal control synthesis algorithm from linear temporal logic specifications, called $\text{STyLuS}^{*}$ for large-Scale optimal Temporal Logic Synthesis, that is designed to solve complex temporal planning problems in large-scale multi-robot systems. Existing planning approaches with temporal logic specifications rely on graph search t… ▽ More This paper proposes a new highly scalable and asymptotically optimal control synthesis algorithm from linear temporal logic specifications, called $\text{STyLuS}^{*}$ for large-Scale optimal Temporal Logic Synthesis, that is designed to solve complex temporal planning problems in large-scale multi-robot systems. Existing planning approaches with temporal logic specifications rely on graph search techniques applied to a product automaton constructed among the robots. In our previous work, we have proposed a more tractable sampling-based algorithm that builds incrementally trees that approximate the state-space and transitions of the synchronous product automaton and does not require sophisticated graph search techniques. Here, we extend our previous work by introducing bias in the sampling process which is guided by transitions in the B$\ddot{\text{u}}$chi automaton that belong to the shortest path to the accepting states. This allows us to synthesize optimal motion plans from product automata with hundreds of orders of magnitude more states than those that existing optimal control synthesis methods or off-the-shelf model checkers can manipulate. We show that $\text{STyLuS}^{*}$ is probabilistically complete and asymptotically optimal and has exponential convergence rate. This is the first time that convergence rate results are provided for sampling-based optimal control synthesis methods. We provide simulation results that show that $\text{STyLuS}^{*}$ can synthesize optimal motion plans for very large multi-robot systems which is impossible using state-of-the-art methods. △ Less

Submitted 7 April, 2020; v1 submitted 21 September, 2018; originally announced September 2018.

Distributed State Estimation Using Intermittently Connected Robot Networks

Authors: Reza Khodayi-mehr, Yiannis Kantaros, Michael M. Zavlanos

Abstract: This paper considers the problem of distributed state estimation using multi-robot systems. The robots have limited communication capabilities and, therefore, communicate their measurements intermittently only when they are physically close to each other. To decrease the distance that the robots need to travel only to communicate, we divide them into small teams that can communicate at different l… ▽ More This paper considers the problem of distributed state estimation using multi-robot systems. The robots have limited communication capabilities and, therefore, communicate their measurements intermittently only when they are physically close to each other. To decrease the distance that the robots need to travel only to communicate, we divide them into small teams that can communicate at different locations to share information and update their beliefs. Then, we propose a new distributed scheme that combines (i) communication schedules that ensure that the network is intermittently connected, and (ii) sampling-based motion planning for the robots in every team with the objective to collect optimal measurements and decide a location for those robots to communicate. To the best of our knowledge, this is the first distributed state estimation framework that relaxes all network connectivity assumptions, and controls intermittent communication events so that the estimation uncertainty is minimized. We present simulation results that demonstrate significant improvement in estimation accuracy compared to methods that maintain an end-to-end connected network for all time. △ Less

Submitted 18 September, 2018; v1 submitted 3 May, 2018; originally announced May 2018.

Control of Magnetic Microrobot Teams for Temporal Micromanipulation Tasks

Authors: Yiannis Kantaros, Benjamin Johnson, Sagar Chowdhury, David J. Cappelleri, Michael M. Zavlanos

Abstract: In this paper, we present a control framework that allows magnetic microrobot teams to accomplish complex micromanipulation tasks captured by global Linear Temporal Logic (LTL) formulas. To address this problem, we propose an optimal control synthesis method that constructs discrete plans for the robots that satisfy both the assigned tasks as well as proximity constraints between the robots due to… ▽ More In this paper, we present a control framework that allows magnetic microrobot teams to accomplish complex micromanipulation tasks captured by global Linear Temporal Logic (LTL) formulas. To address this problem, we propose an optimal control synthesis method that constructs discrete plans for the robots that satisfy both the assigned tasks as well as proximity constraints between the robots due to the physics of the problem. Our proposed algorithm relies on an existing optimal control synthesis approach combined with a novel sampling-based technique to reduce the state-space of the product automaton that is associated with the LTL specifications. The synthesized discrete plans are executed by the microrobots independently using local magnetic fields. Simulation studies show that the proposed algorithm can address large-scale planning problems that cannot be solved using existing optimal control synthesis approaches. Moreover, we present experimental results that also illustrate the potential of our method in practice. To the best of our knowledge, this is the first control framework that allows independent control of teams of magnetic microrobots for temporal micromanipulation tasks. △ Less

Submitted 23 September, 2018; v1 submitted 28 April, 2018; originally announced April 2018.

Probabilistic Motion Planning under Temporal Tasks and Soft Constraints

Authors: Meng Guo, Michael M. Zavlanos

Abstract: This paper studies motion planning of a mobile robot under uncertainty. The control objective is to synthesize a {finite-memory} control policy, such that a high-level task specified as a Linear Temporal Logic (LTL) formula is satisfied with a desired high probability. Uncertainty is considered in the workspace properties, robot actions, and task outcomes, giving rise to a Markov Decision Process… ▽ More This paper studies motion planning of a mobile robot under uncertainty. The control objective is to synthesize a {finite-memory} control policy, such that a high-level task specified as a Linear Temporal Logic (LTL) formula is satisfied with a desired high probability. Uncertainty is considered in the workspace properties, robot actions, and task outcomes, giving rise to a Markov Decision Process (MDP) that models the proposed system. Different from most existing methods, we consider cost optimization both in the prefix and suffix of the system trajectory. We also analyze the potential trade-off between reducing the mean total cost and maximizing the probability that the task is satisfied. The proposed solution is based on formulating two coupled Linear Programs, for the prefix and suffix, respectively, and combining them into a multi-objective optimization problem, which provides provable guarantees on the probabilistic satisfiability and the total cost optimality. We show that our method outperforms relevant approaches that employ Round-Robin policies in the trajectory suffix. Furthermore, we propose a new control synthesis algorithm to minimize the frequency of reaching a bad state when the probability of satisfying the tasks is zero, in which case most existing methods return no solution. We validate the above schemes via both numerical simulations and experimental studies. △ Less

Submitted 23 October, 2017; v1 submitted 16 June, 2017; originally announced June 2017.

Comments: 16 pages, 15 figures

Sampling-Based Optimal Control Synthesis for Multi-Robot Systems under Global Temporal Tasks

Authors: Yiannis Kantaros, Michael M. Zavlanos

Abstract: This paper proposes a new optimal control synthesis algorithm for multi-robot systems under global temporal logic tasks. Existing planning approaches under global temporal goals rely on graph search techniques applied to a product automaton constructed among the robots. In this paper, we propose a new sampling-based algorithm that builds incrementally trees that approximate the state-space and tra… ▽ More This paper proposes a new optimal control synthesis algorithm for multi-robot systems under global temporal logic tasks. Existing planning approaches under global temporal goals rely on graph search techniques applied to a product automaton constructed among the robots. In this paper, we propose a new sampling-based algorithm that builds incrementally trees that approximate the state-space and transitions of the synchronous product automaton. By approximating the product automaton by a tree rather than representing it explicitly, we require much fewer memory resources to store it and motion plans can be found by tracing sequences of parent nodes without the need for sophisticated graph search methods. This significantly increases the scalability of our algorithm compared to existing optimal control synthesis methods. We also show that the proposed algorithm is probabilistically complete and asymptotically optimal. Finally, we present numerical experiments showing that our approach can synthesize optimal plans from product automata with billions of states, which is not possible using standard optimal control synthesis algorithms or off-the-shelf model checkers. △ Less

Submitted 19 June, 2018; v1 submitted 13 June, 2017; originally announced June 2017.

Multi-Robot Data Gathering Under Buffer Constraints and Intermittent Communication

Authors: Meng Guo, Michael M. Zavlanos

Abstract: We consider a team of heterogeneous robots which are deployed within a common workspace to gather different types of data. The robots have different roles due to different capabilities: some gather data from the workspace (source robots) and others receive data from source robots and upload them to a data center (relay robots). The data-gathering tasks are specified locally to each source robot as… ▽ More We consider a team of heterogeneous robots which are deployed within a common workspace to gather different types of data. The robots have different roles due to different capabilities: some gather data from the workspace (source robots) and others receive data from source robots and upload them to a data center (relay robots). The data-gathering tasks are specified locally to each source robot as high-level Linear Temporal Logic (LTL) formulas, that capture the different types of data that need to be gathered at different regions of interest. All robots have a limited buffer to store the data. Thus the data gathered by source robots should be transferred to relay robots before their buffers overflow, respecting at the same time limited communication range for all robots. The main contribution of this work is a distributed motion coordination and intermittent communication scheme that guarantees the satisfaction of all local tasks, while obeying the above constraints. The robot motion and inter-robot communication are closely coupled and coordinated during run time by scheduling intermittent meeting events to facilitate the local plan execution. We present both numerical simulations and experimental studies to demonstrate the advantages of the proposed method over existing approaches that predominantly require all-time network connectivity. △ Less

Submitted 30 October, 2017; v1 submitted 7 June, 2017; originally announced June 2017.

Comments: 14 Pages, 16 figures

Controlling a Robotic Stereo Camera Under Image Quantization Noise

Authors: Charles Freundlich, Yan Zhang, Alex Zihao Zhu, Philippos Mordohai, Michael M. Zavlanos

Abstract: In this paper, we address the problem of controlling a mobile stereo camera under image quantization noise. Assuming that a pair of images of a set of targets is available, the camera moves through a sequence of Next-Best-Views (NBVs), i.e., a sequence of views that minimize the trace of the targets' cumulative state covariance, constructed using a realistic model of the stereo rig that captures i… ▽ More In this paper, we address the problem of controlling a mobile stereo camera under image quantization noise. Assuming that a pair of images of a set of targets is available, the camera moves through a sequence of Next-Best-Views (NBVs), i.e., a sequence of views that minimize the trace of the targets' cumulative state covariance, constructed using a realistic model of the stereo rig that captures image quantization noise and a Kalman Filter (KF) that fuses the observation history with new information. The proposed algorithm decomposes control into two stages: first the NBV is computed in the camera relative coordinates, and then the camera moves to realize this view in the fixed global coordinate frame. This decomposition allows the camera to drive to a new pose that effectively realizes the NBV in camera coordinates while satisfying Field-of-View constraints in global coordinates, a task that is particularly challenging using complex sensing models. We provide simulations and real experiments that illustrate the ability of the proposed mobile camera system to accurately localize sets of targets. We also propose a novel data-driven technique to characterize unmodeled uncertainty, such as calibration errors, at the pixel level and show that this method ensures stability of the KF. △ Less

Submitted 13 January, 2018; v1 submitted 6 June, 2017; originally announced June 2017.

Comments: International Journal of Robotics Research, October 2017

Distributed Hierarchical Control for State Estimation With Robotic Sensor Networks

Authors: Charles Freundlich, Yan Zhang, Michael M. Zavlanos

Abstract: This paper addresses active state estimation with a team of robotic sensors. The states to be estimated are represented by spatially distributed, uncorrelated, stationary vectors. Given a prior belief on the geographic locations of the states, we cluster the states in moderately sized groups and propose a new hierarchical Dynamic Programming (DP) framework to compute optimal sensing policies for e… ▽ More This paper addresses active state estimation with a team of robotic sensors. The states to be estimated are represented by spatially distributed, uncorrelated, stationary vectors. Given a prior belief on the geographic locations of the states, we cluster the states in moderately sized groups and propose a new hierarchical Dynamic Programming (DP) framework to compute optimal sensing policies for each cluster that mitigates the computational cost of planning optimal policies in the combined belief space. Then, we develop a decentralized assignment algorithm that dynamically allocates clusters to robots based on the pre-computed optimal policies at each cluster. The integrated distributed state estimation framework is optimal at the cluster level but also scales very well to large numbers of states and robot sensors. We demonstrate efficiency of the proposed method in both simulations and real-world experiments using stereoscopic vision sensors. △ Less

Submitted 13 January, 2018; v1 submitted 6 June, 2017; originally announced June 2017.

Comments: IEEE Transactions on Control of Network Systems, December 2017

Model-Based Active Source Identification in Complex Environments

Authors: Reza Khodayi-mehr, Wilkins Aquino, Michael M. Zavlanos

Abstract: We consider the problem of Active Source Identification (ASI) in steady-state Advection-Diffusion (AD) transport systems. Unlike existing bio-inspired heuristic methods, we propose a model-based method that employs the AD-PDE to capture the transport phenomenon. Specifically, we formulate the Source Identification (SI) problem as a PDE-constrained optimization problem in function spaces. To obtain… ▽ More We consider the problem of Active Source Identification (ASI) in steady-state Advection-Diffusion (AD) transport systems. Unlike existing bio-inspired heuristic methods, we propose a model-based method that employs the AD-PDE to capture the transport phenomenon. Specifically, we formulate the Source Identification (SI) problem as a PDE-constrained optimization problem in function spaces. To obtain a tractable solution, we reduce the dimension of the concentration field using Proper Orthogonal Decomposition and approximate the unknown source field using nonlinear basis functions, drastically decreasing the number of unknowns. Moreover, to collect the concentration measurements, we control a robot sensor through a sequence of waypoints that maximize the smallest eigenvalue of the Fisher Information matrix of the unknown source parameters. Specifically, after every new measurement, a SI problem is solved to obtain a source estimate that is used to determine the next waypoint. We show that our algorithm can efficiently identify sources in complex AD systems and non-convex domains, in simulation and experimentally. This is the first time that PDEs are used for robotic SI in practice. △ Less

Submitted 31 October, 2018; v1 submitted 6 June, 2017; originally announced June 2017.

Comments: 19 pages, 10 figures, journal

Distributed Active State Estimation with User-Specified Accuracy

Authors: Charles Freundlich, Soomin Lee, Michael M. Zavlanos

Abstract: In this paper, we address the problem of controlling a network of mobile sensors so that a set of hidden states are estimated up to a user-specified accuracy. The sensors take measurements and fuse them online using an Information Consensus Filter (ICF). At the same time, the local estimates guide the sensors to their next best configuration. This leads to an LMI-constrained optimization problem t… ▽ More In this paper, we address the problem of controlling a network of mobile sensors so that a set of hidden states are estimated up to a user-specified accuracy. The sensors take measurements and fuse them online using an Information Consensus Filter (ICF). At the same time, the local estimates guide the sensors to their next best configuration. This leads to an LMI-constrained optimization problem that we solve by means of a new distributed random approximate projections method. The new method is robust to the state disagreement errors that exist among the robots as the ICF fuses the collected measurements. Assuming that the noise corrupting the measurements is zero-mean and Gaussian and that the robots are self localized in the environment, the integrated system converges to the next best positions from where new observations will be taken. This process is repeated with the robots taking a sequence of observations until the hidden states are estimated up to the desired user-specified accuracy. We present simulations of sparse landmark localization, where the robotic team achieves the desired estimation tolerances while exhibiting interesting emergent behavior. △ Less

Submitted 13 January, 2018; v1 submitted 5 June, 2017; originally announced June 2017.

Comments: IEEE Transactions on Automatic Control, June 2017

Temporal Logic Task Planning and Intermittent Connectivity Control of Mobile Robot Networks

Authors: Yiannis Kantaros, Meng Guo, Michael M. Zavlanos

Abstract: In this paper, we develop a distributed intermittent communication and task planning framework for mobile robot teams. The goal of the robots is to accomplish complex tasks, captured by local Linear Temporal Logic formulas, and share the collected information with all other robots and possibly also with a user. Specifically, we consider situations where the robot communication capabilities are not… ▽ More In this paper, we develop a distributed intermittent communication and task planning framework for mobile robot teams. The goal of the robots is to accomplish complex tasks, captured by local Linear Temporal Logic formulas, and share the collected information with all other robots and possibly also with a user. Specifically, we consider situations where the robot communication capabilities are not sufficient to form reliable and connected networks while the robots move to accomplish their tasks. In this case, intermittent communication protocols are necessary that allow the robots to temporarily disconnect from the network in order to accomplish their tasks free of communication constraints. We assume that the robots can only communicate with each other when they meet at common locations in space. Our distributed control framework jointly determines local plans that allow all robots fulfill their assigned temporal tasks, sequences of communication events that guarantee information exchange infinitely often, and optimal communication locations that minimize a desired distance metric. Simulation results verify the efficacy of the proposed controllers. △ Less

Submitted 24 June, 2018; v1 submitted 2 June, 2017; originally announced June 2017.

Simultaneous Intermittent Communication Control and Path Optimization in Networks of Mobile Robots

Authors: Yiannis Kantaros, Michael M. Zavlanos

Abstract: In this paper, we propose an intermittent communication framework for mobile robot networks. Specifically, we consider robots that move along the edges of a connected mobility graph and communicate only when they meet at the nodes of that graph giving rise to a dynamic communication network. Our proposed distributed controllers ensure intermittent connectivity of the network and path optimization,… ▽ More In this paper, we propose an intermittent communication framework for mobile robot networks. Specifically, we consider robots that move along the edges of a connected mobility graph and communicate only when they meet at the nodes of that graph giving rise to a dynamic communication network. Our proposed distributed controllers ensure intermittent connectivity of the network and path optimization, simultaneously. We show that the intermittent connectivity requirement can be encapsulated by a global Linear Temporal Logic (LTL) formula. Then we approximately decompose it into local LTL expressions which are then assigned to the robots. To avoid conflicting robot behaviors that can occur due to this approximate decomposition, we develop a distributed conflict resolution scheme that generates non-conflicting discrete motion plans for every robot, based on the assigned local LTL expressions, whose composition satisfies the global LTL formula. By appropriately introducing delays in the execution of the generated motion plans we also show that the proposed controllers can be executed asynchronously. △ Less

Submitted 22 September, 2016; originally announced September 2016.

Mobile Jammers for Secrecy Rate Maximization in Cooperative Networks

Authors: Dionysios S. Kalogerias, Nikolaos Chatzipanagiotis, Michael M. Zavlanos, Athina P. Petropulu

Abstract: We consider a source (Alice) trying to communicate with a destination (Bob), in a way that an unauthorized node (Eve) cannot infer, based on her observations, the information that is being transmitted. The communication is assisted by multiple multi-antenna cooperating nodes (helpers) who have the ability to move. While Alice transmits, the helpers transmit noise that is designed to affect the ent… ▽ More We consider a source (Alice) trying to communicate with a destination (Bob), in a way that an unauthorized node (Eve) cannot infer, based on her observations, the information that is being transmitted. The communication is assisted by multiple multi-antenna cooperating nodes (helpers) who have the ability to move. While Alice transmits, the helpers transmit noise that is designed to affect the entire space except Bob. We consider the problem of selecting the helper weights and positions that maximize the system secrecy rate. It turns out that this optimization problem can be efficiently solved, leading to a novel decentralized helper motion control scheme. Simulations indicate that introducing helper mobility leads to considerable savings in terms of helper transmit power, as well as total number of helpers required for secrecy communications. △ Less

Submitted 3 March, 2013; originally announced March 2013.

Comments: ICASSP 2013

Spectral Design of Dynamic Networks via Local Operations

Authors: Victor M. Preciado, Michael M. Zavlanos, Ali Jadbabaie

Abstract: Motivated by the relationship between the eigenvalue spectrum of the Laplacian matrix of a network and the behavior of dynamical processes evolving in it, we propose a distributed iterative algorithm in which a group of $n$ autonomous agents self-organize the structure of their communication network in order to control the network's eigenvalue spectrum. In our algorithm, we assume that each agent… ▽ More Motivated by the relationship between the eigenvalue spectrum of the Laplacian matrix of a network and the behavior of dynamical processes evolving in it, we propose a distributed iterative algorithm in which a group of $n$ autonomous agents self-organize the structure of their communication network in order to control the network's eigenvalue spectrum. In our algorithm, we assume that each agent has access only to a local (myopic) view of the network around it. In each iteration, agents in the network peform a decentralized decision process to determine the edge addition/deletion that minimizes a distance function defined in the space of eigenvalue spectra. This spectral distance presents interesting theoretical properties that allow an efficient distributed implementation of the decision process. Our iterative algorithm is stable by construction, i.e., locally optimizes the network's eigenvalue spectrum, and is shown to perform extremely well in practice. We illustrate our results with nontrivial simulations in which we design networks matching the spectral properties of complex networks, such as small-world and power-law networks. △ Less

Submitted 7 September, 2012; v1 submitted 4 December, 2011; originally announced December 2011.

Comments: Submitted for publication

Spectral Control of Mobile Robot Networks

Authors: Michael M. Zavlanos, Victor M. Preciado, Ali Jadbabaie

Abstract: The eigenvalue spectrum of the adjacency matrix of a network is closely related to the behavior of many dynamical processes run over the network. In the field of robotics, this spectrum has important implications in many problems that require some form of distributed coordination within a team of robots. In this paper, we propose a continuous-time control scheme that modifies the structure of a po… ▽ More The eigenvalue spectrum of the adjacency matrix of a network is closely related to the behavior of many dynamical processes run over the network. In the field of robotics, this spectrum has important implications in many problems that require some form of distributed coordination within a team of robots. In this paper, we propose a continuous-time control scheme that modifies the structure of a position-dependent network of mobile robots so that it achieves a desired set of adjacency eigenvalues. For this, we employ a novel abstraction of the eigenvalue spectrum by means of the adjacency matrix spectral moments. Since the eigenvalue spectrum is uniquely determined by its spectral moments, this abstraction provides a way to indirectly control the eigenvalues of the network. Our construction is based on artificial potentials that capture the distance of the network's spectral moments to their desired values. Minimization of these potentials is via a gradient descent closed-loop system that, under certain convexity assumptions, ensures convergence of the network topology to one with the desired set of moments and, therefore, eigenvalues. We illustrate our approach in nontrivial computer simulations. △ Less

Submitted 30 September, 2010; originally announced October 2010.

Comments: http://alum.mit.edu/www/vmp

Distributed Control of the Laplacian Spectral Moments of a Network

Authors: Victor M. Preciado, Michael M. Zavlanos, Ali Jadbabaie, George J. Pappas

Abstract: It is well-known that the eigenvalue spectrum of the Laplacian matrix of a network contains valuable information about the network structure and the behavior of many dynamical processes run on it. In this paper, we propose a fully decentralized algorithm that iteratively modifies the structure of a network of agents in order to control the moments of the Laplacian eigenvalue spectrum. Although t… ▽ More It is well-known that the eigenvalue spectrum of the Laplacian matrix of a network contains valuable information about the network structure and the behavior of many dynamical processes run on it. In this paper, we propose a fully decentralized algorithm that iteratively modifies the structure of a network of agents in order to control the moments of the Laplacian eigenvalue spectrum. Although the individual agents have knowledge of their local network structure only (i.e., myopic information), they are collectively able to aggregate this local information and decide on what links are most beneficial to be added or removed at each time step. Our approach relies on gossip algorithms to distributively compute the spectral moments of the Laplacian matrix, as well as ensure network connectivity in the presence of link deletions. We illustrate our approach in nontrivial computer simulations and show that a good final approximation of the spectral moments of the target Laplacian matrix is achieved for many cases of interest. △ Less

Submitted 22 January, 2010; originally announced January 2010.

Comments: 8 pages, 7 figures