Showing 1–9 of 9 results for author: Ramadan, S

Extended Kalman filter -- Koopman operator for tractable stochastic optimal control

Authors: Mohammad S. Ramadan, Mihai Anitescu

Abstract: The theory of dual control was introduced more than seven decades ago. Although it has provided rich insights to the fields of control, estimation, and system identification, dual control is generally computationally prohibitive. In recent years, however, the use of Koopman operator theory for control applications has been emerging. This paper presents a new reformulation of the stochastic optimal… ▽ More The theory of dual control was introduced more than seven decades ago. Although it has provided rich insights to the fields of control, estimation, and system identification, dual control is generally computationally prohibitive. In recent years, however, the use of Koopman operator theory for control applications has been emerging. This paper presents a new reformulation of the stochastic optimal control problem that, employing the Koopman operator, yields a standard LQR problem with the dual control as its solution. We provide a numerical example that demonstrates the effectiveness of the proposed approach compared with certainty equivalence control, when applied to systems with varying observability. △ Less

Submitted 3 May, 2024; v1 submitted 28 February, 2024; originally announced February 2024.

Comments: 6 pages

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Authors: Mohammad S. Ramadan, Mahmoud A. Hayajnh, Michael T. Tolley, Kyriakos G. Vamvoudakis

Abstract: In this paper we propose a framework towards achieving two intertwined objectives: (i) equipping reinforcement learning with active exploration and deliberate information gathering, such that it regulates state and parameter uncertainties resulting from modeling mismatches and noisy sensory; and (ii) overcoming the huge computational cost of stochastic optimal control. We approach both objectives… ▽ More In this paper we propose a framework towards achieving two intertwined objectives: (i) equipping reinforcement learning with active exploration and deliberate information gathering, such that it regulates state and parameter uncertainties resulting from modeling mismatches and noisy sensory; and (ii) overcoming the huge computational cost of stochastic optimal control. We approach both objectives by using reinforcement learning to attain the stochastic optimal control law. On one hand, we avoid the curse of dimensionality prohibiting the direct solution of the stochastic dynamic programming equation. On the other hand, the resulting stochastic control inspired reinforcement learning agent admits the behavior of a dual control, namely, caution and probing, that is, regulating the state estimate together with its estimation quality. Unlike exploration and exploitation, caution and probing are employed automatically by the controller in real-time, even after the learning process is concluded. We use the proposed approach on a numerical example of a model that belongs to an emerging class in system identification. We show how, for the dimensionality of the stochastic version of this model, Dynamic Programming is prohibitive, Model Predictive Control requires an expensive nonlinear optimization, and a Linear Quadratic Regulator with the certainty equivalence assumption leads to poor performance and filter divergence, all contrasting our approach which is shown to be both: computationally convenient, stabilizing and of an acceptable performance. △ Less

Submitted 26 February, 2024; v1 submitted 18 September, 2023; originally announced September 2023.

A Control Approach for Nonlinear Stochastic State Uncertain Systems with Probabilistic Safety Guarantees

Authors: Mohammad S. Ramadan, Mohammad Alsuwaidan, Ahmed Atallah, Sylvia Herbert

Abstract: This paper presents an algorithm to apply nonlinear control design approaches in the case of stochastic systems with partial state observation. Deterministic nonlinear control approaches are formulated under the assumption of full state access and, often, relative degree one. We propose a control design approach that first generates a control policy for nonlinear deterministic models with full sta… ▽ More This paper presents an algorithm to apply nonlinear control design approaches in the case of stochastic systems with partial state observation. Deterministic nonlinear control approaches are formulated under the assumption of full state access and, often, relative degree one. We propose a control design approach that first generates a control policy for nonlinear deterministic models with full state observation. The resulting control policy is then used to build an importance-like probability distribution over the space of control sequences which are to be evaluated for the true stochastic and state-uncertain dynamics. This distribution serves in the sampling step within a random search control optimization procedure, to focus the exploration effort on certain regions of the control space. The sampled control sequences are assigned costs determined by a prescribed finite-horizon performance and safety measure, which is based on the stochastic dynamics. This sampling algorithm is parallelizable and shown to have computational complexity indifferent to the state dimension, and to be able to guarantee safety over the prescribed prediction horizon. A numerical simulation is provided to test the applicability and effectiveness of the presented approach and compare it to a certainty equivalence controller. △ Less

Submitted 15 September, 2023; originally announced September 2023.

Optimal Trajectories for Propellant-Free Rendezvous Missions

Authors: Mohamed Shouman, Ahmed Atallah, Mohammad S. Ramadan

Abstract: The paper provides a new approach to utilizing space environmental forces in time- and energy-optimal, propellant-less spacecraft rendezvous missions. Considering the nonlinear form of the relative dynamic equations, rendezvous missions are posed as optimal control problems subject to input saturation. We conduct a direct optimal control approach to obtain optimal trajectories and control inputs.… ▽ More The paper provides a new approach to utilizing space environmental forces in time- and energy-optimal, propellant-less spacecraft rendezvous missions. Considering the nonlinear form of the relative dynamic equations, rendezvous missions are posed as optimal control problems subject to input saturation. We conduct a direct optimal control approach to obtain optimal trajectories and control inputs. Initially, we consider the differential drag only and conduct a comprehensive analysis of the effect of altitude on the required control input and achieved cost function. Lorentz forces are then utilized with the differential drag, reducing the time required for time-optimal missions. For energy-optimal missions with combined differential drag and Lorentz forces, a weighting matrix in the cost function is introduced to adjust the relative contributions of these forces. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Monte Carlo Grid Dynamic Programming: Almost Sure Convergence and Probability Constraints

Authors: Mohammad S. Ramadan, Ahmad Al-Tawaha, Mohamed Shouman, Ahmed Atallah, Ming Jin

Abstract: Dynamic Programming (DP) suffers from the well-known ``curse of dimensionality'', further exacerbated by the need to compute expectations over process noise in stochastic models. This paper presents a Monte Carlo-based sampling approach for the state space and an interpolation procedure for the resulting value function, dependent on the process noise density, in a "self-approximating" fashion, eli… ▽ More Dynamic Programming (DP) suffers from the well-known ``curse of dimensionality'', further exacerbated by the need to compute expectations over process noise in stochastic models. This paper presents a Monte Carlo-based sampling approach for the state space and an interpolation procedure for the resulting value function, dependent on the process noise density, in a "self-approximating" fashion, eliminating the need for ordering or set-membership tests. We provide proof of almost sure convergence for the value iteration (and consequently, policy iteration) procedure. The proposed meshless sampling and interpolation algorithm alleviates the burden of gridding the state space, traditionally required in DP, and avoids constructing a piecewise constant value function over a grid. Moreover, we demonstrate that the proposed interpolation procedure is well-suited for handling probabilistic constraints by sampling both infeasible and feasible regions. While the curse of dimensionality cannot be entirely avoided, this approach offers a practical framework for addressing lower-order stochastic nonlinear systems with probabilistic constraints, where traditional DP methods may be intractable or inefficient. Numerical examples are presented to illustrate the convergence and convenience of the proposed algorithms. △ Less

Submitted 26 February, 2024; v1 submitted 10 March, 2023; originally announced March 2023.

Comments: 6 pages, 1 figure

State estimation for control: an approach for output-feedback stochastic MPC

Authors: Mohammad S. Ramadan, Robert R. Bitmead, Ke Huang

Abstract: The paper provides a new approach to the determination of a single state value for stochastic output feedback problems using paradigms from Model Predictive Control, particularly the distinction between open-loop and closed-loop control and between deterministic optimal control and stochastic optimal control. The State Selection Algorithm is presented and relies on given dynamics and constraints,… ▽ More The paper provides a new approach to the determination of a single state value for stochastic output feedback problems using paradigms from Model Predictive Control, particularly the distinction between open-loop and closed-loop control and between deterministic optimal control and stochastic optimal control. The State Selection Algorithm is presented and relies on given dynamics and constraints, a nominal deterministic state-feedback controller, and a sampling based method to select the best state value, based on optimizing a prescribed finite-horizon performance function, over the available candidates provided by a particle filter. The cost function is minimized over the horizon with controls determined by the nominal controller and the selected states. So, the minimization is performed not over the selection of the control other than through the choice of state value to use. The algorithm applies generally to nonlinear stochastic systems and relies on Monte Carlo sampling and averaging. However, in linear quadratic polyhedrally constrained cases the technique reduces to a quadratic program for the state value. The algorithm is evaluated in a set of computational examples, which illustrate its efficacy and limitations. Numerical aspects and the opportunity for parallelization are discussed. The examples demonstrate the algorithm operating, in closed-loop with its attendant particle filter, over the long horizon. △ Less

Submitted 1 March, 2023; originally announced March 2023.

Comments: 15 pages, 10 figures, 2 algorithms

Electrodeposited WS$_2$ Monolayers on Fabricated Graphene Electrodes

Authors: Yasir J Noori, Shibin Thomas, Sami Ramadan, Victoria K. Greenacre, Nema M. Abdelazim, Yisong Han, Jiapei Zhang, Richard Beanland, Andrew L. Hector, Norbert Klein, Gillian Reid, Philip Bartlett, Cornelis H. de Groot

Abstract: The development of scalable techniques to make 2D material heterostructures is a major obstacle that needs to be overcome before these materials can be implemented in device technologies industrially. Electrodeposition is an industrially compatible deposition technique that offers unique advantages in scaling 2D heterostructures. In this work, we demonstrate the electrodeposition of atomic layers… ▽ More The development of scalable techniques to make 2D material heterostructures is a major obstacle that needs to be overcome before these materials can be implemented in device technologies industrially. Electrodeposition is an industrially compatible deposition technique that offers unique advantages in scaling 2D heterostructures. In this work, we demonstrate the electrodeposition of atomic layers of WS$_2$ over graphene electrodes using a single source precursor. Using conventional microfabrication techniques, graphene was patterned to create micro-electrodes where WS$_2$ was site-selectively deposited to form 2D heterostructures. We used various characterisation techniques, including atomic force microscopy, transmission electron microscopy, Raman spectroscopy and x-ray photoelectron spectroscopy to show that our electrodeposited WS$_2$ layers are highly uniform and can be grown over graphene at a controllable deposition rate. This technique to selectively deposit TMDCs over microfabricated graphene electrodes paves the way towards wafer-scale production of 2D material heterostructures for nanodevice applications. △ Less

Submitted 31 August, 2021; originally announced September 2021.

Comments: 9 pages, 5 figures

Maximum Likelihood Recursive State Estimation using the Expectation Maximization Algorithm

Authors: Mohammad S. Ramadan, Robert R. Bitmead

Abstract: A Maximum Likelihood recursive state estimator is derived for non-linear and non-Gaussian state-space models. The estimator combines a particle filter to generate the conditional density and the Expectation Maximization algorithm to compute the maximum likelihood state estimate iteratively. Algorithms for maximum likelihood state filtering, prediction and smoothing are presented. The convergence p… ▽ More A Maximum Likelihood recursive state estimator is derived for non-linear and non-Gaussian state-space models. The estimator combines a particle filter to generate the conditional density and the Expectation Maximization algorithm to compute the maximum likelihood state estimate iteratively. Algorithms for maximum likelihood state filtering, prediction and smoothing are presented. The convergence properties of these algorithms, which are inherited from the Expectation Maximization algorithm, are proven and examined in two examples. It is shown that, with randomized reinitialization, which is feasible because of the algorithm simplicity, these methods are able to converge to the Maximum Likelihood Estimate (MLE) of multimodal, truncated and skewed densities, as well as those of disjoint support. △ Less

Submitted 18 March, 2021; originally announced March 2021.

Comments: 12 pages, 5 figures, 5 algorithms

Large-Area Electrodeposition of Few-Layer MoS2 on Graphene for 2D Material Heterostructures

Authors: Yasir J. Noori, Shibin Thomas, Sami Ramadan, Danielle E. Smith, Victoria K. Greenacre, Nema Abdelazim, Yisong Han, Richard Beanland, Andrew L. Hector, Norbert Klein, Gill Reid, Phillip N. Bartlett, Cornelis H. de Groot

Abstract: Heterostructures involving two-dimensional (2D) transition metal dichalcogenides and other materials such as graphene have a strong potential to be the fundamental building block of many electronic and opto-electronic applications. The integration and scalable fabrication of such heterostructures is of essence in unleashing the potential of these materials in new technologies. For the first time,… ▽ More Heterostructures involving two-dimensional (2D) transition metal dichalcogenides and other materials such as graphene have a strong potential to be the fundamental building block of many electronic and opto-electronic applications. The integration and scalable fabrication of such heterostructures is of essence in unleashing the potential of these materials in new technologies. For the first time, we demonstrate the growth of few-layer MoS2 films on graphene via non-aqueous electrodeposition. Through methods such as scanning and transmission electron microscopy, atomic force microscopy, Raman spectroscopy, energy and wavelength dispersive X-ray spectroscopies and X-ray photoelectron spectroscopy, we show that this deposition method can produce large-area MoS2 films with high quality and uniformity over graphene. We reveal the potential of these heterostructures by measuring the photo-induced current through the film. These results pave the way towards developing the electrodeposition method for the large-scale growth of heterostructures consisting of varying 2D materials for many applications. △ Less

Submitted 22 November, 2020; v1 submitted 18 May, 2020; originally announced May 2020.

Comments: 11 pages and 6 figures

Journal ref: ACS Applied Materials and Interfaces, 2020, 12, 44, 49786-49794