-
Imitation Learning for Intra-Day Power Grid Operation through Topology Actions
Authors:
Matthijs de Jong,
Jan Viebahn,
Yuliya Shapovalova
Abstract:
Power grid operation is becoming increasingly complex due to the increase in generation of renewable energy. The recent series of Learning To Run a Power Network (L2RPN) competitions have encouraged the use of artificial agents to assist human dispatchers in operating power grids. In this paper we study the performance of imitation learning for day-ahead power grid operation through topology actio…
▽ More
Power grid operation is becoming increasingly complex due to the increase in generation of renewable energy. The recent series of Learning To Run a Power Network (L2RPN) competitions have encouraged the use of artificial agents to assist human dispatchers in operating power grids. In this paper we study the performance of imitation learning for day-ahead power grid operation through topology actions. In particular, we consider two rule-based expert agents: a greedy agent and a N-1 agent. While the latter is more computationally expensive since it takes N-1 safety considerations into account, it exhibits a much higher operational performance. We train a fully-connected neural network (FCNN) on expert state-action pairs and evaluate it in two ways. First, we find that classification accuracy is limited despite extensive hyperparameter tuning, due to class imbalance and class overlap. Second, as a power system agent, the FCNN performs only slightly worse than expert agents. Furthermore, hybrid agents, which incorporate minimal additional simulations, match expert agents' performance with significantly lower computational cost. Consequently, imitation learning shows promise for developing fast, high-performing power grid agents, motivating its further exploration in future L2RPN studies.
△ Less
Submitted 18 August, 2024; v1 submitted 29 July, 2024;
originally announced July 2024.
-
Hierarchical Reinforcement Learning for Power Network Topology Control
Authors:
Blazej Manczak,
Jan Viebahn,
Herke van Hoof
Abstract:
Learning in high-dimensional action spaces is a key challenge in applying reinforcement learning (RL) to real-world systems. In this paper, we study the possibility of controlling power networks using RL methods. Power networks are critical infrastructures that are complex to control. In particular, the combinatorial nature of the action space poses a challenge to both conventional optimizers and…
▽ More
Learning in high-dimensional action spaces is a key challenge in applying reinforcement learning (RL) to real-world systems. In this paper, we study the possibility of controlling power networks using RL methods. Power networks are critical infrastructures that are complex to control. In particular, the combinatorial nature of the action space poses a challenge to both conventional optimizers and learned controllers. Hierarchical reinforcement learning (HRL) represents one approach to address this challenge. More precisely, a HRL framework for power network topology control is proposed. The HRL framework consists of three levels of action abstraction. At the highest level, there is the overall long-term task of power network operation, namely, keeping the power grid state within security constraints at all times, which is decomposed into two temporally extended actions: 'do nothing' versus 'propose a topology change'. At the intermediate level, the action space consists of all controllable substations. Finally, at the lowest level, the action space consists of all configurations of the chosen substation. By employing this HRL framework, several hierarchical power network agents are trained for the IEEE 14-bus network. Whereas at the highest level a purely rule-based policy is still chosen for all agents in this study, at the intermediate level the policy is trained using different state-of-the-art RL algorithms. At the lowest level, either an RL algorithm or a greedy algorithm is used. The performance of the different 3-level agents is compared with standard baseline (RL or greedy) approaches. A key finding is that the 3-level agent that employs RL both at the intermediate and the lowest level outperforms all other agents on the most difficult task. Our code is publicly available.
△ Less
Submitted 3 November, 2023;
originally announced November 2023.
-
Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents
Authors:
Malte Lehna,
Jan Viebahn,
Christoph Scholz,
Antoine Marot,
Sven Tomforde
Abstract:
The operation of electricity grids has become increasingly complex due to the current upheaval and the increase in renewable energy production. As a consequence, active grid management is reaching its limits with conventional approaches. In the context of the Learning to Run a Power Network challenge, it has been shown that Reinforcement Learning (RL) is an efficient and reliable approach with con…
▽ More
The operation of electricity grids has become increasingly complex due to the current upheaval and the increase in renewable energy production. As a consequence, active grid management is reaching its limits with conventional approaches. In the context of the Learning to Run a Power Network challenge, it has been shown that Reinforcement Learning (RL) is an efficient and reliable approach with considerable potential for automatic grid operation. In this article, we analyse the submitted agent from Binbinchen and provide novel strategies to improve the agent, both for the RL and the rule-based approach. The main improvement is a N-1 strategy, where we consider topology actions that keep the grid stable, even if one line is disconnected. More, we also propose a topology reversion to the original grid, which proved to be beneficial. The improvements are tested against reference approaches on the challenge test sets and are able to increase the performance of the rule-based agent by 27%. In direct comparison between rule-based and RL agent we find similar performance. However, the RL agent has a clear computational advantage. We also analyse the behaviour in an exemplary case in more detail to provide additional insights. Here, we observe that through the N-1 strategy, the actions of the agents become more diversified.
△ Less
Submitted 17 April, 2023; v1 submitted 3 April, 2023;
originally announced April 2023.
-
Learning to run a Power Network Challenge: a Retrospective Analysis
Authors:
Antoine Marot,
Benjamin Donnot,
Gabriel Dulac-Arnold,
Adrian Kelly,
Aïdan O'Sullivan,
Jan Viebahn,
Mariette Awad,
Isabelle Guyon,
Patrick Panciatici,
Camilo Romero
Abstract:
Power networks, responsible for transporting electricity across large geographical regions, are complex infrastructures on which modern life critically depend. Variations in demand and production profiles, with increasing renewable energy integration, as well as the high voltage network technology, constitute a real challenge for human operators when optimizing electricity transportation while avo…
▽ More
Power networks, responsible for transporting electricity across large geographical regions, are complex infrastructures on which modern life critically depend. Variations in demand and production profiles, with increasing renewable energy integration, as well as the high voltage network technology, constitute a real challenge for human operators when optimizing electricity transportation while avoiding blackouts. Motivated to investigate the potential of AI methods in enabling adaptability in power network operation, we have designed a L2RPN challenge to encourage the development of reinforcement learning solutions to key problems present in the next-generation power networks. The NeurIPS 2020 competition was well received by the international community attracting over 300 participants worldwide.
The main contribution of this challenge is our proposed comprehensive 'Grid2Op' framework, and associated benchmark, which plays realistic sequential network operations scenarios. The Grid2Op framework, which is open-source and easily re-usable, allows users to define new environments with its companion GridAlive ecosystem. Grid2Op relies on existing non-linear physical power network simulators and let users create a series of perturbations and challenges that are representative of two important problems: a) the uncertainty resulting from the increased use of unpredictable renewable energy sources, and b) the robustness required with contingent line disconnections. In this paper, we give the competition highlights. We present the benchmark suite and analyse the winning solutions, including one super-human performance demonstration. We propose our organizational insights for a successful competition and conclude on open research avenues. Given the challenge success, we expect our work will foster research to create more sustainable solutions for power network operations.
△ Less
Submitted 21 October, 2021; v1 submitted 2 March, 2021;
originally announced March 2021.
-
Exploring grid topology reconfiguration using a simple deep reinforcement learning approach
Authors:
Medha Subramanian,
Jan Viebahn,
Simon H. Tindemans,
Benjamin Donnot,
Antoine Marot
Abstract:
System operators are faced with increasingly volatile operating conditions. In order to manage system reliability in a cost-effective manner, control room operators are turning to computerised decision support tools based on AI and machine learning. Specifically, Reinforcement Learning (RL) is a promising technique to train agents that suggest grid control actions to operators. In this paper, a si…
▽ More
System operators are faced with increasingly volatile operating conditions. In order to manage system reliability in a cost-effective manner, control room operators are turning to computerised decision support tools based on AI and machine learning. Specifically, Reinforcement Learning (RL) is a promising technique to train agents that suggest grid control actions to operators. In this paper, a simple baseline approach is presented using RL to represent an artificial control room operator that can operate a IEEE 14-bus test case for a duration of 1 week. This agent takes topological switching actions to control power flows on the grid, and is trained on only a single well-chosen scenario. The behaviour of this agent is tested on different time-series of generation and demand, demonstrating its ability to operate the grid successfully in 965 out of 1000 scenarios. The type and variability of topologies suggested by the agent are analysed across the test scenarios, demonstrating efficient and diverse agent behaviour.
△ Less
Submitted 17 April, 2021; v1 submitted 26 November, 2020;
originally announced November 2020.
-
Continuation of Probability Density Functions using a Generalized Lyapunov Approach
Authors:
S. Baars,
J. P. Viebahn,
T. E. Mulder,
C. Kuehn,
F. W. Wubs,
H. A. Dijkstra
Abstract:
Techniques from numerical bifurcation theory are very useful to study transitions between steady fluid flow patterns and the instabilities involved. Here, we provide computational methodology to use parameter continuation in determining probability density functions of systems of stochastic partial differential equations near fixed points, under a small noise approximation. Key innovation is the e…
▽ More
Techniques from numerical bifurcation theory are very useful to study transitions between steady fluid flow patterns and the instabilities involved. Here, we provide computational methodology to use parameter continuation in determining probability density functions of systems of stochastic partial differential equations near fixed points, under a small noise approximation. Key innovation is the efficient solution of a generalized Lyapunov equation using an iterative method involving low-rank approximations. We apply and illustrate the capabilities of the method using a problem in physical oceanography, i.e. the occurrence of multiple steady states of the Atlantic Ocean circulation.
△ Less
Submitted 11 November, 2020;
originally announced November 2020.
-
Higher-order approximations of the residual-mean eddy streamfunction and the quasi-Stokes streamfunction
Authors:
Jan Viebahn,
Carsten Eden
Abstract:
The series expansion of the residual-mean eddy streamfunction and the quasi-Stokes streamfunction are compared up to third order in buoyancy perturbation, both formally and by using several idealised eddy-permitting zonal channel model experiments. In model configurations with flat bottom, both streamfunctions may be well approximated by the first one or two leading order terms in the ocean interi…
▽ More
The series expansion of the residual-mean eddy streamfunction and the quasi-Stokes streamfunction are compared up to third order in buoyancy perturbation, both formally and by using several idealised eddy-permitting zonal channel model experiments. In model configurations with flat bottom, both streamfunctions may be well approximated by the first one or two leading order terms in the ocean interior, although terms up to third order still significantly impact the implied interior circulations. Further, differences in both series expansions up to third order remain small here. Near surface and bottom boundaries, on the other hand, the leading order terms differ and are initially of alternating sign and of increasing magnitude such that the low order approximate expressions break down there. In more realistic model configurations with significant topographic features, physically inconsistent recirculation cells also appear in the ocean interior and are not effectively reduced by the next higher order terms. A measure indicating an initially increasing or decreasing series expansion is proposed.
△ Less
Submitted 2 May, 2016;
originally announced May 2016.
-
Effects of Drake Passage on a strongly eddying global ocean
Authors:
Jan P. Viebahn,
Anna S. von der Heydt,
Dewi Le Bars,
Henk A. Dijkstra
Abstract:
The climate impact of ocean gateway openings during the Eocene-Oligocene transition is still under debate. Previous model studies employed grid resolutions at which the impact of mesoscale eddies has to be parameterized. We present results of a state-of-the-art eddy-resolving global ocean model with a closed Drake Passage, and compare with results of the same model at non-eddying resolution. An an…
▽ More
The climate impact of ocean gateway openings during the Eocene-Oligocene transition is still under debate. Previous model studies employed grid resolutions at which the impact of mesoscale eddies has to be parameterized. We present results of a state-of-the-art eddy-resolving global ocean model with a closed Drake Passage, and compare with results of the same model at non-eddying resolution. An analysis of the pathways of heat by decomposing the meridional heat transport into eddy, horizontal, and overturning circulation components indicates that the model behavior on the large scale is qualitatively similar at both resolutions. Closing Drake Passage induces (i) sea surface warming around Antarctica due to changes in the horizontal circulation of the Southern Ocean, (ii) the collapse of the overturning circulation related to North Atlantic Deep Water formation leading to surface cooling in the North Atlantic, (iii) significant equatorward eddy heat transport near Antarctica. However, quantitative details significantly depend on the chosen resolution. The warming around Antarctica is substantially larger for the non-eddying configuration (5.5{degree sign}C) than for the eddying configuration (2.5{degree sign}C). This is a consequence of the subpolar mean flow which partitions differently into gyres and circumpolar current at different resolutions. We conclude that for a deciphering of the different mechanisms active in Eocene-Oligocene climate change detailed analyses of the pathways of heat in the different climate subsystems are crucial in order to clearly identify the physical processes actually at work.
△ Less
Submitted 14 October, 2015;
originally announced October 2015.
-
Deep ocean early warning signals of an Atlantic MOC collapse
Authors:
Qing Yi Feng,
Jan P. Viebahn,
Henk A. Dijkstra
Abstract:
The Atlantic Meridional Overturning Circulation (MOC) is a crucial part of the climate system because of its associated northward heat transport. The present-day MOC is sensitive to freshwater anomalies and may collapse to a state with a strongly reduced northward heat transport. A future collapse of the Atlantic MOC has been identified as one of the most dangerous tipping points in the climate sy…
▽ More
The Atlantic Meridional Overturning Circulation (MOC) is a crucial part of the climate system because of its associated northward heat transport. The present-day MOC is sensitive to freshwater anomalies and may collapse to a state with a strongly reduced northward heat transport. A future collapse of the Atlantic MOC has been identified as one of the most dangerous tipping points in the climate system. It is therefore crucial to develop early warning indicators for such a potential collapse based on relatively short time series. So far, attempts to use indicators based on critical slowdown have been marginally successful. Based on complex climate network reconstruction, we here present a promising new indicator for the MOC collapse that efficiently monitors spatial changes in deep ocean circulation. Through our analysis of the performance of this indicator we formulate optimal locations of measurement of the MOC to provide early warning signals of a collapse. Our results imply that an increase in spatial resolution of the Atlantic MOC observations (i.e., at more sections) can improve early detection, because the spatial coherence in the deep ocean arising near the transition is better captured.
△ Less
Submitted 6 May, 2014;
originally announced May 2014.
-
Rotating Einstein-Maxwell-Dilaton Black Holes in D Dimensions
Authors:
Jutta Kunz,
Dieter Maison,
Francisco Navarro-Lerida,
Jan Viebahn
Abstract:
We construct exact charged rotating black holes in Einstein-Maxwell-dilaton theory in $D$ spacetime dimensions, $D \ge 5$, by embedding the $D$ dimensional Myers-Perry solutions in $D+1$ dimensions, and performing a boost with a subsequent Kaluza-Klein reduction. Like the Myers-Perry solutions, these black holes generically possess $N=[(D-1)/2]$ independent angular momenta. We present the global…
▽ More
We construct exact charged rotating black holes in Einstein-Maxwell-dilaton theory in $D$ spacetime dimensions, $D \ge 5$, by embedding the $D$ dimensional Myers-Perry solutions in $D+1$ dimensions, and performing a boost with a subsequent Kaluza-Klein reduction. Like the Myers-Perry solutions, these black holes generically possess $N=[(D-1)/2]$ independent angular momenta. We present the global and horizon properties of these black holes, and discuss their domains of existence.
△ Less
Submitted 1 June, 2006;
originally announced June 2006.
-
Charged Rotating Black Holes in Odd Dimensions
Authors:
Jutta Kunz,
Francisco Navarro-Lerida,
Jan Viebahn
Abstract:
We consider charged rotating black holes in $D=2N+1$ dimensions, $D \ge 5$. While these black holes generically possess $N$ independent angular momenta, associated with $N$ distinct planes of rotation, we here focus on black holes with equal-magnitude angular momenta. The angular dependence can then be treated explicitly, and a system of 5 $D$-dependent ordinary differential equations is obtaine…
▽ More
We consider charged rotating black holes in $D=2N+1$ dimensions, $D \ge 5$. While these black holes generically possess $N$ independent angular momenta, associated with $N$ distinct planes of rotation, we here focus on black holes with equal-magnitude angular momenta. The angular dependence can then be treated explicitly, and a system of 5 $D$-dependent ordinary differential equations is obtained. We solve these equations numerically for Einstein-Maxwell theory in D=5, 7 and 9 dimensions. We discuss the global and horizon properties of these black holes, as well as their extremal limits.
△ Less
Submitted 6 May, 2006;
originally announced May 2006.
