-
Controlling Commercial Cooling Systems Using Reinforcement Learning
Authors:
Jerry Luo,
Cosmin Paduraru,
Octavian Voicu,
Yuri Chervonyi,
Scott Munns,
Jerry Li,
Crystal Qian,
Praneet Dutta,
Jared Quincy Davis,
Ningjia Wu,
Xingwei Yang,
Chu-Ming Chang,
Ted Li,
Rob Rose,
Mingyan Fan,
Hootan Nakhost,
Tinglin Liu,
Brian Kirkman,
Frank Altamura,
Lee Cline,
Patrick Tonker,
Joel Gouker,
Dave Uden,
Warren Buddy Bryan,
Jason Law
, et al. (11 additional authors not shown)
Abstract:
This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments ha…
▽ More
This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.
△ Less
Submitted 14 December, 2022; v1 submitted 11 November, 2022;
originally announced November 2022.
-
Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learning
Authors:
William Wong,
Praneet Dutta,
Octavian Voicu,
Yuri Chervonyi,
Cosmin Paduraru,
Jerry Luo
Abstract:
Reinforcement learning (RL) techniques have been developed to optimize industrial cooling systems, offering substantial energy savings compared to traditional heuristic policies. A major challenge in industrial control involves learning behaviors that are feasible in the real world due to machinery constraints. For example, certain actions can only be executed every few hours while other actions c…
▽ More
Reinforcement learning (RL) techniques have been developed to optimize industrial cooling systems, offering substantial energy savings compared to traditional heuristic policies. A major challenge in industrial control involves learning behaviors that are feasible in the real world due to machinery constraints. For example, certain actions can only be executed every few hours while other actions can be taken more frequently. Without extensive reward engineering and experimentation, an RL agent may not learn realistic operation of machinery. To address this, we use hierarchical reinforcement learning with multiple agents that control subsets of actions according to their operation time scales. Our hierarchical approach achieves energy savings over existing baselines while maintaining constraints such as operating chillers within safe bounds in a simulated HVAC control environment.
△ Less
Submitted 16 September, 2022;
originally announced September 2022.
-
Semi-analytical Industrial Cooling System Model for Reinforcement Learning
Authors:
Yuri Chervonyi,
Praneet Dutta,
Piotr Trochim,
Octavian Voicu,
Cosmin Paduraru,
Crystal Qian,
Emre Karagozler,
Jared Quincy Davis,
Richard Chippendale,
Gautam Bajaj,
Sims Witherspoon,
Jerry Luo
Abstract:
We present a hybrid industrial cooling system model that embeds analytical solutions within a multi-physics simulation. This model is designed for reinforcement learning (RL) applications and balances simplicity with simulation fidelity and interpretability. The model's fidelity is evaluated against real world data from a large scale cooling system. This is followed by a case study illustrating ho…
▽ More
We present a hybrid industrial cooling system model that embeds analytical solutions within a multi-physics simulation. This model is designed for reinforcement learning (RL) applications and balances simplicity with simulation fidelity and interpretability. The model's fidelity is evaluated against real world data from a large scale cooling system. This is followed by a case study illustrating how the model can be used for RL research. For this, we develop an industrial task suite that allows specifying different problem settings and levels of complexity, and use it to evaluate the performance of different RL algorithms.
△ Less
Submitted 26 July, 2022;
originally announced July 2022.
