-
Complementarity Assessment of South Greenland Katabatic Flows and West Europe Wind Regimes
Authors:
David Radu,
Mathias Berger,
Raphaël Fonteneau,
Simon Hardy,
Xavier Fettweis,
Marc Le Du,
Patrick Panciatici,
Lucian Balea,
Damien Ernst
Abstract:
Current global environmental challenges require vigorous and diverse actions in the energy sector. One solution that has recently attracted interest consists in harnessing high-quality variable renewable energy resources in remote locations, while using transmission links to transport the power to end users. In this context, a comparison of western European and Greenland wind regimes is proposed.…
▽ More
Current global environmental challenges require vigorous and diverse actions in the energy sector. One solution that has recently attracted interest consists in harnessing high-quality variable renewable energy resources in remote locations, while using transmission links to transport the power to end users. In this context, a comparison of western European and Greenland wind regimes is proposed. By leveraging a regional atmospheric model specifically designed to accurately capture polar phenomena, local climatic features of southern Greenland are identified to be particularly conducive to extensive renewable electricity generation from wind. A methodology to assess how connecting remote locations to major demand centres would benefit the latter from a resource availability standpoint is introduced and applied to the aforementioned Europe-Greenland case study, showing superior and complementary wind generation potential in the considered region of Greenland with respect to selected European sites.
△ Less
Submitted 2 April, 2019; v1 submitted 5 December, 2018;
originally announced December 2018.
-
Critical Time Windows for Renewable Resource Complementarity Assessment
Authors:
Mathias Berger,
David Radu,
Raphael Fonteneau,
Robin Henry,
Mevludin Glavic,
Xavier Fettweis,
Marc Le Du,
Patrick Panciatici,
Lucian Balea,
Damien Ernst
Abstract:
This paper proposes a systematic framework to assess the complementarity of renewable resources over arbitrary geographical scopes and temporal scales which is particularly well-suited to exploit very large data sets of climatological data. The concept of critical time windows is introduced, and a spatio-temporal criticality indicator is proposed, consisting in a parametrised family of scalar indi…
▽ More
This paper proposes a systematic framework to assess the complementarity of renewable resources over arbitrary geographical scopes and temporal scales which is particularly well-suited to exploit very large data sets of climatological data. The concept of critical time windows is introduced, and a spatio-temporal criticality indicator is proposed, consisting in a parametrised family of scalar indicators quantifying the complementarity between renewable resources in both space and time. The criticality indicator is leveraged to devise a family of optimisation problems identifying sets of locations with maximum complementarity under arbitrary geographical deployment constraints. The applicability of the framework is shown in a case study investigating the complementarity between the wind regimes in continental western Europe and southern Greenland, and its usefulness in a power system planning context is demonstrated. Besides showing that the occurrence of low wind power production events can be significantly reduced on a regional scale by exploiting diversity in local wind patterns, results highlight the fact that aggregating wind power production sites located on different continents may result in a lower occurrence of system-wide low wind power production events and indicate potential benefits of intercontinental electrical interconnections.
△ Less
Submitted 5 December, 2018;
originally announced December 2018.
-
On overfitting and asymptotic bias in batch reinforcement learning with partial observability
Authors:
Vincent Francois-Lavet,
Guillaume Rabusseau,
Joelle Pineau,
Damien Ernst,
Raphael Fonteneau
Abstract:
This paper provides an analysis of the tradeoff between asymptotic bias (suboptimality with unlimited data) and overfitting (additional suboptimality due to limited data) in the context of reinforcement learning with partial observability. Our theoretical analysis formally characterizes that while potentially increasing the asymptotic bias, a smaller state representation decreases the risk of over…
▽ More
This paper provides an analysis of the tradeoff between asymptotic bias (suboptimality with unlimited data) and overfitting (additional suboptimality due to limited data) in the context of reinforcement learning with partial observability. Our theoretical analysis formally characterizes that while potentially increasing the asymptotic bias, a smaller state representation decreases the risk of overfitting. This analysis relies on expressing the quality of a state representation by bounding L1 error terms of the associated belief states. Theoretical results are empirically illustrated when the state representation is a truncated history of observations, both on synthetic POMDPs and on a large-scale POMDP in the context of smartgrids, with real-world data. Finally, similarly to known results in the fully observable setting, we also briefly discuss and empirically illustrate how using function approximators and adapting the discount factor may enhance the tradeoff between asymptotic bias and overfitting in the partially observable context.
△ Less
Submitted 6 February, 2019; v1 submitted 22 September, 2017;
originally announced September 2017.
-
How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
Authors:
Vincent François-Lavet,
Raphael Fonteneau,
Damien Ernst
Abstract:
Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its fin…
▽ More
Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma.
△ Less
Submitted 20 January, 2016; v1 submitted 7 December, 2015;
originally announced December 2015.
-
Benchmarking for Bayesian Reinforcement Learning
Authors:
Michael Castronovo,
Damien Ernst,
Adrien Couetoux,
Raphael Fonteneau
Abstract:
In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the collected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many BRL algorithms have already been proposed, but even though a few toy examples exist in the literature, there are still no extensive or rigorous benchmarks to compare them. The paper addresses t…
▽ More
In the Bayesian Reinforcement Learning (BRL) setting, agents try to maximise the collected rewards while interacting with their environment while using some prior knowledge that is accessed beforehand. Many BRL algorithms have already been proposed, but even though a few toy examples exist in the literature, there are still no extensive or rigorous benchmarks to compare them. The paper addresses this problem, and provides a new BRL comparison methodology along with the corresponding open source library. In this methodology, a comparison criterion that measures the performance of algorithms on large sets of Markov Decision Processes (MDPs) drawn from some probability distributions is defined. In order to enable the comparison of non-anytime algorithms, our methodology also includes a detailed analysis of the computation time requirement of each algorithm. Our library is released with all source code and documentation: it includes three test problems, each of which has two different prior distributions, and seven state-of-the-art RL algorithms. Finally, our library is illustrated by comparing all the available algorithms and the results are discussed.
△ Less
Submitted 14 September, 2015;
originally announced September 2015.
-
Simultaneous Perturbation Algorithms for Batch Off-Policy Search
Authors:
Raphael Fonteneau,
L. A. Prashanth
Abstract:
We propose novel policy search algorithms in the context of off-policy, batch mode reinforcement learning (RL) with continuous state and action spaces. Given a batch collection of trajectories, we perform off-line policy evaluation using an algorithm similar to that by [Fonteneau et al., 2010]. Using this Monte-Carlo like policy evaluator, we perform policy search in a class of parameterized polic…
▽ More
We propose novel policy search algorithms in the context of off-policy, batch mode reinforcement learning (RL) with continuous state and action spaces. Given a batch collection of trajectories, we perform off-line policy evaluation using an algorithm similar to that by [Fonteneau et al., 2010]. Using this Monte-Carlo like policy evaluator, we perform policy search in a class of parameterized policies. We propose both first order policy gradient and second order policy Newton algorithms. All our algorithms incorporate simultaneous perturbation estimates for the gradient as well as the Hessian of the cost-to-go vector, since the latter is unknown and only biased estimates are available. We demonstrate their practicality on a simple 1-dimensional continuous state space problem.
△ Less
Submitted 31 March, 2014; v1 submitted 18 March, 2014;
originally announced March 2014.
-
Min Max Generalization for Two-stage Deterministic Batch Mode Reinforcement Learning: Relaxation Schemes
Authors:
Raphael Fonteneau,
Damien Ernst,
Bernard Boigelot,
Quentin Louveaux
Abstract:
We study the minmax optimization problem introduced in [22] for computing policies for batch mode reinforcement learning in a deterministic setting. First, we show that this problem is NP-hard. In the two-stage case, we provide two relaxation schemes. The first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time. The second relaxati…
▽ More
We study the minmax optimization problem introduced in [22] for computing policies for batch mode reinforcement learning in a deterministic setting. First, we show that this problem is NP-hard. In the two-stage case, we provide two relaxation schemes. The first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time. The second relaxation scheme, based on a Lagrangian relaxation where all constraints are dualized, leads to a conic quadratic programming problem. We also theoretically prove and empirically illustrate that both relaxation schemes provide better results than those given in [22].
△ Less
Submitted 30 October, 2012; v1 submitted 23 February, 2012;
originally announced February 2012.
