text: The power grid is a complex and vital system that necessitates careful reliability management. Managing the grid is a difficult problem with multiple time scales of decision making and stochastic behavior due to renewable energy generations, variable demand and unplanned outages. Solving this problem in the face of uncertainty requires a new methodology with tractable algorithms. In this work, we introduce a new model for hierarchical decision making in complex systems. We apply reinforcement learning (RL) methods to learn a proxy, i.e., a level of abstraction, for real-time power grid reliability. We devise an algorithm that alternates between slow time-scale policy improvement, and fast time-scale value function approximation. We compare our results to prevailing heuristics, and show the strength of our method.
  @cite_37: With the fast-paced changing technologies in the power industry, new power references addressing new technologies are coming to the market. So there is an urgent need to keep track of international experiences and activities taking place in the field of modern unit-commitment (UC) problem. This paper gives a bibliographical survey, mathematical formulations, and general backgrounds of research and developments in the field of UC problem for past 35 years based on more than 150 published articles. The collected literature has been divided into many sections, so that new researchers do not face any difficulty in carrying out research in the area of next-generation UC problem under both the regulated and deregulated power industry.
  @cite_14: There is growing interest in the use of grid-level storage to smooth variations in supply that are likely to arise with an increased use of wind and solar energy. Energy arbitrage, the process of buying, storing, and selling electricity to exploit variations in electricity spot prices, is becoming an important way of paying for expensive investments into grid-level storage. Independent system operators such as the New York Independent System Operator NYISO require that battery storage operators place bids into an hour-ahead market although settlements may occur in increments as small as five minutes, which is considered near "real-time". The operator has to place these bids without knowing the energy level in the battery at the beginning of the hour and simultaneously accounting for the value of leftover energy at the end of the hour. The problem is formulated as a dynamic program. We describe and employ a convergent approximate dynamic programming ADP algorithm that exploits monotonicity of the value function to find a revenue-generating bidding policy; using optimal benchmarks, we empirically show the computational benefits of the algorithm. Furthermore, we propose a distribution-free variant of the ADP algorithm that does not require any knowledge of the distribution of the price process and makes no assumptions regarding a specific real-time price model. We demonstrate that a policy trained on historical real-time price data from the NYISO using this distribution-free approach is indeed effective.
  @cite_22: The valuation of the real option to store natural gas is a practically important problem that entails dynamic optimization of inventory trading decisions with capacity constraints in the face of uncertain natural gas price dynamics. Stochastic dynamic programming is a natural approach to this valuation problem, but it does not seem to be widely used in practice because it is at odds with the high-dimensional natural gas price evolution models that are widespread among traders. According to the practice-based literature, practitioners typically value natural gas storage heuristically. The effectiveness of the heuristics discussed in this literature is currently unknown because good upper bounds on the value of storage are not available. We develop a novel and tractable approximate dynamic programming method that, coupled with Monte Carlo simulation, computes lower and upper bounds on the value of storage, which we use to benchmark these heuristics on a set of realistic instances. We find that these heuristics are extremely fast to execute but significantly suboptimal compared to our upper bound, which appears to be fairly tight and much tighter than a simpler perfect information upper bound; computing our lower bound takes more time than using these heuristics, but our lower bound substantially outperforms them in terms of valuation. Moreover, with periodic reoptimizations embedded in Monte Carlo simulation, the practice-based heuristics become nearly optimal, with one exception, at the expense of higher computational effort. Our lower bound with reoptimization is also nearly optimal, but exhibits a higher computational requirement than these heuristics. Besides natural gas storage, our results are potentially relevant for the valuation of the real option to store other commodities, such as met als, oil, and petroleum products.
  @cite_33: We develop a stochastic dynamic programming model that co-optimizes the use of energy storage for multiple applications, such as energy, capacity, and backup services, while accounting for market and system uncertainty. Using the example of a battery that has been installed in a home as a distributed storage device, we demonstrate the ability of the model to co-optimize services that ‘compete’ for the capacity of the battery. We also show that these multiple uses of a battery can provide substantive value.
  @cite_41: 
  @cite_40: Sustainable energy systems of the future could no longer rely on the current paradigm that energy supply follows demand. Since many of the renewable energy resources do not produce power on demand, there is a need for new market structures that motivate sustainable behaviors by participants. The Power Trading Agent Competition(Power TAC) is a new annual competition that focuses on the design and operation of future retail power markets, specifically in smart grid environments with renewable energy production, smart metering, and autonomous agents acting on behalf of customers and retailers. It uses a rich, open-source simulation platform that is based on real-world data and state-of-the-art customer models. Its purpose is to help researchers understand the dynamics of customer and retailer decision-making, as well as the robustness of proposed market designs. This research contributes to the former, by introducing TacTex'13, the champion agent from the inaugural competition in 2013. TacTex'13 learns and adapts to the environment in which it operates, by heavily relying on reinforcement-learning and prediction methods. We formalize the complex decision-making problem that TacTex'13 faces, and approximate its solution in TacTex'13's constituent components. We examine the success of the complete agent through analysis of competition results.
  @cite_5: In this work we solve the day-ahead unit commitment (UC) problem, by formulating it as a Markov decision process (MDP) and finding a low-cost policy for generation scheduling. We present two reinforcement learning algorithms, and devise a third one. We compare our results to previous work that uses simulated annealing (SA), and show a 27 improvement in operation costs, with running time of 2.5 minutes (compared to 2.5 hours of existing state-of-the-art).
  @cite_15: As more renewable, yet volatile, forms of energy like solar and wind are being incorporated into the grid, the problem of finding optimal control policies for energy storage is becoming increasingly important. These sequential decision problems are often modeled as stochastic dynamic programs, but when the state space becomes large, traditional (exact) techniques such as backward induction, policy iteration, or value iteration quickly become computationally intractable. Approximate dynamic programming (ADP) thus becomes a natural solution technique for solving these problems to near-optimality using significantly fewer computational resources. In this paper, we compare the performance of the following: various approximation architectures with approximate policy iteration (API), approximate value iteration (AVI) with structured lookup table, and direct policy search on a benchmarked energy storage problem (i.e., the optimal solution is computable).
  @cite_34: 
  @cite_20: 
summary: Several works in the literature of power systems, operational research and more recently machine learning offer approaches for solving sequential stochastic problems using dynamic programming. Of which, the majority of these works focus on energy storage @cite_22 @cite_33 @cite_15 @cite_34 , unit commitment @cite_37 @cite_5 @cite_20 , and energy market bidding strategies @cite_41 @cite_40 @cite_14 . To our knowledge, no work has been done to use MDPs for assessing the reliability in power grids.