  This paper develops an efficient multi-agent deep reinforcement learning
algorithm for cooperative controls in powergrids. Specifically, we consider the
decentralized inverter-based secondary voltage control problem in distributed
generators (DGs), which is first formulated as a cooperative multi-agent
reinforcement learning (MARL) problem. We then propose a novel on-policy MARL
algorithm, PowerNet, in which each agent (DG) learns a control policy based on
(sub-)global reward but local states from its neighboring agents. Motivated by
the fact that a local control from one agent has limited impact on agents
distant from it, we exploit a novel spatial discount factor to reduce the
effect from remote agents, to expedite the training process and improve
scalability. Furthermore, a differentiable, learning-based communication
protocol is employed to foster the collaborations among neighboring agents. In
addition, to mitigate the effects of system uncertainty and random noise
introduced during on-policy learning, we utilize an action smoothing factor to
stabilize the policy execution. To facilitate training and evaluation, we
develop PGSim, an efficient, high-fidelity powergrid simulation platform.
Experimental results in two microgrid setups show that the developed PowerNet
outperforms a conventional model-based control, as well as several
state-of-the-art MARL algorithms. The decentralized learning scheme and high
sample efficiency also make it viable to large-scale power grids.
