  The utilization of large-scale distributed renewable energy promotes the
development of the multi-microgrid (MMG), which raises the need of developing
an effective energy management method to minimize economic costs and keep self
energy-sufficiency. The multi-agent deep reinforcement learning (MADRL) has
been widely used for the energy management problem because of its real-time
scheduling ability. However, its training requires massive energy operation
data of microgrids (MGs), while gathering these data from different MGs would
threaten their privacy and data security. Therefore, this paper tackles this
practical yet challenging issue by proposing a federated multi-agent deep
reinforcement learning (F-MADRL) algorithm via the physics-informed reward. In
this algorithm, the federated learning (FL) mechanism is introduced to train
the F-MADRL algorithm thus ensures the privacy and the security of data. In
addition, a decentralized MMG model is built, and the energy of each
participated MG is managed by an agent, which aims to minimize economic costs
and keep self energy-sufficiency according to the physics-informed reward. At
first, MGs individually execute the self-training based on local energy
operation data to train their local agent models. Then, these local models are
periodically uploaded to a server and their parameters are aggregated to build
a global agent, which will be broadcasted to MGs and replace their local
agents. In this way, the experience of each MG agent can be shared and the
energy operation data is not explicitly transmitted, thus protecting the
privacy and ensuring data security. Finally, experiments are conducted on Oak
Ridge national laboratory distributed energy control communication lab
microgrid (ORNL-MG) test system, and the comparisons are carried out to verify
the effectiveness of introducing the FL mechanism and the outperformance of our
proposed F-MADRL.
