  Experience replay is crucial for off-policy reinforcement learning (RL)
methods. By remembering and reusing the experiences from past different
policies, experience replay significantly improves the training efficiency and
stability of RL algorithms. Many decision-making problems in practice naturally
involve multiple agents and require multi-agent reinforcement learning (MARL)
under centralized training decentralized execution paradigm. Nevertheless,
existing MARL algorithms often adopt standard experience replay where the
transitions are uniformly sampled regardless of their importance. Finding
prioritized sampling weights that are optimized for MARL experience replay has
yet to be explored. To this end, we propose MAC-PO, which formulates optimal
prioritized experience replay for multi-agent problems as a regret minimization
over the sampling weights of transitions. Such optimization is relaxed and
solved using the Lagrangian multiplier approach to obtain the close-form
optimal sampling weights. By minimizing the resulting policy regret, we can
narrow the gap between the current policy and a nominal optimal policy, thus
acquiring an improved prioritization scheme for multi-agent tasks. Our
experimental results on Predator-Prey and StarCraft Multi-Agent Challenge
environments demonstrate the effectiveness of our method, having a better
ability to replay important transitions and outperforming other
state-of-the-art baselines.
