  When solving two-player zero-sum games, multi-agent reinforcement learning
(MARL) algorithms often create populations of agents where, at each iteration,
a new agent is discovered as the best response to a mixture over the opponent
population. Within such a process, the update rules of "who to compete with"
(i.e., the opponent mixture) and "how to beat them" (i.e., finding best
responses) are underpinned by manually developed game theoretical principles
such as fictitious play and Double Oracle. In this paper, we introduce a novel
framework -- Neural Auto-Curricula (NAC) -- that leverages meta-gradient
descent to automate the discovery of the learning update rule without explicit
human design. Specifically, we parameterise the opponent selection module by
neural networks and the best-response module by optimisation subroutines, and
update their parameters solely via interaction with the game engine, where both
players aim to minimise their exploitability. Surprisingly, even without human
design, the discovered MARL algorithms achieve competitive or even better
performance with the state-of-the-art population-based game solvers (e.g.,
PSRO) on Games of Skill, differentiable Lotto, non-transitive Mixture Games,
Iterated Matching Pennies, and Kuhn Poker. Additionally, we show that NAC is
able to generalise from small games to large games, for example training on
Kuhn Poker and outperforming PSRO on Leduc Poker. Our work inspires a promising
future direction to discover general MARL algorithms solely from data.
