  Recent advances in multi-agent reinforcement learning (MARL) allow agents to
coordinate their behaviors in complex environments. However, common MARL
algorithms still suffer from scalability and sparse reward issues. One
promising approach to resolving them is automatic curriculum learning (ACL).
ACL involves a student (curriculum learner) training on tasks of increasing
difficulty controlled by a teacher (curriculum generator). Despite its success,
ACL's applicability is limited by (1) the lack of a general student framework
for dealing with the varying number of agents across tasks and the sparse
reward problem, and (2) the non-stationarity of the teacher's task due to
ever-changing student strategies. As a remedy for ACL, we introduce a novel
automatic curriculum learning framework, Skilled Population Curriculum (SPC),
which adapts curriculum learning to multi-agent coordination. Specifically, we
endow the student with population-invariant communication and a hierarchical
skill set, allowing it to learn cooperation and behavior skills from distinct
tasks with varying numbers of agents. In addition, we model the teacher as a
contextual bandit conditioned by student policies, enabling a team of agents to
change its size while still retaining previously acquired skills. We also
analyze the inherent non-stationarity of this multi-agent automatic curriculum
teaching problem and provide a corresponding regret bound. Empirical results
show that our method improves the performance, scalability and sample
efficiency in several MARL environments.
