  Cooperative multi-agent reinforcement learning (CMARL) has shown to be
promising for many real-world applications. Previous works mainly focus on
improving coordination ability via solving MARL-specific challenges (e.g.,
non-stationarity, credit assignment, scalability), but ignore the policy
perturbation issue when testing in a different environment. This issue hasn't
been considered in problem formulation or efficient algorithm design. To
address this issue, we firstly model the problem as a limited policy adversary
Dec-POMDP (LPA-Dec-POMDP), where some coordinators from a team might
accidentally and unpredictably encounter a limited number of malicious action
attacks, but the regular coordinators still strive for the intended goal. Then,
we propose Robust Multi-Agent Coordination via Evolutionary Generation of
Auxiliary Adversarial Attackers (ROMANCE), which enables the trained policy to
encounter diversified and strong auxiliary adversarial attacks during training,
thus achieving high robustness under various policy perturbations. Concretely,
to avoid the ego-system overfitting to a specific attacker, we maintain a set
of attackers, which is optimized to guarantee the attackers high attacking
quality and behavior diversity. The goal of quality is to minimize the
ego-system coordination effect, and a novel diversity regularizer based on
sparse action is applied to diversify the behaviors among attackers. The
ego-system is then paired with a population of attackers selected from the
maintained attacker set, and alternately trained against the constantly
evolving attackers. Extensive experiments on multiple scenarios from SMAC
indicate our ROMANCE provides comparable or better robustness and
generalization ability than other baselines.
