  Value Decomposition (VD) aims to deduce the contributions of agents for
decentralized policies in the presence of only global rewards, and has recently
emerged as a powerful credit assignment paradigm for tackling cooperative
Multi-Agent Reinforcement Learning (MARL) problems. One of the main challenges
in VD is to promote diverse behaviors among agents, while existing methods
directly encourage the diversity of learned agent networks with various
strategies. However, we argue that these dedicated designs for agent networks
are still limited by the indistinguishable VD network, leading to homogeneous
agent behaviors and thus downgrading the cooperation capability. In this paper,
we propose a novel Contrastive Identity-Aware learning (CIA) method, explicitly
boosting the credit-level distinguishability of the VD network to break the
bottleneck of multi-agent diversity. Specifically, our approach leverages
contrastive learning to maximize the mutual information between the temporal
credits and identity representations of different agents, encouraging the full
expressiveness of credit assignment and further the emergence of
individualities. The algorithm implementation of the proposed CIA module is
simple yet effective that can be readily incorporated into various VD
architectures. Experiments on the SMAC benchmarks and across different VD
backbones demonstrate that the proposed method yields results superior to the
state-of-the-art counterparts. Our code is available at
https://github.com/liushunyu/CIA.
