  Utilizing messages from teammates can improve coordination in cooperative
Multi-agent Reinforcement Learning (MARL). Previous works typically combine raw
messages of teammates with local information as inputs for policy. However,
neglecting message aggregation poses significant inefficiency for policy
learning. Motivated by recent advances in representation learning, we argue
that efficient message aggregation is essential for good coordination in
cooperative MARL. In this paper, we propose Multi-Agent communication via
Self-supervised Information Aggregation (MASIA), where agents can aggregate the
received messages into compact representations with high relevance to augment
the local policy. Specifically, we design a permutation invariant message
encoder to generate common information-aggregated representation from messages
and optimize it via reconstructing and shooting future information in a
self-supervised manner. Hence, each agent would utilize the most relevant parts
of the aggregated representation for decision-making by a novel message
extraction mechanism. Furthermore, considering the potential of offline
learning for real-world applications, we build offline benchmarks for
multi-agent communication, which is the first as we know. Empirical results
demonstrate the superiority of our method in both online and offline settings.
We also release the built offline benchmarks in this paper as a testbed for
communication ability validation to facilitate further future research.
