  Communication is crucial in multi-agent reinforcement learning when agents
are not able to observe the full state of the environment. The most common
approach to allow learned communication between agents is the use of a
differentiable communication channel that allows gradients to flow between
agents as a form of feedback. However, this is challenging when we want to use
discrete messages to reduce the message size since gradients cannot flow
through a discrete communication channel. Previous work proposed methods to
deal with this problem. However, these methods are tested in different
communication learning architectures and environments, making it hard to
compare them. In this paper, we compare several state-of-the-art discretization
methods as well as two methods that have not been used for communication
learning before. We do this comparison in the context of communication learning
using gradients from other agents and perform tests on several environments.
Our results show that none of the methods is best in all environments. The best
choice in discretization method greatly depends on the environment. However,
the discretize regularize unit (DRU), straight through DRU and the straight
through gumbel softmax show the most consistent results across all the tested
environments. Therefore, these methods prove to be the best choice for general
use while the straight through estimator and the gumbel softmax may provide
better results in specific environments but fail completely in others.
