  When observing the actions of others, humans make inferences about why they
acted as they did, and what this implies about the world; humans also use the
fact that their actions will be interpreted in this manner, allowing them to
act informatively and thereby communicate efficiently with others. Although
learning algorithms have recently achieved superhuman performance in a number
of two-player, zero-sum games, scalable multi-agent reinforcement learning
algorithms that can discover effective strategies and conventions in complex,
partially observable settings have proven elusive. We present the Bayesian
action decoder (BAD), a new multi-agent learning method that uses an
approximate Bayesian update to obtain a public belief that conditions on the
actions taken by all agents in the environment. BAD introduces a new Markov
decision process, the public belief MDP, in which the action space consists of
all deterministic partial policies, and exploits the fact that an agent acting
only on this public belief state can still learn to use its private information
if the action space is augmented to be over all partial policies mapping
private information into environment actions. The Bayesian update is closely
related to the theory of mind reasoning that humans carry out when observing
others' actions. We first validate BAD on a proof-of-principle two-step matrix
game, where it outperforms policy gradient methods; we then evaluate BAD on the
challenging, cooperative partial-information card game Hanabi, where, in the
two-player setting, it surpasses all previously published learning and
hand-coded approaches, establishing a new state of the art.
