  Multi-agent Reinforcement Learning (MARL) is a powerful tool for training
autonomous agents acting independently in a common environment. However, it can
lead to sub-optimal behavior when individual incentives and group incentives
diverge. Humans are remarkably capable at solving these social dilemmas. It is
an open problem in MARL to replicate such cooperative behaviors in selfish
agents. In this work, we draw upon the idea of formal contracting from
economics to overcome diverging incentives between agents in MARL. We propose
an augmentation to a Markov game where agents voluntarily agree to binding
transfers of reward, under pre-specified conditions. Our contributions are
theoretical and empirical. First, we show that this augmentation makes all
subgame-perfect equilibria of all Fully Observable Markov Games exhibit
socially optimal behavior, given a sufficiently rich space of contracts. Next,
we show that for general contract spaces, and even under partial observability,
richer contract spaces lead to higher welfare. Hence, contract space design
solves an exploration-exploitation tradeoff, sidestepping incentive issues. We
complement our theoretical analysis with experiments. Issues of exploration in
the contracting augmentation are mitigated using a training methodology
inspired by multi-objective reinforcement learning: Multi-Objective Contract
Augmentation Learning (MOCA). We test our methodology in static, single-move
games, as well as dynamic domains that simulate traffic, pollution management
and common pool resource management.
