  Most of the prior work on multi-agent reinforcement learning (MARL) achieves
optimal collaboration by directly controlling the agents to maximize a common
reward. In this paper, we aim to address this from a different angle. In
particular, we consider scenarios where there are self-interested agents (i.e.,
worker agents) which have their own minds (preferences, intentions, skills,
etc.) and can not be dictated to perform tasks they do not wish to do. For
achieving optimal coordination among these agents, we train a super agent
(i.e., the manager) to manage them by first inferring their minds based on both
current and past observations and then initiating contracts to assign suitable
tasks to workers and promise to reward them with corresponding bonuses so that
they will agree to work together. The objective of the manager is maximizing
the overall productivity as well as minimizing payments made to the workers for
ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent
Management Reinforcement Learning (M^3RL), which consists of agent modeling and
policy learning. We have evaluated our approach in two environments, Resource
Collection and Crafting, to simulate multi-agent management problems with
various task settings and multiple designs for the worker agents. The
experimental results have validated the effectiveness of our approach in
modeling worker agents' minds online, and in achieving optimal ad-hoc teaming
with good generalization and fast adaptation.
