  Current approaches to multi-agent cooperation rely heavily on centralized
mechanisms or explicit communication protocols to ensure convergence. This
paper studies the problem of distributed multi-agent learning without resorting
to centralized components or explicit communication. It examines the use of
distribution matching to facilitate the coordination of independent agents. In
the proposed scheme, each agent independently minimizes the distribution
mismatch to the corresponding component of a target visitation distribution.
The theoretical analysis shows that under certain conditions, each agent
minimizing its individual distribution mismatch allows the convergence to the
joint policy that generated the target distribution. Further, if the target
distribution is from a joint policy that optimizes a cooperative task, the
optimal policy for a combination of this task reward and the distribution
matching reward is the same joint policy. This insight is used to formulate a
practical algorithm (DM$^2$), in which each individual agent matches a target
distribution derived from concurrently sampled trajectories from a joint expert
policy. Experimental validation on the StarCraft domain shows that combining
(1) a task reward, and (2) a distribution matching reward for expert
demonstrations for the same task, allows agents to outperform a naive
distributed baseline. Additional experiments probe the conditions under which
expert demonstrations need to be sampled to obtain the learning benefits.
