  Collective human knowledge has clearly benefited from the fact that
innovations by individuals are taught to others through communication. Similar
to human social groups, agents in distributed learning systems would likely
benefit from communication to share knowledge and teach skills. The problem of
teaching to improve agent learning has been investigated by prior works, but
these approaches make assumptions that prevent application of teaching to
general multiagent problems, or require domain expertise for problems they can
apply to. This learning to teach problem has inherent complexities related to
measuring long-term impacts of teaching that compound the standard multiagent
coordination challenges. In contrast to existing works, this paper presents the
first general framework and algorithm for intelligent agents to learn to teach
in a multiagent environment. Our algorithm, Learning to Coordinate and Teach
Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative
multiagent reinforcement learning. Each agent in our approach learns both when
and what to advise, then uses the received advice to improve local learning.
Importantly, these roles are not fixed; these agents learn to assume the role
of student and/or teacher at the appropriate moments, requesting and providing
advice in order to improve teamwide performance and learning. Empirical
comparisons against state-of-the-art teaching methods show that our teaching
agents not only learn significantly faster, but also learn to coordinate in
tasks where existing methods fail.
