  Emergent communication has made strides towards learning communication from
scratch, but has focused primarily on protocols that resemble human language.
In nature, multi-agent cooperation gives rise to a wide range of communication
that varies in structure and complexity. In this work, we recognize the full
spectrum of communication that exists in nature and propose studying
lower-level communication. Specifically, we study emergent implicit signaling
in the context of decentralized multi-agent learning in difficult, sparse
reward environments. However, learning to coordinate in such environments is
challenging. We propose a curriculum-driven strategy that combines: (i)
velocity-based environment shaping, tailored to the skill level of the
multi-agent team; and (ii) a behavioral curriculum that helps agents learn
successful single-agent behaviors as a precursor to learning multi-agent
behaviors. Pursuit-evasion experiments show that our approach learns effective
coordination, significantly outperforming sophisticated analytical and learned
policies. Our method completes the pursuit-evasion task even when pursuers move
at half of the evader's speed, whereas the highest-performing baseline fails at
80% of the evader's speed. Moreover, we examine the use of implicit signals in
coordination through position-based social influence. We show that pursuers
trained with our strategy exchange more than twice as much information (in
bits) than baseline methods, indicating that our method has learned, and relies
heavily on, the exchange of implicit signals.
