  One of the key challenges for multi-agent learning is scalability. In this
paper, we introduce a technique for speeding up multi-agent learning by
exploiting concurrent and incremental experience sharing. This solution
adaptively identifies opportunities to transfer experiences between agents and
allows for the rapid acquisition of appropriate policies in large-scale,
stochastic, homogeneous multi-agent systems. We introduce an online,
distributed, supervisor-directed transfer technique for constructing high-level
characterizations of an agent's dynamic learning environment---called
contexts---which are used to identify groups of agents operating under
approximately similar dynamics within a short temporal window. A set of
supervisory agents computes contextual information for groups of subordinate
agents, thereby identifying candidates for experience sharing. Our method uses
a tiered architecture to propagate, with low communication overhead, state,
action, and reward data amongst the members of each dynamically-identified
information-sharing group. We applied this method to a large-scale distributed
task allocation problem with hundreds of information-sharing agents operating
in an unknown, non-stationary environment. We demonstrate that our approach
results in significant performance gains, that it is robust to noise-corrupted
or suboptimal context features, and that communication costs scale linearly
with the supervisor-to-subordinate ratio.
