  Multi-Agent Path Finding (MAPF) is a crucial component for many large-scale
robotic systems, where agents must plan their collision-free paths to their
given goal positions. Recently, multi-agent reinforcement learning has been
introduced to solve the partially observable variant of MAPF by learning a
decentralized single-agent policy in a centralized fashion based on each
agent's partial observation. However, existing learning-based methods are
ineffective in achieving complex multi-agent cooperation, especially in
congested environments, due to the non-stationarity of this setting. To tackle
this challenge, we propose a multi-agent actor-critic method called Soft
Actor-Critic with Heuristic-Based Attention (SACHA), which employs novel
heuristic-based attention mechanisms for both the actors and critics to
encourage cooperation among agents. SACHA learns a neural network for each
agent to selectively pay attention to the shortest path heuristic guidance from
multiple agents within its field of view, thereby allowing for more scalable
learning of cooperation. SACHA also extends the existing multi-agent
actor-critic framework by introducing a novel critic centered on each agent to
approximate $Q$-values. Compared to existing methods that use a fully
observable critic, our agent-centered multi-agent actor-critic method results
in more impartial credit assignment and better generalizability of the learned
policy to MAPF instances with varying numbers of agents and types of
environments. We also implement SACHA(C), which embeds a communication module
in the agent's policy network to enable information exchange among agents. We
evaluate both SACHA and SACHA(C) on a variety of MAPF instances and demonstrate
decent improvements over several state-of-the-art learning-based MAPF methods
with respect to success rate and solution quality.
