  The networked nature of multi-robot systems presents challenges in the
context of multi-agent reinforcement learning. Centralized control policies do
not scale with increasing numbers of robots, whereas independent control
policies do not exploit the information provided by other robots, exhibiting
poor performance in cooperative-competitive tasks. In this work we propose a
physics-informed reinforcement learning approach able to learn distributed
multi-robot control policies that are both scalable and make use of all the
available information to each robot. Our approach has three key
characteristics. First, it imposes a port-Hamiltonian structure on the policy
representation, respecting energy conservation properties of physical robot
systems and the networked nature of robot team interactions. Second, it uses
self-attention to ensure a sparse policy representation able to handle
time-varying information at each robot from the interaction graph. Third, we
present a soft actor-critic reinforcement learning algorithm parameterized by
our self-attention port-Hamiltonian control policy, which accounts for the
correlation among robots during training while overcoming the need of value
function factorization. Extensive simulations in different multi-robot
scenarios demonstrate the success of the proposed approach, surpassing previous
multi-robot reinforcement learning solutions in scalability, while achieving
similar or superior performance (with averaged cumulative reward up to x2
greater than the state-of-the-art with robot teams x6 larger than the number of
robots at training time).
