  In the multiple unmanned aerial vehicle (UAV)- assisted downlink
communication, it is challenging for UAV base stations (UAV BSs) to realize
trajectory design and resource assignment in unknown environments. The
cooperation and competition between UAV BSs in the communication network leads
to a Markov game problem. Multi-agent reinforcement learning is a significant
solution for the above decision-making. However, there are still many common
issues, such as the instability of the system and low utilization of historical
data, that limit its application. In this paper, a novel graph-attention
multi-agent trust region (GA-MATR) reinforcement learning framework is proposed
to solve the multi-UAV assisted communication problem. Graph recurrent network
is introduced to process and analyze complex topology of the communication
network, so as to extract useful information and patterns from observational
information. The attention mechanism provides additional weighting for conveyed
information, so that the critic network can accurately evaluate the value of
behavior for UAV BSs. This provides more reliable feedback signals and helps
the actor network update the strategy more effectively. Ablation simulations
indicate that the proposed approach attains improved convergence over the
baselines. UAV BSs learn the optimal communication strategies to achieve their
maximum cumulative rewards. Additionally, multi-agent trust region method with
monotonic convergence provides an estimated Nash equilibrium for the multi-UAV
assisted communication Markov game.
