  Finding equilibria via gradient play in competitive multi-agent games has
been attracting a growing amount of attention in recent years, with emphasis on
designing efficient strategies where the agents operate in a decentralized and
symmetric manner with guaranteed convergence. While significant efforts have
been made in understanding zero-sum two-player matrix games, the performance in
zero-sum multi-agent games remains inadequately explored, especially in the
presence of delayed feedbacks, leaving the scalability and resiliency of
gradient play open to questions.
  In this paper, we make progress by studying asynchronous gradient plays in
zero-sum polymatrix games under delayed feedbacks. We first establish that the
last iterate of entropy-regularized optimistic multiplicative weight updates
(OMWU) method converges linearly to the quantal response equilibrium (QRE), the
solution concept under bounded rationality, in the absence of delays. While the
linear convergence continues to hold even when the feedbacks are randomly
delayed under mild statistical assumptions, it converges at a noticeably slower
rate due to a smaller tolerable range of learning rates. Moving beyond, we
demonstrate entropy-regularized OMWU -- by adopting two-timescale learning
rates in a delay-aware manner -- enjoys faster last-iterate convergence under
fixed delays, and continues to converge provably even when the delays are
arbitrarily bounded in an average-iterate manner. Our methods also lead to
finite-time guarantees to approximate the Nash equilibrium (NE) by moderating
the amount of regularization. To the best of our knowledge, this work is the
first that aims to understand asynchronous gradient play in zero-sum polymatrix
games under a wide range of delay assumptions, highlighting the role of
learning rates separation.
