  We consider the problem of decentralized multi-agent reinforcement learning
in Markov games. A fundamental question is whether there exist algorithms that,
when adopted by all agents and run independently in a decentralized fashion,
lead to no-regret for each player, analogous to celebrated convergence results
in normal-form games. While recent work has shown that such algorithms exist
for restricted settings (notably, when regret is defined with respect to
deviations to Markovian policies), the question of whether independent
no-regret learning can be achieved in the standard Markov game framework was
open. We provide a decisive negative resolution this problem, both from a
computational and statistical perspective. We show that:
  - Under the widely-believed assumption that PPAD-hard problems cannot be
solved in polynomial time, there is no polynomial-time algorithm that attains
no-regret in general-sum Markov games when executed independently by all
players, even when the game is known to the algorithm designer and the number
of players is a small constant.
  - When the game is unknown, no algorithm, regardless of computational
efficiency, can achieve no-regret without observing a number of episodes that
is exponential in the number of players.
  Perhaps surprisingly, our lower bounds hold even for seemingly easier setting
in which all agents are controlled by a a centralized algorithm. They are
proven via lower bounds for a simpler problem we refer to as SparseCCE, in
which the goal is to compute a coarse correlated equilibrium that is sparse in
the sense that it can be represented as a mixture of a small number of product
policies. The crux of our approach is a novel application of aggregation
techniques from online learning, whereby we show that any algorithm for the
SparseCCE problem can be used to compute approximate Nash equilibria for
non-zero sum normal-form games.
