  The framework of multi-agent learning explores the dynamics of how individual
agent strategies evolve in response to the evolving strategies of other agents.
Of particular interest is whether or not agent strategies converge to well
known solution concepts such as Nash Equilibrium (NE). Most "fixed order"
learning dynamics restrict an agent's underlying state to be its own strategy.
In "higher order" learning, agent dynamics can include auxiliary states that
can capture phenomena such as path dependencies. We introduce higher-order
gradient play dynamics that resemble projected gradient ascent with auxiliary
states. The dynamics are "payoff based" in that each agent's dynamics depend on
its own evolving payoff. While these payoffs depend on the strategies of other
agents in a game setting, agent dynamics do not depend explicitly on the nature
of the game or the strategies of other agents. In this sense, dynamics are
"uncoupled" since an agent's dynamics do not depend explicitly on the utility
functions of other agents. We first show that for any specific game with an
isolated completely mixed-strategy NE, there exist higher-order gradient play
dynamics that lead (locally) to that NE, both for the specific game and nearby
games with perturbed utility functions. Conversely, we show that for any
higher-order gradient play dynamics, there exists a game with a unique isolated
completely mixed-strategy NE for which the dynamics do not lead to NE. These
results build on prior work that showed that uncoupled fixed-order learning
cannot lead to NE in certain instances, whereas higher-order variants can.
Finally, we consider the mixed-strategy equilibrium associated with
coordination games. While higher-order gradient play can converge to such
equilibria, we show such dynamics must be inherently internally unstable.
