  Learning anticipation is a reasoning paradigm in multi-agent reinforcement
learning, where agents, during learning, consider the anticipated learning of
other agents. There has been substantial research into the role of learning
anticipation in improving cooperation among self-interested agents in
general-sum games. Two primary examples are Learning with Opponent-Learning
Awareness (LOLA), which anticipates and shapes the opponent's learning process
to ensure cooperation among self-interested agents in various games such as
iterated prisoner's dilemma, and Look-Ahead (LA), which uses learning
anticipation to guarantee convergence in games with cyclic behaviors. So far,
the effectiveness of applying learning anticipation to fully-cooperative games
has not been explored. In this study, we aim to research the influence of
learning anticipation on coordination among common-interested agents. We first
illustrate that both LOLA and LA, when applied to fully-cooperative games,
degrade coordination among agents, causing worst-case outcomes. Subsequently,
to overcome this miscoordination behavior, we propose Hierarchical Learning
Anticipation (HLA), where agents anticipate the learning of other agents in a
hierarchical fashion. Specifically, HLA assigns agents to several hierarchy
levels to properly regulate their reasonings. Our theoretical and empirical
findings confirm that HLA can significantly improve coordination among
common-interested agents in fully-cooperative normal-form games. With HLA, to
the best of our knowledge, we are the first to unlock the benefits of learning
anticipation for fully-cooperative games.
