  Securing coordination between AI agent and teammates (human players or AI
agents) in contexts involving unfamiliar humans continues to pose a significant
challenge in Zero-Shot Coordination. The issue of cooperative incompatibility
becomes particularly prominent when an AI agent is unsuccessful in
synchronizing with certain previously unknown partners. Traditional algorithms
have aimed to collaborate with partners by optimizing fixed objectives within a
population, fostering diversity in strategies and behaviors. However, these
techniques may lead to learning loss and an inability to cooperate with
specific strategies within the population, a phenomenon named cooperative
incompatibility in learning. In order to solve cooperative incompatibility in
learning and effectively address the problem in the context of ZSC, we
introduce the Cooperative Open-ended LEarning (COLE) framework, which
formulates open-ended objectives in cooperative games with two players using
perspectives of graph theory to evaluate and pinpoint the cooperative capacity
of each strategy. We present two practical algorithms, specifically \algo and
\algoR, which incorporate insights from game theory and graph theory. We also
show that COLE could effectively overcome the cooperative incompatibility from
theoretical and empirical analysis. Subsequently, we created an online
Overcooked human-AI experiment platform, the COLE platform, which enables easy
customization of questionnaires, model weights, and other aspects. Utilizing
the COLE platform, we enlist 130 participants for human experiments. Our
findings reveal a preference for our approach over state-of-the-art methods
using a variety of subjective metrics. Moreover, objective experimental
outcomes in the Overcooked game environment indicate that our method surpasses
existing ones when coordinating with previously unencountered AI agents and the
human proxy model.
