  While the history of machine learning so far largely encompasses a series of
problems posed by researchers and algorithms that learn their solutions, an
important question is whether the problems themselves can be generated by the
algorithm at the same time as they are being solved. Such a process would in
effect build its own diverse and expanding curricula, and the solutions to
problems at various stages would become stepping stones towards solving even
more challenging problems later in the process. The Paired Open-Ended
Trailblazer (POET) algorithm introduced in this paper does just that: it pairs
the generation of environmental challenges and the optimization of agents to
solve those challenges. It simultaneously explores many different paths through
the space of possible problems and solutions and, critically, allows these
stepping-stone solutions to transfer between problems if better, catalyzing
innovation. The term open-ended signifies the intriguing potential for
algorithms like POET to continue to create novel and increasingly complex
capabilities without bound. Our results show that POET produces a diverse range
of sophisticated behaviors that solve a wide range of environmental challenges,
many of which cannot be solved by direct optimization alone, or even through a
direct-path curriculum-building control algorithm introduced to highlight the
critical role of open-endedness in solving ambitious challenges. The ability to
transfer solutions from one environment to another proves essential to
unlocking the full potential of the system as a whole, demonstrating the
unpredictable nature of fortuitous stepping stones. We hope that POET will
inspire a new push towards open-ended discovery across many domains, where
algorithms like POET can blaze a trail through their interesting possible
manifestations and solutions.
