  Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for
training neural policies to solve complex control tasks. However, these
policies tend to be overfit to the exact specifications of the task and
environment they were trained on, and thus do not perform well when conditions
deviate slightly or when composed hierarchically to solve even more complex
tasks. Recent work has shown that training a mixture of policies, as opposed to
a single one, that are driven to explore different regions of the state-action
space can address this shortcoming by generating a diverse set of behaviors,
referred to as skills, that can be collectively used to great effect in
adaptation tasks or for hierarchical planning. This is typically realized by
including a diversity term - often derived from information theory - in the
objective function optimized by RL. However these approaches often require
careful hyperparameter tuning to be effective. In this work, we demonstrate
that less widely-used neuroevolution methods, specifically Quality Diversity
(QD), are a competitive alternative to information-theory-augmented RL for
skill discovery. Through an extensive empirical evaluation comparing eight
state-of-the-art algorithms (four flagship algorithms from each line of work)
on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the
skills' performance on adaptation tasks, and (iii) the skills' performance when
used as primitives for hierarchical planning; QD methods are found to provide
equal, and sometimes improved, performance whilst being less sensitive to
hyperparameters and more scalable. As no single method is found to provide
near-optimal performance across all environments, there is a rich scope for
further research which we support by proposing future directions and providing
optimized open-source implementations.
