  Quality Diversity (QD) has emerged as a powerful alternative optimization
paradigm that aims at generating large and diverse collections of solutions,
notably with its flagship algorithm MAP-ELITES (ME) which evolves solutions
through mutations and crossovers. While very effective for some unstructured
problems, early ME implementations relied exclusively on random search to
evolve the population of solutions, rendering them notoriously
sample-inefficient for high-dimensional problems, such as when evolving neural
networks. Follow-up works considered exploiting gradient information to guide
the search in order to address these shortcomings through techniques borrowed
from either Black-Box Optimization (BBO) or Reinforcement Learning (RL). While
mixing RL techniques with ME unlocked state-of-the-art performance for robotics
control problems that require a good amount of exploration, it also plagued
these ME variants with limitations common among RL algorithms that ME was free
of, such as hyperparameter sensitivity, high stochasticity as well as training
instability, including when the population size increases as some components
are shared across the population in recent approaches. Furthermore, existing
approaches mixing ME with RL tend to be tied to a specific RL algorithm, which
effectively prevents their use on problems where the corresponding RL algorithm
fails. To address these shortcomings, we introduce a flexible framework that
allows the use of any RL algorithm and alleviates the aforementioned
limitations by evolving populations of agents (whose definition include
hyperparameters and all learnable parameters) instead of just policies. We
demonstrate the benefits brought about by our framework through extensive
numerical experiments on a number of robotics control problems, some of which
with deceptive rewards, taken from the QD-RL literature.
