  Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithms (EA) are
two major paradigms of policy optimization with distinct learning principles,
i.e., gradient-based v.s. gradient-free. An appealing research direction is
integrating Deep RL and EA to devise new methods by fusing their complementary
advantages. However, existing works on combining Deep RL and EA have two common
drawbacks: 1) the RL agent and EA agents learn their policies individually,
neglecting efficient sharing of useful common knowledge; 2) parameter-level
policy optimization guarantees no semantic level of behavior evolution for the
EA side. In this paper, we propose Evolutionary Reinforcement Learning with
Two-scale State Representation and Policy Representation (ERL-Re$^2$), a novel
solution to the aforementioned two drawbacks. The key idea of ERL-Re$^2$ is
two-scale representation: all EA and RL policies share the same nonlinear state
representation while maintaining individual} linear policy representations. The
state representation conveys expressive common features of the environment
learned by all the agents collectively; the linear policy representation
provides a favorable space for efficient policy optimization, where novel
behavior-level crossover and mutation operations can be performed. Moreover,
the linear policy representation allows convenient generalization of policy
fitness with the help of the Policy-extended Value Function Approximator
(PeVFA), further improving the sample efficiency of fitness estimation. The
experiments on a range of continuous control tasks show that ERL-Re$^2$
consistently outperforms advanced baselines and achieves the State Of The Art
(SOTA). Our code is available on https://github.com/yeshenpy/ERL-Re2.
