  Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning
(MARL) have achieved significant successes across a wide range of domains,
including game AI, autonomous vehicles, robotics, and so on. However, DRL and
deep MARL agents are widely known to be sample inefficient that millions of
interactions are usually needed even for relatively simple problem settings,
thus preventing the wide application and deployment in real-industry scenarios.
One bottleneck challenge behind is the well-known exploration problem, i.e.,
how efficiently exploring the environment and collecting informative
experiences that could benefit policy learning towards the optimal ones. This
problem becomes more challenging in complex environments with sparse rewards,
noisy distractions, long horizons, and non-stationary co-learners. In this
paper, we conduct a comprehensive survey on existing exploration methods for
both single-agent and multi-agent RL. We start the survey by identifying
several key challenges to efficient exploration. Beyond the above two main
branches, we also include other notable exploration methods with different
ideas and techniques. In addition to algorithmic analysis, we provide a
comprehensive and unified empirical comparison of different exploration methods
for DRL on a set of commonly used benchmarks. According to our algorithmic and
empirical investigation, we finally summarize the open problems of exploration
in DRL and deep MARL and point out a few future directions.
