  Hierarchical reinforcement learning (HRL) incorporates temporal abstraction
into reinforcement learning (RL) by explicitly taking advantage of hierarchical
structure. Modern HRL typically designs a hierarchical agent composed of a
high-level policy and low-level policies. The high-level policy selects which
low-level policy to activate at a lower frequency and the activated low-level
policy selects an action at each time step. Recent HRL algorithms have achieved
performance gains over standard RL algorithms in synthetic navigation tasks.
However, we cannot apply these HRL algorithms to real-world navigation tasks.
One of the main challenges is that real-world navigation tasks require an agent
to perform safe and interactive behaviors in dynamic environments. In this
paper, we propose imagination-augmented HRL (IAHRL) that efficiently integrates
imagination into HRL to enable an agent to learn safe and interactive behaviors
in real-world navigation tasks. Imagination is to predict the consequences of
actions without interactions with actual environments. The key idea behind
IAHRL is that the low-level policies imagine safe and structured behaviors, and
then the high-level policy infers interactions with surrounding objects by
interpreting the imagined behaviors. We also introduce a new attention
mechanism that allows our high-level policy to be permutation-invariant to the
order of surrounding objects and to prioritize our agent over them. To evaluate
IAHRL, we introduce five complex urban driving tasks, which are among the most
challenging real-world navigation tasks. The experimental results indicate that
IAHRL enables an agent to perform safe and interactive behaviors, achieving
higher success rates and lower average episode steps than baselines.
