  Legged locomotion is a complex control problem that requires both accuracy
and robustness to cope with real-world challenges. Legged systems have
traditionally been controlled using trajectory optimization with inverse
dynamics. Such hierarchical model-based methods are appealing due to intuitive
cost function tuning, accurate planning, generalization, and most importantly,
the insightful understanding gained from more than one decade of extensive
research. However, model mismatch and violation of assumptions are common
sources of faulty operation. Simulation-based reinforcement learning, on the
other hand, results in locomotion policies with unprecedented robustness and
recovery skills. Yet, all learning algorithms struggle with sparse rewards
emerging from environments where valid footholds are rare, such as gaps or
stepping stones. In this work, we propose a hybrid control architecture that
combines the advantages of both worlds to simultaneously achieve greater
robustness, foot-placement accuracy, and terrain generalization. Our approach
utilizes a model-based planner to roll out a reference motion during training.
A deep neural network policy is trained in simulation, aiming to track the
optimized footholds. We evaluate the accuracy of our locomotion pipeline on
sparse terrains, where pure data-driven methods are prone to fail. Furthermore,
we demonstrate superior robustness in the presence of slippery or deformable
ground when compared to model-based counterparts. Finally, we show that our
proposed tracking controller generalizes across different trajectory
optimization methods not seen during training. In conclusion, our work unites
the predictive capabilities and optimality guarantees of online planning with
the inherent robustness attributed to offline learning.
