  We apply reinforcement learning (RL) to robotics tasks. One of the drawbacks
of traditional RL algorithms has been their poor sample efficiency. One
approach to improve the sample efficiency is model-based RL. In our model-based
RL algorithm, we learn a model of the environment, essentially its transition
dynamics and reward function, use it to generate imaginary trajectories and
backpropagate through them to update the policy, exploiting the
differentiability of the model. Intuitively, learning more accurate models
should lead to better model-based RL performance. Recently, there has been
growing interest in developing better deep neural network based dynamics models
for physical systems, by utilizing the structure of the underlying physics. We
focus on robotic systems undergoing rigid body motion without contacts. We
compare two versions of our model-based RL algorithm, one which uses a standard
deep neural network based dynamics model and the other which uses a much more
accurate, physics-informed neural network based dynamics model. We show that,
in model-based RL, model accuracy mainly matters in environments that are
sensitive to initial conditions, where numerical errors accumulate fast. In
these environments, the physics-informed version of our algorithm achieves
significantly better average-return and sample efficiency. In environments that
are not sensitive to initial conditions, both versions of our algorithm achieve
similar average-return, while the physics-informed version achieves better
sample efficiency. We also show that, in challenging environments,
physics-informed model-based RL achieves better average-return than
state-of-the-art model-free RL algorithms such as Soft Actor-Critic, as it
computes the policy-gradient analytically, while the latter estimates it
through sampling.
