  The automatic synthesis of a policy through reinforcement learning (RL) from
a given set of formal requirements depends on the construction of a reward
signal and consists of the iterative application of many policy-improvement
steps. The synthesis algorithm has to balance target, safety, and comfort
requirements in a single objective and to guarantee that the policy improvement
does not increase the number of safety-requirements violations, especially for
safety-critical applications. In this work, we present a solution to the
synthesis problem by solving its two main challenges: reward-shaping from a set
of formal requirements and safe policy update. For the former, we propose an
automatic reward-shaping procedure, defining a scalar reward signal compliant
with the task specification. For the latter, we introduce an algorithm ensuring
that the policy is improved in a safe fashion with high-confidence guarantees.
We also discuss the adoption of a model-based RL algorithm to efficiently use
the collected data and train a model-free agent on the predicted trajectories,
where the safety violation does not have the same impact as in the real world.
Finally, we demonstrate in standard control benchmarks that the resulting
learning procedure is effective and robust even under heavy perturbations of
the hyperparameters.
