  This paper investigates the motion planning of autonomous dynamical systems
modeled by Markov decision processes (MDP) with unknown transition
probabilities over continuous state and action spaces. Linear temporal logic
(LTL) is used to specify high-level tasks over infinite horizon, which can be
converted into a limit deterministic generalized B\"uchi automaton (LDGBA) with
several accepting sets. The novelty is to design an embedded product MDP
(EP-MDP) between the LDGBA and the MDP by incorporating a synchronous
tracking-frontier function to record unvisited accepting sets of the automaton,
and to facilitate the satisfaction of the accepting conditions. The proposed
LDGBA-based reward shaping and discounting schemes for the model-free
reinforcement learning (RL) only depend on the EP-MDP states and can overcome
the issues of sparse rewards. Rigorous analysis shows that any RL method that
optimizes the expected discounted return is guaranteed to find an optimal
policy whose traces maximize the satisfaction probability. A modular deep
deterministic policy gradient (DDPG) is then developed to generate such
policies over continuous state and action spaces. The performance of our
framework is evaluated via an array of OpenAI gym environments.
