  Safe reinforcement learning (RL) aims to find the optimal policy and its
feasible region in a constrained optimal control problem (OCP). Ensuring
feasibility and optimality simultaneously has been a major challenge. Existing
methods either attempt to solve OCPs directly with constrained optimization
algorithms, leading to unstable training processes and unsatisfactory
feasibility, or restrict policies in overly small feasible regions, resulting
in excessive conservativeness with sacrificed optimality. To address this
challenge, we propose an indirect safe RL framework called feasible policy
iteration, which guarantees that the feasible region monotonically expands and
converges to the maximum one, and the state-value function monotonically
improves and converges to the optimal one. We achieve this by designing a
policy update principle called region-wise policy improvement, which maximizes
the state-value function under the constraint of the constraint decay function
(CDF) inside the feasible region and minimizes the CDF outside the feasible
region simultaneously. This update scheme ensures that the state-value function
monotonically increases state-wise in the feasible region and the CDF
monotonically decreases state-wise in the entire state space. We prove that the
CDF converges to the solution of the risky Bellman equation while the
state-value function converges to the solution of the feasible Bellman
equation. The former represents the maximum feasible region and the latter
manifests the optimal state-value function. Experiments show that our algorithm
learns strictly safe and near-optimal policies with accurate feasible regions
on classic control tasks. It also achieves fewer constraint violations with
performance better than (or comparable to) baselines on Safety Gym.
