  We present a Reinforcement Learning (RL) based framework for optimizing
long-term discounted reward problems with large combinatorial action space and
state dependent constraints. These characteristics are common to many
operations management problems, e.g., network inventory replenishment, where
managers have to deal with uncertain demand, lost sales, and capacity
constraints that results in more complex feasible action spaces. Our proposed
Programmable Actor Reinforcement Learning (PARL) uses a deep-policy iteration
method that leverages neural networks (NNs) to approximate the value function
and combines it with mathematical programming (MP) and sample average
approximation (SAA) to solve the per-step-action optimally while accounting for
combinatorial action spaces and state-dependent constraint sets.
  We show how the proposed methodology can be applied to complex inventory
replenishment problems where analytical solutions are intractable. We also
benchmark the proposed algorithm against state-of-the-art RL algorithms and
commonly used replenishment heuristics and find that the proposed algorithm
considerably outperforms existing methods by as much as 14.7\% on average in
various supply chain settings.
  This improvement in performance of PARL over benchmark algorithms can be
attributed to better inventory cost management, especially in inventory
constrained settings. Furthermore, in a simpler back order setting where the
optimal solution is tractable, we find that the RL based policy also converges
to the optimal policy. Finally, to make RL algorithms more accessible for
inventory management researchers, we also discuss a modular Python library
developed that can be used to test the performance of RL algorithms with
various supply chain structures. This library can spur future research in
developing practical and near-optimal algorithms for inventory management
problems.
