  Batch reinforcement learning (RL) aims at leveraging pre-collected data to
find an optimal policy that maximizes the expected total rewards in a dynamic
environment. Nearly all existing algorithms rely on the absolutely continuous
assumption on the distribution induced by target policies with respect to the
data distribution, so that the batch data can be used to calibrate target
policies via the change of measure. However, the absolute continuity assumption
could be violated in practice (e.g., no-overlap support), especially when the
state-action space is large or continuous. In this paper, we propose a new
batch RL algorithm without requiring absolute continuity in the setting of an
infinite-horizon Markov decision process with continuous states and actions. We
call our algorithm STEEL: SingulariTy-awarE rEinforcement Learning. Our
algorithm is motivated by a new error analysis on off-policy evaluation, where
we use maximum mean discrepancy, together with distributionally robust
optimization, to characterize the error of off-policy evaluation caused by the
possible singularity and to enable model extrapolation. By leveraging the idea
of pessimism and under some mild conditions, we derive a finite-sample regret
guarantee for our proposed algorithm without imposing absolute continuity.
Compared with existing algorithms, by requiring only minimal data-coverage
assumption, STEEL significantly improves the applicability and robustness of
batch RL. Extensive simulation studies and one real experiment on personalized
pricing demonstrate the superior performance of our method in dealing with
possible singularity in batch RL.
