Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Evaluating the Robustness of Off-Policy Evaluation   Off-policy Evaluation (OPE), or offline evaluation in general, evaluates the
performance of hypothetical policies leveraging only offline log data. It is
particularly useful in applications where the online interaction involves high
stakes and expensive setting such as precision medicine and recommender
systems. Since many OPE estimators have been proposed and some of them have
hyperparameters to be tuned, there is an emerging challenge for practitioners
to select and tune OPE estimators for their specific application.
Unfortunately, identifying a reliable estimator from results reported in
research papers is often difficult because the current experimental procedure
evaluates and compares the estimators' performance on a narrow set of
hyperparameters and evaluation policies. Therefore, it is difficult to know
which estimator is safe and reliable to use. In this work, we develop
Interpretable Evaluation for Offline Evaluation (IEOE), an experimental
procedure to evaluate OPE estimators' robustness to changes in hyperparameters
and/or evaluation policies in an interpretable manner. Then, using the IEOE
procedure, we perform extensive evaluation of a wide variety of existing
estimators on Open Bandit Dataset, a large-scale public real-world dataset for
OPE. We demonstrate that our procedure can evaluate the estimators' robustness
to the hyperparamter choice, helping us avoid using unsafe estimators. Finally,
we apply IEOE to real-world e-commerce platform data and demonstrate how to use
our protocol in practice.

17
The categories of this paper are: q-fin
19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
12
The categories of this paper are: nucl-th
36
The categories of this paper are: cmp-lg
15
The categories of this paper are: nucl-ex
27
The categories of this paper are: comp-gas
30
The categories of this paper are: q-alg
20
The categories of this paper are: acc-phys
35
The categories of this paper are: mtrl-th
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
9
The categories of this paper are: nlin
5
The categories of this paper are: gr-qc
4
The categories of this paper are: cond-mat
33
The categories of this paper are: plasm-ph