  During online decision making in Multi-Armed Bandits (MAB), one needs to
conduct inference on the true mean reward of each arm based on data collected
so far at each step. However, since the arms are adaptively selected--thereby
yielding non-iid data--conducting inference accurately is not straightforward.
In particular, sample averaging, which is used in the family of UCB and
Thompson sampling (TS) algorithms, does not provide a good choice as it suffers
from bias and a lack of good statistical properties (e.g. asymptotic
normality). Our thesis in this paper is that more sophisticated inference
schemes that take into account the adaptive nature of the sequentially
collected data can unlock further performance gains, even though both UCB and
TS type algorithms are optimal in the worst case. In particular, we propose a
variant of TS-style algorithms--which we call doubly adaptive TS--that
leverages recent advances in causal inference and adaptively reweights the
terms of a doubly robust estimator on the true mean reward of each arm. Through
20 synthetic domain experiments and a semi-synthetic experiment based on data
from an A/B test of a web service, we demonstrate that using an adaptive
inferential scheme (while still retaining the exploration efficacy of TS)
provides clear benefits in online decision making: the proposed DATS algorithm
has superior empirical performance to existing baselines (UCB and TS) in terms
of regret and sample complexity in identifying the best arm. In addition, we
also provide a finite-time regret bound of doubly adaptive TS that matches (up
to log factors) those of UCB and TS algorithms, thereby establishing that its
improved practical benefits do not come at the expense of worst-case
suboptimality.
