  Experimental design is crucial in evidence-based decision-making with
multiple treatment arms, such as online advertisements and medical treatments.
This study investigates the problem of identifying the treatment arm with the
highest expected outcome, referred to as the best treatment arm, while
minimizing the probability of misidentification. This problem has been studied
across numerous research fields, including best arm identification (BAI) and
ordinal optimization. In our experiments, the number of treatment-allocation
rounds is fixed. During each round, a decision-maker allocates a treatment arm
to an experimental unit and observes a corresponding outcome, which follows a
Gaussian distribution with variances that can differ among the treatment arms.
At the end of the experiment, we recommend one of the treatment arms as an
estimate of the best treatment arm based on the observations. To design an
experiment, we first discuss the worst-case lower bound for the probability of
misidentification through an information-theoretic approach. Then, under the
assumption that the variances are known, we propose the
Generalized-Neyman-Allocation (GNA)-empirical-best-arm (EBA) strategy, an
extension of the Neyman allocation proposed by Neyman (1934). We show that the
GNA-EBA strategy is asymptotically optimal in the sense that its probability of
misidentification aligns with the lower bounds as the sample size increases
indefinitely and the differences between the expected outcomes of the best and
other suboptimal arms converge to a uniform value. We refer to such strategies
as asymptotically worst-case optimal.
