  Consider a decision-maker that can pick one out of $K$ actions to control an
unknown system, for $T$ turns. The actions are interpreted as different
configurations or policies. Holding the same action fixed, the system
asymptotically converges to a unique equilibrium, as a function of this action.
The dynamics of the system are unknown to the decision-maker, which can only
observe a noisy reward at the end of every turn. The decision-maker wants to
maximize its accumulated reward over the $T$ turns. Learning what equilibria
are better results in higher rewards, but waiting for the system to converge to
equilibrium costs valuable time. Existing bandit algorithms, either stochastic
or adversarial, achieve linear (trivial) regret for this problem. We present a
novel algorithm, termed Upper Equilibrium Concentration Bound (UECB), that
knows to switch an action quickly if it is not worth it to wait until the
equilibrium is reached. This is enabled by employing convergence bounds to
determine how far the system is from equilibrium. We prove that UECB achieves a
regret of $\mathcal{O}(\log(T)+\tau_c\log(\tau_c)+\tau_c\log\log(T))$ for this
equilibrium bandit problem where $\tau_c$ is the worst case approximate
convergence time to equilibrium. We then show that both epidemic control and
game control are special cases of equilibrium bandits, where $\tau_c\log
\tau_c$ typically dominates the regret. We then test UECB numerically for both
of these applications.
