  Given a sequence of convex functions $f_0, f_1, \ldots, f_T$, we study the
problem of sampling from the Gibbs distribution $\pi_t \propto
e^{-\sum_{k=0}^tf_k}$ for each epoch $t$ in an online manner. Interest in this
problem derives from applications in machine learning, Bayesian statistics, and
optimization where, rather than obtaining all the observations at once, one
constantly acquires new data, and must continuously update the distribution.
Our main result is an algorithm that generates roughly independent samples from
$\pi_t$ for every epoch $t$ and, under mild assumptions, makes
$\mathrm{polylog}(T)$ gradient evaluations per epoch. All previous results
imply a bound on the number of gradient or function evaluations which is at
least linear in $T$. Motivated by real-world applications, we assume that
functions are smooth, their associated distributions have a bounded second
moment, and their minimizer drifts in a bounded manner, but do not assume they
are strongly convex. In particular, our assumptions hold for online Bayesian
logistic regression, when the data satisfy natural regularity properties,
giving a sampling algorithm with updates that are poly-logarithmic in $T$. In
simulations, our algorithm achieves accuracy comparable to an algorithm
specialized to logistic regression. Key to our algorithm is a novel stochastic
gradient Langevin dynamics Markov chain with a carefully designed variance
reduction step and constant batch size. Technically, lack of strong convexity
is a significant barrier to analysis and, here, our main contribution is a
martingale exit time argument that shows our Markov chain remains in a ball of
radius roughly poly-logarithmic in $T$ for enough time to reach within
$\varepsilon$ of $\pi_t$.
