nt, but stochastic gradient ascent is capable of following
such gradients.</p>
</div>
<div id="adaptive-stepsize-sequence" class="section level3 unnumbered">
<h3>Adaptive stepsize sequence</h3>
<p>ADVI uses a finite-memory version of adaGrad <span class="citation">Duchi, Hazan, and Singer (<a href="#ref-Duchi:2011" role="doc-biblioref">2011</a>)</span>. This
has a single parameter that we expose, denoted <code>eta</code>. We now
have a warmup adaptation phase that selects a good value for
<code>eta</code>. 