text: We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout kingma2015 . Stochastic gradient variational Bayes (SGVB) aevb is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, we extend the SGVB estimator with using Alpha-Divergences, which are alternative to divergences to VI' KL objective. The Gaussian dropout can be seen as a local reparametrization trick of the SGVB objective. We extend the Variational Dropout to use alpha divergences for variational inference. Our results compare @math -divergence variational dropout with standard variational dropout with correlated and uncorrelated weight noise. We show that the @math -divergence with @math (or KL divergence) is still a good measure for use in variational inference, in spite of the efficient use of Alpha-divergences for Dropout VI Li17 . @math can yield the lowest training error, and optimizes a good lower bound for the evidence lower bound (ELBO) among all values of the parameter @math .
  @cite_0: We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise that is independent across datapoints in the minibatch. Such parameterizations can be trivially parallelized and have variance that is inversely proportional to the mini-batch size, generally leading to much faster convergence. Additionally, we explore a connection with dropout: Gaussian dropout objectives correspond to SGVB with local reparameterization, a scale-invariant prior and proportionally fixed posterior variance. Our method allows inference of more flexibly parameterized posteriors; specifically, we propose variational dropout, a generalization of Gaussian dropout where the dropout rates are learned, often leading to better models. The method is demonstrated through several experiments.
summary: Our work extends the variational dropout technique originally proposed in @cite_0 . It was suggested that applying a multiplicative random noise to inputs of a fully-connected neural network layer is equivalent to computing the posterior distribution of the pre-activation outputs ( @cite_0 ). Using the reparametrization of the weights @math as @math , where @math is a deterministic weight value and @math is a random variable determining the scale of the @math weight. As described in @cite_0 , it is possible to split the posterior distribution over the weights @math into an additive (i.e., location) parameter @math and a multiplicative (i.e., scale) parameter @math . Formally, @math , using set notation.