  This work considers variational Bayesian inference as an inexpensive and
scalable alternative to a fully Bayesian approach in the context of
sparsity-promoting priors. In particular, the priors considered arise from
scale mixtures of Normal distributions with a generalized inverse Gaussian
mixing distribution. This includes the variational Bayesian LASSO as an
inexpensive and scalable alternative to the Bayesian LASSO introduced in [65].
It also includes a family of priors which more strongly promote sparsity. For
linear models the method requires only the iterative solution of deterministic
least squares problems. Furthermore, for p unknown covariates the method can be
implemented exactly online with a cost of $O(p^3)$ in computation and $O(p^2)$
in memory per iteration -- in other words, the cost per iteration is
independent of n, and in principle infinite data can be considered. For large
$p$ an approximation is able to achieve promising results for a cost of $O(p)$
per iteration, in both computation and memory. Strategies for hyper-parameter
tuning are also considered. The method is implemented for real and simulated
data. It is shown that the performance in terms of variable selection and
uncertainty quantification of the variational Bayesian LASSO can be comparable
to the Bayesian LASSO for problems which are tractable with that method, and
for a fraction of the cost. The present method comfortably handles $n = 65536$,
$p = 131073$ on a laptop in less than 30 minutes, and $n = 10^5$, $p = 2.1
\times 10^6$ overnight.
