  When factorized approximations are used for variational inference (VI), they
tend to underestimate the uncertainty -- as measured in various ways -- of the
distributions they are meant to approximate. We consider two popular ways to
measure the uncertainty deficit of VI: (i) the degree to which it
underestimates the componentwise variance, and (ii) the degree to which it
underestimates the entropy. To better understand these effects, and the
relationship between them, we examine an informative setting where they can be
explicitly (and elegantly) analyzed: the approximation of a Gaussian,~$p$, with
a dense covariance matrix, by a Gaussian,~$q$, with a diagonal covariance
matrix. We prove that $q$ always underestimates both the componentwise variance
and the entropy of $p$, \textit{though not necessarily to the same degree}.
Moreover we demonstrate that the entropy of $q$ is determined by the trade-off
of two competing forces: it is decreased by the shrinkage of its componentwise
variances (our first measure of uncertainty) but it is increased by the
factorized approximation which delinks the nodes in the graphical model of $p$.
We study various manifestations of this trade-off, notably one where, as the
dimension of the problem grows, the per-component entropy gap between $p$ and
$q$ becomes vanishingly small even though $q$ underestimates every
componentwise variance by a constant multiplicative factor. We also use the
shrinkage-delinkage trade-off to bound the entropy gap in terms of the problem
dimension and the condition number of the correlation matrix of $p$. Finally we
present empirical results on both Gaussian and non-Gaussian targets, the former
to validate our analysis and the latter to explore its limitations.
