  Due to the growing adoption of deep neural networks in many fields of science
and engineering, modeling and estimating their uncertainties has become of
primary importance. Despite the growing literature about uncertainty
quantification in deep learning, the quality of the uncertainty estimates
remains an open question. In this work, we assess for the first time the
performance of several approximation methods for Bayesian neural networks on
regression tasks by evaluating the quality of the confidence regions with
several coverage metrics. The selected algorithms are also compared in terms of
predictivity, kernelized Stein discrepancy and maximum mean discrepancy with
respect to a reference posterior in both weight and function space. Our
findings show that (i) some algorithms have excellent predictive performance
but tend to largely over or underestimate uncertainties (ii) it is possible to
achieve good accuracy and a given target coverage with finely tuned
hyperparameters and (iii) the promising kernel Stein discrepancy cannot be
exclusively relied on to assess the posterior approximation. As a by-product of
this benchmark, we also compute and visualize the similarity of all algorithms
and corresponding hyperparameters: interestingly we identify a few clusters of
algorithms with similar behavior in weight space, giving new insights on how
they explore the posterior distribution.
