  Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based
deterministic sampling algorithm. Despite its wide usage, understanding the
theoretical properties of SVGD has remained a challenging problem. For sampling
from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain
Gaussian as long as the initializer is Gaussian. Inspired by this fact, we
undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD
projected to the family of Gaussian distributions via the bilinear kernel, or
equivalently Gaussian variational inference (GVI) with SVGD. We present a
complete picture by considering both the mean-field PDE and discrete particle
systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD
dynamics is proven to converge linearly to the Gaussian distribution closest to
the target in KL divergence. In the finite-particle setting, there is both
uniform in time convergence to the mean-field limit and linear convergence in
time to the equilibrium if the target is Gaussian. In the general case, we
propose a density-based and a particle-based implementation of the
Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from
different perspectives, emerge as special cases of our unified framework.
Interestingly, one of the new particle-based instance from this framework
empirically outperforms existing approaches. Our results make concrete
contributions towards obtaining a deeper understanding of both SVGD and GVI.
