  Gradient inversion attacks are an ubiquitous threat in federated learning as
they exploit gradient leakage to reconstruct supposedly private training data.
Recent work has proposed to prevent gradient leakage without loss of model
utility by incorporating a PRivacy EnhanCing mODulE (PRECODE) based on
variational modeling. Without further analysis, it was shown that PRECODE
successfully protects against gradient inversion attacks. In this paper, we
make multiple contributions. First, we investigate the effect of PRECODE on
gradient inversion attacks to reveal its underlying working principle. We show
that variational modeling introduces stochasticity into the gradients of
PRECODE and the subsequent layers in a neural network. The stochastic gradients
of these layers prevent iterative gradient inversion attacks from converging.
Second, we formulate an attack that disables the privacy preserving effect of
PRECODE by purposefully omitting stochastic gradients during attack
optimization. To preserve the privacy preserving effect of PRECODE, our
analysis reveals that variational modeling must be placed early in the network.
However, early placement of PRECODE is typically not feasible due to reduced
model utility and the exploding number of additional model parameters.
Therefore, as a third contribution, we propose a novel privacy module -- the
Convolutional Variational Bottleneck (CVB) -- that can be placed early in a
neural network without suffering from these drawbacks. We conduct an extensive
empirical study on three seminal model architectures and six image
classification datasets. We find that all architectures are susceptible to
gradient leakage attacks, which can be prevented by our proposed CVB. Compared
to PRECODE, we show that our novel privacy module requires fewer trainable
parameters, and thus computational and communication costs, to effectively
preserve privacy.
