  Federated Learning (FL) enables distributed participants (e.g., mobile
devices) to train a global model without sharing data directly to a central
server. Recent studies have revealed that FL is vulnerable to gradient
inversion attack (GIA), which aims to reconstruct the original training samples
and poses high risk against the privacy of clients in FL. However, most
existing GIAs necessitate control over the server and rely on strong prior
knowledge including batch normalization and data distribution information. In
this work, we propose Client-side poisoning Gradient Inversion (CGI), which is
a novel attack method that can be launched from clients. For the first time, we
show the feasibility of a client-side adversary with limited knowledge being
able to recover the training samples from the aggregated global model. We take
a distinct approach in which the adversary utilizes a malicious model that
amplifies the loss of a specific targeted class of interest. When honest
clients employ the poisoned global model, the gradients of samples belonging to
the targeted class are magnified, making them the dominant factor in the
aggregated update. This enables the adversary to effectively reconstruct the
private input belonging to other clients using the aggregated update. In
addition, our CGI also features its ability to remain stealthy against
Byzantine-robust aggregation rules (AGRs). By optimizing malicious updates and
blending benign updates with a malicious replacement vector, our method remains
undetected by these defense mechanisms. To evaluate the performance of CGI, we
conduct experiments on various benchmark datasets, considering representative
Byzantine-robust AGRs, and exploring diverse FL settings with different levels
of adversary knowledge about the data. Our results demonstrate that CGI
consistently and successfully extracts training input in all tested scenarios.
