  Federated learning (FL) takes a first step towards privacy-preserving machine
learning by training models while keeping client data local. Models trained
using FL may still leak private client information through model updates during
training. Differential privacy (DP) may be employed on model updates to provide
privacy guarantees within FL, typically at the cost of degraded performance of
the final trained model. Both non-private FL and DP-FL can be solved using
variants of the federated averaging (FedAvg) algorithm. In this work, we
consider a heterogeneous DP setup where clients require varying degrees of
privacy guarantees. First, we analyze the optimal solution to the federated
linear regression problem with heterogeneous DP in a Bayesian setup. We find
that unlike the non-private setup, where the optimal solution for homogeneous
data amounts to a single global solution for all clients learned through
FedAvg, the optimal solution for each client in this setup would be a
personalized one even for homogeneous data. We also analyze the privacy-utility
trade-off for this setup, where we characterize the gain obtained from
heterogeneous privacy where some clients opt for less strict privacy
guarantees. We propose a new algorithm for FL with heterogeneous DP, named
FedHDP, which employs personalization and weighted averaging at the server
using the privacy choices of clients, to achieve better performance on clients'
local models. Through numerical experiments, we show that FedHDP provides up to
$9.27\%$ performance gain compared to the baseline DP-FL for the considered
datasets where $5\%$ of clients opt out of DP. Additionally, we show a gap in
the average performance of local models between non-private and private clients
of up to $3.49\%$, empirically illustrating that the baseline DP-FL might incur
a large utility cost when not all clients require the stricter privacy
guarantees.
