  Users are daily exposed to a large volume of harmful content on various
social network platforms. One solution is developing online moderation tools
using Machine Learning techniques. However, the processing of user data by
online platforms requires compliance with privacy policies. Federated Learning
(FL) is an ML paradigm where the training is performed locally on the users'
devices. Although the FL framework complies, in theory, with the GDPR policies,
privacy leaks can still occur. For instance, an attacker accessing the final
trained model can successfully perform unwanted inference of the data belonging
to the users who participated in the training process. In this paper, we
propose a privacy-preserving FL framework for online content moderation that
incorporates Differential Privacy (DP). To demonstrate the feasibility of our
approach, we focus on detecting harmful content on Twitter - but the overall
concept can be generalized to other types of misbehavior. We simulate a text
classifier - in FL fashion - which can detect tweets with harmful content. We
show that the performance of the proposed FL framework can be close to the
centralized approach - for both the DP and non-DP FL versions. Moreover, it has
a high performance even if a small number of clients (each with a small number
of data points) are available for the FL training. When reducing the number of
clients (from 50 to 10) or the data points per client (from 1K to 0.1K), the
classifier can still achieve ~81% AUC. Furthermore, we extend the evaluation to
four other Twitter datasets that capture different types of user misbehavior
and still obtain a promising performance (61% - 80% AUC). Finally, we explore
the overhead on the users' devices during the FL training phase and show that
the local training does not introduce excessive CPU utilization and memory
consumption overhead.
