Privacy–Preserving Online Content Moderation: A Federated Learning Use Case

Users are exposed to a large volume of harmful content that appears daily on various social network platforms. One solution to users’ protection is developing online moderation tools using Machine Learning (ML) techniques for automatic detection or content filtering. On the other hand, the processing of user data by social network platforms requires compliance with privacy policies. Federated Learning (FL) is one of the proposed solutions, where the training of ML models is performed locally on the users’ devices (the FL “clients”), and only the model updates are shared with the central server (the FL central aggregator). Although the raw data never leave the users’ devices, privacy leaks can still occur. One threat is data (or membership) inference attacks, where an attacker accessing the final trained model (the FL output) can successfully perform unwanted inference of the data belonging to the users who participated in the training process. In this paper, we propose a privacy–preserving FL framework for online content moderation that incorporates Differential Privacy (DP). To demonstrate the feasibility of our approach, we focus on detecting harmful content on Twitter – but the overall concept can be generalized to other types of misbehavior. We simulate a text classifier — in a distributed FL fashion — which can detect tweets with harmful content. We show that the performance of the proposed FL framework can be close to the centralized approach – for both the DP and non–DP FL versions. Moreover, it has a high performance even if a small number of clients (each with a small number of data points) are available for the FL training. When reducing the number of clients (from fifty to ten) or the data points per client (from 1K to 0.1K), the classifier can still achieve ∼81% AUC. Furthermore, we extend the evaluation to four other Twitter datasets that capture different types of user misbehavior and still obtain a promising performance (61% – 80% AUC). Finally, we explore the overhead on the users’ devices during the FL training phase and show that the local training does not introduce excessive CPU utilization and memory consumption overhead.