TENSOR EU Project

Part of EU Open Research Repository
Published August 27, 2024 | Version v1
Journal article Open

Stealthy Backdoor Attack Against Federated Learning Through Frequency Domain by Backdoor Neuron Constraint and Model Camouflage

Authors/Creators

  • 1. Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology

Description

Federated Learning (FL) is a beneficial decentralized learning approach for preserving the privacy of local datasets of distributed agents. However, the distributed property of FL and untrustworthy data introducing the vulnerability to backdoor attacks. In this attack scenario, an adversary manipulates its local data with a specific trigger and trains a malicious local model to implant the backdoor. During inference, the global model would misbehave for any input with the trigger to the attacker-chosen prediction. Most existing backdoor attacks against FL focus on bypassing defense mechanisms, without considering the inspection of model parameters on the server. These attacks are susceptible to detection through dynamic clustering based on model parameter similarity. Besides, current methods provide limited imperceptibility of their trigger in the spatial domain. To address these limitations, we propose a stealthy backdoor attack called “Chironex” against FL with an imperceptible trigger in frequency space to deliver attack effectiveness, stealthiness and robustness against various countermeasures on FL. We first design a frequency trigger function to generate an imperceptible frequency trigger to evade human inspection. Then we fully exploit the attacker’s advantage to enhance attack robustness by estimating benign updates and analyzing the impact of the backdoor on model parameters through a task-sensitive neuron searcher. It disguises malicious updates as benign ones by reducing the impact of backdoor neurons that greatly contribute to the backdoor task based on activation value, and encouraging them to update towards benign model parameters trained by the attacker. We conduct extensive experiments on various image classifiers with real-world datasets to provide empirical evidence that Chironex can evade the most recent robust FL aggregation algorithms, and further achieve a distinctly higher attack success rate than existing attacks, without undermining the utility of the global model.

Files

Stealthy_Backdoor_Attack_Against_Federated_Learning_Through_Frequency_Domain_by_Backdoor_Neuron_Constraint_and_Model_Camouflage.pdf

Files (3.8 MB)

Additional details

Identifiers

ISSN
2156-3357

Funding

European Commission
TENSOR - Reliable biomeTric tEchNologies to asSist Police authorities in cOmbating terrorism and oRganized crime 101073920
European Commission
TANGO - Digital Technologies ActiNg as a Gatekeeper to information and data flOws 101070052
European Commission
REWIRE - REWiring the ComposItional Security VeRification and AssurancE of Systems of Systems Lifecycle 101070627

References

  • B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y. Arcas, "Communication-efficient learning of deep networks from decentralized data," in Proc. 20th Int. Conf. Artif. Intell. Statist., 2017, pp. 1273–1282
  • S. Niknam, H. S. Dhillon, and J. H. Reed, "Federated learning for wireless communications: Motivation, opportunities, and challenges," IEEE Commun. Mag., vol. 58, no. 6, pp. 46–51, Jun. 2020.
  • Y. Liu et al., "Vertical federated learning: Concepts, advances, and challenges," IEEE Trans. Knowl. Data Eng., vol. 36, no. 7, pp. 3615–3634, Jul. 2024.
  • L. Fu, H. Zhang, G. Gao, M. Zhang, and X. Liu, "Client selection in federated learning: Principles, challenges, and opportunities," IEEE Internet Things J., vol. 10, no. 24, pp. 21811–21819, Dec. 2023.
  • R. Myrzashova, S. H. Alsamhi, A. V. Shvetsov, A. Hawbani, and X. Wei, "Blockchain meets federated learning in healthcare: A systematic review with challenges and opportunities," IEEE Internet Things J., vol. 10, no. 6, pp. 14418–14437, Aug. 2023
  • J. Zhang, S. Guo, J. Guo, D. Zeng, J. Zhou, and A. Zomaya, "Towards data-independent knowledge transfer in model-heterogeneous federated learning," IEEE Trans. Comput., vol. 72, no. 10, pp. 2888–2901, Oct. 2023
  • H. Li et al., "FedTP: Federated learning by transformer personalization," IEEE Trans. Neural Netw. Learn. Syst., early access, May 23, 2023, doi: 10.1109/TNNLS.2023.3269062
  • M. Kesici, B. Pal, and G. Yang, "Detection of false data injection attacks in distribution networks: A vertical federated learning approach," IEEE Trans. Smart Grid, early access, May 10, 2024, doi: 10.1109/TSG.2024.3399396.
  • I. Dayan et al., "Federated learning for predicting clinical outcomes in patients with COVID-19," Nature Med., vol. 27, no. 10, pp. 1735–1743, 2021.
  • A. Nguyen et al., "Deep federated learning for autonomous driving," in Proc. IEEE Intell. Veh. Symp. (IV), Jun. 2022, pp. 1824–1830
  • G. Baruch, M. Baruch, and Y. Goldberg, "A little is enough: Circumventing defenses for distributed learning," in Proc. Adv. Neural Inf. Process. Syst., vol. 32, 2019, pp. 1–11.
  • A. N. Bhagoji, S. Chakraborty, P. P. Mittal, and S. Calp, "Analyzing federated learning through an adversarial lens," in Proc. 36th Int. Conf. Mach. Learn., May 2019, pp. 634–643
  • C. Xie, K. Huang, P.-Y. Chen, and B. Li, "DBA: Distributed backdoor attacks against federated learning," in Proc. Int. Conf. Learn. Represent., 2019, ppp. 1–19
  • E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, "How to backdoor federated learning," in Proc. Int. Conf. Artif. Intell. Statist., 2020, pp. 2938–2948
  • H. Wang et al., "Attack of the tails: Yes, you really can backdoor federated learning," in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 16070–16084.
  • H. Li et al., "3DFed: Adaptive and extensible framework for covert backdoor attack in federated learning," in Proc. IEEE Symp. Secur. Privacy (SP), May 2023, pp. 1893–1907
  • H. Zhang, J. Jia, J. Chen, L. Lin, and D. Wu, "A3FL: Adversarially adaptive backdoor attacks to federated learning," in Proc. Adv. Neural Inf. Process. Syst., vol. 36, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., New Orleans, LA, USA: Curran Associates, 2023, pp. 61213–61233. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/c07d71ff0bc04 2e4b9acd626a79597fa-Paper-Conference.pdf
  • R. J. G. B. Campello, D. Moulavi, and J. Sander, "Density-based clustering based on hierarchical density estimates," in Proc. Pacific– Asia Conf. Knowl. Discovery Data Mining. Gold Coast, QLD, Australia: Springer, 2013, pp. 160–172.
  • Z. Zhang et al., "Neurotoxin: Durable backdoors in federated learning," in Proc. 39th Int. Conf. Mach. Learn., Jul. 2022, pp. 26429–26446.
  • D. Yin, Y. Chen, R. Kannan, and P. Bartlett, "Byzantine-robust distributed learning: Towards optimal statistical rates," in Proc. Int. Conf. Mach. Learn., Jul. 2018, pp. 5650–5659
  • M. S. Ozdayi, M. Kantarcioglu, and Y. R. Gel, "Defending against backdoors in federated learning with robust learning rate," in Proc. AAAI Conf. Artif. Intell., 2021, vol. 35, no. 10, pp. 9268–9276
  • T. D. Nguyen et al., "FLAME: Taming backdoors in federated learning," in Proc. 31st USENIX Security Symp. (USENIX Security), Boston, MA, USA, Aug. 2022, pp. 1415–1432
  • P. Rieger, T. D. Nguyen, M. Miettinen, and A.-R. Sadeghi, "DeepSight: Mitigating backdoor attacks in federated learning through deep model inspection," in Proc. Netw. Distrib. Syst. Secur. Symp., 2022, pp. 1–18
  • D. Yin, R. G. Lopes, J. Shlens, E. D. Cubuk, and J. Gilmer, "A Fourier perspective on model robustness in computer vision," in Proc. Adv. Neural Inf. Process. Syst., vol. 32, 2019
  • Z.-Q. J. Xu, Y. Zhang, and Y. Xiao, "Training behavior of deep neural network in frequency domain," in Proc. Int. Conf. Neural Inf. Process., Sydney, NSW, Australia. Springer, 2019, pp. 264–274
  • T. Wang, Y. Yao, F. Xu, S. An, H. Tong, and T. Wang, "An invisible black-box backdoor attack through frequency domain," in Proc. Eur. Conf. Comput. Vis. Tel Aviv-Yafo, Israel: Springer, 2022, pp. 396–413
  • Y. Feng, B. Ma, J. Zhang, S. Zhao, Y. Xia, and D. Tao, "FIBA: Frequency-injection based backdoor attack in medical image analysis," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2022, pp. 20876–20885
  • K. Bonawitz et al., "Practical secure aggregation for federated learning on user-held data," 2016, arXiv:1611.04482
  • B. Zhao, P. Sun, T. Wang, and K. Jiang, "FedInv: Byzantine-robust federated learning by inversing local model updates," in Proc. AAAI Conf. Artif. Intell., 2022, vol. 36, no. 8, pp. 9171–9179
  • T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," in Proc. 3rd Mach. Learn. Syst. Conf., vol. 2, 2020, pp. 429–450
  • Y. Liu et al., "A communication efficient collaborative learning framework for distributed features," 2019, arXiv:1912.11187
  • T. Li, S. Hu, A. Beirami, and V. Smith, "Ditto: Fair and robust federated learning through personalization," in Proc. Int. Conf. Mach. Learn., 2021, pp. 6357–6368
  • V. Tolpegin, S. Truex, M. E. Gursoy, and L. Liu, "Data poisoning attacks against federated learning systems," in Proc. Eur. Symp. Res. Comput. Secur. Guildford, U.K.: Springer, 2020, pp. 480–501.
  • Y. Dai and S. Li, "Chameleon: Adapting to peer images for planting durable backdoors in federated learning," in Proc. 40th Int. Conf. Mach. Learn., 2023, pp. 6712–6725
  • A. Reisizadeh, F. Farnia, R. Pedarsani, and A. Jadbabaie, "Robust federated learning: The case of affine distribution shifts," in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 21554–21565
  • V. Shejwalkar and A. Houmansadr, "Manipulating the Byzantine: Optimizing model poisoning attacks and defenses for federated learning," in Proc. Netw. Distrib. Syst. Secur. Symp., 2021, pp. 1–19.
  • X. Cao, J. Jia, and N. Z. Gong, "Provably secure federated learning against malicious clients," in Proc. AAAI Conf. Artif. Intell., 2021, vol. 35, no. 8, pp. 6885–6893
  • X. Cao, M. Fang, J. Liu, and N. Z. Gong, "FLTrust: Byzantine-robust federated learning via trust bootstrapping," in Proc. Netw. Distrib. Syst. Secur. Symp., 2021, pp. 1–18
  • C. Xie, M. Chen, P.-Y. Chen, and B. Li, "CRFL: Certifiably robust federated learning against backdoor attacks," in Proc. Int. Conf. Mach. Learn., 2021, pp. 11372–11382.
  • Z. Zhang, X. Cao, J. Jia, and N. Z. Gong, "FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients," in Proc. 28th ACM SIGKDD Conf. Knowl. Discovery Data Mining, Aug. 2022, pp. 2545–2555.
  • X. Fang and M. Ye, "Robust federated learning with noisy and heterogeneous clients," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2022, pp. 10072–10081.
  • P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, "Machine learning with adversaries: Byzantine tolerant gradient descent," in Proc. Adv. Neural Inf. Process. Syst., vol. 30, 2017, pp. 1–11
  • E. M. El Mhamdi, R. Guerraoui, and S. Rouault, "The hidden vulnerability of distributed learning in Byzantium," in Proc. 35th Int. Conf. Mach. Learn., 2018, pp. 3521–3530
  • Z. Sun, P. Kairouz, A. T. Suresh, and H. B. McMahan, "Can you really backdoor federated learning?" 2019, arXiv:1911.07963
  • J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, "signSGD: Compressed optimisation for non-convex problems," in Proc. 35th Int. Conf. Mach. Learn., J. Dy and A. Krause, Eds., vol. 80, Jul. 2018, pp. 560–569
  • A. Panda, S. Mahloujifar, A. N. Bhagoji, S. Chakraborty, and P. Mittal, "SparseFed: Mitigating model poisoning attacks in federated learning with sparsification," in Proc. Int. Conf. Artif. Intell. Statist., vol. 151, G. Camps-Valls, F. J. R. Ruiz, and I. Valera, Eds., Mar. 2022, pp. 7587–7624
  • M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. N. Rotaru, and B. Li, "Manipulating machine learning: Poisoning attacks and countermeasures for regression learning," in Proc. IEEE Symp. Security Privacy, Oct. 2018, pp. 19–35
  • Y. LeCun. (1998). The Mnist Database of Handwritten Digits. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  • H. Xiao, K. Rasul, and R. Vollgraf, "Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms," 2017, arXiv:1708.07747.
  • A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," Univ. Toronto, Toronto, ON, Canada, Tech. Rep. 0, 2009.
  • S. Caldas et al., "LEAF: A benchmark for federated settings," 2018, arXiv:1812.01097
  • O. Russakovsky et al., "ImageNet large scale visual recognition challenge," Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, Dec. 2015
  • K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770–778.
  • X. Chen, C. Liu, B. Li, K. Lu, and D. Song, "Targeted backdoor attacks on deep learning systems using data poisoning," 2017, arXiv:1712.05526.
  • V. Shejwalkar, A. Houmansadr, P. Kairouz, and D. Ramage, "Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning," in Proc. IEEE Symp. Secur. Privacy (SP), May 2022, pp. 1354–1371