Federated influencer learning for secure and efficient collaborative learning in realistic medical database environment.
Collaborative learning
Federated learning
Knowledge distillation
On-device
Privacy
Security
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
30 Sep 2024
30 Sep 2024
Historique:
received:
01
07
2024
accepted:
22
09
2024
medline:
1
10
2024
pubmed:
1
10
2024
entrez:
30
9
2024
Statut:
epublish
Résumé
Enhancing deep learning performance requires extensive datasets. Centralized training raises concerns about data ownership and security. Additionally, large models are often unsuitable for hospitals due to their limited resource capacities. Federated learning (FL) has been introduced to address these issues. However, FL faces challenges such as vulnerability to attacks, non-IID data, reliance on a central server, high communication overhead, and suboptimal model aggregation. Furthermore, FL is not optimized for realistic hospital database environments, where data are dynamically accumulated. To overcome these limitations, we propose federated influencer learning (FIL) as a secure and efficient collaborative learning paradigm. Unlike the server-client model of FL, FIL features an equal-status structure among participants, with an administrator overseeing the overall process. FIL comprises four stages: local training, qualification, screening, and influencing. Local training is similar to vanilla FL, except for the optional use of a shared dataset. In the qualification stage, participants are classified as influencers or followers. During the screening stage, the integrity of the logits from the influencer is examined. If the integrity is confirmed, the influencer shares their knowledge with the others. FIL is more secure than FL because it eliminates the need for model-parameter transactions, central servers, and generative models. Additionally, FIL supports model-agnostic training. These features make FIL particularly promising for fields such as healthcare, where maintaining confidentiality is crucial. Our experiments demonstrated the effectiveness of FIL, which outperformed several FL methods on large medical (X-ray, MRI, and PET) and natural (CIFAR-10) image dataset in a dynamically accumulating database environment, with consistently higher precision, recall, Dice score, and lower standard deviation between participants. In particular, in the PET dataset, FIL achieved about a 40% improvement in Dice score and recall.
Identifiants
pubmed: 39349569
doi: 10.1038/s41598-024-73863-1
pii: 10.1038/s41598-024-73863-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
22729Informations de copyright
© 2024. The Author(s).
Références
Krupa, A. J. D., Dhanalakshmi, S., Lai, K. W., Tan, Y. & Wu, X. An iomt enabled deep learning framework for automatic detection of fetal qrs: A solution to remote prenatal care. In Journal of King Saud University-Computer and Information Sciences34, 7200–7211 (2022) (Elsevier).
doi: 10.1016/j.jksuci.2022.07.002
Achiam, J. et al. Gpt-4 technical report. In arXiv preprint arXiv:2303.08774 (2023).
Li, Q. et al. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. In IEEE Transactions on Knowledge and Data Engineering (IEEE, 2021).
Choi, S. J., Johnson, M. E. & Lee, J. An event study of data breaches and hospital it spending. In Health Policy and Technology9, 372–378 (2020) (Elsevier).
doi: 10.1016/j.hlpt.2020.04.008
Ali, M., Naeem, F., Tariq, M. & Kaddoum, G. Federated learning for privacy preservation in smart healthcare systems: A comprehensive survey. IEEE journal of biomedical and health informatics27, 778–789 (2022).
doi: 10.1109/JBHI.2022.3181823
El Ouazzani, Z., El Bakkali, H. & Sadki, S. Privacy preserving in digital health: main issues, technologies, and solutions. In Research Anthology on Privatizing and Securing Data, 1503–1526 (IGI Global, 2021).
McMahan, B., Moore, E., Ramage, D., Hampson, S. & y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282 (PMLR, 2017).
Wu, X. et al. A novel centralized federated deep fuzzy neural network with multi-objectives neural architecture search for epistatic detection. In IEEE Transactions on Fuzzy Systems (IEEE, 2024).
Rieke, N. et al. The future of digital health with federated learning. In NPJ digital medicine3, 119 (2020) (Nature Publishing Group UK London).
doi: 10.1038/s41746-020-00323-1
Tan, A. Z., Yu, H., Cui, L. & Yang, Q. Towards personalized federated learning. In IEEE Transactions on Neural Networks and Learning Systems (IEEE, 2022).
Kairouz, P. et al. Advances and open problems in federated learning. In Foundations and Trends® in Machine Learning14, 1–210 (2021) (Now Publishers, Inc.).
doi: 10.1561/2200000083
Liu, B., Lv, N., Guo, Y. & Li, Y. Recent advances on federated learning: A systematic survey. Neurocomputing 128019 (2024).
Zhao, Y. et al. Federated learning with non-iid data. In arXiv preprint arXiv:1806.00582 (2018).
Qi, P. et al. Model aggregation techniques in federated learning: A comprehensive survey. In Future Generation Computer Systems (Elsevier, 2023).
Lyu, L., Yu, H. & Yang, Q. Threats to federated learning: A survey. In arXiv preprint arXiv:2003.02133 (2020).
Tolpegin, V., Truex, S., Gursoy, M. E. & Liu, L. Data poisoning attacks against federated learning systems. In Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14–18, 2020, Proceedings, Part I 25, 480–501 (Springer, 2020).
Mammen, P. M. Federated learning: Opportunities and challenges. In arXiv preprint arXiv:2101.05428 (2021).
Wu, X., Wei, Y., Jiang, T., Wang, Y. & Jiang, S. A micro-aggregation algorithm based on density partition method for anonymizing biomedical data. In Current Bioinformatics14, 667–675 (2019) (Bentham Science Publishers).
doi: 10.2174/1574893614666190416152025
Li, T. et al. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems2, 429–450 (2020).
Li, Q., He, B. & Song, D. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10713–10722 (2021).
Mendieta, M. et al. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8397–8406 (2022).
Karimireddy, S. P. et al. Scaffold: Stochastic controlled averaging for federated learning. In International Conference On Machine Learning, 5132–5143 (PMLR, 2020).
Li, T., Hu, S., Beirami, A. & Smith, V. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, 6357–6368 (PMLR, 2021).
Zhang, J. et al. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, 26311–26329 (PMLR, 2022).
Li, Z., Zhang, J., Liu, L. & Liu, J. Auditing privacy defenses in federated learning via generative gradient leakage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10132–10142 (2022).
Li, J. et al. Ressfl: A resistance transfer framework for defending model inversion attack in split federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10194–10202 (2022).
Li, D. & Wang, J. Fedmd: Heterogenous federated learning via model distillation. In arXiv preprint arXiv:1910.03581 (2019).
Lin, T., Kong, L., Stich, S. U. & Jaggi, M. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems33, 2351–2363 (2020).
Shen, T. et al. Federated mutual learning. In arXiv preprint arXiv:2006.16765 (2020).
Gong, X. et al. Ensemble attention distillation for privacy-preserving federated learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15076–15086 (2021).
Zhang, L., Shen, L., Ding, L., Tao, D. & Duan, L.-Y. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10174–10183 (2022).
Wang, H. et al. Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20412–20421 (2023).
Wang, L. & Yoon, K.-J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. In IEEE Transactions On Pattern Analysis and Machine Intelligence, vol. 44, 3048–3068 (IEEE, 2021).
Gou, J., Yu, B., Maybank, S. J. & Tao, D. Knowledge distillation: A survey. In International Booktitle of Computer Vision129, 1789–1819 (2021) (Springer).
doi: 10.1007/s11263-021-01453-z
Li, T., Sanjabi, M., Beirami, A. & Smith, V. Fair resource allocation in federated learning. In arXiv preprint arXiv:1905.10497 (2019).
Fang, X. & Ye, M. Robust federated learning with noisy and heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10072–10081 (2022).
Ma, X., Zhang, J., Guo, S. & Xu, W. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10092–10101 (2022).
Duan, J.-h., Li, W., Zou, D., Li, R. & Lu, S. Federated learning with data-agnostic distribution fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8074–8083 (2023).
Zhang, Y., Xiang, T., Hospedales, T. M. & Lu, H. Deep mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4320–4328 (2018).
Guo, Q. et al. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11020–11029 (2020).
Wu, G. & Gong, S. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence35, 10302–10310 (2021).
doi: 10.1609/aaai.v35i12.17234
Li, S. et al. Distilling a powerful student model via online knowledge distillation. In IEEE Transactions on Neural Networks and Learning Systems (IEEE, 2022).
Irvin, J. et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence33, 590–597 (2019).
Krizhevsky, A. et al. Learning multiple layers of features from tiny images (ON, Canada, Toronto, 2009).
Menze, B. H. et al. The multimodal brain tumor image segmentation benchmark (brats). In IEEE Transactions on Medical Imaging34, 1993–2024 (2014) (IEEE).
doi: 10.1109/TMI.2014.2377694
Andrearczyk, V. et al. Overview of the hecktor challenge at miccai 2021: automatic head and neck tumor segmentation and outcome prediction in pet/ct images. In 3D head and neck tumor segmentation in PET/CT challenge, 1–37 (Springer, 2021).
Van de Ven, G. M. & Tolias, A. S. Three scenarios for continual learning. In arXiv preprint arXiv:1904.07734 (2019).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (2016).