Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models

Aydin Abadi; Vishnu Asutosh Dasu; Sumanta Sarkar

Paper 2024/1151

Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models

Aydin Abadi

, Newcastle University

Vishnu Asutosh Dasu, Pennsylvania State University

Sumanta Sarkar

, University of Warwick

Abstract

Deduplication is a vital preprocessing step that enhances machine learning model performance and saves training time and energy. However, enhancing federated learning through deduplication poses challenges, especially regarding scalability and potential privacy violations if deduplication involves sharing all clients’ data. In this paper, we address the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD). It efficiently removes duplicates from multiple clients’ datasets without compromising data privacy. EP-MPD is constructed in a modular fashion, utilizing two novel variants of the Private Set Intersection protocol. Our extensive experiments demonstrate the significant benefits of deduplication in federated learning of large language models. For instance, we observe up to 19.61% improvement in perplexity and up to 27.95% reduction in running time. EP-MPD effectively balances privacy and performance in federated learning, making it a valuable solution for large-scale applications.

Metadata

Available format(s): PDF
Category: Applications
Publication info: Preprint.
Keywords: Federated Learning Private Set Intersection Privacy-Preserving Machine Learning
Contact author(s): aydin abadi @ ncl ac uk
vdasu @ psu edu
sumanta sarkar @ warwick ac uk
History: 2024-07-19: approved; 2024-07-15: received; See all versions
Short URL: https://ia.cr/2024/1151
License: CC BY

BibTeX

@misc{cryptoeprint:2024/1151,
      author = {Aydin Abadi and Vishnu Asutosh Dasu and Sumanta Sarkar},
      title = {Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models},
      howpublished = {Cryptology ePrint Archive, Paper 2024/1151},
      year = {2024},
      note = {\url{https://eprint.iacr.org/2024/1151}},
      url = {https://eprint.iacr.org/2024/1151}
}