Accurate Differentially Private Deep Learning on the Edge

Han, Rui; Li, Dong; Ouyang, Junyan; Liu, Chi Harold; Wang, Guoren; Wu, Dapeng Oliver; Chen, Lydia Y.

doi:10.1109/TPDS.2021.3064345

Accurate Differentially Private Deep Learning on the Edge

Title

Accurate Differentially Private Deep Learning on the Edge

Author

Han, Rui (Beijing Institute of Technology)
Li, Dong (Beijing Institute of Technology)
Ouyang, Junyan (Beijing Institute of Technology)
Liu, Chi Harold
Wang, Guoren (Beijing Institute of Technology)
Wu, Dapeng Oliver (University of Florida)
Chen, Lydia Y. (TU Delft Distributed Systems)

Date

2021

Abstract

Deep learning (DL) models are increasingly built on federated edge participants holding local data. To enable insight extractions without the risk of information leakage, DL training is usually combined with differential privacy (DP). The core theme is to tradeoff learning accuracy by adding statistically calibrated noises, particularly to local gradients of edge learners, during model training. However, this privacy guarantee unfortunately degrades model accuracy due to edge learners' local noises, and the global noise aggregated at the central server. Existing DP frameworks for edge focus on local noise calibration via gradient clipping techniques, overlooking the heterogeneity and dynamic changes of local gradients, and their aggregated impact on accuracy. In this article, we present a systematical analysis that unveils the influential factors capable of mitigating local and aggregated noises, and design PrivateDL to leverage these factors in noise calibration so as to improve model accuracy while fulfilling privacy guarantee. PrivateDL features on: (i) sampling-based sensitivity estimation for local noise calibration and (ii) combining large batch sizes and critical data identification in global training. We implement PrivateDL on the popular Laplace/Gaussian DP mechanisms and demonstrate its effectiveness using Intel BigDL workloads, i.e., considerably improving model accuracy by up to 5X when comparing against existing DP frameworks.

Subject

Biological system modeling
Data models
Deep learning
Differential privacy
differential privacy
federated learning
model accuracy
Privacy
Sensitivity
Servers
Training
Model accuracy
Federated learning