We propose an ensemble learning framework with Poisson sub-sampling to effectively train a collection of teacher models to issue some differential privacy (DP) guarantee for training data. Through boosting under DP, a student model derived from the training data suffers little model degradation from the models trained with no privacy protection. Our proposed solution leverages upon two mechanisms, namely: (i) a privacy budget amplification via Poisson sub-sampling to train a target prediction model that requires less noise to achieve a same level of privacy budget, and (ii) a combination of the sub-sampling technique and an ensemble teacher-student learning framework that introduces DP-preserving noise at the output of the teacher models and transfers DP-preserving properties via noisy labels. Privacy-preserving student models are then trained with the noisy labels to learn the knowledge with DP-protection from the teacher model ensemble. Experimental evidences on spoken command recognition and continuous speech recognition of Mandarin speech show that our proposed framework greatly outperforms existing state-of-the-art DP-preserving algorithms in both ASR tasks.
Yang C.-H.H., Qi J., Siniscalchi S.M., Lee C.-H. (2022). An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition. In 2022 13th International Symposium on Chinese Spoken Language Processing, ISCSLP 2022 (pp. 1-5). 345 E 47TH ST, NEW YORK, NY 10017 USA : Institute of Electrical and Electronics Engineers Inc. [10.1109/ISCSLP57327.2022.10038060].
An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition
Siniscalchi S. M.Co-ultimo
Writing – Original Draft Preparation
;
2022-01-01
Abstract
We propose an ensemble learning framework with Poisson sub-sampling to effectively train a collection of teacher models to issue some differential privacy (DP) guarantee for training data. Through boosting under DP, a student model derived from the training data suffers little model degradation from the models trained with no privacy protection. Our proposed solution leverages upon two mechanisms, namely: (i) a privacy budget amplification via Poisson sub-sampling to train a target prediction model that requires less noise to achieve a same level of privacy budget, and (ii) a combination of the sub-sampling technique and an ensemble teacher-student learning framework that introduces DP-preserving noise at the output of the teacher models and transfers DP-preserving properties via noisy labels. Privacy-preserving student models are then trained with the noisy labels to learn the knowledge with DP-protection from the teacher model ensemble. Experimental evidences on spoken command recognition and continuous speech recognition of Mandarin speech show that our proposed framework greatly outperforms existing state-of-the-art DP-preserving algorithms in both ASR tasks.File | Dimensione | Formato | |
---|---|---|---|
An_Ensemble_Teacher-Student_Learning_Approach_with_Poisson_Sub-sampling_to_Differential_Privacy_Preserving_Speech_Recognition.pdf
Solo gestori archvio
Descrizione: main document
Tipologia:
Versione Editoriale
Dimensione
412.4 kB
Formato
Adobe PDF
|
412.4 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
2210.06382v1.pdf
accesso aperto
Tipologia:
Pre-print
Dimensione
372.4 kB
Formato
Adobe PDF
|
372.4 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.