Print Email Facebook Twitter Design Guidelines for Inclusive Speaker Verification Evaluation Datasets Title Design Guidelines for Inclusive Speaker Verification Evaluation Datasets Author Hutiri, Wiebke (TU Delft Information and Communication Technology) Gorce, Lauriane (Open North) Ding, Aaron Yi (TU Delft Information and Communication Technology) Date 2022 Abstract Speaker verification (SV) provides billions of voice-enabled devices with access control, and ensures the security of voice-driven technologies. As a type of biometrics, it is necessary that SV is unbiased, with consistent and reliable performance across speakers irrespective of their demographic, social and economic attributes. Current SV evaluation practices are insufficient for evaluating bias: they are over-simplified and aggregate users, not representative of usage scenarios encountered in deployment, and consequences of errors are not accounted for. This paper proposes design guidelines for constructing SV evaluation datasets that address these short-comings. We propose a schema for grading the difficulty of utterance pairs, and present an algorithm for generating inclusive SV datasets. We empirically validate our proposed method in a set of experiments on the VoxCeleb1 dataset. Our results confirm that the count of utterance pairs/speaker, and the difficulty grading of utterance pairs have a significant effect on evaluation performance and variability. Our work contributes to the development of SV evaluation practices that are inclusive and fair. Subject auditbiasdesign guidelinesevaluationfairnessspeaker verificationvoice biometrics To reference this document use: http://resolver.tudelft.nl/uuid:3dd6fc8b-3abf-4d57-8241-0acabedc027d DOI https://doi.org/10.21437/Interspeech.2022-10799 Embargo date 2023-07-01 ISSN 2308-457X Source Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022-September, 1293-1297 Event 23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022, 2022-09-18 → 2022-09-22, Incheon, Korea, Republic of Bibliographical note Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. Part of collection Institutional Repository Document type journal article Rights © 2022 Wiebke Hutiri, Lauriane Gorce, Aaron Yi Ding Files PDF toussaint22_interspeech.pdf 573.7 KB Close viewer /islandora/object/uuid:3dd6fc8b-3abf-4d57-8241-0acabedc027d/datastream/OBJ/view