SG
S. Goldie
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
1 records found
1
Speech deepfake detectors can perform extremely well when they are trained and tested on one benchmark, yet perform poorly on recordings from other sources. This raises a concern: a detector may learn differences between datasets rather than a general distinction between genuine and deepfake speech. This thesis tests whether three detectors—RawNet2, LCNN, andWav2Vec-AASIST—use such dataset-related shortcuts.
First, a data audit of four public datasets shows that simple classifiers can identify which dataset a recording came from using several kinds of audio measurements. This establishes that source-related clues are available, but not that the deepfake detectors use them. To test detector reliance directly, the thesis introduces the mixing ratio protocol. The protocol changes which of two datasets supplies the genuine and deepfake recordings during training, while the evaluation recordings remain fixed. If a detector uses dataset source as a shortcut, reversing this relationship should cause its decisions to change.
This is what occurs. Across all three detectors, reversing the training composition moves the equal error rate (EER) on the same test recordings from 0.003–0.028 to 0.78–0.87. An EER of 0 is perfect, 0.5 is chance, and a value above 0.5 means that the predictions are systematically inverted. A similar response appears with a second dataset pair. By contrast, the results are much flatter when the same procedure is applied to two groups of speakers from one dataset. This control shows that the mixing procedure does not create the main response by itself.
Further experiments test whether one audio property explains the dependence. Reducing the frequency range, matching the average signal level, and removing low-frequency energy do not remove it. Dataset source can also be predicted from several different kinds of measurements. These findings are consistent with source information being represented through several overlapping or untested properties, but they do not identify which properties each detector used. Overall, dataset identity is available as an exploitable shortcut in speech deepfake detection. Future benchmarks should therefore include diverse recording and generation pipelines in both genuine and deepfake classes. ...
First, a data audit of four public datasets shows that simple classifiers can identify which dataset a recording came from using several kinds of audio measurements. This establishes that source-related clues are available, but not that the deepfake detectors use them. To test detector reliance directly, the thesis introduces the mixing ratio protocol. The protocol changes which of two datasets supplies the genuine and deepfake recordings during training, while the evaluation recordings remain fixed. If a detector uses dataset source as a shortcut, reversing this relationship should cause its decisions to change.
This is what occurs. Across all three detectors, reversing the training composition moves the equal error rate (EER) on the same test recordings from 0.003–0.028 to 0.78–0.87. An EER of 0 is perfect, 0.5 is chance, and a value above 0.5 means that the predictions are systematically inverted. A similar response appears with a second dataset pair. By contrast, the results are much flatter when the same procedure is applied to two groups of speakers from one dataset. This control shows that the mixing procedure does not create the main response by itself.
Further experiments test whether one audio property explains the dependence. Reducing the frequency range, matching the average signal level, and removing low-frequency energy do not remove it. Dataset source can also be predicted from several different kinds of measurements. These findings are consistent with source information being represented through several overlapping or untested properties, but they do not identify which properties each detector used. Overall, dataset identity is available as an exploitable shortcut in speech deepfake detection. Future benchmarks should therefore include diverse recording and generation pipelines in both genuine and deepfake classes. ...
Speech deepfake detectors can perform extremely well when they are trained and tested on one benchmark, yet perform poorly on recordings from other sources. This raises a concern: a detector may learn differences between datasets rather than a general distinction between genuine and deepfake speech. This thesis tests whether three detectors—RawNet2, LCNN, andWav2Vec-AASIST—use such dataset-related shortcuts.
First, a data audit of four public datasets shows that simple classifiers can identify which dataset a recording came from using several kinds of audio measurements. This establishes that source-related clues are available, but not that the deepfake detectors use them. To test detector reliance directly, the thesis introduces the mixing ratio protocol. The protocol changes which of two datasets supplies the genuine and deepfake recordings during training, while the evaluation recordings remain fixed. If a detector uses dataset source as a shortcut, reversing this relationship should cause its decisions to change.
This is what occurs. Across all three detectors, reversing the training composition moves the equal error rate (EER) on the same test recordings from 0.003–0.028 to 0.78–0.87. An EER of 0 is perfect, 0.5 is chance, and a value above 0.5 means that the predictions are systematically inverted. A similar response appears with a second dataset pair. By contrast, the results are much flatter when the same procedure is applied to two groups of speakers from one dataset. This control shows that the mixing procedure does not create the main response by itself.
Further experiments test whether one audio property explains the dependence. Reducing the frequency range, matching the average signal level, and removing low-frequency energy do not remove it. Dataset source can also be predicted from several different kinds of measurements. These findings are consistent with source information being represented through several overlapping or untested properties, but they do not identify which properties each detector used. Overall, dataset identity is available as an exploitable shortcut in speech deepfake detection. Future benchmarks should therefore include diverse recording and generation pipelines in both genuine and deepfake classes.
First, a data audit of four public datasets shows that simple classifiers can identify which dataset a recording came from using several kinds of audio measurements. This establishes that source-related clues are available, but not that the deepfake detectors use them. To test detector reliance directly, the thesis introduces the mixing ratio protocol. The protocol changes which of two datasets supplies the genuine and deepfake recordings during training, while the evaluation recordings remain fixed. If a detector uses dataset source as a shortcut, reversing this relationship should cause its decisions to change.
This is what occurs. Across all three detectors, reversing the training composition moves the equal error rate (EER) on the same test recordings from 0.003–0.028 to 0.78–0.87. An EER of 0 is perfect, 0.5 is chance, and a value above 0.5 means that the predictions are systematically inverted. A similar response appears with a second dataset pair. By contrast, the results are much flatter when the same procedure is applied to two groups of speakers from one dataset. This control shows that the mixing procedure does not create the main response by itself.
Further experiments test whether one audio property explains the dependence. Reducing the frequency range, matching the average signal level, and removing low-frequency energy do not remove it. Dataset source can also be predicted from several different kinds of measurements. These findings are consistent with source information being represented through several overlapping or untested properties, but they do not identify which properties each detector used. Overall, dataset identity is available as an exploitable shortcut in speech deepfake detection. Future benchmarks should therefore include diverse recording and generation pipelines in both genuine and deepfake classes.