ASR Systems for Deaf/Hard-of Hearing People

Abstract

Automatic Speech Recognition (ASR) systems perform worse on the speech of deaf and hard-of-hearing (d/Dhh) individuals than on standard speech, creating a substantial accessibility barrier. Improving ASR for this demographic is typically made complicated by severe data scarcity and high inter-speaker heterogeneity. This thesis investigates strategies to reduce the Word Error Rate (WER) for Dutch d/Dhh speech without relying on large, specialized training datasets. First, a prosodic analysis contrasting d/Dhh speakers with standard-hearing controls identifies key deviations in speech duration, loudness, and pitch. To mitigate data scarcity, standard Dutch read speech is augmented to mirror these identified prosodic differences, followed by parameter-efficient fine-tuning (LoRA) on the modified data. While this targeted prosodic augmentation successfully closes the acoustic gap between standard and d/Dhh speech, it does not improve WER, indicating that other differences are the primary drivers of recognition errors. Conversely, transitioning from population-level adaptation to speaker-level personalization yields substantial improvements. Fine-tuning a base model on a small sample of an individual speaker's voice reduces WER, with the most gains observed for the more difficult-to-transcribe speakers.