Title
Audio-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis
Author
Chen, Hang (University of Science and Technology of China)
Du, Jun (University of Science and Technology of China)
Dai, Yusheng (University of Science and Technology of China)
Lee, Chin Hui (Georgia Institute of Technology)
Siniscalchi, Sabato Marco (Georgia Institute of Technology; University of Enna Kore)
Watanabe, Shinji (Carnegie Mellon University)
Scharenborg, O.E. (TU Delft Multimedia Computing) 
Chen, Jingdong (iFlytek)
Yin, Bao Cai (iFlytek)
Pan, Jia (iFlytek)
Date
2022
Abstract
In this paper, we present the updated Audio-Visual Speech Recognition (AVSR) corpus of MISP2021 challenge, a large-scale audio-visual Chinese conversational corpus consisting of 141h audio and video data collected by far/middle/near microphones and far/middle cameras in 34 real-home TV rooms. To our best knowledge, our corpus is the first distant multi-microphone conversational Chinese audio-visual corpus and the first large vocabulary continuous Chinese lip-reading dataset in the adverse home-tv scenario. Moreover, we make a deep analysis of the corpus and conduct a comprehensive ablation study of all audio and video data in the audio-only/video-only/audiovisual systems. Error analysis shows video modality supplement acoustic information degraded by noise to reduce deletion errors and provide discriminative information in overlapping speech to reduce substitution errors. Finally, we also design a set of experiments such as frontend, data augmentation and end-to-end models for providing the direction of potential future work. The corpus and the code are released to promote the research not only in speech area but also for the computer vision area and cross-disciplinary research.
Subject
Audio-visual
data augmentation
speech enhancement
speech recognition
To reference this document use:
http://resolver.tudelft.nl/uuid:f245d102-4dec-4da2-b913-0a632dc1b769
DOI
https://doi.org/10.21437/Interspeech.2022-10483
Embargo date
2023-07-01
ISSN
2308-457X
Source
Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2022-September, 1766-1770
Event
23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022, 2022-09-18 → 2022-09-22, Incheon, Korea, Republic of
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
Part of collection
Institutional Repository
Document type
journal article
Rights
© 2022 Hang Chen, Jun Du, Yusheng Dai, Chin Hui Lee, Sabato Marco Siniscalchi, Shinji Watanabe, O.E. Scharenborg, Jingdong Chen, Bao Cai Yin, Jia Pan