Audio-Visual Speech Recognition in MISP2021 Challenge
Dataset Release and Deep Analysis
Hang Chen (University of Science and Technology of China)
Jun Du (University of Science and Technology of China)
Yusheng Dai (University of Science and Technology of China)
Chin-Hui Lee (Georgia Institute of Technology)
Sabato Marco Siniscalchi (Georgia Institute of Technology, University of Enna Kore)
Shinji Watanabe (Carnegie Mellon University)
Odette Scharenborg (TU Delft - Multimedia Computing)
Jingdong Chen (iFlytek)
Bao Cai Yin (iFlytek)
Jia Pan (iFlytek)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
In this paper, we present the updated Audio-Visual Speech Recognition (AVSR) corpus of MISP2021 challenge, a large-scale audio-visual Chinese conversational corpus consisting of 141h audio and video data collected by far/middle/near microphones and far/middle cameras in 34 real-home TV rooms. To our best knowledge, our corpus is the first distant multi-microphone conversational Chinese audio-visual corpus and the first large vocabulary continuous Chinese lip-reading dataset in the adverse home-tv scenario. Moreover, we make a deep analysis of the corpus and conduct a comprehensive ablation study of all audio and video data in the audio-only/video-only/audiovisual systems. Error analysis shows video modality supplement acoustic information degraded by noise to reduce deletion errors and provide discriminative information in overlapping speech to reduce substitution errors. Finally, we also design a set of experiments such as frontend, data augmentation and end-to-end models for providing the direction of potential future work. The corpus and the code are released to promote the research not only in speech area but also for the computer vision area and cross-disciplinary research.