NTCD-TIMIT

NTCD-TIMIT: A New Database and Baseline for Noise-robust Audio-visual Speech Recognition

Although audio-visual speech is well known to improve the robustness properties of automatic speech recognition (ASR) systems against noise, the realm of audio-visual ASR (AV-ASR) has not gathered the research momentum it deserves. This is mainly due to the lake of audio-visual corpora and the need to combine two fields of knowledge: ASR and computer vision. This paper describes the NTCD-TIMIT database and baseline that can overcome these two barriers and attract more research interests to AV-ASR. The database has been created by adding six noise types at a range of signal-to-noise ratios to the speech material of the recently published TCD-TIMIT corpus. The database also includes visual features that have been extracted from the TCD-TIMIT video recordings using the visual front-end presented in this paper. NTCD-TIMIT contains Kaldi scripts for training and decoding audio-only, video-only, and audio-visual ASR models. The baseline experiments and results obtained using these scripts are detailed in this paper.