CHiME-7 UDASE evaluation data

Leglaive, Simon; Fraticelli, Matthieu; ElGhazaly, Hend; Borne, Léonie; Sadeghi, Mostafa; Wisdom, Scott; Pariente, Manuel; Hershey, John; Pressnitzer, Daniel; Barker, Jon

doi:10.5281/zenodo.10418311

Published February 2024 | Version v1

Dataset Open

CHiME-7 UDASE evaluation data

Description

This repository contains audio and CSV files used in the evaluation of the UDASE task of the 7th CHiME challenge. In particular, it contains the audio samples and subjective ratings of an ITU-T P.835 listening test.

If you use this material in your research, please cite the following paper:

Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge

Simon Leglaive, Matthieu Fraticelli, Hend ElGhazaly, Léonie Borne, Mostafa Sadeghi, Scott Wisdom, Manuel Pariente, John R. Hershey, Daniel Pressnitzer, Jon P. Barker

Computer Speech & Language, vol. 89, 2025

General information

The folder 'listening_test' contains:

the audio files used for the ITU-T P.835 listening test ('data' subfolder);
the subjective evaluation results:
- 'raw_results_listening_test.csv' contains the individual ratings of each participant (identified by the file name in the 'csv' column);
- 'MOS_results_listening_test.csv' files contains the mean opinion scores (MOS) computed from the participants' ratings.

The folder 'objective_evaluation' contains:

a subset of the output audio files of the baseline and submitted speech enhancement methods ('data' subfolder);
the objective evaluation results ('results_objective_evaluation.csv' file).

Additional details

'listening_test/data/ref' contains truncated versions of the ITU-T P.501 (2017) test signals for use in telephonometry. We downloaded the files from Microsoft's P.808 Toolkit and we modified them to keep only the first utterance of each audio file and to normalize the loudness.
'listening_test/data/C0' contains audio segments extracted from the binaural recordings of the CHiME-5 dataset ('eval' set). It corresponds to the unprocessed noisy speech condition.
'listening_test/data/{C1, C2, C3, C4}' contains denoised versions of the audio files in 'listening_test/data/C0', where
- condition 'C1' corresponds to the 'CMGAN-FT' system;
- condition 'C2' corresponds to the 'ISDS1' system;
- condition 'C3' corresponds to the 'N&B' system;
- condition 'C4' corresponds to the 'RemixIT-VAD' system.
'objective_evaluation/data/<METHOD>/CHiME-5' contains denoised versions of the noisy speech signals in the segmented CHiME-5 dataset ('eval/{1, listening_test}' subsets only).
We do not share the denoised mixtures of the reverberant LibriCHiME-5 dataset (which should be in 'objective_evaluation/data/<METHOD>/reverberant-LibriCHiME-5'). These mixtures were created using clean speech utterances of the LibriSpeech dataset, noise signals from the CHiME-5 dataset, and room impulse responses from the VoiceHome dataset. Unfortunately we cannot distribute material derived from both the CHiME-5 and VoiceHome datasets because of incompatible ShareAlike licenses (CC BY-SA 4.0 and CC BY-NC-SA 4.0).

Licences

The files shared in this repository are licensed under a CC BY-SA 4.0 license. They were derived from the following datasets:

CHiME-5, CC BY-SA 4.0 license.
ITU-T P.501 (2017) test signals for use in telephonometry,: license available in 'listening_test/data/ref/itu_license_text_from_P501.txt'.

Files