CitySpeechMix: A Simulated Dataset of Speech and Urban Sound Mixtures from LibriSpeech and SONYC-UST

CitySpeechMix is a simulated audio dataset that mixes speech excerpts from LibriSpeech with environmental recordings from SONYC-UST to create controlled mixtures of voice and background noise. Each audio file is accompanied by the corresponding LibriSpeech transcription and the SONYC-UST sound class labels. A mapping is also provided between the selected SONYC-UST sound classes and their corresponding AudioSet categories.

📊 Dataset Overview

The dataset consists of 742 audio clips , each 10 seconds long:
- 371 mixtures  of speech over urban background noise
- 371 voice-free  urban environmental recordings

🛠️ Dataset Construction

The dataset, included in the `cityspeechmix.zip` archive, is constructed as follows:

- Environmental sounds are selected from the SONYC-UST v2 evaluation set. Only clips annotated with exactly one of the following seven sound classes are retained: `engine`, `jackhammer`, `chainsaw`, `car horn`, `siren`, `music`, and `dog`.
- The resulting SONYC subset is balanced to 742 clips (106 per class, selected randomly when more clips are available). Of these, 371 clips are retained for mixing (sonyc_librispeech_mixtures folder), and 371 clips are peak-normalized and left untouched (sonyc_unmixed_subset) .
- 371 speech clips (approximately 10 seconds each) are randomly selected from the LibriSpeech evaluation set and matched randomly to the 371 SONYC audio files selected for mixing.
- Each pair of SONYC and LibriSpeech clips is resampled to 44.1 kHz  and put at the same RMSE . To simulate realistic background noise conditions, the SONYC signal is attenuated by 6 dB prior to mixing.
- The resulting mixtures are peak-normalized.

📁 Folder Structure

Inside the `cityspeechmix/` folder:

- `sonyc_librispeech_mixtures/` — 371 speech + background noise mixtures  
- `sonyc_unmixed_subset/` — 371 voice-free environmental recordings  

The source stems (individual speech and background files for each mixture) are available separately in `stems.zip`.

📄 Metadata File Description

Each row in `metadata.csv` corresponds to a 10-second audio clip from the CitySpeechMix dataset. The columns are defined as follows:

- `fname` — Filename of the resulting audio file (either a mixture or a reference clip).
- `sonyc_file` — Filename of the SONYC-UST environmental recording used.
- `librispeech_file` — Filename of the LibriSpeech audio sample used in the mixture. This field is `NaN` for voice-free clips.
- `script` — Transcription of the spoken content from the LibriSpeech file. This field is `NaN` for voice-free clips.
- `label1_sonyc` — First SONYC sound class label (e.g., `siren`, `dog`, `engine`) associated with the environmental recording.
- `label1_audioset` — Corresponding AudioSet-compatible label for `label1_sonyc`.
- `label2_sonyc` — Second SONYC label, corresponding to the voice label of SONYC-UST. This field is `NaN` for voice-free clips.
- `label2_audioset` — Corresponding AudioSet-compatible label for `label2_sonyc`. This field is `NaN` for voice-free clips.

🔎 Suggested Applications

- Speech anonymization systems
- Robust automatic speech recognition (ASR)
- Urban sound tagging in presence of voice

📚 Source Datasets

- LibriSpeech  
  Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015).  
  Librispeech: An ASR corpus based on public domain audio books.  
  [Paper] • [Dataset]

- SONYC-UST V2  
  Cartwright, M., Cramer, J., Bello, J. P., McFee, B., Cartwright, M., & Salamon, J. (2020).  
  SONYC-UST V2: An Urban Sound Tagging Dataset with Spatiotemporal Context.  
  [Paper] • [Dataset]

📎 Citation

If you use CitySpeechMix in your research, please cite it as:

> CitySpeechMix: A Simulated Dataset of Speech and Urban Sound Mixtures from LibriSpeech and SONYC-UST  
> Modan Tailleur, Mathieu Lagrange, Pierre Aumond, Vincent Tourre. 2025.  
> Zenodo. https://doi.org/10.5281/zenodo.15405950

@misc{tailleur2025cityspeechmix,
  title        = {CitySpeechMix: A Dataset of Speech and Urban Sound Mixtures},
  author       = {Tailleur, Modan and Lagrange, Mathieu and Aumond, Pierre and Tourre, Vincent},
  year         = 2025,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.15405950},
  url          = {https://doi.org/10.5281/zenodo.15405950}
}