GENEA Challenge 2022 Dataset Files

This Zenodo repository contains the main dataset for the GENEA 2022 challenge, which is based on the Talking With Hands 16.2M data.

Contents:

The "trn" and "val" zip files contain audio files (in WAV format), time-aligned transcriptions (in TSV format), and motion files (in BVH format) for the training and validation datasets, respectively.

The "tst" zip file contains audio files (in WAV format) and transcriptions (in TSV format) for the test set, but no motion. The corresponding test motion is available at:

https://doi.org/10.5281/zenodo.6976463

Each zip file also contains a "metadata.csv" file that contains information for all files regarding the speaker ID and whether or not the motion files contain finger motion.

Note that the speech audio in the data sometimes has been replaced by silence for the purpose of anonymisation.

Data processing scripts:

We provide a number of optional scripts for encoding and processing the challenge data:

Audio: Scripts for extracting basic audio features, such as spectrograms, prosodic features, and mel-frequency cepstral coefficients (MFCCs) can be found at this link.

Text: A script to encode text transcriptions to word vectors using FastText is available. tsv2wordvectors.py

Motion: If you wish to encode the joint angles from the BVH files to and from an exponential map representation, you can use scripts by Simon Alexanderson based on the PyMo library, which are available here:

• bvh2features.py
• features2bvh.py

Attribution:

If you use this material, please cite our latest paper on the GENEA Challenge 2022. At the time of writing (2022-08-16), that is our ACM ICMI 2022 paper:

Youngwoo Yoon, Pieter Wolfert, Taras Kucherenko, Carla Viegas, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter. 2022. The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI '22). ACM.

You can find the latest information and a BibTeX file on the project website:

https://youngwoo-yoon.github.io/GENEAchallenge2022/

Also cite the paper about the original dataset from Meta Research:

Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S. Srinivasa, and Yaser Sheikh. 2019. Talking With Hands 16.2M: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV ’19). IEEE, 763–772.

The motion and audio files are based on the Talking With Hands 16.2M dataset at https://github.com/facebookresearch/TalkingWithHands32M/. All material is available under a CC BY NC 4.0 international license, with the text provided in LICENSE.txt.

To find more GENEA Challenge 2022 material on the web, please see:

* https://youngwoo-yoon.github.io/GENEAchallenge2022/

* https://genea-workshop.github.io/2022/challenge/

If you have any questions or comments, please contact:

* The GENEA Challenge & Workshop organisers <genea-contact@googlegroups.com