Audio-tactile speech EEG Dataset

Dataset setting out to investigate behavioural and EEG responses to continuous speech coupled with tactile pulses. Support functions to process data and derive neural responses to both sensory features can be found on this repository. The raw behavioural and EEG data processed to this end is provided here, as well as the stimuli used in the behavioural and electrophysiological experiments.

Note : For a library that features encoding and decoding from EEG data but less specific about the data format, please see this repository.

#Introduction

This dataset contains the behavioural response to one speech comprehension task in which subjects received both audio and tactile stimuli. It also contains the EEG response to 2 sessions of audio-book listening accompanied by those same tactile stimuli. Finally, the stimuli for both experiments are also available in the dataset. There were 18 subjects.

Naming Conventions:

• The subjects IDs are :  'al', 'yr', 'alio', 'chap', 'sep', 'phil', 'lad', 'calco', 'hudi', 'nima', 'ogre', 'raqu', 'nikf', 'zartan', 'naga', 'miya', 'elios', 'olio'
• The labels of the condition generally have three fields : type, delay and correlation. The type indicates the nature of stimuli (audio, tactile or audio-tactile). The delay indicates the delay between the two stimuli in the case there are two, it is fixed at 0 otherwise. The correlation indicates whether the tactile stimuli is correlated to the audio or not, it is at True by default, except for the sham stimuli described in the paper.

#Content

The dataset contains a .zip named after each subject : these are the eeg recordings for each of them. It also contains files called Stimuli_EEG and Stimuli_Behavioural which contains the stimuli for each experiment. Finally Behavioural_Experiment.zip contains the data from the behavioural experiment. Their respective content is described in more details down below. 

• Behavioural Stimuli: This folder contains files named 'sent_4k__xxxyyy.zzz'. sent_4k refers to sent stimuli at 44.1kHz. xxx refers to the number of the sentence. yyy.zzz refers to the nature and format of the file: TextGrid, wav, audio.npy, phone.npy. TextGrid are text files indicating the timing of each phoneme and words, they can be vizualized alongside the audio file using Praat. .wav are simple audio files at 44.1kHz. audio.npy is just the Audio file put in an .npy format and resampled at the internal rate of the TDT RX8 system used to send the stimuli (39062kHz). phone.npy are the tactile pulses at the onset of vowels and resampled accordingly.

• EEG Stimuli: This folder contains files named 'Odin_x_y_zzz.vvv'. x and y refers to the chapter in the story and the part of that chapter. zzz.vvv refers to the nature and format. Most notations are the same as for the Behavioural Stimuli file. In addition, there are sparse matrices containing the timing of all phonetic features and phonemes in the .npz format. Finally, this folder also contains comprehension questions and their answers for each part of each chapter.

• Behavioural Experiment: This folder contains two types of files : 'xxx_subjective.pkl' and 'xxx_total_behavioural.csv' where xxx refers to the subject ID. 'xxx_subjective.pkl' is a dictionary containing the average comfort score for each condition. 'xxx_total_behavioural.csv' is a table containing for each trial: the sentence number (same notation as in Behavioural Stimuli), the type, delay and correlation nature of the stimuli, the real sentence and the ratio of understood words of the subjects.

• EEG Experiment: All of the 'xxx.zip' files where xxx refers to the subject ID. Each folder contains one folder for each of the two sessions (xxx_EEG_1 and xxx_EEG_2). Inside of those, .eeg, .vhdr and .vmrk files can be found. These correspond to the recorded data. More information on these file formats can be found on the Brian vision webpage. It also contains a .csv file containing for each segment of ~2mn30: the chapter and part sent to the subject, the type, delay and correlation nature of the stimuli, the onset of the target tactile stimuli (for the behavioural task in the tactile-only condition) and answers of the subjects to the comprehension questions (for the behavioural task in the audio-only condition). Finally, the order of chapter and parameters in a .npy format is also provided although redundant with the .csv file.

#EEG Data Format

We recorded continuous EEG from the participants over an hour at 1kHz. 

In addition to the 63 channels, there is also Stimtrack channel labelled as 'Sound' which was tracking an addition of the auditory and tactile stimuli and allowed to track the sent stimuli as well as the alignment. The corresponding files can be found using the provided .csv file.

 The .vmrk file provided contains the timing of the triggers sent at the start of each 2mn30 segment and can help with tracking the progress in the experiment. Overall .vmrk and stimtrack are used together to align the EEG response with the corresponding stimuli and checking the state of the experiment.

The subjects could also press a button to signal that they detected the tactile pattern they were ask to find, this response can be found as an additional channel called 'Button'.

#Note 

Dysfunctions of the equipment were detected and should be taken into account:

• The button experienced malfunction when 'chap' was using it during the tactile pattern detection task
• The channels 'CPz', 'FCz', 'FC6', 'AF3', 'AF7' needed to be interpolated.
• For some subjects, the recording was interrupted and therefore the recording was separated in half. The missing sessions were as follow: 'yr' : 12, 'naga' : 0, 13, 'elios' : 14. Note that in those cases, you might need to modify the trigger timing extracted from the .vmrk file.