EEG data recorded during spoken and imagined speech interaction with a simulated robot

Dataset Description

This dataset consists of Electroencephalography (EEG) data recorded from 15 healthy subjects with a 64-channel EEG headset during spoken and imagined speech interaction with a simulated robot.

Citation

The dataset recording and study setup are described in detail in the following publication:

Rekrut, M., Selim, A. M., & Krüger, A. (2022, October). Improving Silent Speech BCI Training Procedures Through Transfer from Overt to Silent Speech. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 2650-2656). IEEE.

If you use this dataset, please consider citing this work.

Study Design

Participants were seated in a chair and controlled the simulated robot on a screen in a game-like setup through a maze. They were presented with a birds-view of the robots’ surroundings with the robot in the middle. Participants had to decide about its next step and interact for one part of the study via overt and in the second part via imagined speech. The interaction consisted of moving the robot in 3 different directions resulting in the command words ”left”, ”right” and ”up” and picking up screws and pushing boxes out of the way by the words ”pick” and ”push”. Whenever the user had made a decision about the next command they could press the spacebar to indicate the desire for interaction. After the spacebar was pressed, the screen turned black for 2 seconds to give the participant time to prepare the input. After the 2 seconds, a fixation cross appeared, which indicated to start speaking or producing imagined speech of the desired command, depending on the current condition. After 2 seconds, the fixation cross disappeared, and the few switched back to the robot with its updated position.

Our participants were advised to speak the word out loud once during the overt condition and, in the imagined speech part, to repeat the word once silently in their head, just like reading it to themselves, without any movement of the articulatory muscles. The input of the user did not have any impact on the systems output, the robot always performed the correct action, an aspect our participants were informed about. Those requirements were made in order to minimize stress, confusion, or other mental states and prevent impacts on the EEG recording.

The game was split up into 4 parts to allow sufficient breaks in between each session for the participant to rest and prevent inducing too much cognitive load. Furthermore, those breaks were used to check the impedances of the EEG headset. Each participant started with a block of overt speech, followed by a silent speech part, continued with overt speech and did a final block of silent speech. This shift was chosen mainly to keep the participants attentive and provide some sort of variety over the duration of the experiment but also to prevent the blockwise recording of the two paradigms.

We recorded 80 repetitions per word and paradigm, meaning for the 5 words, 400 imagined and 400 spoken repetitions per participant, resulting in 800 repetitions overall. Furthermore, we needed to integrate breaks into the experiment, as doing all repetitions in one session would, in the best case take around 70 minutes (5-6 sec per task times 800 tasks), far too long to remain focused. Therefore, we decided to split up the task into levels of 25 interactions, including 5 repetitions of each word in a random order without repeating a word directly. For our 800 repetitions, this meant that we had to create 32 unique levels with a random order of the 5 different commands with 5 repetitions of each word in each level. Those 32 unique levels were then split into 4 parts, two for imagined and two for overt repetitions. The order of the parts during the experiment was overt, imagined, overt, imagined, again to prevent recording the data per paradigm blockwise, accidentally resulting in classifying arbitrary brain states rather then cognitive processes. Additionally, a tutorial level was created to let the participants practice the interaction and make them familiar with the task to feel comfortable during interaction.

Subjects

We conducted the study with 15 healthy subjects, 11 male and 4 female, with an average age of 26.8 years, all with normal or corrected-to-normal vision and right-handed. All subjects were non-native English speakers but fluent and experienced with the language, as our command words were selected to be English. Each subject was introduced to the task, and informed consent was obtained from all subjects for the scientific use of the recorded data. The study was approved by the ethical review board of the Faculty of Mathematics and Computer Science at Saarland University.

Recording

The data was acquired in a dimly lit room with minimized distractions like external sound, mobile devices, and others. The voluntary participants were asked to sit in a comfortable chair to prevent unnecessary muscle movements and reduce noise and artefacts in the EEG, which could emerge from mental stress, unrelated sensory input, physiological motor activity and electrical interference. EEG signals were recorded using a wireless 64-channel electroencephalograph system, namely Brain Products Live Amp 64. The sampling rate was set to 500 Hz. The 10-20 International System of electrode placement was used to cover the whole scalp resulting in the capturing of spatial information from the brain recordings effectively. The robot game was compiled and executed on the same Windows PC as the recording software of the EEG-Headset to allow synchronization of the data and events recorded in the game, e.g. keyboard press or fixation cross.

Data format

Data was recorded in one single file, including the breaks between sessions in .fif format. This format contains a list of events which look as follows: