Dataset: AFFEC - Advancing Face-to-Face Emotion Communication Dataset

Overview

The AFFEC (Advancing Face-to-Face Emotion Communication) dataset is a multimodal dataset designed for emotion recognition research. It captures dynamic human interactions through electroencephalography (EEG), eye-tracking, galvanic skin response (GSR), facial movements, and self-annotations, enabling the study of felt and perceived emotions in real-world face-to-face interactions. The dataset comprises 84 simulated emotional dialogues, 72 participants, and over 5,000 trials, annotated with more than 20,000 emotion labels.

Dataset Structure

The dataset follows the Brain Imaging Data Structure (BIDS) format and consists of the following components:

Root Folder:

• sub-* : Individual subject folders (e.g., sub-aerj, sub-mdl, sub-xx2)
• dataset_description.json: General dataset metadata
• participants.json and participants.tsv: Participant demographics and attributes
• task-fer_events.json: Event annotations for the FER task
• README.md: This documentation file

Subject Folders (sub-<subject_id>):

Each subject folder contains:

• Behavioral Data (beh/): Physiological recordings (eye tracking, GSR, facial analysis, cursor tracking) in JSON and TSV formats.
• EEG Data (eeg/): EEG recordings in .edf and corresponding metadata in .json.
• Event Files (*.tsv): Trial event data for the emotion recognition task.
• Channel Descriptions (*_channels.tsv): EEG channel information.

Data Modalities and Channels

1. Eye Tracking Data

• Channels: 16 (fixation points, left/right eye gaze coordinates, gaze validity)
• Sampling Rate: 62 Hz
• Trials: 5632
• File Example: sub-<subject>_task-fer_run-0_recording-gaze_physio.json

2. Pupil Data

• Channels: 21 (pupil diameter, eye position, pupil validity flags)
• Sampling Rate: 149 Hz
• Trials: 5632
• File Example: sub-<subject>_task-fer_run-0_recording-pupil_physio.json

3. Cursor Tracking Data

• Channels: 4 (cursor X, cursor Y, cursor state)
• Sampling Rate: 62 Hz
• Trials: 5632
• File Example: sub-<subject>_task-fer_run-0_recording-cursor_physio.json

4. Face Analysis Data

• Channels: Over 200 (2D/3D facial landmarks, gaze detection, facial action units)
• Sampling Rate: 40 Hz
• Trials: 5680
• File Example: sub-<subject>_task-fer_run-0_recording-videostream_physio.json

5. Electrodermal Activity (EDA) and Physiological Sensors

• Channels: 40 (GSR, body temperature, accelerometer data)
• Sampling Rate: 50 Hz
• Trials: 5438
• File Example: sub-<subject>_task-fer_run-0_recording-gsr_physio.json

6. EEG Data

• Channels: 63 (EEG electrodes following the 10-20 placement scheme)
• Sampling Rate: 256 Hz
• Reference: Left earlobe
• Trials: 5632
• File Example: sub-<subject>_task-fer_run-0_eeg.edf

7. Self-Annotations

• Trials: 5807
• Annotations Per Trial: 4
• Event Markers: Onset time, duration, trial type, emotion labels
• File Example: task-fer_events.json

Experimental Setup

Participants engaged in a Facial Emotion Recognition (FER) task, where they watched emotionally expressive video stimuli while their physiological and behavioral responses were recorded. Participants provided self-reported ratings for both perceived and felt emotions, differentiating between the emotions they believed the video conveyed and their internal affective experience.

The dataset enables the study of individual differences in emotional perception and expression by incorporating Big Five personality trait assessments and demographic variables.

Usage Notes

• The dataset is formatted in ASCII/UTF-8 encoding.
• Each modality is stored in JSON, TSV, or EDF format as per BIDS standards.
• Researchers should cite this dataset appropriately in publications.

Applications

AFFEC is well-suited for research in:

• Affective Computing
• Human-Agent Interaction
• Emotion Recognition and Classification
• Multimodal Signal Processing
• Neuroscience and Cognitive Modeling
• Healthcare and Mental Health Monitoring

Acknowledgments

This dataset was collected with the support of brAIn lab, IT University of Copenhagen. 
Special thanks to all participants and research staff involved in data collection.

License

This dataset is shared under the Creative Commons CC0 License.

Contact

For questions or collaboration inquiries, please contact [brainlab-staff@o365team.itu.dk].