REFLEX Dataset: A Multimodal Dataset of Human Reactions to Robotic Failures and Subsequent Robotic Explanations.

REFLEX Dataset is a comprehensive collection of multimodal Human Behavioral reactions to Robot Failures and Explanations.

This version 1.0 is a representative sample of this dataset with the reactions from 5 users out of a total 55 users.
Currently the metadata has been anonymized since the corresponding manuscript is under submission. 

Please refer to the Readme in the zipped file for following and further information:

 This data was recorded from a user study and has been processed for anonymization.

About Data

This description gives a detailed process on how the data was collected. It should describe the conditions under which the data was recorded and also the devices used to record the data.

Data Organisation

The data is structured by strategy and participant, as shown below:

Strategy Dir/
  -Participant Dir/
    - analysis
    - questonnaire
    - facetorch
    - openface
    - gaze
    - hume
    - body
    - voice
    - time
    - video_cam1
    - video_cam2

We employed five different strategies (C1, C2, C3, D1, D2), collecting data from 11 participants for each strategy. The data for each participant is organized within a corresponding folder.

Participants are labeled based on their assigned strategy. For example, data from the first participant under the “Fixed Low” (C1) strategy can be found in the C1-1 subfolder within the C1 directory.

Collected Data

Each participant folder contains various datasets related to different modalities. All visual data are collected using the camera 1 video. The collected data are outlined below:

• Anonymized Videos (video_cam1.mp4, video_cam2.mp4) - Visual Representation:

  • Video from camera 1 (robot side of view)
  • Video from camera 2 (experiment side of view)

• Analysis (analysis.csv) - Failure Instance Description:

  • Failure type
  • Explanation strategy
  • Explanation level
  • Phase (Pre, Failure, Explanation, Resolution)
  • Start/End frame and time of failure
  • Task Resolved

• Questionnaire (questionnaire.csv) - Failure Instance Description:

  • Participant Data (Age, Gender, etc)
  • Answers of explanation-satisfaction rate question for rounds and overall experiment

• Facetorch (facetorch.csv) - Facetorch - Face:

  • Arousal/Valence levels
  • Presence of Facial Action Units (AUs)
  • Dominant Emotion (Out of six basic emotions and neutral)

• OpenFace (openface.csv) - OpenFace - Face, Gaze, Head:

  • Eye Gaze (2D and 3D Landmarks)
  • Eye Direction (vector and in radians)
  • Head Pose Estimation (Pose Estimation, Rotation)
  • Face Landmarks (2D and 3D Landmarks)
  • Facial Action Units (0.0-1.0 intensity scores, occurrences)

• Gaze (gaze.csv) - Gaze:

  • Eye Gaze Classification (e.g., Robot, Task, Miscellaneous)

• Hume (hume.csv) - Hume Expression Measurement API - Face:

  • 48 Emotion likelihoods
  • Facial Action Units (0.0-1.0 score)
  • Facial Descriptions (0.0-1.0 score)

• Voice (speech.csv) - Hume Expression Measurement API - Speech:

  • Speech conversation data
  • Emotional likelihoods inferred from prosody

• Body (body.csv) - MediaPipe Pose Landmark Detection - Body:

  • Pose classifications (e.g., crossed arms, arms behind back)
  • 2D and 3D Pose Landmarks

• Time (time.csv) - MediaPipe Pose Landmark Detection:

  • Associated timestamp and time for each frame of camera 1 video.

How to Visualize Participant Data

To visualize the participant data, navigate to the folder HRI-Dataset/Visualize, install the requirements and run the main.py with the participant code:

# Navigate to the code-set for visualization (~/HRI-Dataset/Visualize)
cd Visualize
# Install the necessary libraries
pip install -r requirements.txt  
# Run the code
python main.py --participant PARTICIPANT_CODE # specific participant (e.g. C1-2, etc) or default participant by not using the argument