Published December 5, 2024 | Version v1
Dataset Open

HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour

Creators

  • 1. ROR icon University of Southampton
  • 2. ROR icon University of Warwick

Description

This is the dataset for the paper "HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour" – available at the Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction.
 
The dataset captures various modalities of user behaviour (facial landmarks, facial action units, head pose, body pose landmarks, depth footage) exhibited by participants interacting with a TIAGo Steel robot following different behaviour models on a collaborative manipulation "Burger Assembly" task. Details of the task can be found in the paper. The non-verbal modalities are complemented by explicit feedback signals (verbal dialogue transcripts), robot joint movements data, interaction event labels and self-assessed questionnaires (pre-study and post-interaction questionnaires) on sociodemographics and perceived user impressions (e.g. frustration, satisfaction).
 
The dataset's contents have been collected from over 6 hours of verbal and physical human-robot interactions in over 146 sessions with 18 participants.

Data Modalities

User facial expression data (facial landmarks, facial action units and head pose) have been calculated by OpenFace:
  • head location: pose_Tx, pose_Ty, pose_Tz
  • head rotation: pose_Rx, pose_Ry, pose_Rz
  • facial landmark 3d locations: X_0, ... X_67, Y_0,...Y_67, Z_0,...Z_67
  • facial action units intensity (r) and presence (c): AU01_r, AU02_r,...AU25_r, AU26_r, AU45_r, AU01_c, AU02_c,...AU28_c, AU45_c
User pose landmarks have been calculated by MediaPipe Pose Landmarker:
  • 3d coordinates of 33 pose landmarks
Verbal dialogue transcripts have been produced by OpenAI's Whisper model.
 
Depth information and robot joint data has been recorded by the TIAGo robot's sensors.
 
Self-assessed questionnaire data has been collected using a 5-point Likert scale, following question items and practices established in previous Human-Robot Interactions and Psychology works. Details of this can be found in the paper.
 
For more details on the data collection and processing pipeline, please see the paper.

Contents

The dataset is organised as:
 
  • README.md
  • questionnaire-data/
    • pre-study-questionnaire/
      • pre-study-questionnaire.pdf
      • pre-study-questionnaire-responses.csv: participant-id,A1-age,A2-occupation,B1,B2,B3,B4,B5,B6,C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12
    • post-interaction-questionnaire/
      • post-interaction-questionnaire.pdf
      • post-interaction-questionnaire-responses.csv: participant-id,model-id,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8,Q9,Q10,Q11,Q12,Q13,Q14,Q15,Q16,Q17
  • sensory-data/
    • depth-data/
      • P[participant_id]-M[model_id]-[trial_number].mp4
    • user-face-data/
      • P[participant_id]-M[model_id]-[trial_number].csv: timestamp,confidence,success,pose_Tx,pose_Ty,pose_Tz,pose_Rx,pose_Ry,pose_Rz,X_0,X_1,X_2,X_3,X_4,X_5,X_6,X_7,X_8,X_9,X_10,X_11,X_12,X_13,X_14,X_15,X_16,X_17,X_18,X_19,X_20,X_21,X_22,X_23,X_24,X_25,X_26,X_27,X_28,X_29,X_30,X_31,X_32,X_33,X_34,X_35,X_36,X_37,X_38,X_39,X_40,X_41,X_42,X_43,X_44,X_45,X_46,X_47,X_48,X_49,X_50,X_51,X_52,X_53,X_54,X_55,X_56,X_57,X_58,X_59,X_60,X_61,X_62,X_63,X_64,X_65,X_66,X_67,Y_0,Y_1,Y_2,Y_3,Y_4,Y_5,Y_6,Y_7,Y_8,Y_9,Y_10,Y_11,Y_12,Y_13,Y_14,Y_15,Y_16,Y_17,Y_18,Y_19,Y_20,Y_21,Y_22,Y_23,Y_24,Y_25,Y_26,Y_27,Y_28,Y_29,Y_30,Y_31,Y_32,Y_33,Y_34,Y_35,Y_36,Y_37,Y_38,Y_39,Y_40,Y_41,Y_42,Y_43,Y_44,Y_45,Y_46,Y_47,Y_48,Y_49,Y_50,Y_51,Y_52,Y_53,Y_54,Y_55,Y_56,Y_57,Y_58,Y_59,Y_60,Y_61,Y_62,Y_63,Y_64,Y_65,Y_66,Y_67,Z_0,Z_1,Z_2,Z_3,Z_4,Z_5,Z_6,Z_7,Z_8,Z_9,Z_10,Z_11,Z_12,Z_13,Z_14,Z_15,Z_16,Z_17,Z_18,Z_19,Z_20,Z_21,Z_22,Z_23,Z_24,Z_25,Z_26,Z_27,Z_28,Z_29,Z_30,Z_31,Z_32,Z_33,Z_34,Z_35,Z_36,Z_37,Z_38,Z_39,Z_40,Z_41,Z_42,Z_43,Z_44,Z_45,Z_46,Z_47,Z_48,Z_49,Z_50,Z_51,Z_52,Z_53,Z_54,Z_55,Z_56,Z_57,Z_58,Z_59,Z_60,Z_61,Z_62,Z_63,Z_64,Z_65,Z_66,Z_67,AU01_r,AU02_r,AU04_r,AU05_r,AU06_r,AU07_r,AU09_r,AU10_r,AU12_r,AU14_r,AU15_r,AU17_r,AU20_r,AU23_r,AU25_r,AU26_r,AU45_r,AU01_c,AU02_c,AU04_c,AU05_c,AU06_c,AU07_c,AU09_c,AU10_c,AU12_c,AU14_c,AU15_c,AU17_c,AU20_c,AU23_c,AU25_c,AU26_c,AU28_c,AU45_c
    • user-pose-data/
      • P[participant_id]-M[model_id]-[trial_number]-[camera_id].csv: time,L0-x,L0-y,L0-z,L1-x,L1-y,L1-z,L2-x,L2-y,L2-z,L3-x,L3-y,L3-z,L4-x,L4-y,L4-z,L5-x,L5-y,L5-z,L6-x,L6-y,L6-z,L7-x,L7-y,L7-z,L8-x,L8-y,L8-z,L9-x,L9-y,L9-z,L10-x,L10-y,L10-z,L11-x,L11-y,L11-z,L12-x,L12-y,L12-z,L13-x,L13-y,L13-z,L14-x,L14-y,L14-z,L15-x,L15-y,L15-z,L16-x,L16-y,L16-z,L17-x,L17-y,L17-z,L18-x,L18-y,L18-z,L19-x,L19-y,L19-z,L20-x,L20-y,L20-z,L21-x,L21-y,L21-z,L22-x,L22-y,L22-z,L23-x,L23-y,L23-z,L24-x,L24-y,L24-z,L25-x,L25-y,L25-z,L26-x,L26-y,L26-z,L27-x,L27-y,L27-z,L28-x,L28-y,L28-z,L29-x,L29-y,L29-z,L30-x,L30-y,L30-z,L31-x,L31-y,L31-z,L32-x,L32-y,L32-z
    • robot-joint-data/
      • P[participant_id]-M[model_id]-[trial_number].csv: time, arm_1_joint, arm_2_joint, arm_3_joint, arm_4_joint, arm_5_joint, arm_6_joint, arm_7_joint 
    • dialogue-transcript/
      • P[participant_id]-M[model_id]-[trial_number].csv: start,end,text
  • interaction-event-labels/
    • P[participant_id]-M[model_id]-[trial_number].csv: event,start,end

Limitations

Due to recording sensor malfunctions and processing library limitations, not all interaction scenario data contains all modalities. Verbal dialogue transcripts or user pose data may be incomplete or missing for a small number of interactions or recording angles.
 

Acknowledgements

This work was supported by UK Research and Innovation [EP/S024298/1].

Files

HRI-SENSE.zip

Files (3.6 GB)

Name Size Download all
md5:cc3919211f77c969dcc409622ea5684d
3.6 GB Preview Download