RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments

License + Attribution

This dataset is licensed under CC BY-NC-SA 4.0. Commercial usage is not permitted. If you use this dataset or the code in a scientific publication, please cite the following paper:

@inproceedings{FischerECCV2018,
author = {Tobias Fischer and Hyung Jin Chang and Yiannis Demiris},
title = "{RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments}",
booktitle = {European Conference on Computer Vision},
year = {2018},
month = {September},
pages = {339--357}
}

This work was supported in part by the Samsung Global Research Outreach program, and in part by the EU Horizon 2020 Project PAL (643783-RIA).

More information can be found on the Personal Robotic Lab's website: https://www.imperial.ac.uk/personal-robotics/software/.

Overview

The dataset consists of two parts: 1) One where the eyetracking glasses were worn (and thus ground truth labels for head-pose and eye gaze are available; suffix _glasses), and 2) One with natural appearances (no eyetracking glasses are worn; suffix _noglasses). The _noglasses images were used to train subject-specific GANs, and these GANs were used to inpaint the region covered by the eyetracking glasses in the _glasses images.

There is code accompanying this dataset: https://github.com/Tobias-Fischer/rt_gene. Please use the issue tracker in the code respository if you have questions regarding the dataset.

Subjects / 3-Fold evaluation

15 participants were recorded in 17 sessions. Session 014 is a second recording of participant 002, and session 015 is a second recording of participant 005 (different days and different camera poses were used).

We used a 3-fold evaluation, with the three folds consisting of the following sessions (test on one of the groups, training with the remaining two groups):

1. 's001', 's002', 's008', 's010'
2. 's003', 's004', 's007', 's009'
3. 's005', 's006', 's011', 's012', 's013'

The validation set consists of sessions 's014', 's015' and 's016'.

While the MATLAB script (prepare_dataset.m; see code repository) creates train and test images for each subject, all images were used for the evaluation (see evaluate_model.py).

Labeled dataset (sXYZ_glasses)

The file for each subject contains the following information:

• label_combined.txt This is the main file containing labels. The formatting is as follows:
  seq_number, [head pose: right(pos) / left(neg), up (pos) / down(neg)], [gaze: right(pos) / left(neg), up(pos) / down(neg)], timestamp
• label_headpose.txt This file contains more detail about the head pose of the subject.
  seq_number, [head pose translation: further(pos) / closer(neg), left(pos) / right(neg), up(pos) / down(neg)], [head pose rotation: roll right(pos) / roll left(neg), down(pos) / up(neg), rotate left(pos), rotate right(neg)], timestamp
• kinect2_calibration.yaml
  The kinect2_calibration.yaml file contains the camera projection matrix in ROS format (this file should not be required).
• kinect2_pose.txt
  The kinect2_pose.txt file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).
• "original" folder
  • The face_before_inpainting folder contains the face with a large margin to the left and right.
  • The mask folder contains images indicating the regions of the eyetracking glasses, aligned with the images in the face_before_inpainting folder.
  • The overlay folder contains images where the mask was overlaid on the face_before_inpainting images.
  • The face folder contains the face image extracted using MTCNN with a tighter margin.
  • The left and right folders contain the left and right eye image areas.
• The face, left and right images were used as baseline comparison in the paper (Fig. 7 without inpainting).
• "inpainted" folder
  • The face_after_inpainting folder contains images corresponding to the ones in the face_before_inpainting folder after applying the inpainting.
  • Then, the images contained in the face, left and right folders were extracted using MTCNN as above.

Unlabeled dataset (sXYZ_noglasses)

• kinect2_calibration.yaml
  This file contains the camera projection matrix in ROS format (this file should not be required).
• kinect2_pose.txt
  This file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).
• "face" folder
  This folder contains the faces that can be used to train the GANs (without eyetracking glasses being worn).