Dataset Open Access

RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments

Fischer, Tobias; Chang, Hyung Jin; Demiris, Yiannis

License + Attribution

This dataset is licensed under CC BY-NC-SA 4.0. Commercial usage is not permitted. If you use this dataset or the code in a scientific publication, please cite the following paper:

@inproceedings{FischerECCV2018,
author = {Tobias Fischer and Hyung Jin Chang and Yiannis Demiris},
title = "{RT-GENE: Real-Time Eye Gaze Estimation in Natural Environments}",
booktitle = {European Conference on Computer Vision},
year = {2018},
month = {September},
pages = {339--357}
}

This work was supported in part by the Samsung Global Research Outreach program, and in part by the EU Horizon 2020 Project PAL (643783-RIA).

More information can be found on the Personal Robotic Lab's website: https://www.imperial.ac.uk/personal-robotics/software/.

Overview

The dataset consists of two parts: 1) One where the eyetracking glasses were worn (and thus ground truth labels for head-pose and eye gaze are available; suffix _glasses), and 2) One with natural appearances (no eyetracking glasses are worn; suffix _noglasses). The _noglasses images were used to train subject-specific GANs, and these GANs were used to inpaint the region covered by the eyetracking glasses in the _glasses images.

There is code accompanying this dataset: https://github.com/Tobias-Fischer/rt_gene. Please use the issue tracker in the code respository if you have questions regarding the dataset.

Subjects / 3-Fold evaluation

15 participants were recorded in 17 sessions. Session 014 is a second recording of participant 002, and session 015 is a second recording of participant 005 (different days and different camera poses were used).

We used a 3-fold evaluation, with the three folds consisting of the following sessions (test on one of the groups, training with the remaining two groups):

  1. 's001', 's002', 's008', 's010'
  2. 's003', 's004', 's007', 's009'
  3. 's005', 's006', 's011', 's012', 's013'

The validation set consists of sessions 's014', 's015' and 's016'.

While the MATLAB script (prepare_dataset.m; see code repository) creates train and test images for each subject, all images were used for the evaluation (see evaluate_model.py).

Labeled dataset (sXYZ_glasses)

The file for each subject contains the following information:

  • label_combined.txt This is the main file containing labels. The formatting is as follows:
    seq_number, [head pose: down(pos) / up (neg), left(pos) / right(neg)], [gaze: up(pos) / down(neg), right(pos) / left(neg)], timestamp
  • label_headpose.txt This file contains more detail about the head pose of the subject.
    seq_number, [head pose translation: further(pos) / closer(neg), left(pos) / right(neg), up(pos) / down(neg)], [head pose rotation: roll right(pos) / roll left(neg), down(pos) / up(neg), rotate left(pos), rotate right(neg)], timestamp
  • kinect2_calibration.yaml
    The kinect2_calibration.yaml file contains the camera projection matrix in ROS format (this file should not be required).
  • kinect2_pose.txt
    The kinect2_pose.txt file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).
  • "original" folder
    • The face_before_inpainting folder contains the face with a large margin to the left and right.
    • The mask folder contains images indicating the regions of the eyetracking glasses, aligned with the images in the face_before_inpainting folder.
    • The overlay folder contains images where the mask was overlaid on the face_before_inpainting images.
    • The face folder contains the face image extracted using MTCNN with a tighter margin.
    • The left and right folders contain the left and right eye image areas.
  • The face, left and right images were used as baseline comparison in the paper (Fig. 7 without inpainting).
  • "inpainted" folder
    • The face_after_inpainting folder contains images corresponding to the ones in the face_before_inpainting folder after applying the inpainting.
    • Then, the images contained in the face, left and right folders were extracted using MTCNN as above.

Unlabeled dataset (sXYZ_noglasses)

  • kinect2_calibration.yaml
    This file contains the camera projection matrix in ROS format (this file should not be required).
  • kinect2_pose.txt
    This file contains the pose of the Kinect with respect to the motion capture system (this file should not be required).
  • "face" folder
    This folder contains the faces that can be used to train the GANs (without eyetracking glasses being worn).

This work was supported in part by the Samsung Global Research Outreach program, and in part by the EU Horizon 2020 Project PAL (643783-RIA).
Files (47.7 GB)
Name Size
LICENSE.txt
md5:7931e7a7beaa761140368e83f55181c2
16.7 kB Download
README_Dataset.md
md5:b4d901b6cfcb1e3280fa8e29c6e84d99
4.7 kB Download
s000_glasses.tar
md5:5133796344086cab7421adb87a067cd4
2.0 GB Download
s000_noglasses.tar
md5:66a1bef70557a794e82a999928e9df3f
798.5 MB Download
s001_glasses.tar
md5:003443f333e0f3313213c49d0b87d4d9
4.0 GB Download
s001_noglasses.tar
md5:fb79a8d6e2f87e7262dcacfeb8c450bd
590.4 MB Download
s002_glasses.tar
md5:049edf16f5912c393b9a40b46eca2ad6
2.0 GB Download
s002_noglasses.tar
md5:b16ee2b7c18d773a274a58b0cd8993cd
596.3 MB Download
s003_glasses.tar
md5:58c50ec9d3fbb893c677c154062aff0b
4.4 GB Download
s003_noglasses.tar
md5:a7903b887f139461d1bd349ad90264e0
225.7 MB Download
s004_glasses.tar
md5:01cb190e984cf97b78993bd3cf028ace
1.8 GB Download
s004_noglasses.tar
md5:f046a5316b6db55d567bf7b8c937ab61
298.6 MB Download
s005_glasses.tar
md5:49f8ca811b3485fa141346856f860633
654.8 MB Download
s005_noglasses.tar
md5:fc84b337f90c71083c8ee9e93af978b0
344.9 MB Download
s006_glasses.tar
md5:2c1fd43bf7069c2b0cd2540ba0c3fde9
2.6 GB Download
s006_noglasses.tar
md5:98147f7db20f14080bf319f42a7e5698
472.2 MB Download
s007_glasses.tar
md5:52935565e781b25225e085f572fd4e8d
2.0 GB Download
s007_noglasses.tar
md5:5f1d86f8ec152b1334d3812a686f02c7
122.2 MB Download
s008_glasses.tar
md5:48298971c9a80f5112bcbf7247c724b0
1.9 GB Download
s008_noglasses.tar
md5:ef870a89441bc3498bea6765840cbce2
417.2 MB Download
s009_glasses.tar
md5:518ecc5d2045581c96a055157a047132
1.3 GB Download
s009_noglasses.tar
md5:77bfc049a55fbef60b4758298b1e6e1c
264.2 MB Download
s010_glasses.tar
md5:e0aee15af925bd604229a5d6f75e313b
3.8 GB Download
s010_noglasses.tar
md5:755a33f6d9d6d06ab15e5aa725030de9
1.2 GB Download
s011_glasses.tar
md5:970002164be3b55204756059bb87efaa
3.2 GB Download
s011_noglasses.tar
md5:a44edb49bff0b84fc44e6896e8586e90
891.8 MB Download
s012_glasses.tar
md5:106bccc117f4664383916420cd594303
437.4 MB Download
s012_noglasses.tar
md5:7367837c1f3c6047d24f020c79b1aa8f
217.5 MB Download
s013_glasses.tar
md5:b5fbcf8c8513f7781988ba01e9209e81
3.8 GB Download
s013_noglasses.tar
md5:5bcc872060612c7435b575f1838f6f05
606.0 MB Download
s014_glasses.tar
md5:00c87ff4e83d7080f13cdedecd7f86d4
2.3 GB Download
s014_noglasses.tar
md5:95442bc63f2575a70a861bce3834b88d
360.4 MB Download
s015_glasses.tar
md5:2ea51833a1a3b5b422908a09316367b8
1.6 GB Download
s015_noglasses.tar
md5:1a96d411ad297cdbe802e7ac992d3a90
137.8 MB Download
s016_glasses.tar
md5:7a3ea57f066904e98c4ee3c3a1032eaa
2.0 GB Download
s016_noglasses.tar
md5:8b50676c9b297539cc9e57acb04f42e3
310.7 MB Download
1,216
11,547
views
downloads
Views 1,216
Downloads 11,547
Data volume 24.7 TB
Unique views 1,059
Unique downloads 1,264

Share

Cite as