Wrist-mounted IMU data towards the investigation of in-meal human eating behavior - the Food Intake Cycle (FIC) dataset

Introduction

The Food Intake Cycle (FIC) dataset was created by the Multimedia Understanding Group towards the investigation of in-meal eating behavior. The FIC dataset contains the triaxial acceleration and orientation velocity signals (\(6\) DoF) from \(21\) meal sessions provided by \(12\) unique subjects. All meals were recorded in the restaurant of Aristotle University of Thessaloniki using a commercial smartwatch, the Microsoft Band \(2\)™ for ten out of the twenty-one meals and the Sony Smartwatch \(2\)™ for the remaining meals. In addition, the start and end moments of each food intake cycle as well as of each micromovement are annotated throughout the FIC dataset.

Description

A total of \(12\) subjects were recorded while eating their launch at the university’s cafeteria. The total duration of the \(21\) meals sums up to \(246\) minutes, with a mean duration of \(11.7\) minutes. Each participant was free to select the food of their preference, typically consisting of a starter soup, a salad, a main course and a desert. Prior to the recording, the participant was asked to wear the smartwatch to the hand that he typically uses in his everyday life to manipulate the fork and/or the spoon. A GoPro™ Hero \(5\) camera was already set at the table of the participant using a small, \(23\) cm in height, tripod facing the participant, including both the food tray and upper body part in it’s field of view. The purpose of video recording was to obtain ground truth data by manually annotating the IMU sequences based on the video stream. Participants were also asked to perform a clapping hand movement both at the start and end of the meal, for synchronization purposes (as this movement is distinctive in the accelerometer signal). No other instructions were given to the participants. It should be noted that the FIC dataset does not contain instances related with liquid consumption or eating without the fork, knife and spoon (e.g. eating directly with hands). The accompanying python script viz_dataset.py will visualize the IMU signals and food intake cycle (i.e., bite) ground truth intervals for each of the recordings. Information on how to execute the Python scripts can be found below.

# The script(s) and the pickle file must be located in the same directory.
# Tested with Python 3.6.4
# Requirements: Numpy, Pickle and Matplotlib

# Visualize signals and ground truth
$ python viz_dataset.py

FIC is also tightly related to FreeFIC, a dataset we created in order to investigate the in-the-wild eating behavior. More information on FreeFIC can be found here and here.

Annotation

Micromovements

For all recordings, the start and end points of all \(6\) micromovements of interest were manually labeled. The micromovements of interest include:

• pick food, wrist manipulates a fork to pick food from the plate
• upwards, wrist moves upwards, towards the mouth area
• downwards, wrist moves downwards, away from the mouth area
• mouth, wrist inserts food in mouth
• no movement, wrist exhibits no movement
• other movement, every other wrist movement

The annotation process was performed in such a way that the start and end times of each micro-movement span the whole meal session, without overlapping each other.

Food intake cycles

For all recordings, we annotated the start and end points for each intake cycle (i.e. every bite). Each food intake cycle starts with a p, ends with a d and contains an m micromovement.

Publications

If you plan to use the FIC dataset or any of the resources found in this page, please cite our work:

@article{kyritsis2019modeling,
    title={Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data},
    author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
    journal={IEEE journal of biomedical and health informatics},
    year={2019},
    publisher={IEEE}}

@inproceedings{kyritsis2017food,
  title={Food intake detection from inertial sensors using lstm networks},
  author={Kyritsis, Konstantinos and Diou, Christos and Delopoulos, Anastasios},
  booktitle={International Conference on Image Analysis and Processing},
  pages={411--418},
  year={2017},
  organization={Springer}}

@inproceedings{kyritsis2017automated,
    title={Automated analysis of in meal eating behavior using a commercial wristband IMU sensor},
    author={Kyritsis, Konstantinos and Tatli, Christina Lefkothea and Diou, Christos and Delopoulos, Anastasios},
    booktitle={2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)},
    pages={2843--2846},
    year={2017},
    organization={IEEE}}

Technical details

We provide the FIC dataset as a pickle. The file can be loaded using Python in the following way:

import pickle as pkl
import numpy as np

with open('./FIC.pkl','rb') as fh:
    dataset = pkl.load(fh)

The dataset variable in the snipet above is a dictionary with \(6\) keys. Namely:

• 'subject_id'
• 'session_id'
• 'signals_raw'
• 'signals_proc'
• 'meal_gt'
• 'bite_gt'

The contents under a specific key can be obtained by:

sub = dataset['subject_id'] # for the subject id
ses = dataset['session_id'] # for the session id
raw = dataset['signals_raw'] # for the raw IMU signals
proc = dataset['signals_proc'] # for the processed IMU signals
mm = dataset['mm_gt'] # for the micromovement ground truth
bite = dataset['bite_gt'] # for the bite ground truth

The sub, ses, raw, proc, mm and gt variables in the snipet above are lists with a length equal to \(21\). Elements across all lists are aligned; e.g., the 3rd element of the list under the 'session_id' key corresponds to the 3rd element of the list under the 'signals_proc' key.

sub: list
Each element of the sub list is a scalar (integer) that corresponds to the unique identifier of the subject that can take values between \(1\) and \(12\). Moreover, the subject identifier in FIC is in-line with the subject identifier in the FreeFIC dataset (information available here and here); i.e., FIC’s subject with id equal to 2  is the same person as FreeFIC’s subject with id equal to 2.

ses: list
Each element of this list is a scalar (integer) that corresponds to the unique identifier of the session that can range between 1 and \(3\). It should be noted that not all subjects have the same number of sessions.

raw: list
Each element of this list is dictionary with the 'acc', 'gyr' and 'offset' keys.
The data under the 'acc' key is a \(N_{acc}\times4\) numpy.ndarray that contains the timestamps in seconds (first column) and the \(3D\) raw accelerometer measurements in \(g\) (second, third and forth columns - representing the \(x, y\) and \(z\) axis, respectively). The data under the 'gyr' key is a \(N_{gyr} \times 4\) numpy.ndarray that contains the timestamps in seconds (first column) and the \(3D\) raw gyroscope measurements in \(degrees/second\)(second, third and forth columns - representing the \(x, y\) and \(z\) axis, respectively). All sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). Finally, the length of the raw accelerometer and gyroscope numpy.ndarrays is different \(N_{acc} \neq N_{gyr}\). This behavior is predictable and is caused by the Android/MS Band platforms. The offset key contains a float that is used to align the IMU sensor streams with the videos that were used for annotation purposes (videos are not provided).

proc: list
Each element of this list is an \(M \times 7\)  numpy.ndarray that contains the timestamps, \(3D\) accelerometer and gyroscope measurements for each meal. Specifically, the first column contains the timestamps in seconds, the second, third and forth columns contain the \(x,y\) and \(z\) accelerometer values in \(g\) and the fifth, sixth and seventh columns contain the \(x, y\) and \(z\) gyroscope values in \(degrees/second\). Unlike elements in the raw list, processed measurements (in the proc list) have a constant sampling rate of 100 Hz and the accelerometer/gyroscope measurements are aligned with each other. In addition, all sensor streams are transformed in such a way that reflects all participants wearing the smartwatch at the same hand with the same orientation, thusly achieving data uniformity. This transformation is in par with the signals in the FreeFIC dataset (information available here and here). No other preprocessing is performed on the data; e.g., the acceleration component due to the Earth's gravitational field is present at the processed acceleration measurements. The potential researcher can consult the article "Modeling Wrist Micromovements to Measure In-Meal Eating Behavior from Inertial Sensor Data" by Kyritsis et al. on how to further preprocess the IMU signals (i.e., smooth and remove the gravitational component).

mm: list
Each element of this list is a \(K \times 3\) numpy.ndarray. Each row represents a single micromovement interval. The first column contains the timestamps of the start moments in seconds, the second column the timestamps of the end moments in seconds and the third column a number representing the type of the micromovement. The identifier to micromovement mapping is provided below:
\([1] \rightarrow\) no movement
\([2] \rightarrow\) upwards
\([3] \rightarrow\) downwards
\([4] \rightarrow\) pick food
\([5] \rightarrow\) mouth
\([6] \rightarrow\) other movement

bite: list
Each element of this list is a \(L\times2\) numpy.ndarray. Each row represents a single food intake event (i.e., a bite). The first column contains the start moments while the second column contains the end moments of each intake event. Both the start and end moments are provided in seconds.

Ethics and funding

Informed consent, including permission for third-party access to anonymised data, was obtained from all subjects prior to their engagement in the study. The work has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No 727688 - BigO: Big data against childhood obesity.

Contact

Any inquiries regarding the FIC dataset should be addressed to:

Dr. Konstantinos KYRITSIS

Multimedia Understanding Group (MUG)
Department of Electrical & Computer Engineering
Aristotle University of Thessaloniki
University Campus, Building C, 3rd floor
Thessaloniki, Greece, GR54124

Tel: +30 2310 996359, 996365 
Fax: +30 2310 996398
E-mail: kokirits [at] mug [dot] ee [dot] auth [dot] gr