Impact of delayed response on Wearable Cognitive Assistance

This dataset contains the data associated with our research project titled Impact of delayed response on Wearable Cognitive Assistance. A preprint of the associated paper can be found at https://arxiv.org/abs/2011.02555.

-------------------
GENERAL INFORMATION
-------------------

1. Title of Dataset: Impact of delayed response on Wearable Cognitive Assistance

2. Author Information

First Author Contact Information
    Name: Manuel Olguín Muñoz
    Institution: School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology
    Address: Malvinas väg 10, Stockholm 11428, Sweden
    Email: molguin@kth.se
    Phone Number: +46 73 652 7628
    
Author Contact Information
    Name: Roberta L. Klatzky
    Institution: Department of Psychology, Carnegie Mellon University
    Address: 5000 Forbes Ave, Pittsburgh, PA 15213
    Email: klatzky@cmu.edu
    Phone Number: +1 412 268 8026

Author Contact Information
    Name: Mahadev Satyanarayanan
    Institution: School of Computer Science, Carnegie Mellon University
    Address: 5000 Forbes Ave, Pittsburgh, PA 15213
    Email: satya@cs.cmu.edu
    Phone Number: +1 412 268 3743

Author Contact Information
    Name: James R. Gross
    Institution: School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology
    Address: Malvinas väg 10, Stockholm 11428, Sweden
    Email: jamesgr@kth.se
    Phone Number: +46 8 790 8819

---------------------
DATA & FILE OVERVIEW
---------------------

Directory of Files:
    A. Filename: accelerometer_data.csv
        Short description: Time-series accelerometer data. Each row corresponds to a sample.

    B. Filename: block_aggregate.csv
        Short description: Contains the block- and slice-level aggregates for each of the metrics and statistics present in this dataset. Each row corresponds to either a full block or a slice of a block, see below for details.

    C. Filename: block_metadata.csv
        Short description: Contains the metadata for each block in the task for each participant. Each row corresponds to a block.
    
    D. Filename: bvp_data.csv
        Short description: Time-series blood-volume-pulse data. Each row corresponds to a sample.

    E. Filename: eeg_data.csv
        Short description: Time-series electroencephalogram data, represented as power per band. Each row corresponds to a sample; power was calculated in 0.5 second intervals.

    F. Filename: frame_metadata.csv
        Short description: Contains the metadata for each video frame processed by the cognitive assistant. Each row corresponds to a processed frame.

    G. Filename: gsr_data.csv
        Short description: Time-series galvanic skin response data. Each row corresponds to a sample.

    H. Filename: task_step_metadata.csv
        Short description: Contains the metadata for each step in the task for each participant. Each row corresponds to a step in the task.

    I. Filename: temperature_data.csv
        Short description: Time-series thermometer data. Each row corresponds to a sample.

Additional Notes on File Relationships, Context, or Content
(for example, if a user wants to reuse and/or cite your data,
what information would you want them to know?):

- The data contained in these CSVs was obtained from 40 participants in a study performed with approval from the Carnegie Mellon University Institutional Research Board. In this study, participants were asked to interact with a Cognitive Assistant while wearing an array of physiological sensors. The data contained in this dataset corresponds to the actual collected data, after some preliminary preprocessing to convert from sensors readings into meaningful values.

- Participants have been anonymized using random integer identifiers.

- block_aggregate.csv can be replicated by cross-referencing the start and end timestamps of each block in  block_metadata.csv and the timestamps for each desired metric.

- The actual video frames mentioned in frame_metadata.csv are not included in the dataset since their contents were not relevant to the research.

File Naming Convention: N/A

-----------------------------------------
DATA DESCRIPTION FOR: accelerometer_data.csv
-----------------------------------------

1. Number of variables: 7

2. Number of cases/rows: 1844688

3. Missing data codes: N/A

4. Variable list:

    A. Name: timestamp
        Description: Timestamp of the sample.

    B. Name: x
        Description: Acceleration reading from the x-axis of the accelerometer in g-forces [g].

    C. Name: y
        Description: Acceleration reading from the y-axis of the accelerometer in g-forces [g].
        
    D. Name: z
        Description: Acceleration reading from the z-axis of the accelerometer in g-forces [g].

    E. Name: ts
        Description: Time difference with respect to first sample.

    F. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

    G. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

-----------------------------------------
DATA DESCRIPTION FOR: block_aggregate.csv
-----------------------------------------

1. Number of variables: 16

2. Number of cases/rows: 2520

3. Missing data codes:
    - Except for the 'slice' columns, empty cells mean that the data is not applicable or was removed from the dataset due to noise or instrument failure.
    - For the 'slice' column, a missing value indicates that the row corresponds to the whole block as opposed to a slice of it.

4. Variable List:

    A. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

    B. Name: block_seq
        Description: Denotes the position of the block in the task. Ranges from 1 to 21.

    C. Name: slice
        Description: Index of the 4-step slice of the block over which the data was aggregated. Ranges from 0 to 2, however higher values are only applicable for blocks of appropriate length (i.e. blocks of length 4 only have a 0-slice, length 8 have 0 and 1, and length 12 have slices from 0 to 2). A missing value indicates that this row instead contains aggregate values for the whole block.

    D. Name: block_length
        Description: Length of the block. Valid values are 4, 8 and 12.

    C. Name: block_delay
        Description: Delay applied to the block, in seconds.

    F. Name: start
        Description: Timestamp marking the start of the block or slice.

    G. Name: end
        Description: Timestamp marking the end of the block or slice.

    H. Name: duration
        Description: Duration of the block or slice, in seconds.

    I. Name: exec_time_per_step_mean
        Description: Mean execution time for each step in the block or slice.

    J. Name: bpm_mean
        Description: Mean heart rate, in beats-per-minute, for the block or slice.

    K. Name: bpm_std
        Description: Standard deviation of the heart rate, in beats-per-minute, for the block or slice.

    L. Name: gsr_per_second
        Description: Galvanic skin response in microsiemens, summed and then normalized by block or slice duration.

    M. Name: movement_score
        Description: Movement score for the block or slice. The movement score is calculated as the sum of the magnitude of all the acceleration vectors in the block or slice, divided by duration in seconds.

    N. Name: eeg_alpha_log_mean
        Description: Log of the average EEG power for the alpha band for the, block or slice.

    O. Name: eeg_beta_log_mean
        Description: Log of the average EEG power for the beta band for the, block or slice.

    P. Name: eeg_total_log_mean
        Description: Log of the average EEG power for the complete EEG signal, for the block or slice.

-----------------------------------------
DATA DESCRIPTION FOR: block_metadata.csv
-----------------------------------------

1. Number of variables: 8

2. Number of cases/rows: 880

3. Missing data codes: N/A

4. Variable list:

    A. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

    B. Name: seq
        Description: Index of the block in the task, ranging from 0 to 21. Note that block 0 is not to be included in aggregate calculations.

    C. Name: length
        Description: Length of the block in number of steps.

    D. Name: delay
        Description: Delay applied to the block.

    E. Name: start
        Description: Timestamp marking the start of the block.

    F. Name: end
        Description: Timestamp marking the end of the block.

    G. Name: duration
        Description: Duration of the block as a timedelta.

    H. Name: exec_time
        Description: Execution time of the block as a timedelta.

-----------------------------------------
DATA DESCRIPTION FOR: bvp_data.csv
-----------------------------------------

1. Number of variables: 8

2. Number of cases/rows: 3683504

3. Missing data codes: Columns bpm and ibi only contain values for rows corresponding to a sample taken at a heartbeat.

4. Variable list:

    A. Name: ts
        Description: Time difference with respect to first sample.

    B. Name: timestamp
        Description: Timestamp of the sample.

    C. Name: bvp
        Description: Blood-volume-pulse reading, in millivolts.

    D. Name: onset
        Description: Boolean indicating if this sample corresponds to the onset of a pulse.

    E. Name: bpm
        Description: Instantaneous beat-per-minute value.

    F. Name: ibi
        Description: Instantaneous inter-beat-interval value.

    G. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

    H. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

-----------------------------------------
DATA DESCRIPTION FOR: eeg_data.csv
-----------------------------------------

1. Number of variables: 16

2. Number of cases/rows: 114169

3. Missing data codes: N/A

4. Variable list:

    A. Name: ts
        Description: Time difference with respect to first sample.

    B. Name: delta_0
        Description: Power in the delta band, measured at Fp1 (left temple).

    C. Name: theta_0
        Description: Power in the theta band, measured at Fp1 (left temple).

    D. Name: alpha_0
        Description: Power in the alpha band, measured at Fp1 (left temple).

    E. Name: beta_0
        Description: Power in the beta band, measured at Fp1 (left temple).

    F. Name: gamma_0
        Description: Power in the gamma band, measured at Fp1 (left temple).

    G. Name: total_0
        Description: Total power in the signal captured by the electrode at Fp1 (left temple).

    H. Name: delta_1
        Description: Power in the delta band, measured at Fp2 (right temple).

    I. Name: theta_1
        Description: Power in the theta band, measured at Fp2 (right temple).

    J. Name: alpha_1
        Description: Power in the alpha band, measured at Fp2 (right temple).

    K. Name: beta_1
        Description: Power in the beta band, measured at Fp2 (right temple).

    L. Name: gamma_1
        Description: Power in the gamma band, measured at Fp2 (right temple).

    M. Name: total_1
        Description: Total power in the signal captured by the electrode at Fp2 (right temple).

    N. Name: timestamp
        Description: Timestamp of the sample.

    O. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

    P. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

-----------------------------------------
DATA DESCRIPTION FOR: frame_metadata.csv
-----------------------------------------

1. Number of variables: 6

2. Number of cases/rows: 116841

3. Missing data codes: N/A

4. Variable list:

    A. Name: seq
        Description: Absolute sequence number of the frame. Not contiguous, as frames were captured at 24 FPS but processing was only able to achieve around 15 FPS at best, so a large number of frames were dropped and not included in this dataset.

    B. Name: submitted
        Description: Timestamp marking when the frame was submitted to the application for processing.

    C. Name: processed
        Description: Timestamp marking when processing for the frame finished.

    D. Name: returned
        Description: Timestamp marking when the result of processing was presented to the user.

    E. Name: result
        Description: Result of the processing of the frame. Possible values:
            - CV_ERROR: Frame was discarded due to image processing failure (e.g. because the LEGO board was not visible).
            - JUNK_FRAME: Similar to CV_ERROR. Processing succeeded but frame included no usable information. Frame was discarded.
            - NO_CHANGE: Processing succeeded, but frame was detected as capturing the exact same task state as the previous SUCCESS frame. Frame was discarded.
            - LOW_CONFIDENCE: Processing succeeded but CV algorithm had low confidence as to the captured task state. Frame was discarded.
            - SUCCESS: Processing succeeded and a correct task state was detected. A new instruction was generated and presented to the user.
            - TASK_ERROR: Processing succeeded, but detected task state was incorrect. A corrective instruction was generated and presented to the user.
            

    F. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

-----------------------------------------
DATA DESCRIPTION FOR: gsr_data.csv
-----------------------------------------

1. Number of variables: 8

2. Number of cases/rows: 230172

3. Missing data codes: Column amplitude only contains values for rows corresponding to peaks.

4. Variable list:

    A. Name: timestamp
        Description: Timestamp of the sample.

    B. Name: ts
        Description: Time difference with respect to first sample.

    C. Name: gsr
        Description: Raw GSR reading in microsiemens.

    D. Name: peak
        Description: Boolean indicating if the sample corresponds to a peak in the GSR signal or not.

    E. Name: onset
        Description: Boolean indicating if the sample corresponds to the onset of a peak in the GSR signal or not.

    F. Name: amplitude
        Description: Amplitude of the sample with respect to the baseline.

    G. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

    H. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

-----------------------------------------
DATA DESCRIPTION FOR: task_step_metadata.csv
-----------------------------------------

1. Number of variables: 9

2. Number of cases/rows: 6760

3. Missing data codes: N/A

4. Variable list:

    A. Name: abs_seq
        Description: Absolute index of the step in the task.

    B. Name: start
        Description: Timestamp marking the start of the step.

    C. Name: end
        Description: Timestamp marking the end of the step.

    D. Name: duration
        Description: Duration of the step in time delta format.

    E. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

    F. Name: block
        Description: Index of the block to which the step belongs.

    G. Name: block_seq
        Description: Index of the step in the block.

    H. Name: exec_time
        Description: Calculated execution time for the step.

    I. Name: participant
        Description: Denotes the numeric ID representing each individual participant.
        
-----------------------------------------
DATA DESCRIPTION FOR: temperature_data.csv
-----------------------------------------

1. Number of variables: 4

2. Number of cases/rows: 230016

3. Missing data codes: N/A

4. Variable list:

    A. Name: timestamp
        Description: Timestamp of the reading.

    B. Name: temp
        Description: Measured temperature in degrees Celsius.

    C. Name: delay
        Description: Delay that was being applied on the task when this reading was obtained in time delta format.

    D. Name: participant
        Description: Denotes the numeric ID representing each individual participant.

--------------------------
METHODOLOGICAL INFORMATION
--------------------------

1. Software-specific information:

Name: Empatica E4 Streaming Server
Version: N/A
System Requirements: Windows 7+
Open Source? (Y/N): N

Executable URL: http://get.empatica.com/win/EmpaticaBLEServer.html
Source Repository URL: https://github.com/empatica/ble-client-windows
Developer: Empatica
Product URL: http://developer.empatica.com/windows-streaming-server.html
Software source components: N/A

Additional Notes (such as, will this software not run on  certain operating systems?): Windows only. Used to capture data from the Empatica E4 Wristband.

Name: Python
Version: 3.7
System Requirements: Linux
Open Source? (Y/N): Y

Executable URL: https://www.python.org/downloads/release/python-370/
Source Repository URL: https://www.python.org/downloads/release/python-370/
Developer: The Python Software Foundation
Product URL: https://www.python.org/downloads/release/python-370/
Software source components: N/A

Additional Notes (such as, will this software not run on  certain operating systems?):

The Python 3.7 language was used to:

- Develop a Wearable Cognitive Assistant for the experiment.
- Execute participant-specific experiment configurations.
- Capture the data from the E4 Streaming Server and the OpenBCI Ganglion Biosensing Board.
- Process the data into meaningfull tables and values.
- Write the data into cross-platform CSV files.

2. Equipment-specific information:

Manufacturer: Empatica
Model: E4 Wristband

Embedded Software / Firmware Name: N/A
Embedded Software / Firmware Version: N/A
Additional Notes: Used with the Empatica E4 Streaming Server to obtain GSR, Temperature and Accelerometer data.

Manufacturer: OpenBCI
Model: Ganglion Biosensing Board + Headband Kit

Embedded Software / Firmware Name: N/A
Embedded Software / Firmware Version: N/A
Additional Notes: Employed to capture EEG data, using custom Python code.

3. Date of data collection (single date, range, approximate date) <suggested format YYYYMMDD>: 2019-10-01 - 2019-11-30