SENSE-42: A multimodal human-computer interaction dataset for neurocognitive user state evaluation
Authors/Creators
-
1.
Queen Mary University of London
Contributors
Data collector (2):
-
1.
Queen Mary University of London
Description
The dataset includes multimodal signals across the following channels:
Behavioural Data
Participant interaction logs collected via PsychoPy, including task performance metrics, reaction times, and mouse & keyboard status per frame.
Electroencephalography (EEG)
32-channel scalp EEG recordings collected with the BioSemi ActiveTwo system at 1024 Hz. EEG signals are stored in .bdf format. We further shared the cleaned EEG files in .set format compatible with EEGLab in this repository.
Electrocardiography (ECG)
3-lead ECG signals extracted from the external channels of the EEG system. Electrode placement follows a modified lead configuration (see diagram in the GitHub repository).
Respiration Data
Respiratory cycles captured using a thoracic breathing belt, collected with BioSemi ActiveTwo system, processed with RespInPeace and stored as .wav files at 32 Hz.
Webcam Recordings
High-resolution facial video recordings were obtained for facial expression and pupilometry analysis.
All participants agreed to share their data for authorised research use.
Abstract
SENSE-42 is a publicly available, multimodal dataset designed to support the study of user state monitoring during extended computer interaction sessions via neurocognitive, physiological and behavioural responses. Combining high-resolution neurophysiological recordings with behavioural and subjective data, this dataset enables research on the alternations of attention, mental/physical fatigue, cognitive workload, and related subjective indices at a very early stage.
The dataset was collected from 42 participants over a 2-hour continuous interaction session, during which participants engaged in a series of designed tasks on a desktop computer with a mouse and keyboard. The experimental tasks were conducted within a fully simulated desktop operating system environment, designed to closely mimic real-world computer usage scenarios. This setup mirrors how people typically use computers in daily life. The simulated experiment program also enables the comprehensive capture of the mouse and keyboard data, synchronised with a high refresh rate monitor at 144 Hz. Recordings were collected in a noise-insulated room, minimising external interruptions from the environment or the experimenter.
Other
We introduce a multimodal dataset named Simulated Environment for Neurocognitive State Evaluation (SENSE-42), designed to study early alertness state fluctuations during realistic computer interaction in a desktop operating system environment. To accommodate user preferences while maintaining a unified experimental design, we developed precise replicas of two major operating system interfaces (macOS and Windows) to facilitate the study of task-related behavioural responses. Both interface styles were presented to each participant during the session, allowing within-subject comparisons across familiar and unfamiliar environments. The dataset includes synchronised high-resolution neurocognitive responses in multiple modalities, including physiological, behavioural, and self-reported data, from 42 participants within a simulated desktop operating system. These responses were collected over continuous 2-hour sessions for each participant, providing rich measurements of spontaneous fluctuations in user states. Structured tasks were designed to emulate everyday computer use—file management, typing, browsing, and application switching—under tightly controlled conditions. Participants self-reported their alertness state at five-minute intervals using predefined scales, including the Karolinska Sleepiness Scale (KSS), NASA Task Load Index (NASA-TLX), and attentiveness ratings. Physiological data were collected to profile internal user states from EEG, ECG and respiration signals, and behavioural data were recorded for keyboard and mouse dynamics. Additionally, trait-level measures such as the Epworth Sleepiness Scale (ESS) and the Pittsburgh Sleep Quality Index (PSQI) were collected to contextualise inter-individual variability in daytime sleepiness and nighttime sleep patterns.
The SENSE-42 dataset is collected in a time-locked experimental setting to replicate real-world computer use scenarios. It provides a structured resource for examining tonic alertness dynamics during human-computer interaction and may be used to explore relationships between various multimodal indicators of the user state and their personal traits. The dataset includes components relevant to research in areas such as cognitive state monitoring, user behaviour analysis, and physiological computing.
Methods
A total of 42 participants were recruited in our study, consisting of 21 males and 21 females. The average age of participants is 26.8 +/- 4.5 years, ranging from 18 to 40. Participants represented diverse ethnic backgrounds, education levels, and occupations. To preserve ecological validity, they were instructed to maintain their usual intake of substances that could influence alertness, such as caffeine or medication. The study was approved by the Queen Mary Ethics of Research Committee (Reference Number: PSY2025-32), and the procedures were designed in accordance with the Data Protection Privacy Notice of Queen Mary University of London. Detailed explanations of the study and its procedures were provided to the participants, and informed consent for data publication with optional data-sharing preferences was obtained from all participants. Participants received £30 for taking part in the study.
Notes
The dataset is organised in a flattened structure to facilitate ease of download and processing for individual or combined modalities. Data preprocessing and analysis pipeline was implemented using MNE, NumPy, Pandas, and SciPy. The source code for the experimental software is available under the ‘experiment’ branch in the GitHub repository, while the main branch contains the analysis pipeline for data verification and further exploration.
Designed around computer users, this dataset supports the investigation of tonic alertness dynamics during human-computer interaction. It enables the study for physiological fingerprints of behavioural performance and supports analysis of within- and between-subject factors in spontaneous user-state fluctuations. Beyond user state analysis, the dataset allows for user behaviour analysis based on individual traits and graphical interface design, offering opportunities to explore the physiological and behavioural patterns related to the experience, habits, preferences of computer usage and cross-system design patterns. Furthermore, through the integration of physiological signals with interface interactions, this dataset may further support the development of physiological computing studies, where signals from the brain and body are transformed into adaptive inputs to optimise or personalise human-computer interfaces.
Files
Restricted
The record is publicly accessible, but files are restricted. <a href="https://zenodo.org/account/settings/login?next=https://zenodo.org/records/20328099">Log in</a> to check if you have access.
Request access
If you would like to request access to these files, please fill out the form below.
You need to satisfy these conditions in order for this request to be accepted:
Thank you for your interest in the SENSE-42 dataset. Access is granted for approved research use under the terms set out below. Please complete the request, and we will review your request as promptly as possible. Upon approval, your access will be granted for 1 year.
If you are not logged in via your institutional account, please provide an institutional email address or other verifiable identity information to support the approval process.
Conditions of access
By submitting this request, you confirm on behalf of yourself and any collaborators that you agree to the following:
1. Permitted use. You may use the dataset for internal research and analysis, including academic research, methodological development, replication studies, educational use, and commercial research and development, provided that such use does not breach any other term of this agreement. Publishing analyses, releasing trained models or algorithms derived from the data, and sharing aggregate or non-identifiable results are permitted, provided no primary data are redistributed and no participant is identifiable in the outputs.
2. No redistribution. You will not share, transfer, publish, mirror, upload, or otherwise redistribute the dataset or any subset of it, including derived files that retain participant-level information, to any third party. Collaborators who require access must submit their own request through this platform.
3. No re-identification. You will not attempt to identify, contact, or otherwise establish the personal identity of any participant, whether directly from the data or by linking it to other datasets, public records, social media content, or biometric matching services. You will report any inadvertent identification to the dataset authors immediately.
4. Restrictions on identifiable content. Webcam recordings, video frames, screenshots, and any other visually identifiable material from this dataset must not be reproduced in publications, presentations, posters, preprints, teaching materials, public talks, social media, blog posts, or any other public or semi-public output. Anonymised illustrations (e.g. blurred, schematic, or non-photographic figures) are permitted only if no participant is identifiable.
5. Secure storage. You will store the dataset on devices and infrastructure that meet the data security standards of your institution and the requirements of UK GDPR and the Data Protection Act 2018, including access control, encryption at rest where practicable, and protection against unauthorised access. You will not store the dataset on personal cloud accounts or unmanaged removable media.
6. Retention and deletion. You will retain the dataset only for as long as needed to complete the stated research purpose. Upon completion, withdrawal from the research, or upon request by the dataset authors, you will permanently delete all copies of the dataset and confirm deletion in writing if requested. Aggregate or fully anonymised derived outputs are not subject to this deletion requirement provided they do not enable participant re-identification.
7. Citation. You will cite the SENSE-42 dataset and its accompanying data descriptor in any publication, preprint, presentation, or report that uses the data, in the form provided on the dataset landing page.
8. Compliance with ethics and law. You will use the dataset in accordance with your own institution's research ethics policies, applicable data protection law, and the terms of the original ethics approval (Queen Mary University of London, PSY2025-32). If your intended use requires separate ethics review at your institution, you will obtain it before beginning work with the data.
9. Breach notification. You will notify the dataset authors within 72 hours of becoming aware of any actual or suspected loss, theft, unauthorised access, or other security incident involving the dataset.
10. Participant withdrawal. Participants retain the right to withdraw their data from the dataset. If a participant withdraws, you will be notified and required to delete that participant's records from your local copy within a reasonable period.
11. No warranty. The dataset is provided as-is, without warranty of any kind. The authors and Queen Mary University of London accept no liability for any loss or damage arising from its use.
12. Revocation of access. The dataset authors reserve the right to revoke access if any term of this agreement is breached, in which case you will delete all copies of the dataset as set out in clause 6.
13. Intellectual property. This agreement does not grant or transfer any intellectual property rights in the dataset, in any derived works, or in any pre-existing rights of the user. Existing legal protections continue to apply independently of this agreement.
By submitting this form, you confirm that you have read, understood, and agree to be bound by these conditions, and that you have authority to enter into this agreement on behalf of any collaborators you intend to share working access with within your institutional environment.
You are currently not logged in. Do you have an account? Log in here
Additional details
Identifiers
Related works
- Is cited by
- Conference paper: 10.1145/3772363.3798603 (DOI)
- Is published in
- Preprint: 10.1101/2025.09.03.673947 (DOI)
Funding
- China Scholarship Council
- PhD Scholarship 202309210085
Software
- Repository URL
- https://github.com/Catherine9811/HCI-SENSE-42
- Programming language
- Python
- Development Status
- Active