UCOPhyRehab++: Multi-modal and multi-view dataset for human rehabilitation analysis

Dataset Description

UCOPhyRehab++ is a multi-view, multi-modal video dataset designed for research on physical rehabilitation analysis, human motion understanding, and computer vision–based biomechanical assessment. The dataset captures subjects performing a set of standardized rehabilitation exercises under controlled conditions, enabling the study of motion patterns from both visual and kinematic perspectives.

Data Acquisition and Structure

The dataset consists of recordings from multiple synchronized camera views, with a consistent hierarchical organization by subject, exercise, and camera. Each exercise is performed following predefined protocols and includes repetitions for both left and right sides when applicable. This structure allows reproducible experimentation across views, modalities, and subjects.

Included Modalities

This Zenodo release provides the following visual modalities, computed from the original RGB recordings:

• sils: Binary silhouette representations preserving full-frame spatial context.

• sils_cropped: Cropped silhouette images centered on the subject to reduce background influence and focus on body motion.

• of_gmflow: Optical flow fields computed using GMFlow, capturing dense motion information between consecutive frames.

• of_tvl1: Optical flow estimated using the classical TV-L1 method, included for comparison with learning-based motion representations.

• semantic_segmentation: Pixel-wise semantic segmentation maps of the human body, providing structured spatial information beyond binary silhouettes.

• samples: A folder that contains a unique subject with all the modalities for quick inspection purposes.

All modalities preserve the original dataset structure to facilitate multimodal and multi-view experimentation.

Ground-Truth and Metadata

In addition to the visual data, the dataset includes two key annotation files:

• dataset_3d_with_angles.json - Contains 3D ground-truth joint information obtained using an OptiTrack motion capture system. The file provides joint coordinates and derived joint angles for the articulation involved in each rehabilitation exercise, enabling quantitative biomechanical analysis and validation.

• ucophyrehab_data.jsonl - A comprehensive metadata file aggregating all additional information associated with this dataset version, including subject attributes, exercise descriptors, view identifiers, modality availability, and other relevant annotations. Each entry corresponds to a single recording instance, facilitating programmatic access and large-scale analysis.

Code availability

The code required to reproduce the data processing pipeline and the experimental results associated with this dataset will be made publicly available on GitHub at: UCOPhyRehab++