PianoVAM: A Multimodal Piano Performance Dataset

The multimodal nature of music performance has driven increasing interest in data beyond the audio domain within the music information retrieval (MIR) community. This paper introduces PianoVAM, a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata. The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions, alongside synchronized top-view videos in realistic and varied performance conditions. Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering detection algorithm. We discuss the challenges encountered during data collection and the alignment process across different modalities. Additionally, we describe our fingering detection method based on hand landmarks extracted from videos. Finally, we present experimental results on both audio-only and audio-visual piano transcription using the PianoVAM dataset for benchmarking purposes and discuss other potential applications.