Predicting Flutist Onset Timing in Duet Performance: A Multimodal Analysis of Gesture and Breath Cues

In ensemble performances, musicians use gesture and breath cues to synchronize their initial notes at the beginning of a piece, but the precise relationship between these cues and onset timing remains under-explored. This study investigates how flutists' gesture and breath cues encode the timing information for the initial note onset. This research consists of four components: (1) Collection of a cue dataset containing synchronized video and audio recordings of flute-piano duets, (2) Identification of cue candidate points through facial movement curves and breath onset-offset analysis, (3) Verification of predicted onset accuracy using linear regression on these cues compared to human onset asynchronies and (4) Introduction and exploration of a `trigger' concept, defined as immediate, clearly perceivable gestures (such as stopping or raising the head) indicating the precise moment of onset. Our findings suggest a dual-cue system: preparatory cues broadly predict onset timing, while precise triggers refine the exact onset. We compared the time difference between the predicted and piano onsets with the flute–piano asynchronies and verified the concepts of cue and trigger through expert interviews. This research contributes to a deeper understanding of the complex phenomena of musical cues during performance through multimodal analysis. This paper provides an open-access cue dataset, which can be found on the accompanying website.