Capturing Violin Performance Gestures with a Single Camera System

In this work we present a novel approach in capturing violin performance gestures by employing both direct and indirect acquisition methods based on low-cost equipment that is available with any modern personal device. It is a multidisciplinary study that covers several research fields, including audio signal processing, machine learning, motion capture and computer vision. According to the bibliography, the majority of the proposals employs a direct way for the acquisition of musical gestures, by measuring the physical variables with the support of sensors which are placed on the instrument or on the performer. However, these systems invoke some kind of intrusiveness that affects the performance procedure. An alternative approach is to apply indirect acquisition from the analysis of the audio signal. The main difficulty of this method is to develop robust detection algorithms and provide accurate measurements similar to the sensors. Therefore, our goal was to implement a hybrid audio-informed system that utilizes the built-in web camera and microphone of a laptop, in order to provide qualitative feedback to the performer. This was achieved by developing an algorithm that employs video frame analysis, augmented reality and audio signal processing methods. After computing the various features from the video and audio domains, we unified the retrieved information into a single dataset in order to apply feature selection and machine learning techniques for investigating the regression prediction between the audio descriptors and the bowing controls, as they were computed from our analysis algorithm. The results are promising, since they present high correlation rates between the Bow Inclination parameter and the audio features, with maximum accuracy of 97%.