Collaborative Musical Performances with Automatic Harp Based on Image Recognition and Force Sensing Resistors

In this paper, collaborative performance is defined as the performance of the piano by the performer and accompanied by an automatic harp. The automatic harp can play music based on the electronic score and change its speed according to the speed of the performer. We built a 32-channel automatic harp and designed a layered modular framework integrating both hardware and software, for experimental real-time control protocols. Considering that MIDI keyboard lacking information of force (acceleration) and fingering detection, both of which are important for expression, we designed force-sensor glove and achieved basic image recognition. They are used to accurately detect speed, force (corresponding to velocity in MIDI) and pitch when a performer plays the piano.