Deployment of LSTMs for Real-Time Hand Gesture Interaction of 3D Virtual Music Instruments with a Leap Motion Sensor

The aim of this paper is to explore deep learning architectures for the development of a real-time gesture recognizer
for the Leap Motion Sensor that will be able to continuously classify sliding windows into targeted gesture classes
related to responsive interactions that are used in controlling the performance with a virtual 3D musical instrument.
In terms of responsiveness it is assumed that the gestures can be recognized within a small time interval, while the
employed gestures are inspired by interaction with real world percussive and string instruments. The proposed method uses a Long Short-Term Memory (LSTM) network on top of a feature embedding layer of the raw data sequences as input, to map the input sequence to a vector of fixed dimensionality which is subsequently passed to a dense layer for  classification among the targeted gesture classes. The performance evaluation of the proposed system has been carried out on a dataset of hand gestures of 8 classes with 11 participants. We report a recognition rate of 92.62% for a 10-fold cross-validation setup and 85.50% for a cross-participant setup. We also demonstrate that the later recognition rate can be further improved by adapting the trained model with the addition of few user gesture samples in the training set.