Hello everyone. I’m Haruya TAKASE from Nagoya Institute of Technology in Japan.
Today, I’d like to talk about Support System for Improvisational Ensemble Based on Long Short-Term Memory Using Smartphone Sensor.
Let me explain this study's purpose.
In our laboratory, we have developed an improvisational ensemble support system based on user’s physical gesture for music beginners.
When music beginners play improvisational performance, the tonality is difficult.
It is difficult for beginners to play melody having tonality on a particular chord.
So, our system corrects tonality of melody by outputting consonant tones.
On the other hand, rhythm and pitch contour represented by up-and-down of melody are simple elements.
So, we develop the system inputting rhythm and pitch contour by physical gesture.
Let me explain the performance systems developed in our laboratory so far.
There are two systems: one is an accompaniment performance system that manipulates the chord progression.
The other is a melody performance system that can play melodies harmonized with the chord progression.
There are three implementations of the melody performance system.
In this study, we picked up the playing method by the up-and-down and shaking movements of the smartphone, which are circled in red.
You can see how they played with the existing systems.
An existing melody performance system with swinging motion was implemented based on a Bayesian network.
The elements that implemented the system were attack timing, vertical motion and pitch.
As a problem with the existing system,
it was raised that "when the attack timing is estimated by the shaking motion, the estimation accuracy falls"
and "the melody sometimes moves in the opposite direction to the up and down motion of the smartphone".
Therefore, the purpose of this study is to improve the accuracy
by "estimation based on Long Short-Term Memory" and "reconsidering the training data shaping method".
Next, let me explain the system architecture of this study.
The smartphone sensor values obtained by the user moving the smartphone to the accompaniment sound of the background music
are input to the attack timing estimation LSTM and the vertical motion estimation LSTM, and the attack timing and the vertical motion are estimated.
Next, the chord progression information of the background music and the estimated value of the vertical motion estimation LSTM
are input to the pitch estimation LSTM and the pitch is estimated.
Finally, the estimated pitch is output as a melody at the estimated attack timing.
This study is a system that uses a smartphone, which is now widely used, and is currently targeted at Android smartphones.
The sampling rate of the smartphone sensor is 200Hz and the Android Sensor Framework is used to obtain the sensor values.
Next, let me explain the method of collecting training data for each LSTM model.
The data required for learning are body movement data and music information text data.
For the body movement data, various sensor values and hand height values are recorded when the smartphone
is moved to the melody of the music data that is easy to perform body movements prepared in advance.
On the other hand, music information text data is extracted from MIDI data in the RWC music database to show the melody track and NoteOn time.
In addition, we extract the code name and NoteOn time from the annotation data in the database.
This video was taken when we recorded the body movement data.
Here is an overview of the inputs and outputs of the three LSTM models.
We conducted a preliminary experiment to determine these optimal feature sets.
Let me explain how to shaping training data of the attack timing estimation LSTM.
Previously, the training data was in two states of phonetic state or not.
However, there was a misjudgment due to an outlier.
Therefore, in this study, we divided the training data into three sections,
"swing-down section", "stop section", and "other motion section“,
and made the sound output at the timing of the shift from the swing-down section to the stop section.
In order to create three sections, we need to find the start time of each section
from a certain period of time before and after the MIDI sound source's NoteOn time.
Since the start time of the swing-down is as the time when the smartphone is swung up the most,
the start time is defined as the time when the Rotation(z) is the minimum.
The start time of the stop is defined as the time when the acceleration is at its maximum
using the magnitude of the change in acceleration caused by the stop of the smartphone.
The start time of the other movement is the time when the smartphone stops
and then swung up to some extent, so it is the time that satisfies the formula shown.
Training data for the vertical motion estimation model is collected using Kinect 3D camera.
It records the thumb height of the hand holding the smartphone.
The model estimates the relative vertical movements of each frame,
and classifies the movement into three categories: "up," "no change," and "down" based on a threshold.
Let me explain how to create training data for the pitch motion estimation model.
The system creates a one-hot vector of the input "chord names".
It also creates a one-hot vector of MIDI note numbers for the output "pitches".
The pitch estimation is done for each attack timing.
We verified the accuracy of each model.
For the attack timing estimation, there were no estimation errors and no useless estimation,
and the Mean absolute error was within one frame, which is very accurate.
Dividing it into three sections led to the high accuracy of this model.
For the vertical motion estimation, the low accuracy of No change is noticeable.
Insufficient scrutiny of the no change threshold is an issue for the future.
For the pitch estimation, the accuracy was very low.
the same pitch was continuously estimated, pulled by a previously estimated pitch.
I’d like to conclude my presentation.
The estimation of vertical motion and the pitch are insufficient,
but it is now possible to specify the attack timing accurately by shaking the smartphone.
In the future, we will improve the accuracy of vertical estimation and pitch estimation,
and implement an Android application using the three models and questionnaire evaluation using the application.
Thank you for listening.