Hi,
today we're going to talk about ExSampling,
a system for the real-time ensemble performance of field-recorded environmental sounds.
This reserch is worked by Atsuya kobayashi and Reo Anzai and Nao Tokui
from Keio University Computational Creativity Lab.
So, we're gonna start with overview
and show some videos
and explain about sound classification
and explain about the performance use case.
So, we proposed ExSampling
on integrated system of recording application
and deep learning improvement for real-time music performance
of environmental sounds sample by field-recording.
Automated sound mapping to Ableton Live
tracks by deep learning and the enables field-recording
to be applied to real-time performance
and create interactions among sound recorders, composers and performers.
So nowadays,
collecting specific sound by field-recordings
and the using this sound becomes general techniques.
However, when demonstrating computer music performance using these environmental sound,
artists must select sound sample from recorded sound
by listening to them
and post them such as equalizing and map them to music scene and musical instrument.
So it takes a long time for artists before starting to use those sound in their music.
For this reason,
the number with a specific sound sample that can be used is limited
and the real-time nature of the performance is lost.
This system needs the first prototype
the application of the machine learning for employee field-recorded sound in real-time musical expression.
So, we're gonna show you guys some demo videos
by solo player and recorder.
Next, this is a demo video of using our system
by multiple participants while doing two meetings.
So the system consists of the flowing three modules.
The first module is a Web-based sound recorder for collecting environmental sound.
The recording function is implemented using recorded.js,
a rubber library for the Web Audio API.
On the sound recorded by the device,
microphone can be immediately sent to the performance computer.
By publishing the recorder application
to internet through network port forwarding using ngrok,
anyone can participate in the recording of the performance.
The second module is a Web API server
and the deep learning execution environmental written in Python
This module also functions as a backend for providing a sound recorder.
After counting the part of the received sound file of less than 20 disposed,
the module convert to the spectral image
and impeded to the deep learning model to estimate the type of the sound.
The sound file path the estimated sound type
and the detected sound pitch under location information
are sent to the Max/MSP by OSC.
The third module, the Max/MSP patch program,
link tracked in a dll to the sample sound based on the transmitted sound file
on classification result,
the buffered sound file is played to the note inform the map MIDI track.
The user interface of the recorder is implemented in p5.js,
and it's designed to make the operation of the field-recording easy to understand.
Since the log message from the local web server displays the recognized class over the sound,
recording participants can figure out how the sound will be used in the performance.
The Max for Live Device had
map type user interface that allow the performance
to check where the sound were recorded.
We implemented this interface with the web audio application API and map box API
For the classification environmental sound,
we adopted MobileNetV2,
a lightweight convolutional neural network architecture.
For low latency real-time interface
on the local laptop computer,
to use this CNN model,
the sound data were processed using the Python library librosa.
The recorded environmental sound cut into five second increment
each of which is converted into a spectrum
that model reads them at the three channel image matrix
and making inference of the class.
We train the model using sound data set with unrelate 41 classes
that were used and publishing the sound recognition competition at kaggle.
We implemented world training an interface workflow
based on open source tensorflow sample source code.
Sound mapping patterns among the classification result
and mini instrument tracks can be set by writing configuration file
on Python server directory
or changing max full live device options.
This figure shows how we mapped classification
result of instrument tracks in Ableton Live.
So, this is Performance Use Case: Timeline.
So firstly performance create MIDI note clips beforehand
or create one during the performance to play the recorded sound
and share QR code which it's issued to access the web recorder.
Now participants can access the recorder on the internet via the QR code.
When participants recorded various environmental sound with the recorders,
the sound data and location information automatically sent to the perform local computer
and as soon as it received deep learning inference about class a matting
Instrument mapping based on the classification result,
it performed on Max and samples assigned to meet tracks and switch in real-time.
So performer can start using this sound with their music in real time.
The ExSampling system made it possible
to apply field-recorded environmental sound to real-time performance.
We believe that method we proposed here
encouraged more mission to use few recorded sound in depth live performance.
Based on the methods,
performance also can sample or assigned audio source
to samplers and recence machine automatically on stage
and to mix them into pre-recorded tracks they play.
Automation of the sound cut up and mappings on the instrument using machine learning
gives opportunity for performers to use real-time recorded sound.
The main contribution of this research
is that by using this system,
we propose a new form of interaction
in which many recordings participants create a musical performance
and culture of live music.
Thank you for listening.