Hello everyone
The title of my presentation is Al-terity: Non-Rigid Musical Instrument with Artificial Intelligence
Applied to Real-Time Audio Synthesis. I'm Koray Tahiroğlu and this work is produced with Miranda Kastemaa and Oskar Koli.
Al-terity is a new musical instrument that we have been working on since last autumn. It is a
deformable and non-rigid musical instrument. The instrument itself is a close folded shape and its physical deformability
becomes an opening of the instrument's folded shape, as you can see in this video clip, and the physical manipulation
changes the control parameters in a sample based granular synthesis audio module.
Rigid instruments can take various different shapes with there many features, so building audio characteristics for such an interface
behavior could be challenging and in that sense
what is really interesting in Al-terity instrument is that the instrument sound synthesis module is based on GANsynth deep learning
model that generates
audio samples for real-time audio synthesis.
So when the performer applies a handheld physical action the instrument generates new audio samples and distributes them to the
particular contact points on the surface for further audio processing.
One of the main benefits we find using
GANsynth is the ability to produce a realistic range of timbre based on the training data set as well as the ability
to generate new audio samples in real time in the moment of playing with
sound characteristics that the performer has never heard before.
I guess it's more obvious today deep learning and artificial intelligence
tools are currently shaping our relationship with music and they do enable new possibilities of
utilizing new musical instruments and interfaces.
This is something we know that is tightly connected with machine learning
models. Machine learning models are commonly used in practices with new musical instruments in the NIME community.
Artificial intelligence methods applied to music can be traced back
centuries in relation to its automated algorithmic features,
but today the growing use of AI in music practices through more advanced computational features brings in new
opportunities that are not limited to only
automation.
In this project we combined our interest in deep learning with audio and our
experience in our earlier work in which we developed a shape
retaining, freely deformable interface, and as a result this has allowed us to focus more on the
real time synthesis of sound by creating
completely new sounding sounds. The idea for the physicality of the Al-terity instrument appears to be more
related to how the latent space in GANsynth model is organized as a
spatial distribution of timbre features of audio samples. The shape of this instrument is based on a
visualization, or the representation, of such high dimensional space which is extracted from our approach to represent
the latent space as unevenly and geometrically formed surfaces of a folded paper.
Unfolding the flexible shaped paper let us create 3D drawings shown on this slide, and
then we 3D printed to object with a mix of photo polymer
material which allowed us to set the level of stiffness that we wanted to have in this instrument.
The shape of this instrument makes it possible to access the instrument in a number of different ways.
It might be important to note here that the audio samples used in the audio
synthesis in this instrument are not created or prepared beforehan. Instead,
GANsynth generates an audio sample each time the instrument receives direct handheld contact on the surface.
Audio synthesis
modules here in this instrument is organized in three layers. The bottom layer synthesis module is based on three granular
synthesizers each processing a separate audio sample
manipulating the parameters to generate a mix of sustained sounds. The temporal features in this synthesis photo can be described as a
wavy drone sound. The middle layer
synthesis model is another granular synthesizer. A significant difference here
is that it outputs more percussive sounds. An output of the top layer synthesis model generates different
waveforms. This part of the instrument is
coupled with smaller sized grains with shorter durations. When the performer opens up the folded parts
the audio module responds with subtle but sharp sounds more percussive than the middle region
responses.
So what does GANsynth do for us in this project?
GANsynth is an audio synthesis algorithm based on generative adversarial networks, GANs,
introduced by google magenta team in January last year.
GANsynth is designed for generating musical notes at specified pitches
GANsynth uses better audio quality and also synthesizes thousands of times faster
compared to their previous deep learning model that they use in the GANsynth project
It's much improved speed makes the algorithm suitable for interactive purposes, potentially even near real-time
applications for us.
You can find more about
GANsynth in our paper and its author's papers, but very briefly the GANsynth network learns
to represent timbre as vectors in a higher dimensional latent space.
It is also conditioned on one representation of pitch to allow
independent control of pitch and timbre in the synthesis process
here you can see the interpolation of the audio
samples that are organized in random order in GANsynth latent space.
The ability to generate new audio samples in real time with the
characteristics of the sound that the performer ever heard before and the such structure of audio synthesis that continuously changes its behavior
right in the moment of playing or performing
I would say is unusual compared to many other instruments with predetermined sound producing features.
This gives the performer the possibility to develop a particular relationship with the instrument that is
idiomatic to the instruments unusual behavior, which allows massive flexibility and further exploration of the instrument's playability.
Our particular approach on the digital idiomaticity of this instrument has been the
possibility to be able to engage in an
unusual state of playing with Al-terity instrument, in which the performer remains in a state of
continuous uncertainty. Remaining in such uncertainty creates a particular form of idiomatic relationship between the
unfamiliar, unusual musical instrument and the performer
We composed the piece
"Uncertainty" dramatically reflecting the unusual behavior of the Al-terity instrument keeping the performer remaining in an
unconfident state of performance.
This composition turns into a continued state of playing
reformulating our relationship with the Al-terity instrument and opening up a new variety of musical demands.
We argue hear in this paper that these advanced and
intelligence features on the audio synthesis level could allow us to explore performing music with particular features that define the
instrument's digital idiomaticity and allow us to reinvent the instrument in the act of music making
performance. Reinventing the instrument here might sound like an overwhelming demand, but surely we can be challenged to consider this. Here
what we mean by
reinventing is exploring
the instrument's potential to reshape its idiomatic features,
discovering new ways of playing the could emerge from the instrument itself.
Most likely this reshaping could happen in audio synthesis modules for instance
granular synthesis is about modifying the underlying patterns of audio samples a
particular timbre of sound in this synthesis could be very fundamental to the
characteristics of such a digital musical instrument. Changing the characteristics of the audio samples could result in a very
distinctive flavor for the output sounds.
So what we have learned: we acknowledge the potential of generative adversarial networked GANS for further
developments of NIMEs as they are capable of generating realistic and at the same time with new characteristics or sounds for a given data set.
AI models are appearing more in the audio domain and these tools could open up new opportunities
for the practices in NIME community.
One possible direction that we mention in the paper that this project could take in the near future is dealing with the
preparation of our own data set with samples from various alternative sources and train the checkpoint with that dataset.
In this current version of the Al-terity instrument that you have seen in this video clip earlier
we use the existing train checkpoint provided by the
Magenta team. This is the timbre characteristic for the generated audio samples.
Now we have checkpoints
generated through various different types of datasets and they're all available on our GitHub page that you will see at the link on the
last slide.
We have also mentioned in the paper that its in our research interest to develop further features for conditional
generation of all your samples with a potential GAN model. The current GANsynth
model is conditioned to a pitch function and this allows us to only set the pitch variable for audio samples.
After the submission of this paper
we have already spent a good amount of time on modifying the current GANsynth algorithms to implement more
conditions trying to make the model respond to other conditions besides pitch.
But it was always after a while as GANsynth also responded us
GANsynth could not reliably learn such higher-level
semantic characteristics: we try to add 10 new labels as further audio features
to the generator input and its related
classification so the discriminator ever such conditions are not really possible to be implemented with GANsynth architecture
We didn't give up: we came across another GAN. It
functions on image domain and it provided us promising results looking into latent space and putting a list of
most dominant latent vectors organized in that space which represents the most common features for the
characteristics of that a medium. So we managed to merge our work on GANsynth with that and
generated further
conditions for GAN in the audio domain. The final output of this part of the work is actually just happening while I'm making this video
presentation, so more will be presented later on. You can check our GitHub and at the moment it might look like it's not
that well-organized, but you can come back after the holidays, most likely in autumn, and we will have more
detailed instructions for this project. And
before I thank you all, I would like to mention that Al-terity instrument with its "Uncertainty"
composition will be presented at this year's Ars Electronica Festival with its new features from the conditional GAN. Thank you