Hello everyone, my name is Gilberto Bernardes.
I work at the University of Porto in INESC TEC,
and today I have the great pleasure of addressing the NIME
community to present the revamp of earGram, a long term music software project
I've been developing since 2009.
EarGram aims at repurposing musical audio by interfacing with audio descriptors.
The talk will be split into four major parts.
I'll start by briefly introducing the concept of computation in the creative applications of descriptors.
I'll then motivate the current revamp of the earGram software
as well as its major analysis synthesis methods and
finally, I'll conclude by summarising the main contributions of the work
Audio descriptors can be understood as a specific case of machine listening,
as they provide some contextual information
about the content of an audio signal.
They can be expressed as either symbolic or numeric data and
its extraction process can be done manually or automatically.
In my work, I'll focus on automatic methods for extracting
descriptor data as continuous and discrete numerical values.
Audio descriptors are typically split into categories according to their level of abstraction,
from low to high levels.
Low level descriptors focus on physical or sensorial information
such as duration and timbre,
mid-level descriptors on perceptual and formal information
such as key and dynamics,
and high-level descriptors on cognitive traits
such as style and composer.
Starting from an audio signal waveform on the left
we typically compute audio descriptors by isolating a slice of the audio,
which we may need to transform to the frequency domain,
and then we compute descriptions from the resulting representation
as if reducing the information of the distribution.
Audio descriptors can be found at the back end of many
state-of-the-art musical applications,
namely for creative workflows.
Representative examples include
concatenative sound synthesis tools,
beat matching,
harmonic mixing DJ software, and
interfaces for promoting active listening and music discovery.
The slide shows two screenshots of the early and current version of earGram.
The accumulated experience of playing with the early earGram software
drove my motivation to adopt, in a recent version, a human-in-the-loop approach,
decreasing the amount of automation,
and aiming to enhance the creative potential of expert users.
The spectral morphological descriptor scheme in the earlier version
was replaced by a more efficient set of descriptors,
rendered in prominent qualities of music notation:
at the rhythmic, harmonic, dynamics, and timbral levels.
These multiple audio viewpoints can be then
selected to create a finite alphabet,
from which a factor oracle structure is drawn
to reveal temporal morphological patterns in the audio.
Rhythm is the first description stage in earGram.
It aims to segment the audio source into a collection of
component audio units on prominent sound event onsets,
either notes or any other sonic objects, and beat locations.
A user-defined variable,
defining the lengths of the bar or phrase pattern,
is then used to automatically label each audio unit
to denote temporal regularities.
EarGram uses the tonal interval space to capture the harmonic affinity
between keys, chords, and notes.
Roughly, the space is computed as the DFT of chroma vectors.
Two continuous phase spaces
are adopted to reduce the information to two dimensions,
and then a clustering algorithm
partitions the units into multiple discrete groups.
Quality threshold clustering is adopted using
user-defined criteria for the distant threshold between
units and the minimum number of units per cluster.
At the key level, express relations in terms of distances
are similar to those of the circle of fifths.
At a chordal level, perceptually related chords
such as those sharing functional categories
are at smaller distances.
Finally, at the note level,
intervals understood as perceptually related within Western tonal music,
such as fifth and thirds,
are equally located at smaller distances.
Continuous descriptions of energy
are then converted into a user-defined number of discrete dynamics
using a division into quartiles.
MFCC is a standard descriptor to capture timbral qualities from audio.
I wrote this strategy in the continuous space
which are then discretized using the quality threshold clustering technique.
Discrete matching algorithm factor oracle is adopted in earGram to unveil
the audio source temporal morphology.
Links connect repeating sequences or optimal continuations between units.
We can extrapolate, from the representations,
sections in the temporal dimension of the audio source,
which the user can then use to navigate in the units collection.
To expose the deployment potential of the proposed hierarchical audio scheme
in generating musical content,
I will now present three strategies for repurposing musical audio
using descriptors from low to high levels of abstraction.
The opening video of my presentation and the example video whose sound we hear in the background
show the potential of the timbral and harmonic continuous description spaces
in creating sonic trajectories at various density rates
and with highly controllable accuracy of timbral and harmonic qualities.
[audio example]
Mid-level categorical descriptions can be explored
to easily create sequencer-based interfaces
with multiple degrees of automation.
The video example shows the adoption of audio clusters in a Euclidean rhythms generator.
[audio example]
Finally the long-term and morphological structure description in factor oracle
is adopted as a strategy to recreate infinite variations of the audio source,
one warranting some degree of sonic and formal structural retainment.
We will hear two audio sources:
1) an electronic dance music track
2) and a soundscape.
[audio example]
I've just presented a revamp of the earGram software
for the fluid repurpose of musical audio
using generative strategies, while retaining sonic and formal attributes of the source.
It is a minimal audio descriptor scheme in continuous and discrete spaces,
which mostly target expert users.
The descriptors are highly contextual and have minimal assumptions from the source.
And finally, it allowed for the adoption of generative strategies
at the various levels of descriptors' instruction.
Thanks for your attention.
I'd just like to point out that earGram is available for free
as an open-source tool for Pure Data,
and can be easily reached from a Google search.