Hi I'm Lia Mice and in this video I'm
talking about my NIME 2020 paper "From miming
to NIMEing: the development of idiomatic
gestural language on large-scale DMIs.
This is research I conducted at Queen
Mary University of London with Andrew
McPherson. We're interested in finding
out the ways that the design of a
digital musical instrument influences
the performer and in turn the music
created using the instrument. In
particular we're interested in
discovering how the instrument size and
the location of tones influence the
performer. So why does this interest us?
Every year new digital musical
instruments are created. Right now, what's
trending commercially in the world of
digital musical instrument design is
small desktop sized boxes and miniature
reissue synths like the Roland Boutique
Series. These often require performance
gestures that are familiar but
scaled-down. Non-commercially, artists are
designing new instruments that can be
virtually any size and shape and are
performable by virtually any gesture,
and this is all possible because of
recent advancements in sensor and
microcomputer technology, and this really
allows new digital musical instruments
to be virtually any size and shape. So
with this relatively new wealth of
options, we're curious to know what are
the hidden influences of an instrument's
size, shape, materiality, and tonal layout?
When we change the physical aspects of
instrument design we are in effect
changing the gestures required to
perform the instrument. So to look at the
impact of these changes we decided to
explore gesture. When we say gesture we
mean "any human action used to generate
sounds" and this includes "grasping,
manipulation, non-contact movements, and
general voluntary body movements". Gesture
that's well suited to an instrument and
feels natural is called "idiomatic".
Tahiroglu found that when composing from the
basis of idiomatic gestures, the result
is performances rich in style and
meaning, which is arguably the foundation
of valued musical repertoire.
Atau Tanaka points out when we have a
new instrument with a lack of musical
history
"what is idiomatic has not been defined
yet". So musicians may think when
performing an instrument they have full
agency over their gestural and
compositional decisions, the existence of
idiomatic gestures reveals there's more
at play. And this is how music composed
on an instrument can be made up of
"seemingly prefabricated patterns". If
you've ever played a piano, chances are
you've swept a finger down all of the
white keys. You might have played a
composition that only features black
keys. These are both examples of
idiomatic patterns that the instrument
suggests to the performer. And this is
how idiomatic patterns influence the
repertoire composed on an instrument.
The instrument design features affordances
that can make you feel confined by the
instrument and can give you a feeling
that your natural bodily interaction is
constrained by an instrument.
Tuuri et al. call this a "push effect". Likewise,
the instrument affordances can enable
spontaneous and natural movement. This is
what to Tuuri et al. call "pull effects".
So when trying to understand how the size
of an instrument influences the
performers choices, we decided to look at
"push effects", "idiomatic gestures" and
"idiomatic patterns". We designed a new
large instrument for the purpose of
monitoring musicians interacting with it.
We wanted to understand the formation of
idiomatic gestures and patterns, so we
designed this instrument to be
unfamiliar to the musicians because this
way they would not already have the
skills to be able to perform it.
[Music]
The instrument is performable using
mallets, and how it works is there are
analog accelerometers embedded in each
pendulum. In the code, each pendulum is
assigned a different fundamental tone.
The sound model is based on the Karplus
Strong algorithm and is assigned
overtones so it sounds like an electric
guitar with some bell-like qualities.
We created two versions of the instrument:
A and B. The versions sound exactly the same.
The only difference is the layout of the
tones. Take a look at our paper to read
about how the instrument was constructed
including the hardware, software and how
the sensors are configured. For the study,
we invited five musicians to perform
Version A, and five musicians to perform
Version B. The participants were all
classically trained on various
instruments but their musical styles
were diverse. Each study session went
like this: the participant would arrive
and be shown the instrument.
The instrument was not turned on.
The participant was told to imagine how the
instrument sounds while trying to play
it for five minutes and at the end of
this they were asked to create a
one-minute improvised performance on the
instrument while it remained turned off.
The participant then heard how the
instrument sounded and was asked to repeat
the previous task however now with the
knowledge of the sounds the instrument
could make. And finally, the instrument
was turned on and amplified and the
participant was invited to perform it
for five minutes and then perform a
one-minute improvisation.
Some of the findings that came out of
this study were not expected.
We found a strong correlation between
the materiality of the instrument and
how the performers imagined the
instrument to sound. 9 out of 10 of the
participants imagined a sound related to
the materiality. Some imagined a metallic
instrument, others imagined a güero.
One participant decided not to imagine a
sound during the task but when he first
encountered the instrument he did think
it was metal and he imagined it was a
bell. When the sound was on all
participants created new performance
gestures. Some participants chose to move
to certain sections of the instrument.
With the sound off they had performed
some pendulums. With the sound on they
chose to focus on others. This is an
example of the push effects created by
sound design. When an instrument is this
large, the push effect is controlling the
performer's entire body.
[Music]
There were various participants who said
they chose certain pendulums to perform
because of their location - because they
were ergonomic or easier to reach.
One participant said she played some bass
tones but she wasn't sure if she played
the tones because she liked the tones or
if she liked the tones because they were
the easiest for her to reach. With each
task the participants expanded their
gestural languages, adding new
performance gestures. Looking at the data
we could see that many of the
participants used the same gestures so we
decided these are the idiomatic gestures
of the instrument.
Similarly, we notice trends of
participants performing the same
patterns of pendulums in order. This is
interesting because the tones are in a
different order on Version A and Version B
of the instrument. Yet participants
played these patterns regardless of the
version of the instrument they were
assigned. We therefore propose that these
are the idiomatic patterns of the
instrument.
[Music]
Additionally, we found evidence that the
physicality and visual aesthetic of the
instrument actually influences the
performers perception of the tones.
Various participants reported that they
thought the order of tones was clustered
due to the quadrant layout of the
instrument. They thought that the tones
in a quadrant all had a certain
characteristic, but this wasn't the case
in reality. The visual aesthetic of
arranging the tones in quadrants with 5
in each quadrant influenced how the
participants perceived the tones and in
turn whether they chose to perform them
or not. So what did we learn about the
factors that influence the performances
made on large digital instruments?
We learned that the sound design can
influence which tones are performed.
But a larger factor is the relationship
between the human body and the
instrument's physical layout - so the
relationship between the human body,
the tonal layout and the instrument size.
This relationship influenced the
improvisations created, and even how
the performer perceives the sound of the
instrument.
[Music]
This study explored the improvised
performances created by musicians during
their first encounter with this
instrument.
Our next research will monitor musicians
as they become familiarized with the
instrument while they create and perform
fixed compositions over several sessions.