Hello, I’m Diemo Schwarz from the Sound–Music–Movement Interaction
Team at Ircam, and I’ll present the results of a Survey
on the Use of 2D Touch Interfaces for Musical Expression.
Conducted together with Abby Wanyu Liu and Frédéric Bevilacqua.
Touch surfaces are an interface of choice for musical expression.
They debuted in the 90s without pressure sensitivity and only single touch,
followed by a phase of prototypes or niche products with pressure-sensing and multi-touch
and have now become adopted by the electronic musical instrument industry
In parallel, computer input peripherals or tablet computers have appeared and have
been used for music performance, and at the same time, experimentation is still
ongoing with many Kickstarter projects These industrially produced touch devices
have become really useful, because they are ubiquitous, robust, fast, have a high resolution
and a low price, and are easily replaceable.
This fortunate situation leads many researchers and practitioners to turn to touch interaction
for music performance , but also poses questions of how they are using these devices, with
which gestures, mappings, control layout, and sound generators.
For instance, touch surfaces allow for embodied musical performance using control paradigms
of timbre spaces , symbolic and continuous gestural parameter control, hand-shape recognition,
or gesture following and recognition. While these interfaces can be approached intuitively,
exploiting their full potential for expressiveness remains an open question. In this regard,
the technical implementation, gesture processing and sound mapping, as well as the human gesture
learning must be addressed conjointly. To better understand these questions we launched
an online survey that inquired about the participants’ devices, their strengths and weaknesses, the
layout of control dimensions, the used gestures and mappings, the synthesis software or hardware
and the use of audio descriptors and machine learning.
The survey was conducted online using Google Forms, with mostly free-form and multiple-response
questions We inquired about the demographics of the
participants, the touch input device and the synthesis software
or hardware, and about the control layout, the used gestures,
and machine learning.
The answers were analyzed qualitatively via one or two rounds of coding to uncover common
topics and their frequencies of occurrences.
We received 37 responses from participants almost exclusively from Europe and North America.
Only 3 were women, and their age is rather high with a median of 46 years.
The answers about background and occupation revealed 30% “pure” professional musicians,
the rest are involved with “academia” (i.e. researchers, teachers, designers, students).
Participants had a median of 19 years of formal musical training, and 20 years of experience
in digital audio. The main genres they perform are avant-garde/experimental/contemporary,
and electro-acoustic music as expected, but there are also mentions of “exotic” genres
like Blues, Country, and Rap. The performance habits are shown at the bottom,
with just over one quarter performing more than 10 times per year, venue size equally
distributed, and strong prevalence of group practice.
Now, the second part of the survey about the 2D touch input device revealed that 80% use
a commercial device, 20% a noncommercial one. We see a large variety of devices used, with
the most ubiquitous and longest existing ones coming up top: iPad, touchscreen, trackpad,
Wacom, followed by DIY or recent specialised devices Sensel, Roli Lightpad, Touché, Trill.
Not shown here are single mentions of other devices like the Linnstrument, Soundplane,
Joué, and others. The device shape is 80% rectangular, and the
distribution of the physical size is quite broad, as you can see top right, between 7
and 120 cm with a median of 24. People use two hands slightly more often than
one hand, even with small device sizes, but above 30 cm, it is exclusive, and 68% use
fingertips others the palms of the hand, etc.
Finally, we asked about what the participants liked the most about their devices.
We coded the responses into topics and further grouped the topics into the groups to the
right. The colour-codes correspond to feature classes
of a-priori “hardware object” features, functional features, and features pertaining
to the interaction. We see that the versatility-group is on top,
followed by ease of use and nuance tied, for the liked features,
and for the disliked feature groups, low nuance, difficult setup, unreliability.
Interestingly, small size appears in the disliked as well as the liked features.
In this section, we inquired about the synthesis software or hardware used with the touch device.
As always, multiple responses were possible. We see an almost pervasive use of programmable
interactive environments like Max, PureData, or Supercollider, but also a higher than expected
part of hardware synthesisers. The synthesis methods span the last 5 decades
of research, with concatenative synthesis in 3rd place.
That’s why we asked if audio descriptors were considered useful for the control of
synthesis on a Likert scale, with positive results.
We asked which descriptors were used, resulting in the big 4 of Loudness, Pitch, Brilliance,
Noisiness on top, and what additional descriptors were suggested. This last question yielded
answers stressing high-level musical features, derived, textural and spatial descriptors
OK, now we’ll pass to the the next section, where we asked about the layout or positioning
of sounds or parameters on the devices’ 2D touch surface.
First of all, 58% of respondents use one 2D space, but 36% use more than two or a variable
number during a performance. People use continuous or mixed continuous
and discrete layout, and more manual than automatic layout.
We then asked about the type and character of gestures or movements that are used on
the touch device(s). An overwhelming part of the respondents find
gestural interaction useful. When asked to describe the gestures they are using, the
participants came up with topics pertaining to their character (continuous, discontinuous,
slow), size (micro, meso, macro), execution (tap, swipe, press), and one metaphorical
description: instrumental.
For the following 11 Likert-scale questions on the influence of gesture use on the creative
process, we can see high ratings for the first group of creativity-related questions, but
less high ratings for the second group of efficiency-related ones. Especially, the results
of the last group of questions on gesture learning can be interpreted that for about
half the respondents, gestures are easy to learn, but not so easy to teach to others,
but definitely lead to expert or virtuoso playing. This hints at a highly individual
approach to gestural control of performance, where each practitioner improves herself,
with more difficulty to teach others her art and replicate the gained knowledge.
At the end of the gesture section, we inquired about the use of any type of machine learning
techniques to control sound. Only 16% answered positively. In the follow-up
question how they could imagine using machine learning in the future, the most prominent
topic were gestures and generativity, automatic 2D layout and spatialisation, but several
also mentioned technical limitations they were encountering, or said outright they were
not interested. The results of these two questions hint at
a too high informational and technical barrier for the musicians and performing academics
who participated in the study to take up machine learning.
Finally, we asked about their imagined ideal touch input device and how they would like
to use it for control of sound synthesis or transformation.
The topics occurring in the “dream device” mention mainly technological improvements
like having a screen as the touch surface, its material, haptic feedback, a bigger size,
and better resolution and data rate. Only some propositions go further, proposing flexible
devices, different shapes, 3D, and transparency.
Together with the responses to the accompanying gestures and synthesis devices, we can observe
that a large part of respondents are satisfied with their own device, or desire only incremental
improvements. This is interesting, as many of them are academic researchers/designers/teachers
who should at least know, or even develop themselves prototypes of new devices. However,
for their practical musical use, they would prefer their tried and tested instruments,
if they were just a little bigger. This might also hint at the interfaces transitioning
from being seen as prototypical controllers towards actual musical instruments, into which
time needs to be invested to master them, thus needing them to be mostly unchanging.
We get the following interesting results: First, many participants favour easily available,
well established general computing devices over devices
specially developed for musical expression, accepting the loss of some expressive input
dimensions, but it is balanced by a desire for versatility and nuance.
This result leads us to predict a future drift towards more use of specialised devices as
these become better established.
Somehow unexpected was the high use of two-handed interaction, even on the smaller devices.
audio descriptors are used often or always by almost half the participants.
In the gesture-related questions, we found a clear adherence to their usefulness for
musical expression and creativity, while recognising difficulties in their learnability and teachability.
This question should be further examined.
To conclude, the forward-looking questions can inform the design of future devices and
gesture analysis software, and brought to light a gap in the actual practical use of
machine learning for expressive interaction, a gap that might easily be overlooked given
the over-presence of trendy machine learning in publications and social media.
An important limitation of the study lies on expected biases in the representation of
the respondents, and the relatively small size of the group. Therefore, this study should
be completed by further investigations, in particular to better target currently underrepresented
groups in our study (for example the number of female respondents is very small)
The survey is still open, so if you want to help correcting those biases or simply missed
the invitation, go to this url, or look it up in the paper.
Thank you for your attention.