Hello, my name is Alberto Boem,
and I will present the paper
“Non-rigid musical interfaces: Exploring practices takes and future perspective.”
This work was done in collaboration with
Giovanni Troiano, and Victor Zappi from Northeastern University,
Giacomo Lepri from Queen Mary University of London, and myself.
Through this work, we aim to map and analyze
the field of non-rigid interfaces for musical expression.
Non-rigid interfaces in the context of NIME
currently represent a niche,
consisting of highly experimental
and very scattered work,
but characterized by strong musical potentials.
In the first part of this presentation,
we will define what non-rigid interfaces are
and provide a review of existing musical
interfaces that capitalize on deformable input.
In the second part,
we will present the results of a survey
conducted among 11 experts and pioneers in the field.
Based on such expert’s input
and through the review of the field,
we suggest possible next steps
for advancing the research
of non-rigid musical interfaces.
While in NIME non-rigid interfaces are still a minority,
in HCI we have observed an
increasing interest in this research area.
But what do we mean by non-rigid interfaces?
Non-rigid interfaces can be generally described as
physical interfaces that use
dynamic changes in a device’s
physical shape for input and output.
On one side,
non-rigid interface devices characterized
by deformable input,
such as flexible controllers
or bendable displays.
On the other side,
non-rigid interfaces use the change of
shape to dynamically represent
information and data through self-actuation.
These two ideally converge into
the idea of having a sort of digital clay,
where the digital and physical are
synchronized together through
surface’s deformation.
As a first contribution,
we provide a global overview of the field
to the NIME community.
To give a comprehensive view of the field,
we first surveyed the literature of musical
interfaces by looking at proceedings
of NIME and related publications.
We found around 30 papers.
Using the analysis proposed by Boem and Troiano,
we mapped such examples by analyzing
their salient aspects like shapes, materials,
sensing technologies, mapping, and applications.
As you can see from these examples,
we can notice that non-rigid interfaces are everything but
uniform, they show a great variety of shapes
(from organic to display-like shapes)
and materials (from textiles to silicone).
Unfortunately, we have no time to dive more
deeply into the analysis, so please
refer to the paper for more details.
However, differently from HCI,
none of them are shape-changing or
self-actuated but are characterized
by deformable input only.
The analysis of the literature left us
with many open questions since we found
a great variety of examples,
but at the same time we felt a lack
of a global understanding of such kind of interfaces.
Moreover, what does it mean to design
non-rigid musical interfaces?
What is the value of such interfaces for musical expression?
Which are their limitations?
Therefore, we decided to ask experts
about their experience and provide ideas for future directions.
Inspired by recent works published at NIME,
we interviewed designers and researchers
who previously published their work
in peer-reviewed conferences and journals.
We invited such experts of non-rigid musical interfaces to:
share their personal experiences on designing and using those interfaces with us.
reflect more broadly on the impact of such interfaces on NIME and musical activities.
share insights and personal visions on future research perspectives for non-rigid musical interfaces.
Such work was conducted through an online survey on a protected Qualtrics account.
11 experts replied and agreed to take part
in the survey from the original pool
of people we contacted.
As you can see, some examples are
well-known in the NIME community,
such as Sonic Banana or The spine.
Some are more recent, like IllumiWear.
We have also to notice that P8 replied
about two interfaces but reported two
different experiences.
We analyzed the results of the survey through
a thematic and content analysis.
Based on grounded theory,
we identified various emergent themes
and organized them into six categories.
Then, we developed a discussion
on open challenges and how we might tackle them.
Our goal is to provide a generalization
of the results beyond the existing examples
of non-rigid interfaces for music.
The first category maps the motivations behind the development
of non-rigid musical interfaces,
which are characterized mainly by:
the desire to experiment new forms of control in music,
especially exploring continuous and multidimensional input
the creation of more enjoyable and
easy-to-use musical interfaces through non-rigid materials
challenge the aesthetics of existing controllers,
by creating interfaces that are unique
in their shapes, materials and tactile feeling
The second category is about applications.
Here we considered not only the context
in which such interfaces have been used
but also the different applications that the authors
had in mind or had considered for future works.
The majority of interfaces have been designed for performative music activities.
Also, the sonification of deformation
was considered for installations and for the augmentation of physical interactions.
Some have been used for music and disabilities.
In general non-rigid interfaces have been considered for
musical applications that require a high level of physical engagement.
In the third category,
we grouped the reflections developed by the experts
about the current status of their interfaces.
most of these interfaces are currently unused,
and only a few went through further development
the most significant impact of their work
is perceived to be limited to academic and artistic research.
We should also notice
that the situation we are observing
in non-rigid interfaces
is in line with what is happening
in the NIME community in general.
Later, we tried to map the issues and challenges
encountered by the experts,
The first and biggest challenge is
crafting the interface
This is probably the most problematic
of the whole process of the designing and
using a non-rigid interface
This is because of the complex interplay between
sensors and materials, and the absence of
comprehensive design guidelines and documented processes.
Then, it seems unclear to many authors
how to properly design the mapping of continuous
deformations to sound, because of the lack of
protocols, and especially when the interface
provides several degrees of freedom,
and different types of signals must be interpreted.
According to many experts,
these issues are perceived as the main reason
why it is tough to develop
more idiosyncratic performance practices
with non-rigid musical interfaces.
The fifth category tries to understand
which are -according to the experts-
the main aspects needed for the research to advance.
The first is the popularization of non-rigid interfaces.
It’s difficult to see non-rigid interfaces in action.
Most of the experts agree on
“Getting devices into the hands of end-users
and out of the research lab”
as a much-needed step.
The second relates to the need for understanding
and developing the learning process
specific to non-rigid musical interfaces.
All the participants highlighted the high potentials
of non-rigid musical interfaces for enabling
expressive control of sound
However, we found that there is no agreement
between experts on the TYPE of control
these interfaces are capable of supporting.
Some consider deformation fundamental
and useful for precise and detailed control of sound,
while others interpret non-rigid interfaces
more suited for exploratory activities and larger actions.
Some expressed the need to find
new technologies and techniques for fabricating
and better designing non-rigid interfaces.
They identified the urgency to collaborate
with other disciplines to solve problems
like embedding electronics,
and interpreting and analyzing sensors’ data.
To push forward the reflection of non-rigid interfaces,
we asked experts to elaborate on the possibility
of incorporating shape-change
and self-actuation in the context of musical expression.
Most expressed excitement,
linked to the idea of gaining access
to an infinite space of expressive possibilities
However, important criticalities were also underlined by the participants,
as each time an instrument changes shape
a new playing technique must be developed,
compromising what we currently consider the path towards virtuosity.
The analysis of the survey conducted
with experts gave us some better insights
into the state of the art of non-rigid musical interfaces.
However, in the recent years HCI moved forward
with follow up studies, user studies, definitions of applications, and interaction techniques.
Conversely, in NIME this field seems
to be “stuck” in the exploration phase,
and the impact is very limited.
After the analysis of the literature and the experts' survey
we found that most non-rigid musical interfaces
don't go over the prototype stage and
many problems are yet to be solved.
We identified The first reason with the burden of the design.
Non-Rigid interfaces are difficult to build and maintain.
This seems to be a significant problem.
Despite considering the development of musical practice
and pedagogy being of utmost importance,
most of the effort seems to be oriented
on the design and implementation phase of the interface.
In line with the general NIME effort
to separate performer from designer,
we suggest more user-studies in the wild,
in which performers can focus on the exploration of ready-made interfaces.
Regarding the improvement of the design itself,
our results stress out a call
for a stronger interdisciplinary effort.
For instance, the collaboration with material scientists
and experts in nanotechnologies may provide
access to novel deformable materials
and help embed electronics in non-rigid structures.
Some examples of this interdisciplinary collaborative effort
can be seen in recent work done in HCI.
Also, collaboration with experts from the AI
and Machine Learning communities
can lead to better methods for interpreting
and dealing with continuous and multidimensional signals.
Another topic that deserves more attention
is the issue of mapping
and cross-modal correspondences in non-rigid interfaces.
When dealing with interfaces that can
simultaneously support different input gestures
(such as squeeze and stretch, or bend twist and push)
we need to understand which combinations are feasible
as well as advantageous in musical context,
and which mapping strategies perform best.
To tackle these open issues, we suggest
leveraging psychophysical methods and investigating
how sensory and motor abilities of performers
align with multidimensional non-rigid input and continuous sound control.
Differently from HCI,
we don't see any consistent research on shape-changing
non-rigid musical interfaces.
When stimulated, the experts
that we interviewed generated both interesting ideas
and critical considerations,
directly inspired by composition and performance experience.
We think that the NIME community can provide
a more critical approach to the development
of non-rigid interfaces
using shape-change and self actuation,
and offer some specific scenarios and applications not considered in HCI until now.
Thank you for your attention,
and we are looking forward for questions
If you need more information please check the website
deformableui.com
and thank you again!