Hi, my name is Olivier Capra, and I'm very pleased to attend my first NIME conference.
With my colleagues Florent Berthaut and Laurent Grisoni
we work on the understanding and the improvement of the experience of spectators of digital interactions.
I’m very pleased to give you a quick look at our last work and its main results.
I hope you will enjoy the video !
Digital interactions are at the heart of electronic music.
Compared with rehearsal or studio recordings, those interactions during live performances
are not only the interest of the sole musician.
The spectators of these interactions are part of the global experience of live performance.
How could we define the experience of spectators to study it and improve it ?
As it is an elusive phenomena, we propose to decompose it in 3 main dimensions easier to address in experiments.
The subjective experience is a component that can be addressed with questions like :
“ Did you like the performance ?” or “How much would you rate the virtuosity of the artist ?”
The objective comprehension can be addressed with questions like
“Who from the musician or the computer is controlling that particular sound ?”
And finally, we think it can be insightful to consider the subjective comprehension.
This component is more about the confidence spectators have in their inner representations of an interaction.
We address it with questions like “If the musician committed an error, would you notice it ?”
When performing with Digital Musical Instruments, the interactions are not always visible,
and when they are, the physical link between the gesture of the musician and the production of sound is broken.
Thus, researchers and artists developed techniques to improve the experience of spectators.
Bin et al. pointed the limited efficiency of explanations about the instrument before the concert,
and Benford et al recently proposed visual illustrations and a mobile application.
Today, we will focus on a technique called visual augmentations.
As you can see on this video of Berthaut et al.,
the interactions of the musician are graphically represented and updated in real time.
A first study formally confirmed the positive role of visual augmentations on multiple components of the spectator experience.
The impact on the overall subjective experience is well supported.
The objective comprehension triggered less sharp contrasts and would require further refinement.
The biggest impact we measured was on the subjective comprehension.
We will cover in more detail this dimension in a minute.
So to refine what we know about visual augmentations, we would like to address 2 points.
How much and what kind of information should visual augmentations provide ?
And should the expertise of spectators be taken into account ?
We think those questions are crucial to better understand how to improve the experience
and avoid the risk to decrease it
So we conducted a study and I would like to tell you more about it.
In this study, 18 participants were distributed after an expertise questionnaire in 2 groups : Novices and Experts
We asked them to watch videos of short music performances along with their visual augmentations with various levels of detail.
Let’s have a closer look at the visual augmentations used in this study.
Here are the 7 LODs we used.
LOD SENSORS displays the sensors representation only,
LOD PROCESSUS, the audio processes only.
The next LOD is a combination of Sensors and Processus.
Then we add the representations of the mappings in LOD MAPPINGS.
The FULL COMBINED reuses the SENSORS and adds extra details in the mappings and the processes.
With the final LOD, the full audio graph is represented.
Each processus is finely described with each effect applied and their parameter values.
In this study, we want to control the objective contribution of the musician and the one from the computer
to assess the ability of the participants to evaluate it.
So we designed 3 different mappings leading to 3 different levels of contribution of the musician :
a completely manual mapping,
a semi automated and an highly automated mapping.
To sum up, we wanted to evaluate the effect of levels of detail of the visual augmentations on 3 dimensions of the spectator experience, with respect to the expertise.
Two groups followed the same tasks distributed in 2 blocks.
In the first block, the LODs were fixed whereas in the second participants could freely modify the LODs
of the visual augmentations used in the music performances they were watching.
In block 1, The 7 LODs multiplied by the 3 contribution levels ended in a set of 21 short videos.
After each one them, we assessed the 3 components of the experience.
We asked the participants their personal rating of the performance
and the virtuosity of the musician.
We also asked them who from the musician, the computer or both contributed the most in the performance ?
And then we evaluated their subjective comprehension
thanks to questions based on communication design issues.
These design issues were presented by Victoria Bellotti in the context of user-computer interactions.
They were then transposed by Fyans et al. to the perspective of spectators of interactions.
In our turn, we transposed them to an evaluation perspective and add a new item that we call "Association".
We won’t present these Bellotti/Fyans challenges into great details.
Just note that they are related to the feeling of knowing
something rather than an objective knowledge.
For instance, the association targets the ability of a device to allow observers to see the contribution of the musician and the one of the computer.
The results of block 1 show that regardless of the LOD,
Experts considered that the musician contributed more than the computer
in 62% of the stimuli compared to 45.5% for Novices.
More interesting are the results for the subjective comprehension.
Feel free to read the paper for a detailed analysis of the results.
Here we just wanted to underline that globally,
experts reported higher evaluations of the subjective questions
and 2 LODs were particularly efficient,
the one displaying only the sensors and the one displaying the full abstracted mechanisms (FULL_COMB)
In the second block, participants could change with the mouse scroll wheel
the LOD of the augmentations as the video was playing.
In a first task, they watched 3 short (60s) performances
and were asked to select the LOD that gave them the best experience, that they preferred.
In a second task, they watched the same performances and were asked instead to choose the LOD
that allowed them to understand best what the musician was doing.
Both tasks, experience and comprehension,
show comparable evolution characterised by a minimum for the control condition NONE
and a maximum for the higher LODs
A decisive effect of LODs was found but
with no difference between the groups.
The FULL_COMB LOD was the favorite for both groups.
Interestingly, even if they preferred the FULL_COMB LOD,
novices massively chose the FULL_GRAPH LOD when considering a better comprehension.
Once again, the interested spectator is invited to read the paper
for other interesting contrasts between experts and novices.
To sum up, we saw that specific levels of detail contribute to a better experience,
especially when considering subjective comprehension, and even when the objective comprehension is low.
We completed the controlled study with interviews
that revealed various strategies where the ability to change the LOD throuhgout the performance
or even to stop visual augmentations is an important feature.
A last word to tell you that we worked on a taxonomy to facilitate the inventory and the analyses
of Spectator Experience Augmentation Techniques, like visual augmentations and many others.
Our next step is to pursue our research on spectator experience
with cognitive science protocols involving electrophysiological measurements.
We hope to see you soon IRL to tell you more about these projects !
We hope you liked this video, and that it make you want to know more about our work and to read our papers.