Hello, I'm Sihwa Park a PhD student in video arts technology University of California, Santa Barbara.
In this talk, I will present my paper:
Collaborative Mobile Instruments in a shared AR space: a Case of ARlooper
This paper presents, ARlooper which is an augmented reality
Mobile interface, that allows multiple users to record sound and perform together in a shared inner space.
Given the advances in mobile computing the possibility of the use of mobile devices
In NIME has been explored by a body of researchers, artists and designers,
to suggest new performance paradigms and
to create new forms of musical interfaces by utilizing embedded sensors.
Meanwhile augmented reality
also has been involved with the increased computing power of mobile devices in particular,
remarkable improvements in SLAM,
simultaneous localizing and mapping techniques,
have enabled mobile AR to build a 3d model the users local environment without any prior mapping information
and
with the networking functionality of mobile devices,
it is possible to create a synchronized airspace by sharing the map with multiple devices in real time.
Now the creation of collaborate AR is more feasible with mobile devices
By using development frameworks such as Google's ARCore and Apples ARkit.
Since the small screen of mobile devices
become the main space for musical tasks,
performers need to more pay attention to the screen itself, not collaboration.
It will be difficult to have other layers of nonverbal interactions that can be found in face-to-face collaboration.
However, with collaborative AR
mobile instrument could have seamless interaction between the task space and
communication space, by extending the desk space from the 2d screen to the real world.
Also a mobile instrument could enhance the capability of
communication for musical collaboration by enabling performers to see each other's activities
along with future content.
In this regard ARLooper
aims to explore these possibilities of collaborate AR in designing new mobile instruments.
In addition to providing UI elements needed for sound recording and playback,
the design goal of ARLooper is to help performers with the intuitive
understanding of collaboration context and to enhance
nonverbal cues for communication among collaborators.
ARLoopers sound recording interaction is similar to drawing 3d lines with real-time audio data.
When the user taps our recording button on the screen, AR looper begins to record sound from a device's microphone.
The sound is visualized as a 3d tube like shape,
the amplitude determines the radius and brightness of the shape
and the position of the shape follows the center of the devices screen.
ARLooper attaches a set of sound filters and effects to each recorded sound.
To enable the user to control the parameters of these sound features and effect ARLooper presents a slider based GUI.
When the user taps a waveform,
It presents the GUI having a conventional 2d waveform viewer, control buttons and sliders.
When the user taps the waveform again or taps an empty space on the screen the GUI disappears.
For more immediate sound control, ARLooper also has a multi-touch GUI, with the concept of
catch-and-release interaction.
When the user double taps a waveform, the waveform is fixed at the center of the screen and dragable
spheres appear within a boundary circle around the wave form.
The circles are mapped to parameters that are chosen up to five in the slider based GUI.
The distance between the screen center and each circle is proportionally mapped to each parameters value.
When the usual double taps the waveform again,
the waveform moves back to the original location.
To represent the change away from status, ARlooper also
visualizes a playback position and a change of some parameters as a waveform animation effect.
As part of design elements to support collaboration, ARlooper uses colour to distinguish the ownership of waveforms.
Each user is assigned an ID color, this ID color is used to represent
this ID color is used to represent,
who created, who has selected, who is controlling or playing waveforms.
The user can change their ID color through the color setting GUI.
To resolve a possible conflict in
accessing the shared virtual resources,
ARLooper has a dominant user protocol that allows only one user to control a waveform at a time.
On top of that, if the user leaves the
waveform after playing it and the other user can remotely change its sound parameters.
These rules can make a collaborative
situation wherein, user A with purple ID,
manipulates a waveform which is recorded by user B,
with yellow ID whilst it's sound is coming out from the device of user C who has a blue ID.
ARLooper is developed with the Swift programming language
with apples ARkit and all Audiokit which is a sound synthesis processing and analysis framework for iOS.
For multi-user mode there are two steps to be followed,
First,
One user need to be a host and the host user has to create and advertise their AR session
after scanning a given space and
building a map of the space.
In the next step, other users join the session as guests and
we relocalise the shared map received from the host.
Once the peer-to-peer networking is established among, the users the activities of the users such as,
the creation selection and manipulation of the waveforms are synchronized across all connected devices.
To evaluate the user experience with ARLooper,
I conducted a user study attempting to address questions:
How can ARLooper support performers in a shared airspace?
How does ARLooper enhance nonverbal cues for communication?
The study,
consisted of two sessions,
questionnaires and a semi structured group interview followed by data analysis.
In total, five participants were recruited from my graduate program and
they were divided into three groups of two players.
All had experience with collaborative music performance.
Only three participant had prior experience with a mobile music instrument.
The questionnaire research showed that, ARLopper was not difficult to play.
The participants were highly aware of collaborators and felt a positive sense of participation.
Whilst they were satisfied with collaborative AR experience,
they also indicated the design of ARLooper for collaboration was effective.
However, the sound was still the main cue for communication.
The participants were also asked to choose the design element of ARLooper that were helpful for communication.
They all selected
shared waveforms and user ID colors as the best helpful elements.
But they indicated, the design of the remote sound control feature was less helpful for collaborative communication.
In summary,
ARLooper demonstrated the potential of utilizing
collaborative mobile AR in creating new mobile instrument. For future work there could be several items
based on the user study
research and design reflection.
First, more stable peer-to-peer networking is required.
Second, adding more animation effects,
could provide the user with a better understanding of collaborative context.
Third, improving the bounding box design with additional information could enhance
communication in using shared AR resources
Sound specialization would be a reasonable step to be taken for
spatial interaction design in AR.
Last, it is necessary to consider a relevant stage set up for actual music perfrormances with ARLooper.
Thank you.