Hi there my name is Charles Martin I'm
from the Australian National University
and I'm here to present this research
Sonic Sculpture: Activating Engagement
with head mounted augmented reality.
So my co-authors on this project are Zeruo Liu,
Yichen Wang, Weannan He and Henry Gardner and
my co-authors are also associated with
Data 61 CSIRO or the Commonwealth
Scientific and Industrial Research
Organisation here in Australia.
Now in this research we aim to explore how a
head-mounted augmented reality system
can be used to present a sound artwork
that accompanies a physical sculpture
I'm going to talk a bit about using AR
for a sound art installation a bit about
our particular piece and the two
different interaction modes which we
designed then I'll take talk a bit about
what we learned developing and testing this work.
So there's been many different
sonic artworks created as part of
sculptural installations or gallery
exhibitions including many discussed and
shown here at NIME. Broadly there's two
technologies used to get the sounds to
visitors. First: mobile devices. We know
that you can develop amazing sonic art
pieces and interactive music systems
with mobiles the great advantage here is
that every visitor will have their own
experience and you might be able to use
some of the mobile sensors to help drive
the sound and of course many artists
simply place loudspeakers around
artworks and deliver sound to visitors
this is simple and effective but
everybody has to hear the same thing and
you have to physically install it at the
start of the installation.
So in our work, we're interested in using head mounted
mixed reality devices such as
Microsoft's HoloLens to deliver an
interactive sonic art experience to
individual visitors. So these devices
have a transparent visor so they can
display virtual objects within the
visual field of the wearer. They're
loaded up with cameras and sensors to
figure out where they are in an
environment and what the wearer is
looking at. They can track your hand
gestures so that you can interact with
virtual objects and they have speakers
in the headband that work quite well to
provide sounds of the wearer so compared
to phones the Hololens has more
capacity to be spatially aware.
Our interactive system can then be aware of
the wearer's location, where they're
looking, and what they're actually
touching; and compared to a physical
system of speakers and
con trollers we can build a HoloLens
interaction with no physical components
and in fact just add this secret layer
to an existing environment now we wanted
to explore how we could use the hololens
to create an interactive sound artwork
that would accompany and surround an
existing piece of public art and we had
two goals with our design we wanted our
system to reveal a hidden interactive
sound world around an existing part of
an everyday environment so the hololens
will let us place virtual objects and
sound sources around this physical
sculpture for visitors to explore and
discover secondly we wanted to explore
how this might actually activate
engagement with a public sculpture so I
think we've all had this experience
where with public art where it fades a
little bit into the background of our
everyday life and this is particularly
the case if the world is changing around
an artwork so this was kind of the case
with the sculpture we were working
around so this sculpture is called
Il Grande Ascolto---The Great Listening
and it's by an Arnaldo Pomodoro and was
created in 1968. It depicts these two
bronze cones which kind of recall radio
telescopes listening in to radio signals
in this era of digital communications
and the Information Age so this
sculpture actually has kind of a curious
history it was placed outside the old
CSIRO headquarters in Canberra in 1970
and it stood there until 2014 when
someone actually stole one of the cones
luckily the cone was recovers but the
headquarters was moved and the sculpture
ended up being placed inside a new CSIRO
building but it's actually kind of
hidden under the stairs and doesn't have
the same prominent position that it used to have
So the sound artwork was created
to reflect the idea of listening to
different worlds in the sculpture's
location. We developed three six-minute
pieces from field recordings which would
serve as layers for visitors to explore.
The first layout was of the natural
world around Black Mountain here in
Canberra which is where the new building
was located; the second layer was of the
human environment around this new
building and the
university again near that location and
the third layout was the hidden
radio frequency world of computers which
I recorded with telephone pickups from
my computer on my desk at ANU the idea
was that visitors might be able to hear
these three different sonic perspectives
on the sculptor's location and to play
with mixing between them while engaging
with the sculpture itself.
So we developed two interaction models for
this work in order to test out how we
could best use the HoloLens in this context.
These were developed in Unity
with Microsoft's Mixed Reality Toolkit
first we created a mixer user interface
that visitors could manipulate to
control the sound layers individually we
also create as a locative interaction
where each sound file was placed at a
different location in the room and
visitors could mix them just by walking
around so I'll show you a little video
about how this looked in practice
[Field Recordings]
Now, we've done informal trials of our
two interaction schemes at the sculpture
during our development process and I'm
going to talk a little bit about the
design issues and practical concerns
that popped up during development and these trials.
So for the UI mixer
interface we experimented with a few
locations for this mixer I thought it
might work well on the plinth in front
of the sculpture but because of the
field of view was a bit difficult to see
so we place it on the floor in front of
the sculpture this meant that you
couldn't actually see the mixer and the
sculpture together
with one eye fall so it kind of felt a
bit disconnected the HoloLens hand
tracking turned out to be a little bit
tricky for some users and although I
would normally use a fader up-and-down
gesture there was a little bit difficult
with the field of view of the HoloLens
because your hand would run outside the
field of view so it's actually easier to
go on the side and then use this left to
right mixing interaction or I guess DJ style mixing.
So for the locative audio
interaction we found that this was
easier for some users to explore because
you didn't have to use hand gestures and
there were clear markers to guides the
user to interesting locations the audio
was actually more convincing in this
interaction because the audio sound
files were spatialized from these
locations in the floor which you could
stand right on top of them so you have a
nice feeling of immersion in the
environment with the mixer UI the audio
came out of the sculpture itself or
appeared to come out of the sculpture
which wasn't as convincing so in terms
of engagement visitors needed to view
the sculpture from three sides to
explore it which was good and when they
were at actually looking at the sculpture
they could also see one or two of the
other markers and that was also good!
So I've told you about our mixed reality
sonic artwork developed to go around
this piece of public art now it seems to
us from this experience that the locative
interaction was the most promising in
terms of convincing experiences in this
hidden sound world it also seems
promising in terms of activating
engagement with an artwork that was
stuck under a staircase.
So far we haven't found a tactile sonic
interaction that worked well with the
first generation HoloLens
due to these issues with field-of-view
and the the difficulty of using the hand
gesture pointer.
So this system that we put together was practical enough with
the HoloLens but we look forward to
continuing our work with the HoloLens 2
which has an improved field of view and
much better hand tracking and we think
that this could open up some more
interactive possibilities we also
weren't satisfied with the interactive
audio tools available in Unity so we
have another project to build a new
synthesis toolkit for unity that we are
using in our research currently so I
just want to thank Matt Adcock and
Stuart Anderson from Data 61 CSIRO as
well as Ben Swift from the ANU and the
other students in our Code Creativity
Culture Lab. I'll note that this was
supported by the CSIRO Science and Industry
Endowment Fund. Thanks for listening and
I'll catch you next time!