Hello, my name is Doga Buse Cavdir from CCRMA and I will be presenting Felt Sound, a shared musical experience for Deaf
and Hard of Hearing
In Felt Sound, we present a musical interface that incorporates non-musical communicative gestures (as input) and
connects those gestures to low-frequency high amplitude sounds designed to produce physical sensations (as output)
This instruments borrows from American Sign Language gestures
potentially with or without their overt meaning.
When designing Felt Sound, we are inspired by Dana Murphy’s original performance, Resonance.
She states that her inspiration is from ASL song
interpretations and the experience of music for those who are deaf and hard of hearing
Continuing her exploration of such a musical experience,
we wanted to challenge the practice of music as a solely auditory experience and
deliver music through physical sensations with in air vibrations.
We wanted to offer a musical experience
that can be shared by deaf and hard of hearing listeners as well as those who are not beyond popular music of the hearing communities.
Now, I would like to play an excerpt from our performance sessions.
Please, use headphones when listening and remember that the performance is designed to be listened with a surround subwoofer system
Some elements are lost in the videos since the focus of the piece is as much in the feel as in the sound.
This version of the instrument is the first prototype of the interface.
It is composed of four different 3D printed modules. An accelerometer on the back of the hand
finger modules with force sensitive resistors and hall-effect sensors on the finger tips
magnet modules in multiple locations depending on the desired interaction and a controller module.
Here is another view of the interface and two common gestural interaction
These modules allowed us to customize instrument in multiple ways to capture different gestures.
The finger gestures and palm opening and closing gestures were used to control the sound engines in multiple ASL words.
These sensors are embedded into the small modules during the 3D printing process to design a more compact variable interface.
The sensor modules are designed to be combined based on the gestural composition
For example, for the current gestural composition the touch sensing model is combined in both hands with the magnet module
to capture the fingers closing and the pinch gestures.
We designed the gesture-to-sound mapping based on five different ASL words. For example
the music word and associated hand waving or wobbling gesture can be frequently seen in the performance
This gesture controls the intensity of the low-frequency beating with the accelerometer data
Here are the rest of the gestures from the performances
Another example is the poetry word. It controls the frequency change combined with the accelerometer data
it can create sudden frequency changes.
The showed word and its gesture is captured by bringing the whole effect sensor on the fingertips close to the magnet module
And it triggers a drone sound engine
Please note that the composition is not intended to offer a narrator
But rather a poetic gestural composition using ASL words and its gestures as metaphors.
The music gesture for example was frequently used as a main interaction mechanism and a transition to the other gestures
We held three short performance sessions with an exit questionnaire these sessions included small groups of audience an
eight channel
surround subwoofer system and
and music delivered through in air vibrations and visual and kinesthetic feedback.
Eight audience members volunteered their thoughts on the physical aural visual and tactile
experience of Felt Sound in an open-ended qualitative questionnaire.
In the performances and qualitative assessment sessions, we explored three main design considerations
creating expressive physical sensations through through in air sound waves
offering a shared experience for all audience members with or without hearing impairment
and aesthetically coupling the sound-mediated kinesthetic sensations with gestural
composition. All listeners supported some physical sensations through low-frequency sounds one audience member
Reported that they felt like they were using their torso to listen to music rather than their ears
The vibrations seemed to be felt in and out of their torso
another one said that the Felt Sound
highlighted the moments of silence for them more so than traditional sound they felt lights and free in those moments and
active in the moments of more intense vibrations
One emphasized the body-to-body connection the gestural interaction provided.
and another one reported the effects of the validity of the gestures in the performance and
emphasized the metaphor of the ASL words and the corresponding gesture-to-sound mapping
This current work is our first prototype of the Felt Sound interface and his performance
We would like to continue further on developing this instrument
One of the biggest limitations draws from the difficulty of reaching the deaf and hard of hearing communities
although we collaborated with and consulted to an ASL speaker and a hard-of-hearing musician composer in
our performances we couldn't host any audience members from deaf and hard of hearing communities
in the future
We would like to collaborate with them and use their feedback when both designing the gesture controller and the performance
We also received a critical feedback from our audience
members that non ASL speaker listeners were challenged by the context and they reported that
Whole body movements and facial expressions could enrich the performance
We would like to address these issues in the next iteration of our design
To summarize. I would like to give a few takeaway points on Felt Sound
first of all, the performance offered a shared musical space for the deaf and hard-of-hearing and those who are not
it was also appreciated by renowned ASL speakers, and they
Reported that they tried to put themselves into ASL speakers and deaf and hard of hearing listeners’ place
Even though they felt challenged by the context
The performance also communicated physical sensations through in airwaves and gestural interaction
Finally the audience reported dance like
movement quality descriptors as in Laban efforts delivered through music and adjust your interaction
Thank you for listening for more details. Please see our paper
If you have any questions
Please feel free to email me
And if you like to see the full video from performance excerpts this link is also provided in the paper