Designing the Balance between Sound and Touch: Methods for Multimodal Composition

In this paper, we discuss key research questions generated from a collaborative workshop, during which our aim was to explore the potential of a wearable device to produce novel audio-haptic sensory experiences. The main intention of this research is to enable users with any type of hearing profile to appreciate a body-centered listening experience. The multimodal harness, developed by the co-authors, integrates nine voice coil actuators into a wearable structure, stimulating both the auditory and tactile senses via extratympanic sound conduction and vibrotactile stimulation of the skin on the upper body (spine, clavicles, ribs). We need to create our own interfaces in order to judge its capacity to elicit three modes of vibratory sound perception: auditory, tactile and bi-modal. To this end, we used the Max environment to make several preliminary authoring tools, whose compositional features allow us to explore three main themes of sensory composition: multimodal music listening experiences, spatialization of audio-haptic signals on the body, and sensory equalization. Feedback from trial sessions, along with current constraints due to the wearable and interface design, give
us direction for our future work: iterate to improve the sensory experience of the multimodal harness, and apply these tools experimentally in order to contribute to multi-sensory processing research.