Feeling the music: exploring emotional effects of auditory-tactile musical experiences

Tactile pairing with auditory stimulation has been shown to enhance various capabilities, including the intensity of the stimulus, its location, and its comprehensibility in noise. However the effect of adding haptics on emotional state is still poorly understood, despite the key role of bodily experiences on emotional states. In the current study we aimed to investigate the impact of a multisensory audio-tactile music experience on emotional states and anxiety levels. For this purpose we developed an in-house algorithm and hardware, converting audio information to vibration perceivable through haptics, optimized for music. We compare participants’ emotional experiences of music when provided with audio only versus audio-tactile feedback. We further investigate the impact of enabling participants to freely select their music of choice on the experience. Results indicate multisensory music significantly increases positive mood and decreased state anxiety when compared to the audio only condition. These findings underscore the potential of multisensory stimulation and sensory-enhanced music-touch experiences, specifically for emotional regulation. The results are further enhanced when participants are given the autonomy to choose the musical content. We discuss the importance of multisensory enhancement and embodied experiences on emotional states. We further outline the potential of multisensory experiences in producing robust representations, and discuss specific use cases for technologies that enable controlled multisensory experiences, particularly VR headsets, which increasingly incorporate multifrequency haptic feedback through their controllers and APIs.