Embodied Virtual Instruments in Web-Based Multi-User VR: A Case Study with a 3D Drum Kit and Web Audio Modules

WAM Jam Party is a web-based, multi-user VR application that enables participants to collaboratively build musical experiences by visually manipulating an audio signal chain composed of interactive audio nodes. Users can create, modify, and route sound by adding or connecting these nodes, each being a WAM (Web Audio Module) plugin with a 3D GUI. WAMs are often described as the web equivalent of VST plugins. Available 3D WAMs include synthesizers, samplers, audio effects, and pattern-based music note generators such as piano rolls or step sequencers, though traditional instrument metaphors have not yet been incor- porated.

This article explores our efforts to enhance interactivity within this environment by introducing embodied virtual instruments —interfaces designed to replicate real-world in- strumental interaction as closely as possible. Our initial focus is the development of a virtual drum kit (composed of several 3D meshes) that generates MIDI notes (includ- ing velocity), connected to a 3D drum sampler WAM and implemented using the Havok physics engine used in many AAA video games, for realistic collision detection with the virtual drumsticks that follow the user’s movements. We aimed to simulate a drummer’s physical gestures using VR controllers, while also addressing audio responsiveness, haptic feedback, and synchronized visual cues (i.e animating the 3D meshes during impacts) to reinforce the sense of embodied performance. During the demonstration, participants will be equipped with VR headsets to interact directly with the virtual drum kit environment. A secondary display will mirror the VR user’s viewpoint, enabling observers to monitor the real-time immersive experience.