Design and Interaction of 3D Virtual Music Instruments for STEAM Education Using Web Technologies

In this paper we present our current work on the development of a web-based system that allows users to design and interact with virtual music instruments in a virtual 3D environment, providing three natural and intuitive means of interaction (physical, gestural and mixed). By employing the Leap Motion sensor, we benefit from its high performance on providing accurate finger tracking data. The proposed system is integrated as a creative tool of a novel STEAM education platform that promotes science learning through musical activities. Our approach models two families of music instruments (stringed and percussion), with realistic sonic feedback by utilizing a physical model-based sound synthesis engine. Consequently, the proposed interface meets the performance requirements of real-time interaction systems and is implemented strictly with web technologies that allow platform independent functionality.