Design and Evaluation of a Gesture Controlled Singing Voice Installation

We present a system that allows users to experience singing without singing using gesture-based interaction techniques. We designed a set of body-related interaction and multi-modal feedback techniques and developed a singing voice synthesizer system that is controlled by the user's mouth shapes and arm gestures. Based on the adaption of a number of digital media-related techniques such as face and body tracking, 3D rendering, singing voice synthesis and physical computing, we developed a media installation that allows users to perform an aria without real singing and provide the look and feel from a 20th century performance of an opera singer. We evaluated this system preliminarily with users.