SwimTunes: A gamified music performance system for co-creating with a novice audience

This paper presents SwimTunes, a prototype game system designed for novice multi-user music-making in live performance settings. The system features a digital game and a public web app that allows audience members to participate using their mobile devices. After connecting via QR code, participants create and pilot virtual fish that generate music as they bump into one another. The performer then enters the game as a shark, using camera-based hand tracking to chase and consume the participants' fish. The result is a performance dynamic that evolves from playful co-creation to one of gameful contest between the performer and audience. SwimTunes explores how this shifting interaction context can shape the instantiation of a set of musical parameters, and further how performers can harness gameplay metaphors to conduct live audiences in shared acts of musical expression. The paper details the design considerations and conceptual motivations that informed SwimTunes before describing its implementation via Node.js, Open Sound Control, Unreal Engine 5, and MetaSounds. It discusses technical challenges and opportunities unearthed during development and outlines future directions for the project and gamified music performance at large.