Published July 2, 2020 | Version v1

Towards Assisted Interactive Machine Learning: Exploring Gesture-Sound Mappings Using Reinforcement Learning

  • 1. Luleå University of Technology
  • 2. Goldsmiths, University of London, UK

Description

We present a sonic interaction design approach that makes use of deep reinforcement learning to explore many
mapping possibilities between input sensor data streams and sound synthesis parameters. The user can give
feedback to an artificial agent about the mappings proposed by the latter while playing the synthesiser and
trying the new mappings on the fly. The design approach we adopted is inspired by the ideas established
by the interactive machine learning paradigm, as well as by the use of artificial agents in computer music
for exploring complex parameter spaces. We refer to this interaction design approach as Assisted Interactive
Machine Learning (AIML). We describe the architecture of an AIML system prototype, a typical workflow
for interacting with the agent and obtain gesture-sound mappings. We then present feedback data collected
during a demonstration and discuss perspectives for developing the AIML paradigm further, pointing out
current limitations. In light of the feedback obtained and the considerations arisen following the use of the
system in a multimedia performance piece, we suggest that the implementation and evaluation of new features
should take into consideration established creative workflows and take place close to actual artistic practice.

Files

icli2020_1_Visi.pdf

Files (3.2 MB)

Name Size
md5:275a5464b8fdf7cda6e98309650090a4
3.2 MB Preview Download

Additional details

Related works

Is part of
Conference paper: 2663-9041 (ISSN)