Conference paper Open Access

Designing Gestures for Continuous Sonic Interaction

Tanaka, Atau; Di Donato, Balandino; Zbyszynski, Michael; Roks, Geert

DCAT Export

<?xml version='1.0' encoding='utf-8'?>
<rdf:RDF xmlns:rdf="" xmlns:adms="" xmlns:dc="" xmlns:dct="" xmlns:dctype="" xmlns:dcat="" xmlns:duv="" xmlns:foaf="" xmlns:frapo="" xmlns:geo="" xmlns:gsp="" xmlns:locn="" xmlns:org="" xmlns:owl="" xmlns:prov="" xmlns:rdfs="" xmlns:schema="" xmlns:skos="" xmlns:vcard="" xmlns:wdrs="">
  <rdf:Description rdf:about="">
    <dct:identifier rdf:datatype=""></dct:identifier>
    <foaf:page rdf:resource=""/>
        <rdf:type rdf:resource=""/>
        <foaf:name>Tanaka, Atau</foaf:name>
        <rdf:type rdf:resource=""/>
        <foaf:name>Di Donato, Balandino</foaf:name>
        <foaf:familyName>Di Donato</foaf:familyName>
        <rdf:type rdf:resource=""/>
        <foaf:name>Zbyszynski, Michael</foaf:name>
        <rdf:type rdf:resource=""/>
        <foaf:name>Roks, Geert</foaf:name>
    <dct:title>Designing Gestures for Continuous Sonic Interaction</dct:title>
    <dct:issued rdf:datatype="">2019</dct:issued>
    <dct:issued rdf:datatype="">2019-06-01</dct:issued>
    <owl:sameAs rdf:resource=""/>
        <skos:notation rdf:datatype=""></skos:notation>
    <dct:isVersionOf rdf:resource=""/>
    <dct:isPartOf rdf:resource=""/>
    <dct:description>This paper presents a system that allows users to quickly try different ways to train neural networks and temporal modeling techniques to associate arm gestures with time varying sound. We created a software framework for this, and designed three interactive sounds and presented them to participants in a workshop based study. We build upon previous work in sound-tracing and mapping-by-demonstration to ask the participants to design gestures with which to perform the given sounds using a multimodal, inertial measurement (IMU) and muscle sensing (EMG) device. We presented the user with four techniques for associating sensor input to synthesizer parameter output. Two were classical techniques from the literature, and two proposed different ways to capture dynamic gesture in a neural network. These four techniques were: 1.) A Static Position regression training procedure, 2.) A Hidden Markov based temporal modeler, 3.) Whole Gesture capture to a neural network, and 4.) a Windowed method using the position-based procedure on the fly during the performance of a dynamic gesture. Our results show trade-offs between accurate, predictable reproduction of the source sounds and exploration of the gesture-sound space. Several of the users were attracted to our new windowed method for capturing gesture anchor points on the fly as training data for neural network based regression. This paper will be of interest to musicians interested in going from sound design to gesture design and offers a workflow for quickly trying different mapping-by-demonstration techniques.</dct:description>
    <dct:accessRights rdf:resource=""/>
      <dct:RightsStatement rdf:about="info:eu-repo/semantics/openAccess">
        <rdfs:label>Open Access</rdfs:label>
    <dct:license rdf:resource=""/>
        <dcat:accessURL rdf:resource=""></dcat:accessURL>
        <dcat:downloadURL rdf:resource=""></dcat:downloadURL>
All versions This version
Views 9595
Downloads 8181
Data volume 30.9 MB30.9 MB
Unique views 9191
Unique downloads 7171


Cite as