Piano&Dancer: Interaction Between a Dancer and an Acoustic Instrument

Piano&Dancer is an interactive piece for a dancer and an electromechanical acoustic piano. The piece presents the dancer and the piano as two performers on stage whose bodily movements are mutually interdependent. This interdependence reveals a close relationship between physical and musical gestures. Accordingly, the realisation of the piece has been based on creative processes that merge choreographic and compositional methods. In order to relate the expressive movement qualities of a dancer to the creation of musical material, the piece employs a variety of techniques. These include methods for movement tracking and feature analysis, generative algorithms for creating musical structures, and the application of non-conventional scales and chord transformations to shape the modal characteristics of the music. The publication contextualises Piano&Dancer by relating its creation to concepts of embodiment, interactivity and musical structure and by discussing opportunities for creative cross-fertilisation between dance choreography and musical composition. It also provides some details about the challenges and potentials of integrating a mechanical musical instrument into an interactive setting for a dance performance. Finally, the paper highlights some of the technical and aesthetic principles that were used in order to connect expressive qualities of body movements to the creation of music structures.


INTRODUCTION
Piano&Dancer is a dance piece for a single human dancer and an electromechanical acoustic piano (Disklavier). is piece has been realised as a collaboration between the two authors of this publication and the choreographer and dancer Muriel Romero. During the performance, both the dancer and the Disklavier are present on stage. e music of the performance is produced through the piano's mechanical movements. ese movements are generated in real-time through a combination of compositional algorithms, stochastic functions, swarm simulations, and mappings according to speci c musical modes, all of which are in uenced by the dancer's bodily movements. As a result, the piano and dancer are connected with each other through three layers of relationships: they are both physically present on stage and exhibit bodily movements, their respective movements are correlated through an algorithmic intermediate layer, and they exhibit in the musical and bodily domain a strong correspondence in expressivity.
One of the central motivations for the realisation of this piece is based on the assumption that a pianist's bodily movements during a musical performance represents a choreography.
is choreography is at the same time very sophisticated but also highly constrained. Furthermore, the experiential aspects of this choreography are revealed through the resulting musical forms. ese forms are shaped by the intrinsic physical properties of the piano and by the characteristics of the gestures performed by the player.
Starting from these considerations, the piece Piano&Dancer undertakes an artistic investigation into the choreographic qualities of instrumental gestures and how these qualities can be creatively extended while at the same time maintaining their inherent music producing functionality. ese investigations form part of a larger research context concerning the functional and aesthetic relationships between body gestures and musical gestures. Some research questions within this context are: Can musical structures convey acoustically some of the qualitative and expressive characteristics of bodily movement? Which intimate aspects of a dancer's bodily activities that are normally hidden to an external observer can be conveyed through acoustic feedback? What formal and aesthetic mechanisms need to be established to mediate between the bodily and musical domains of gestural movements? How can the important role of kinaesthetic body awareness be taken advantage of for the control and perception of interactive music? How can compositional and choreographic approaches inform each other in the creation of performance pieces for dancing instrumentalists?
For the realisation of Piano&Dancer, an interactive se ing was chosen that abolishes the necessity for a direct tactile manipulation of the acoustic instrument and thereby relaxes the traditional functional constrains of piano gestures. In addition, a direct causality between the physical aspects of the dancer's body movements and piano actuation was avoided in favour of interaction techniques that take the expressive aspects of bodily movement into account and that integrate compositional algorithms as part of their mapping mechanisms. Some of the technology that has been employed was developed in the context of the European H2020 ICT research projects DANCE and WHOLODANCE. e DANCE project studies how music can convey qualities of body movement and enables a cross-modal transfer of movement perception into the acoustic domain.
e WHOLODANCE project develops and applies technologies that provide educational tools for dance practitioners.

BACKGROUND
is section provides a condensed overview about some of the topics that are of relevance for this project. ese topics include: the development of design principles for interactive music systems that draw from embodied cognition research, the mutual crossfertilisation between creative approaches in dance and music, the computational analysis of higher level movement features from raw sensory data, and the correlation of movement features with the articulation of sonic gestures. Other relevant topics such as the combination of machines and performers on stage and the application of algorithmic and generative approaches in music composition have been discussed in a previous publication [6].

Image Schema, Movement alities and Sound Morphology
Music in its origins is inseparable from the manipulation of physical objects with our body in order to produce sound. e coordination of movements that are necessary to control an acoustic instrument corresponds in itself to a very sophisticated choreography that requires a thorough awareness of body position and body motion, both of which have large e ects on the resulting sounds. eories from the eld of embodied cognition emphasise the important role of bodily experience and engagement with the world for the development of cognitive capabilities. e notion of image schemas refers to cognitive representations of the intimate and preconscious relationships between recurring pa erns in perception and action that occur when we interact with and move through our environments [16]. It is assumed that many of these representation form the foundations for higher level cognitive capabilities. Exploratory and intuitive forms of engagement with interactive systems very likely are guided by image schemas [7,14]. Similarly, certain key musical principles that inform the organisation of sound arise from a preconscious understanding of our body and environment [17]. For example, the perception of rhythms is based on a disparity of durations, intensities and pitches which in order to be recognised requires at least two sonic events to occur. As in human walking, the rst event is an elevation and the second is a fall. In musical terms, this corresponds to an upbeat and a downbeat, and a tension and a release [21]. Musical principles emerging from image schema related to verticality are also very prevailing. Moreover, as Graham and Bridges explain in [13], these image schemas can be considered to be closely linked to the archetypal sonic dynamic morphologies that Smalley describes in his prominent theory of Spectromorphology [29]. is relationship is relevant in the context of Piano&Dancer, since the sound motions portrayed by Smalley such as ow, push/drag, rise, oat all have clear connotations in the bodily domain. Aspects of Laban analytic notation [23] that serve to describe the stress pa erns and internal ow of body movements can also be related to the articulation of music and sound. In traditional western music notation, these qualitative aspects are concerned mainly with how the sounds are connected or stressed, the legato and staccato and how these two are combined. ese aspects are also important in the articulation of spoken language. Here, the stress pa erns and other components belong to what is de ned as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [3]. In a similar way, a movement quality carries information about the dancer's expressivity and may possess a degree of independence that allows it to be imposed on di erent movement trajectories and thereby alter its perceived characteristics [15]. Although the Laban framework has been used as a way to conceptualise movement in an abstract manner, it has also enriched the study of the phenomenological aspects embodiment. e Laban framework provides a complementary point of view from cognitive theories in that is emphasises the experiential aspects of bodily awareness and their role in building up personal experience [18]. As Levishon and Schiphorst state in their contribution, human computer interaction design may bene t from the study of how movement is experienced in rst person, concentrating on the qualitative and expressive nature of movement. is approach is connected to dance studies such as somatics and complements the image schema model. ese previous considerations help us to understand how metaphorical relationships between embodied cognition principles and the analysis of natural morphologies of sound can inform the design of mapping strategies for interactive music. Furthermore, these considerations can lead over to inform creative strategies for converging and complementing aesthetic principles in dance and music. And more speci cally, they provide inspirations for how the qualitative aspects of bodily articulations can be transformed into an articulation of sound objects [32].

Cross-Fertilisation Between Movement and Sonic Arts
One of the sources of inspiration for Piano&Dancer is the assumption of the existence of common principles underlying dance and music. Probably the most ancient but very evident fact of this relationship can be traced back to the 4500 BC. A technique named Chironomy [1], a term deriving from the Greek word quiro which translates to hand, illustrates how music was created in the ancient Egypt empire. In this technique, a system of gestures involving the posture and spatial trajectories of the hand, ngers, arms, as well as the choice of le and right, were used to control every musical detail performed by one or several instrument players. e vertical spatial trajectories of these singing hands may be regarded as the basis for western music notation systems [24]. is very old technique has served as an inspiration for the development of one of the scenes in Piano&Dancer during which the dancers performs Chironomy-based gestures that are translated in real time into music gestures by the piano (see Figure 1).

Figure 1: Performance Scene that is Inspired by the Ancient Egyptian Chironomy Technique
Other spatial techniques of choreographic nature have inspired music composition throughout history and specially during the twentieth century. One of the most evident examples is how the sonic spatialisation techniques in electroacustic music have introduced the geometrical space as a compositional dimension. An overview of spatialisation approaches in musical composition is provided in the PhD thesis by Bates [2]. Wishart's seminal text On Sonic Art [32] provides a thorough description of spatial motions for sonic objects. Some of these motions could be interpreted as choreographic cues. Terms employed to describe these motions are: direct, cyclical, symmetric, looping, zig-zag, forward, backward, elastic, bounced, frame rotations, accelerating, and many more.
Also relevant in the context of this publication are gesture-based music interfaces that have been speci cally designed to allow unconstrained motions. Contrary to more conventional interfaces, these gesture-based interfaces o er musicians a freedom of movement and dexterous control which is not unlike that of dancers [25]. A good example of this approach are non-tactile interfaces. For the successful use of such interfaces, an advanced sense of proprioception and kinaesthetic awareness needs to be acquired by the musician [27]. Accordingly, the study of dancers might provide important inspirations for musicians who plan to design and control these type of interfaces [19]. An example of an interface that serves the speci c purpose of translating dance into music is the Motion Composer [4]. is interface integrates a camera-based motion tracking system and is meant to aid people with physical disabilities to improve the movement awareness. Another exemplary interface is Coniglio's MidiDancer [11]. is interface consists of a sensor equipped suit that allows the dancer to control music. e intended use of this interface is to encourage dancers to move like musicians in order to interact with the musical system.
In Piano&Dancer, the interactive relationship between dancer and electromechanical piano is based on the concept of a non-tactile exchange of mechanical energy, through which the dancer's movements control the actuators that set into motion the piano hammers. By allowing the performer to control the piano through other means than direct tangible interaction, the functional constraints of sound producing gestures and the immediacy of their e ects on the musical result are dissolved. is provides the opportunity to invent novel and diversi ed relationships between physical and musical gestures. As a result, the dancer's bodily movements can be shaped according to choreographic criteria. Furthermore, through algorithmic means, the relationship between movement and music can be expanded to involve interactive control over the compositional process itself. e transfer of music compositional techniques into choreographic principles is also not uncommon in contemporary dance. A movement or a melody are objects extended in time that can be transformed through a variety of means. For example, the traditional transformations of a melodic pa ern, its inversion, its retrograde and retrograde inversion have become movement generation strategies for William Forsythe and other choreographers following his example [10] . It is important to mention in this context how these popular melodic transformation techniques have actually originated from the spatial domain, since they are derived from symmetric group transformations of the rectangle. ere exist many more transformation possibilities since a melodic line or a movement can be rotated through any angle, as Forsythe has showed choreographically in [10] or Xenakis in the musical domain with his Arborescences theory [30]. Another popular technique is musical counterpoint. In music, this term refers to the establishment of harmonic relationships between two melodic lines whose rhythm and contour are di erent. In dance, the counterpoint principle can be applied to align and merge choreographed sections into coherent unities that di er in their use of space, time, or movement. is technique has been extensively used by choreographers such as William Forsythe, Marius Petipa, George Balanchine, Trisha Brown, Jonathan Burroughs, and Pablo Ventura [31]. e adoption of these musical principles as a basis for developing choreographic structures has played an important role in the merging of choreographic and music compositional methods that guided the creative process of Piano&Dancer.

REALISATION
In this section, we describe the technical implementations that form part of the realisation of Piano&Dancer. A major focus is placed on the developments that enable the dancer to control the piano without direct tangible interaction.
is remote mode of interaction with an acoustic piano is based on an analysis of certain qualities of the dancer's movements which serve as input for the generation and control of the sounding material. Accordingly, we describe in some detail which aspects of the dancer's movement are extracted, how these aspects are mapped to control the mechanical movements of the piano, and how the resulting sounds re ect and combine in their properties both the extrinsic aspects of the dancer's movements and the intrinsic properties of the piano's sound producing mechanism. e real time data we gather from the dancer's movements can be organised in two layers [9]. e rst layer corresponds to the physical aspects of movement which is directly measured by the sensing devices (rotational accelerations, position of joints, etc.). e second layer deals with higher level properties of movement that can be obtained by analysing the raw sensory input. Movement properties from both the rst and second layer are then further processed by an additional layer which comprises the generative algorithms such as stochastic distributions or swarm simulations that underly the automated composition of the music. As a result of these relationships, the choreographic and music compositional principles are tightly interconnected. e composition of Piano&Dancer emerges from the integration of movement gestures and corresponding sonic gestures that mirror the dancer's expressivity. A more detailed description of the choreographic and musical aspects of Piano&Dancer can be found in [6].

Initial Considerations and Challenges
One of the challenges of developing this work concerns the establishment of relationships between the aesthetic principles of a dancing human body and that of a mechanical piano. e dancer exhibits a high degree of freedom and continuity in her movements. e piano on the other hand represents a highly constrained musical mechanism whose properties have been optimised to match the formalised principles of western notated music. Accordingly, we face the challenge of how to transfer a gestural activity that evolves in a multidimensional continuum into another gestural activity that is con ned into a one-dimensional and discrete la ice. An additional challenge concerns the fact that the dancer lacks haptic feedback from the instrument that are regular pianist experiences. As Gillespie explains in [12], the hearing sense informs the performer of the acoustical behaviour of an instrument, but haptic feedback conveys information about the instrument's mechanical behaviour. e perception of this mechanical behaviour plays a quite important role in the process of learning and playing an instrument. As compensation mechanism, kineasthetic awareness becomes an extremely important experiential aspect when interacting with musical systems that don't provide haptic feedback [19]. In such situations, kineasthetic feedback provides an important means of assessment for training the motor control needed to master the system. However, compared to haptic feedback, kinaesthetic awareness plays an inferior role for musicians in that it is less suitable for achieving accurate and repeatable musical results [27]. e other side of this relationship is also of crucial importance, namely, how or to what degree does the morphology of a sounding object depend on the morphology of the performer's kinetic energy. To address this topic, it is useful to iterate that every sound-event possesses an intrinsic and an extrinsic morphology [32]. e intrinsic morphology of a sound is shaped by the sound producing mechanisms of an acoustic instrument. Apart from the fact that the sound has to be triggered through a singular impulse by the player, its intrinsic properties are independent from the performer. Sounds whose extrinsic morphology is dominant are produced through a continued excitation through which the morphology of the sound producing activity of the human performer is imprinted. Sounds that possess predominately an intrinsic morphology are produced by instruments such as the piano, or by instruments that are plucked or percussive. Sounds that possess predominately an extrinsic morphology are produced by instruments that are blowed or bowed. e consequence from this observation is that once a dance movement sets into motion a piano key, the performer has no control on the morphology of the resulting discrete sound. Continuous modulation is only possible in a very limited manner through the control of the damping and mute pedals.
However, an alternative means to provide the dancer with a seemingly continuous control over the extrinsic morphology of a piano sound is based on the creation of a acoustic stream in which discrete note events fuse into a single perceptual unit. is is the case for iterative and continuous sound morphologies such as fast trills, tremolos or glissandi, in which the adjacent notes that composed the temporal sequence are no longer distinguishable as individual sonic events but rather become perceptually merged. As we will see below such modes of sound continuation can be used as the sonic substrate that can be shaped by certain qualitatives aspects of movement.

Movement Sensing and Feature Analysis
For almost all the scenes in Piano&Dancer, interactivity is based on sensing the dancer's movements with inertial measurement units (IMU) that are a ached to the dancer's body.
is technique is complemented only during particular moments by a camera-based tracking system. e camera based tracking systems provides an allocentric and absolute frame of reference whereas IMU sensors provide an egocentric and therefore relative frame of reference.
e camera system is used to enable the dancer to set into motion the piano keys or to accentuate notes based on the spatial position of her joints and regardless of whether an IMU is a ached to that particular joint or not. e IMU devices that are used for the piece are named Xosc and are provided by the company x-io Technologies. ese devices integrate a gyroscope, an accelerometer and a magnetometer, each of them providing three degrees of freedom. Furthermore, these devices o er Wi -based wireless connectivity whose bandwidth and latency is excellent even when multiple IMU devices are used concurrently. For the piece, four of these devices are employed to track the movements of four joints on the dancer's body (two wrists and two ankles). Interactivity is based on both the acquisition of raw sensory data as well as on the analysis of higher level movement features such as Smoothness, Weight, Energy and Dynamic Symmetry [9]. e characteristics of these movement features are inspired by the de nitions proposed by choreographer Rudolf Laban [23] but occasionally deviate in their exact implementation from these de nitions. By integrating methods for higher level feature analysis into an interactive system, this system becomes capable of detecting and subsequently responding to movement qualities that are also salient for the dancer and the human audience [28]. is helps to alleviate one of the problems when applying interactive technology for dance: the constraining of dance movements through technological prerequisites and the shi ing of the dancer's a ention away from intentionality and expressivity towards the purely functional execution of movement [20]. e analysis of higher level movement features is implemented in the EyesWeb programming environment. As part of two EU H2020 ICT projects named DANCE and WHOLODANCE in which the authors of this paper are participating, a custom EyesWeb-based so ware application has been developed.
is application processes the raw IMU sensor data and extracts low and higher level movement features. ese features are sent via the open sound communication protocol (OSC) to the composition and piano control so ware. is la er so ware has been developed in the Supercollider programming environment. A third piece of so ware is a custom developed swarm simulation program. is program has also been speci cally developed for the piece and serves mediates semi-autonomous system between movement qualities and piano gestures. Figure Figure 2 depicts a schematic representation of this technical setup.

Soni cation of Movement alities
As mentioned previously, the realisation of Piano&Dancer was strongly motivated by the desire to address the problem of how movement qualities can be related to the morphology and articulation of sounds in such a way that the dancer's movement can be inferred from the resulting sounds. Each of the composed scenes in Piano&Dancer develops a particular movement quality. ese movement qualities are always linked to a particular musical material or algorithmic approach to composition. Moreover, the choreographic structure highlights these relationships. For example by focusing on certain body parts that are associated to the actuation of a group of piano keys. A detailed description of the choreographic and musical aspects of the piece may be found in [6].
Of particular importance in the piece are the movement qualities Energy, Weight, Smoothness, and Dynamic Symmetry of Smoothness. ese qualities can be easily associated to particular embodied metaphors or image schemas. For instance Energy is tightly connected to embodied metaphors of force and constitutes a transversal aspect of movement. Energy also forms a key aspect of any human gesture that gives rise to sound morphologies. Energy can be combined with other qualities in order to obtain a movement analysis that is more in agreement with an intuitive appreciation of a dancer's gestural actions.
e Weight quality is based on Laban's weight e ort and describes how gravity in uences a movement. Accordingly, Weight is directly connected with the verticality image schema that has been previously mentioned. Two of the aforementioned qualities, Smoothness and Dynamic Symmetry of Smoothness, as de ned within the framework of the EU H2020 ICT projects DANCE and WHOLODANCE [9], will now be described in more detail in combination with the soni cation and choreographic strategies that have been developed to exploit them.

Smoothness.
Smoothness corresponds to a concept from biomechanics and is de ned as minimum jerk [9]. e movement of a joint is considered smooth when no abrupt changes in acceleration occur. When taking into consideration the activity of multiple joints, smooth movement would correspond to a coordinated wavelike propagation pa ern through several body joints [26]. e nature of this quality is closely related with the notions of continuity, uidity and predictability in case of high levels of Smoothness and conversely to discontinuity, shakiness, unpredictability, accident or surprise for low levels of Smoothness. In Piano&Dancer, the analysis of Smoothness is performed on the acceleration data obtained from the dancer's wrist movements. e dancer develops choreographic material to alternate between high and low levels of Smoothness.
ese levels are linked to the organisation of musical pitch. By doing so, a gamut of contrasting musical entities can be created, ranging from smooth glissandi to jumps across distant degrees which split or break the continuity of music lines. e contrasting quality of the resulting music can be clearly seen in the real time score transcription of the corresponding scene in the performance (see Figure 3). Smoothness is mapped to a sequence of evolving harmonic elds. e rst eld is purely chromatic to clearly expose the relationship between the quality and the response of the piano keys. Very smooth movements only set into motion adjacent keys.
is chromatic eld evolves into a non-octavating scale and its inversion, a sieve in the Xenakian conception that has been specially devised for the piece. In an non-octavating scale, notes do not repeat a er covering the span of an octave but they maintain intervallic consistency. is o ers the interesting possibility of not only assigning di erent sections of the scale to the movement of di erent joints of the dancer's body but also to create echoing sonorities as the dancer creates slides or jumps across the octaves of the keyboard. Continuous smooth movements are not only re ected in the pitch but also also amplitude domain as accelerando-deccelerando or crescendo-decrecendo. In the corresponding scene, the dancer stands behind and above the piano (see Figure Figure4). is setup puts into focus the strong connection between the dancer's bodily movements and the movements of the piano keys. e movement of the keys visually convey the Smoothness quality in that keys are actuated only locally when the dancer's movements are smooth whereas jerky movements trigger the depression of multiple keys that spread over the entire keyboard.

Dynamic
Symmetry. Dynamic Symmetry is a higher level quality that di ers from static or postural symmetry in that it includes dynamic and temporal aspects. Dynamic Symmetry is based on an analysis of the degree of coordination and dynamics of multiple moving body parts [8]. e skill necessary to maintain high or low levels of Dynamic Symmetry is important not only in  contemporary dance but also in classical ballet. In Piano&Dancer, Dynamic Symmetry is derived from a comparison of the level of Smoothness between the acceleration of pairs of joints. e piano soni cation mirrors in the auditory domain the degree of Dynamic Symmetry in the dancer's movements. e piano soni cation model developed for this quality is based on the equivalence of symmetry and periodicity, where symmetry corresponds to repetitions in space and periodicity to repetitions in time. e model explores multiple possible states from periodicity to aperiodicity in order to mirror the degree of Dynamic Symmetry in the dancer's movements. ese contrasting states can easily be distinguished in a real time score transcription (see Figure Figure5). e degree of Dynamic Symmetry between body parts is re ected musically not only in the resulting pitch and amplitude dynamics but also in the inner structure of the evolving musical modes that shape the harmonic content and color of the music. ese structures correspond to a selection of modes of limited transposition [22] that are composed of symmetric modes in which the last note of each group is always the rst one of the next group. e degree of symmetry of a mode is determined by the number of transpositions it allows before it arrives at its starting position. For instance mode 1 (the famous whole tone scale) has only two transpositions, whereas mode 6 has six transpositions. In the corresponding scene of Piano&Dancer, the level of Dynamic Symmetry is mapped to the degree of symmetry of a mode, ranging from mode 2 (three transpositions) to mode 6 (six transpositions). Accordingly, the dancer can modulate from one mode to another based on the similarity of the smoothness of her wrist movements.

Intermediate Algorithmic Layers: Swarm
Simulations, Stochastic Functions and Musical Abstractions.
By providing an interaction scenario that mediates between the kinaesthetic and visual domain of body movement on one hand and the acoustic domain of algorithmic composition and piano playback on the other hand, the dancer becomes able to transfer articulated gestural expressions into music without the necessity for her to consciously pay a ention to and plan the musical consequences of her movements. Rather, it is up to the music composer to specify the compositional and sonic principles of the music, the automation of which is incorporated and nally delegated to the algorithmic and mechanical mechanisms of the computer and piano, respectively. As a result, the dancer can focus on the creative and aesthetic principles that lie fully within her own area of expertise while relying on the compositional expertise embedded in the interactive musical system to respond in a musically meaningful way to her own performance. e incorporation of algorithmic and generative methods into the mediating layers between movement and music allows the choreographic and compositional elements of the performance to preserve their intrinsic aesthetic principles while at the same time remain connected through a strong causal relationship. ese algorithmic abstractions are in uenced or perturbed in di erent ways by the bodily movements of the dancer. e design of these algorithmic abstractions has been carried out in such a way that the qualities of the dancer's movements are transferred into compositional structures and sonic morphologies that convey aesthetically and metaphorically similarly qualities in the acoustic domain. A more complete description of these mediating algorithmic layers and musical abstractions can be found in [6].
Some of these algorithms explore the properties of nite groups. is for instance is the case for the automated creation of inverted transpositions on the same fundamental note. As an example, the application of automated transposed inversions depends on the rotation of the dancer's ankles and wrists. For each joint rotation, the algorithm returns a new version of an array of numbers corresponding to an inverted transposition with the same fundamental note. Based on the direction of a wrist's rotation, the resulting chord is either in a closed or open position.
Other approaches make use of probability distributions such as Beta or Exponential distributions. ese distributions are used for example for the creation of random walks, which are controlled by the level of Smoothness [9] of the dancer's current movements. In this case, the step size of the random walks is proportional to the jerkiness of the dancer's movement. Furthermore, the direction of the random walks depends on the direction of the angular acceleration of the movement. e set of probability distributions are employed to shape the density of events and also to distribute them across di erent harmonic elds. Each distribution has its own speci c characteristics and can be associated with di erent musical aspects. For example, Cauchy and Gaussian distributions are combined and superposed and their means are assigned to harmonic nodes or tonal centers that are associated to speci c body joints. e levels of movement qualities of a particular joint may be used to perturb the spread or deviation from the mean that de nes these distributions. is causes the musical output to oscillate between harmonic and more dissonant states. A similar application may be developed in the temporal domain to organise durations that move between predictable and unpredictable or random rhythms. Of musical interest are also Beta distributions since they can be used to generate values that cluster at the lower and upper boundary of the distribution range. e mapping of these values on the pitch of notes causes a frequent appearance of highest and lowest notes. ese notes are of critical importance in the way music is perceived. e dancer's movement qualities can be used to control the shape of the beta probability distribution and thereby a ect the frequency of occurrence of extreme note pitches.
A more sophisticated algorithmic layer is provided by a swarm simulation that has been implemented using the ISO programming library [5]. By integrating swarm simulations as intermediary level between movement analysis and piano control, the interactive systems gains the capability to create complex musical material in a self-organised manner. In this setup, the dancer is no longer able to directly and fully control the behaviour of the algorithmic layer but rather assumes the role of an improvisation partner with an arti cial and autonomous musical agency. e repertory of swarm behaviours that are provided by the programming library has been expanded with several additional behaviours.
ese behaviours have been developed to meet the speci c requirements to provide control data for the creation of discrete and note based musical forms (see Figure6). e most important of these behaviours are: A discretisation behaviour that maps any continuous agent parameter such as position or velocity to a prede ned set of discretised values. A cohesion behaviour that permits the speci cation of axis aligned o sets among the positions of neighbouring agents. is e ect permits the realising of chord-like groupings within a swarm. A neighbourhood behaviour that encodes the positions of neighbouring agents in spherical coordinates in order to simplify the distinction between distance and orientation relationships within a swarm. e distance can be used for instance to control the intervallic relationships between notes whereas the orientation can control the permutations of chords. A sequencing behaviour that triggers a timed series of modi cations to a particular agent parameter. e purpose of the sequencing behaviour is to generate control data that exhibits a motivic form.
An example of how the swarm simulation mediates between bodily activity and piano control is provided by a scene in which the choreography is restricted to the hands which oscillate between high and low degrees of Dynamic Symmetry. In this scene, the Dynamic Symmetry of Smoothness between both wrists of the dancer controls the cohesion of a swarm whose agents' positions are mapped to piano keys. Due to the fact that the swarm gathers into oscillating clusters when the cohesion among agents is high whereas it spreads through outwards trajectories when cohesion is low, its mapping onto piano keys causes the piano to mirror both visually and acoustically the dancer's level of Dynamic Symmetry.
All the algorithms that have been described so far map their numerical output onto one of several prede ned harmonic modal systems before being translated into pitch values. ese modal systems shape part of the aesthetic character of the work. A more detailed description of these musical structures may be found in [6].

DISCUSSION AND CONCLUSION
Music performance is a complex and multidimensional phenomena that involves not only a capability for sophisticated conceptualisation and formal structuring on the part of the composer but also highly skilled body awareness, movement co-ordination, and aesthetic sensitivity on the part of the instrument player. Our chosen approach of creating an interactive gesture-based relationship between dancer and piano serves to expand the creative and expressive freedom of movement on the side of the performer while maintaining a clear correspondence between movement and expression in the kinaesthetic, visual and acoustic domain.
By connecting high level movement qualities that play an important role in the dancer's body awareness to the generation of sonic material, the acoustic feedback provided by the interactive system may help to emphasise and foreground otherwise hidden choreographic aspects for both the dancer and the audience. is is made possible mainly because of two approaches: the abolishment of a direct touch-based interaction with the piano and the extraction of higher level qualities from the dancer's movements. We consider these two approaches to constitute a minimum requirement for allowing choreography to free itself from the constraints of normal musical playing gestures while at the same time to maintain a clear correlation between physical gesture and musical gesture.
Furthermore, we discussed how image schemas that form the basis for our higher level cognitive capabilities can help to ground design principles for interactive musical systems in the bodily familiarity of its users. In the context of dance, such an approach enables the dancer to fully rely on her movement expertise while moving through a sensing and sounding interactive environment. e paper has also brie y introduced a combination of algorithmic and swarm-based generative techniques for creating note-based musical structures. is algorithmic layer establishes a network of direct and indirect cause and e ect relationships that connect the dancer's movements to a wide spectrum of di erent sonic results. e usage of such an algorithmic layer is a ractive since it provides both precise compositional control over the musical result while also o ering the possibility for the emergence of complex and surprising musical pa erns.
Finally, we would like to mention that the realisation of Pi-ano&Dancer constitutes for us a bold step away from our previous artistic activities. Beforehand, we had always employed within our collaboration exclusively computer-based synthetic musical instruments for creating the interactive musical aspects of the performance. One goal of this publication was to clarify both to the reader but also for ourselves the artistic motivation and considerations that led to our undertaking of this new creative adventure. One of the most exciting aspects in the creation of this new piece concerns the design of an interactive sensing environment in which the physical presence and gestural qualities of the dancer and the acoustic instrument complement and enrich each other in both a straight forward and striking manner. Furthermore, the richness of the natural acoustic sounds, the residual noises caused by the movements of the piano keys, the large and nuanced movements of the dancer, and the visual relationship and friction between a mechanical instrument and a human dancer create a stage situation that is densely populated by audiovisual events without ever becoming overwhelming. ese observations provided an additional motivation and increased engagement for realising Piano&Dancer as compared to the realisation of more strongly computer-based interactive media contexts.