Choreomorphy: a whole-body interaction experience for dance improvisation and visual experimentation

Choreomorphy is inspired by the Greek words "choros" (dance) and "morphe" (shape). Visual metaphors, such as the notion of transformation, and visual imagery are widely used in various movement and dance practices, education, and artistic creation. Motion capture and comprehensive movement representation technologies, if appropriately employed can become valuable tools in this field. Choreomorphy is a system for a whole-body interactive experience, using Motion Capture and 3D technologies, that allows the users to experiment with different body and movement visualisations in real-time. The system offers a variety of avatars, visualizations of movement and environments which can be easily selected through a simple GUI. The motivation of designing this system is the exploration of different avatars as "digital selves" and the reflection on the impact of seeing one's own body as an avatar that can vary in shape, size, gender and human vs. non-human characteristics, while dancing and improvising. Choreomorphy is interoperable with different motion capture systems, including, but not limited to inertial, optical, and Kinect. The 3D representations and interactions are constantly updated through an explorative co-design process with dance artists and professionals in different sessions and venues.


INTRODUCTION
Interactive motion capture technologies create new opportunities for reflecting on movement through different modalities. In particular, the use of full body motion capture and. 3D display of the movement allows the dance practitioner to view the movement in its full 3D dimensionality. The Choreomorphy interaction system offers 3D display of the movement in real-time, as well as the option of viewing recorded motion captured sequences. Through a simple interface, the users themselves or an assistant independent of technical background, can alter the environment and avatar and add special effects such as trails and traces, customizing characteristics like fade-out duration time, allowing the dancer to focus on specific aspects of movement such as shapes in space and trajectories.
Choreomorphy provides an interactive virtual environment which gives the possibility to select and customize the visualizations of the dancer's body and her movement, facilitating self-reflection and experimentation with different visualizations and avatars. According to the feedback from the dance experts who participated in the co-design and iterative evaluation sessions, the interactive experience allows selfreflection, stimulates imagination and raises many questions related to body perception, such as if and how they identify the avatar as their own reflection or as another "partner" or "creature", depending on its shape and figure.
Firstly, in Section 2 we frame the Choreomorphy system, in the context of the state of the art in the field and relevant work and we highlight its contribution. In Section 3 we describe the experience and present the technical aspects of the setting and the system. In Section 4, we present the co-design and formative evaluation process with dance practitioners that we have followed. We describe the characteristics of dance practice, and in particular contemporary dance and improvisation. In Section 5, we explain why the visual representation of the body and movement in digital spaces is important and how we took this into account. In the same section we describe the virtual environment and different options given by the system, and we present the user requirements which are derived from the codesign and evaluation sessions. Finally, in Section 6, we discuss the outcomes of the process, including open issues and future work.

DANCE IN DIGITAL ENVIRONMENTS
Motion and its characteristics can be translated into different visualizations, sound and virtual objects that can provide new insights into dance teaching, learning and creation of new movement sequences and choreographies.
Whereas in performance and dance production, technology is being widely used, it is still absent from the dance studios where dancers rehearse, learn and experiment, according to Molina et al. [17]. Designing digital environments to satisfy actual needs of dance practitioners, is a challenge, since there are no mainstream tools or best practices that can guide the development to such experiences [6]. The low affordability and portability of motion capture state-of-the-art technologies, in combination with the fact that few dance practitioners have experienced such technologies, makes the elicitation of concrete user requirements and user-centered design of this type of experience a really challenging task.
The need for a multidisciplinary approach and actual involvement of dance experts and practitioners in the process of design is crucial.
A number of recent technologies have explored the concept of an augmented mirror to enhance the dance learning experience. "Super Mirror" by Marquardt, Zoe, et al. [14] is a Kinect-based system that combines the functionality of studio mirrors with instructional feedback in real-time. The results of its usability evaluation with ballet students [20], has shown a potential for its use in ballet education. Hachimura et al. [10] describe a prototype dance training support system called "Just Follow Me", with motion capture and mixed reality technologies. Molina-Tanco et al. [17], propose a simple technology setting consisting of a recording mirror which reproduces the image with a few seconds of delay.
Another common characteristic of the existing digital dance practice environments is that their approach follows the paradigm of learning by "seeing and doing" [11][10], a teaching approach which is also known as mimesis. In this case, these systems approach the use of interactive technologies for dance learning by teaching a choreography through an avatar, asking the learner to follow and provide feedback on the "correctness" of the movements. Anderson et al. [2] introduce "YouMove", a system for learning full body movements, comprised of a Kinectbased recording system, and a corresponding training system. Aristidou et al. [3] introduces a prototype virtual reality simulator in which users can preview segments of folk dance performed by a 3D avatar and repeat them. Alexiadis et al. [1] describe a prototype system that automatically evaluates dance performances against a dance professional performance and provides visual feedback to the performer in a 3D virtual environment.
In dance learning, which is by nature multimodal and diverse, using visual, kinaesthetic or even poetic imagery and metaphors to enhance the performance of an exercise is a very common practice, even in genres were the kinetic vocabulary and technique is very precise, such as classical Ballet. Franklin [7] presents hundreds of imagery examples which can be used during dance technique and improvisation classes to enhance performance, while some of them have been evaluated in order to measure if and how they affect the performance of dancers in a real-life context [12]. In contemporary dance, the use of visual metaphors is widely used to inspire new kinetic material in the contexts of improvisation and creativity. Merce Cunningham, Trisha Brown, William Forsythe, and Wayne McGregor, "are only some of the well-known choreographers who focus on the creation of innovative movement through the use of mental imagery related to sensation, space, meaning and emotion" [15]. In addition, some somatic practices, such as "Skinner Releasing Technique TM " [19] which are still applied and have inspired contemporary dance learning practices, fully rely on the use of kinaesthetic and even also poetic metaphors. These have also been investigated in digital and virtual spaces by R. Gibson [9].
Contemporary dance approaches and learning techniques focus not only on the produced shape of the movement, but also on the movement qualities, the dynamic or the way the motion is performed. Fdili Alaoui et. al. has explored the relationship of movement qualities and physical models, through a long-term collaboration with the choreographer Emio Greco [8]. Camurri et al. [5] have developed a computational framework for movement qualities applied both on dance and other forms of non-verbal communication.
While several digital environment systems have been introduced for dance practice, and while some valuable contributions have been made the last years in investigating the relation between abstract movement visualizations and movement qualities such as those by Bisig et. al, [4], Camurri et. al [5], and Fdili et. al [8], little discussion has been dedicated to the use of anthropomorphic 3D avatars and how their characteristics could affect the experience of dance improvisation or teaching.
Summarizing the above we identify two types of systems: one extending the paradigm of "augmented mirror", and one where the scenario of use relies on seeing and following a prototype movement on an avatar -teacher. In parallel to the two aforementioned types of systems for practicing dance, there are two points of using avatars 1) showing the teachers' movement, 2) seeing oneself as an avatar for self-reflection, while moving. Although Choreomorphy system can be applied for both types of avatars, the co-design and evaluation with users have been mainly focused on the latter perspective. As defined by Loke et. al [13] in such interactive digital experiences the body can be seen through two perspectives: the mover's, referring to the perspective of "first person experience of the moving body" vs. the observer's, referring to the "the view of the body from the outside". In the experience of seeing one's self as an avatar while moving, the user becomes a mover and an observer at the same time.

SYSTEM AND SETTING
For this exploratory research we focused on the effect of "seeing oneself, as a moving digital body". Thus, the pipeline setup of Choreomorphy resembles a real-time inertial motion capture session. Independently of the venue where Choreomorphy is deployed, the setting should be a darkened room with space layout of at least 12m 2 so that the user can move freely. It is recommended to use a projector or a monitor of at least 24" for display purposes. The bigger the screen or the projector used, the more the avatar becomes closer to a life size figure and the whole experience becomes more immersive.
The user firstly has to wear the motion capture suit and calibrate it and then she is able to move freely in space. The system livestreams the motion to an avatar which simulates the movement on the screen (or projection), as shown in Fig.1. The user can customize the visualization settings (avatar, scene, effect etc.) or ask an assistant to do so. The user can initialize and fine tune different visual parameters including avatar textures, particle systems, motion trails and motion traces, and even the virtual environment itself. The user can also switch between several viewpoints of the avatar, including a first-person perspective. The interface is available through the PC or laptop, as shown in Fig.3. and Fig.4, or a mobile device and has been inspired by a typical character customization interface [16].

Figure 3. Graphical User Interface and avatar with motion trails and traces -screenshot of the application
Choreomorphy as a system is developed to be interoperable with a number of input and output devices. As input we consider 1) the motion data coming from the motion capture suit, and 2) the user input from the controls. During the co-design sessions Synertial's "Cobra IGS C-420" inertial motion capture suit was mainly used as the input for motion data. Other motion capture devices have been tested and can be used, including the Microsoft Kinect.
Choreomorphy can also be used for displaying pre-recorded movement sequences, thus providing a desktop experience of seeing the motion recording in the different visualization options the tool offers for real time visualization. The architecture of the system is shown in Fig.2. More intuitive and less disruptive ways of controlling the visualization parameters are being considered and tested, including voice and/or gesture recognition.

CO-DESIGN AND EVALUATION
The design and evaluation of the Choreomorphy tool follows a user-centered iterative approach. Taking into account the exploratory research aspects of the approach and the need to investigate the complex relation of the dancer and her virtual body in an improvisation, but also an educational context, the team worked closely with dance practitioners, both members of the core Choreomorphy research team and external ones.
The tool, from its early requirements and design phases, has been informed by constant interaction with experts, initially in the form of design ideation interviews and focus group sessions and later through interactive sessions both in the lab and in other settings, where testing the current version of the tool became the stimulus for more concrete input and insights by the practitioners.
The co-design sessions were organized with the following main objectives: a) to provide an insight about desired features for the interface and avatars (human vs. non-human shape, human vs. non-human articulation, face characteristics), b) to explore the potential and impact of such a tool in an artistic, creative, and also educational context for dance practitioners that wish to experiment with visual metaphors and imagery, and c) to evaluate the experience as whole, since motion capture technologies are not widely used by dance practitioners and for the majority of them this was a completely new digital experience.
During this process, the team collected feedback also on the more technical interaction aspects of the tool, constantly improving thus the interface and interaction paradigm to better serve a whole-body interaction experience with movement being its main objective.
The co-design and evaluation activities organized can be roughly divided in three main groups: 1. Laboratory sessions with 3 dance practitioners that closely collaborated with our group and were able to provide continuous feedback. These included a professional ballet and contemporary practitioner, also experienced in other dance

Scenes
Smartphone Mapping genres, one ballet/contemporary practitioner and one contemporary dance performer. They regularly experimented with different versions of the system and offered their input in the form of informal interviews and on-going commentary while using the system.

Presentation and interactive sessions in three science fairs
where the system has been presented interactively to a wider audience, including children. These sessions included the professional performer who demonstrated the use of the system while members of the audience could try it through a Kinect device. Feedback was gathered through brief interviews.
3. Two Workshops with dance practitioners have been organized, with a total of 12 participants. In this case the participants had the chance to experiment with the tool at length and then provide their feedback in the form of a questionnaire and semi-structured interviews.
The next paragraphs present in more detail the two Workshop participants profile and setting.

Choreomorphy co-design sessions
The first Workshop group, part of it shown in Fig.5 consisted of 5 female and 2 male dancers, all with extensive experience in contemporary dance and improvisation. In particular, the participants were: one female contemporary teacher, choreographer and professional performer, one male professional performer and choreographer, two (male-female) graduate dance students and professional performers, two female advanced, amateur dancers with experience in contemporary dance, ballet and dance theater, one female professional performer/actor with background in contemporary dance and somatic theater. The second Workshop group, part of it shown in Fig.6 in a different venue, consisted of four female participants, and one male. In particular, the group included one experienced dancer 1 User Experience Questionnaire, http://www.ueq-online.org/ and educator in Greek folk, one choreographer and certified Skinner Release Technique TM teacher, one professional contemporary dancer, choreographer and ballet teacher, two contemporary performers one of the with background in visual arts. Only two out of the twelve participants in both sessions had some experience with motion capture before.
The participants were firstly briefly introduced to the purpose of the experiment and asked to sign a consent form agreeing to be recorded through video, motion capture and/or audio. After wearing the mocap suit, each participant individually spent on average 60 minutes experimenting with the tool through movement improvisation. No music was used in order to avoid additional bias and influence on movement.
After the movement sessions a face-to-face interview followed with each of the participants and they were asked to complete a questionnaire to provide additional input. For the user experience and usability part of the questionnaire, we used the UEQ -User Experience Questionnaire 1 which foresees 6 scales measured through 26 questions (seven-point Likert-scale from -3 to 3). For our evaluation we used the five of the six scales as most relevant, Attractiveness, Perspicuity, Dependability, Stimulation, Novelty. Attractiveness is a pure valence dimension.

User experience and usability
As it is shown also in Table 1 presenting the UEQ results, users were enthusiastic with the UX aspect of the tool. As one them noted: "I felt interest and curiosity, it was a completely new experience!". As it will be shown later in this section, the participants felt the experience was engaging and attractive, absorbing them in a novel way of experimentation through interacting with a virtual altered and augmented self.
Choremorphy: a Whole-body Interaction Experience for Dance Improvisation and Visual Experimentation AVI 2018, May-June 2018, Grosseto, ITALY.
Again, as shown from the results, the users, although positive were more reserved with the pragmatic aspects of the tool (perspicuity and dependability), due to two main reasons, identified in the interviews. Firstly, the environment of the lab and the idea of performing with the evaluators present, to them an audience not focused in the performance itself but on the technology was to a certain degree daunting and did not make them feel at ease. Secondly and most importantly, the users did not feel in complete control of the experience. With the objective to have them focus in the performance, the evaluators were handling the controls of the experience, with the participant's instructions. Most of the participants would have liked to be allowed to control the experience themselves and have some "alone time" with what was for them an exciting new piece of technology.
Impact and potential for dance performance As already mentioned, the main strong point of the approach was that the different avatars were seen as an augmented mirrored-self that allows the users to see their movement but at the same time to distance themselves from their own self, emerging thus as a new character or moving creature. Therefore, as we expected, the different avatars created for the dance practitioners a creative, immersive experience which stimulated their movement improvisation. Most of the participants explained that this marginal perception of the avatar as their own self was a motivation to new movement patterns.  It was interesting that a kinetic relation was forming between the physical and virtual self, as sometimes they felt that the avatar was leading and vice versa, although in fact it was always their own movement that was reflected. One of the participants noted: "At some points I felt like the avatar was not me, but rather that I was just initiating its movement. I became more and more curious about moving in different ways and trying to affect the visualizations." Another participant adds, "I was surprised when I saw myself through the avatars. I was trying to move according to the shape which I was seeing every time. Each avatar triggered me in a different way. So, my movement was affected by the avatar. I really liked it, because I was discovering all the time new movements." An interesting aspect to investigate further is the perception of the "empathic" relation gradually forming between the dancer and the avatar. As one of the performers notes: "At the beginning the emotions where not connected at all between the avatars and me. We were kind of separate, the avatar and me. But then, very empathic, narcissistic, playful and equal emotions started to trigger between the avatars and me." Some participants reflected on how the Choreomorphy interactive experience allowed them to move in a new body, with different size or different gender. One of the professional female performers, who is also a choreographer and dance teacher, notice: "I'm a small shaped dancer, so seeing myself as a bulky avatar with big volume, was an interesting experience and triggered me to move in new ways". While a young male choreographer and dancer comments: "Seeing myself dancing as a female avatar was an interesting and strange feeling. It was amazing! I like the fact that most of the avatars were gender neutral. I was mostly intrigued by some avatars where the human shape was distorted. I would like to play more with this aspect".
In addition, dance experts commented on the way the visualizations allowed them to actually see, what in the real world they can only imagine through imagery techniques: "It was great playing with the traces and the trails. Usually in the dance studio you get instructions of trying to imagine the shape of your movement traces, the trajectories, but with this tool I could actually see them!" The moment of changing from one avatar to another was always a moment of excitement and a focus of attention for the dancers. Last but not least, it was interesting to see how the usage of the system by each one of the individuals, revealed something about her/his dance practice. For example, one of the performers who never uses the mirror in his practice, preferred to not look directly on the screen while dancing but to use more his peripheral vision. The performer who had also an acting background, improvised with voice and speech and explained that she would like as an additional feature to be able to produce different voices and sounds while moving through the avatars.

DIGITAL BODY REPRESENTATIONS AND ENVIRONMENT
Through the co-design and evaluation sessions, the use of a variety of avatar types, environments and visual effect characteristics is being explored resulting in new visualizations added to the system.

Avatar Variations
As the co-design sessions confirmed, avatars are not simply digital-virtual bodies, but also graphical representations of identities. In this case the avatar not only conveys the movement, but depending on its shape, size, anthropomorphism vs. abstractness, gender, articulation, color and texture, it may convey particular movement and character qualities. In the case of the dancers seeing themselves as a moving avatar, instead of his/her real physical body in the mirror, may create a novel and captivating experience as the kinesthetic feedback is matched with a brand new visual feedback. Taking into account the deep connection between body representation and dance practices, and also the need to go beyond the stereotypical representations and explore the potential of moving in a different body, the question of designing appropriate avatars for such experiences becomes critical. Through the constant interaction with users, different avatar types and variations have been explored in an ongoing effort to categorize them and identify patterns as to their suitability for different contexts and practitioners. The Choreomorphy avatar library at the moment contains two main avatar groups, a) Abstract Anthropomorphic and b) Cartoonistic. The first category can be distinguished by the lack of face characteristics combined with humanoid body shape, as shown in Fig.8 The basic characteristic of the second category is the resemblance to a cartoon character that maintains the familiarity of the human body shape but also abstains from realism. Both basic avatar categories cover a wide range of character representations that could match the user's preferences for their dance improvisation. It is worth mentioning that, despite the fact that the avatars share the same basic characteristic of the category that they belong to, they differ in other features. As a result, we divided them to subcategories according to more detailed qualities. In Table 2, we present a categorization of the implemented avatars based on their characteristics.
The different avatars provide a variety of body representation and aim to bring to the surface the impact of seeing oneself as a different digital body and to identify patterns among characteristics of avatars, movement qualities and emotional reaction. During the workshop the participants were able to provide their feedback on a poster including a variety of avatars, as shown on Fig. 9.  Figure 9. Feedback on the poster of Choreomorphy during the workshop with the dance experts.

Motion and Scene Visualizations
The scenes are divided in themes and each one of them stimulates the imagination of the user to choose it according to their aesthetics and current needs. Some of the scenes are pictured in Fig.10. The strength of the system is that by combining the different features such as the avatar, scene, trails or traces, the resulting environment can effortlessly offer a very different atmosphere and aesthetic, transforming from a colorful, pop, cartoonistic, to a more abstract and poetic one.  The co-design sessions with users, not only highlight the importance of the graphical representation of the body in digital environments for dance learning and creativity, but also show the possibility for an emerging tool for artistic creation and exploration of movement beyond the conventional ways of moving, which are deeply rooted in the way we perceive our own body with its particular shape, size, gender characteristics, etc. All of the participants felt that they would possibly or definitely use such a tool in their practice if they would have the chance to deal with the practicalities (hardware availability and cost).
The interactive avatar is the reflection of the performer, but at the same time the avatar depicts a character, a dance partner which can inspire the dancer to explore different ways of moving. Both within the framework of artistic experimentation, performance and creativity, and in the context of education, visual metaphors of movement, shape and qualities can be a powerful tool and raise many scientific and research questions.
Through the dedicated sessions with dance professionals in the lab, all participants commented on the fact that each avatar inspired them to move in particular form or quality, and overall their mood has been largely influenced by the avatar chosen at the time in the way they moved and improvised. Nevertheless, during the co-design and evaluation sessions, it was hard to identify any particular patterns of movement related to the avatar on the visualization, neither to the preference towards a very specific avatar.
Another interesting outcome was that the majority of the participants commented on the fact that each avatar created different emotions and that depending on the avatar they would identify with the avatar as a "mirrored-self" or not. According to the feedback received, the abstract anthropomorphic avatars were closer to being identified as a reflection, or shadow, rather than a separate entity. Taking into account the analysis of the video recordings, and also the feedback during the interviews, the abstract anthropomorphic avatars created a more mysterious and esoteric mood which was usually reflected through smaller, cautious movements and stillness, while it also made the users wonder "what is this, is it my shadow?". In the case of the cartoonistic avatars, they frequently led to laughter, bigger and more rhythmic movements and a more relaxed and playful mood, while most of the participants had the feeling of moving or puppeteering another creature, different from themselves. The moment of transformation was always a very intense moment in terms of reaction, a fact which might be connected with a surprise of seeing it for the very first time. We are continuously investigating different interactions and metaphors which will take advantage of this movement of transformation in a more meaningful way. The Kinect version of the tool which has been used in a performance session for wider audience including children, indicates a big potential to be explored in this direction.
Finally, another interesting part of the user feedback, is the wider discussion on whether the experience of seeing oneself as a digital avatar different from oneself diminishes or amplifies the "narcissistic" effect of the mirror, i.e., the obsession of constantly seeing one's own reflection on a two-dimensional surface (mirror, screen or projection).

DISCUSSION AND FUTURE WORK
In this paper we presented Choreomorphy, a novel interactive system which is compatible with a variety of motion tracking and capture systems and allow to explore time. The system is in its prototype form and reflects a long-term co-design process and experimentation with experienced dance practitioners and also demonstrated and used in events for the wider public of different ages. This process led to the definition of a number of design decisions, based on user requirements. As a digital environment for dance practice, Choreomorphy satisfies the following principles: 1) Allow the dance practitioner (user) to deepen the understanding and perception of movement through self-reflection and multimodal exploration. 2) Follow a teaching approach which goes beyond the traditional or mimetic teaching approach, where the teacher or choreographer makes all the decisions. Since digital learning and practicing environments have the advantage of allowing the practitioner to self-practice without the anxiety of being judged by an austere dance teacher or choreographer, it is important that the feedback provides rich material for self-reflection and goes beyond the "right/wrong" paradigm. 3) Inspire the dance practitioners to generate their own kinetic material and embody the different principles and concepts of movement based on their own sensorimotor abilities, learning or artistic objectives and self-expression potential. 4) Provide tools to explore and enrich their movement vocabulary, movement qualities and expressivity range through imagery, instead of mimicking particular steps and movement forms and structures. 5) Create an experience that would not be possible in the real world. One way to address digital dance environments for learning and practice is by implementing a digital environment which aims at simulating the real studio experience. On the other hand, there is a dynamic potential in stimulating dance practitioners' imagination through designing experiences where the dancers can test with whatever is unfeasible in the real world, extending the physical limitations of the body, space and time. Consequently, creating digital environments and prototypes, where dance experts can explore a wide range of visualization options, experiment with them and then reflect on the experience becomes a necessity in order to take interactive digital dance improvisation and education tools closer to the actual needs of advanced dance practitioners.
In addition, taking into account the importance of imagery in dance practice, Choreomorphy can be applied to digital improvisation sessions to advance research on dance movement by exploring the impact of body and movement representation in digital environments and draft requirements regarding the avatar characteristics (anthropomorphism, textures, shape, gender, etc.) and their influence on movement improvisation.
Our on-going work is focused in the visualizations themselves, to understand which characteristics are more interesting to the practitioners and how they affect their practice. It is also focused in exploring alternative ways to make the interface between the dancer and their digital counter-part, the avatar, even more transparent, through the use of different interaction paradigms and possible augmented and/or virtual reality equipment. Finally, we plan further user sessions with dance experts. Comparing to any other professionals, experienced dance practitioners have great ability of analyzing movement, and sensitivity in perceiving body and space, and therefore can provide valuable feedback and set requirements that would be applicable in a wider range of applications and movementrelated-areas.