Exploring Visualizations in Real-time Motion Capture for Dance Education

In this paper, we describe an ongoing work towards developing a whole-body interaction interface for exploring different visualizations of movement, using real-time motion capture and 3D models, to apply in dance learning and improvisation within a creative, gamified context. A full inertial motion capture system is used by the performer while a simple user interface provides the option to the user to experiment with different avatars, and visualizations e.g., trace of motions on different parts of the body and to interact with virtual objects. The 3D simulation provides a real-time visual feedback for the movement. The interaction follows the paradigm of moving from mimicking kinetic material into a self-reflection teaching approach. The interactive avatar is the reflection of the performer, but on the same time the avatar depicts a character, a dance partner which can inspire the user who moves to explore different ways of moving. Either within the framework of artistic experimentation and creativity, or in the context of education, the visual metaphors of movement shape and qualities consist a powerful tool and raise many scientific and research questions.


INTRODUCTION
In a typical dance class, the mirror is one of the standard ways to self-reflect on the performance of the movement and promote selfcorrection. Interactive motion capture technologies create new opportunities for reflecting on movement through different modalities. Motion and its characteristics can be translated into different visualizations, sound (sonifications) and virtual objects that can provide new insights into dance teaching, learning and making of new movement sequences and choreographies [4]. Interactive technologies provide the tools to design and implement experiments for researching the connection between imagery and the enhancement of one's movement performance and technique, while computational models and algorithms have been developed and evaluated in measuring movement qualities [14] [15] as both dance characteristics and non-verbal communication elements.On the other hand, Augmented and Virtual environments can be used as means to realize and experiment with imagery examples which are widely used in dance practice. For great choreographers such as Merce Cunningham, Trisha Brown, William Forsythe, and Wayne McGregor, whose interest was also the creation of innovative movement, instead of repeating forms, the use of mental imagery related to sensation, space, meaning and emotion is also widespread, as May, Jon, et al. [13] reports. Franklin [5] presents hundreds of imagery examples which can be used during dance technique and improvisation classes to enhance performance. Though these visual and kinesthetic examples are widely used in practice, and are in fact adopted by dance practitioners and teachers through experience, as Heiland's [8] study explains there are no research results that combine specific examples of imagery with specific enhancements in particular actions and movements. In our experiment we set an installation and interface (Section 3) which allows us to experiment with different visualizations of motion as kinetic metaphors realizations (Section 4).

RELATED WORK
Recent research papers describe relevant experiments of adopting mo-cap technologies and interactive interfaces in screen, augmented and virtual reality settings. In the following section we describe related work relevant to dance education which use different technologies of motion capture, such suits with optical mo-cap, and Kinect. Hachimura et. al [6] describe a prototype dance training support system (Just Follow Me) with motion capture and mixed reality technologies. The system uses an intuitive "ghost" metaphor and a first-person viewpoint for Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. PCI '16, November 10-12, 2016 effective motion training. Using the ghost metaphor (GM), it visualizes the motion of the trainer in real time as a ghost (initially superimposed on the trainee) that emerges from one's own body. The trainee who observes the motion from the first-person viewpoint "follows" the ghostly master as closely as possible to learn the motion. Anderson et. al.
[1] introduce "YouMove" a system for learning full body movements with a direct application in dance education. It allows users to record and learn physical movement sequences through a simple recording system that allows anyone to create and share training content. The training system uses recorded data to train the user using a large-scale augmented reality mirror. The system trains the user through a series of stages that gradually reduce the user's reliance on guidance and feedback. YouMove is comprised of a Kinect-based recording system, and a corresponding training system. Aristidou et. al. [6], introduces a motion analysis and comparison framework that is based on Laban Movement Analysis (LMA), used also in the context of teaching folk dances, and a prototype virtual reality simulator in which users can preview segments of folk dance performed by a 3D avatar and repeat them. Alexiadis et al. [3] describe a prototype system that automatically evaluates dance performances against a dance professional performance and provides visual feedback to the performer in a 3D virtual environment. The system acquires the motion of a performer via Kinect-based human skeleton tracking, making the approach viable for a wide range of users, including home enthusiasts. SuperMirror by Marquardt, Zoe, et al. [12] is a Kinect-based system that combines the functionality of studio mirrors and prescriptive images to provide the user with instructional feedback in real-time. The results of its usability evaluation with ballet students [16], show a potential for its use in ballet education but improvements of Super Mirror are needed to comply with the standardized subject matter expert's criteria.

EQUIPMENT AND INSTALLATION
In this section we briefly describe the technologies which have been used and developed in this ongoing work. The performer (user) wears the motion capture (mo-cap) suit which contains a wireless HUB and is connected to the local network, via its MAC address. Then, in the mocap software (Animate) we connect the suit and Animate transmits the mocap data to Unity software 1 . The motion capture system that we use for our experiment is inertial. Each sensor measures rotational rates. These rotations are translated to a skeleton in the Animate software, which transmits the data to Unity, where the 3D model mapping takes place.
While the performer moves in the environment, the system live streams his/her motion to an avatar. The avatar simulates the movement of the user on the screen while the user moves in the real world. Inertial motion capture system presents several advantages for our specific application in comparison with the optical ones, including the following characteristics: a) portability: since the data are transmitted through the wireless hub, the performer wearing the suit, can be anywhere within the distance of twenty meters from the server, and there is no need for a special set up of cameras like in the case of optical motion capture, b) no spatial setting is needed: Since the motion is captured through the change of the inertia and not the optical result, there is no need to set up a particularly large space, c) lower cost comparable to other optical systems, which makes it more affordable and thus more feasible for a realistic setting, e.g., a dance school. Any other person or object can be close to the performer without affecting the motion capture. The main limitation, however, of inertial motion capture in comparison to the optical is the interpretation of feet relationship to a reference surface e.g., ground in movements such as jumping and sitting, though the rotations of the body are captured.
Through the interface, the user, i.e., the performer wearing the suit, or the teacher is able to alter several parameters which are described in the following section.

VISUALIZATION VARIATIONS
The user, whether this is the performer alone or the teacher, can make some initializations and choose among different options of visualizing the different parameters. These parameters mostly are particle systems [17] and motion trails [18].Our particle systems consist of a large number of primitive 3D shapes. Their behavior replicates phenomena such as smoke, fire etc. and they are produced by an emitter. The emitter is an invisible spot which operates as the source of the particles that are being rendered. Our Motion trails are entities that consist of large number of sprites that are too close to each other and replicate the effect of a light trail.

User Interface (UI)
Through this menu the User can change several settings during the session. For the time being we have implemented controls such as buttons for changing the camera view , morphing to another avatar ,taking screenshots, recording videos, restarting the scene and hiding the menu itself, switches for activating/deactivating particles and sliders for adjusting the shape of the motion trails and the duration that they remain rendered .However, there are corresponding hotkeys for quicker actions. (Figure 1) The user can decide on which joint of the avatar he/she will put the particle emitters, and therefore decide to bring the performer's focus and attention on a particular body part or joint. The switches and sliders allow the user to change the mode of emitting style. This can inspire different qualities of movement. In the following section we describe in detail what these different options and variations for the visualizations are.

Avatar variations
One of the main questions when it comes to teaching dance through animation, is how the avatar looks like and how the characteristics of this avatar reflect the body image related with the dance practice or context [5]. In this work, through the simple interface, the user can choose different avatars -predefined 3D models. In the following figures, four of the different anthropomorphic Unity avatars, which have been used in the experiments are shown. In addition, we have collaborated with a 3D artist, for creating additional customized 3D models depicting anthropomorphic but more abstract shapes. Figure 2 shows one of these anthropomorphic prototypes which depicts a "sharp" character with spikes on its back. It is true however, that all the examples shown in the Figures, reflect a variety of mainstream or archetypical characters and more abstract body representations. The different avatars allow the interactive experience to range from a gamified, creative experience, which can be addressed also to children, to an improvisation environment where the looks of the character can guide the performers' movement qualities into a "motion story".

Avatar emits particles from its hands and feet (optionally).
Particles and motion trails contribute to a better understanding of the user's movement and also offer vividness and beauty to the movement of the avatar which can be used for dancing performance and education. In fact, they can work as an extension to the body limbs and clearly depict the path of the motion and the virtual shapes that the mover draws in space. This metaphor has been used in other artistic and research installations, also using neural networks [6]. In our interface there an option of adding or removing this effect, as well as controlling some of the properties of the emission effect. Alternatively, it could be controlled by specific hand gestures that will activate/deactivate the particle emission. In addition, every emitter can be attached to any one of the joints of the avatar. For the experiment that we describe in this paper, we have used several types of particles and motion trails that the models emit:

Capsule shaped particles
Capsule-shaped particles that are emitted linearly and through the progress of their lifespan, they rotate and change their color. The current gradient color is yellowish to red (gradient of fire). This emitter doesn't contain any collider component, which means that its particles don't interact with other projects of the visual world. (Figure 3)

Sphere-shaped particles
Sphere-shaped particles that are rambling around the hands of the avatars. They contain collider component and the fact that the hand also contains a collider component, keeps them nearly together and they may escape from the group of particles only through violent movement of the avatar. Also, they may interact with the visual world, which means that they bounce on the floor etc. Color gradient: white -> yellow -> red (gradient of fire) (Figure 4)

Square-shaped particles
Square-shaped particles are emitted slowly and harmonically around the whole avatar and create a "cloud". The color of the particles is white and when the avatar is moving, so does the emitter, because it's a "child" object to a joint of the avatar and inherits the world transform coordinates of its "parent". Also, it maintains its local transform coordinates. The particles don't actually trace the movement of the avatar but they create the effect of movement tracing. Basically they follow the first law of Newton. When the avatar moves its hand, also the corresponding emitter moves but every particle that is emitted in every specific moment, maintains its initial movement. (Figure 4)

Motion Trails
They are trails of polygons that are rendered behind the moving limbs of the avatar. They emphasize the feeling of motion to the moving avatar. Through the UI, the User can change the width of the source and the tail of the trails and also adjust the time that the trails will remain rendered to the scene. For each avatar there are specific color-themed motion trails. (Figure 5)

Figure 5. Avatar (a) with motion trails
Also, there are be three emitting modes:

Continuous Mode
Particles are pre-emitted during the session and when the User changes the avatar, the flow of the particles doesn't start again from scratch. This mode is currently used for the particles.

Fade-in Mode
Particles start to be emitted when the session starts. Also when the User changes avatar, the emitter starts again from scratch and a fade-in effect is created. This mode is currently used for the particles.

Trigger Mode
Motion trails are rendered only when the Avatar moves the joint where the emitter is attached to. When a specific joint stops moving, its corresponding emitter stops rendering. The fade-in and fade-out effects apply here too. This mode is currently used for the motion trails.

Avatar can interact with objects of the visual world.
There are objects such as wall of crates,stacks of crates and a ball, that the avatar can interact with. There are colliders in every object (including the avatars), thus the interactivity can be achieved via collision detection. Collider components define the shape of an object for the purposes of physical collisions, they are however, invisible. Also, when collisions occur, the physics engine calls functions with specific names on any scripts attached to the objects involved. For example, the User throws an air punch in the real world, but in the visual world her/his avatar punches a crate and drops it from the top of the stack. We have attached collider component to every joint of the avatars in order to be able to interact with every one of them. Also we have added colliders to the objects with which we want to interact with. In order to conform to the laws of physics we added rigid body components and physics materials to the objects that we interact with (crates and ball). For example, the ball will not bounce if it doesn't contain any physics material component and will not be affected by gravity if it doesn't contain a rigid body component. (Figure 6 and 7) The User can switch from one camera to another one to have different viewpoints of the avatar. For this set-up we have defined the following cameras:

Main Camera
The main camera, with which the User can roam freely in the visual world. In other words, the User can control (translate, tilt, pan and zoom) the camera and configure the Point of View that suits to her/his session. (Figure 6)

First Person Camera
First Person Camera, in which the User sees through the eyes of the avatar itself. In the FPS camera mode, the application works fine with Virtual Reality kits e.g., Oculus Rift, Google cardboard etc. (Figure 8)

Cinematic Camera
Cinematic camera, where the camera does a 360 degree movement around the target. This movement is fixed given the fact that it has been key-framed. It can be used for performances. (Figure 5)

Screenshot and Video recording.
The system provides recording options for archiving and further analysis of the movement of each performer in relation to the different options of visualizations. The user (performer) can record her/his actions by recording the session as an image sequence (which can be edited to a video with a video editing software) in order to see how its avatar moves, how it interacts with the visual world and how the particles visualize his/her movement, after finishing the dance learning task, or improvisation. Also, a recording can be made in the Animate software, where it will be exported as a .bvh file which contains mocap data and will show the movement of the skeleton, which can be later edited in an 3D animation software.

DISCUSSION-INNOVATING DANCE TEACHING
One common way to approach the use of interactive technologies for dance learning is by teaching a choreography through an avatar, ask the learner to follow the movements and provide feedback on the "correctness" of the movements. This approach follows the paradigm of learning by "seeing and doing", a teaching approach which is also known as mimesis. Though, this teaching approach have been criticized by many as restrictive, Harbonnier-Topin [9] highlights that there is much more in it than just a mimesis of motions. As she explains, the learners do not only imitate the steps, but they also perceive and embrace other elements of the teacher's personality, such as the movement and voice qualities. Moreover, the teacher usually combines what she/he does with verbal instructions. Experiments [5] also show that observation of movement can enhance movement mainly to students who have previous experience on this types of movement, but not on beginners and less experienced dancers. In our experiment, we propose a more reflective approach of teaching, where the interactive experience works as a reflection to own movement, as the performer sees her/his own movement on the screen through augmented with visual effects and different avatars, a fact which inspire different qualities and ways of moving. The different avatars in this case, can be seen as a costume or a mask that allow the user to see his/her movement but at the same time allow him/her to distant from own self emerging into a character. Therefore, the installation allows to investigate and explore the whole-body interaction and visualization as gamified, creative experience for movement improvisation.

CONCLUSIONS AND FUTURE WORK
In this paper we have described the installation of a system which allows us to investigate the use of different 3D avatars in effects for interactive gamified experiences for dance learning. One of the future works includes a full user evaluation of the system with performers and dancers. For this version of the installation we have conducted a pilot evaluation with three users with the following profile 1) a professional dance theater performer, 2) an amateur dancer, and 3) a higher education dance graduate. All of the the users have characterized the installation as "engaging" and "excited" and characterized the installation "as more suitable as a gamified experience for creativity". These first outcomes were very positive, they stress however for the need for further evaluation and investigation to assess the educational purpose of the installation, along with its entertaining and "highly stimulating for creativity". From the pilot evaluation and interviews it was very clear that the different avatar affects the way the users "felt like moving".
In future versions, we investigate the implementation of "transform your avatar" using your movement scenario. In this scenario the qualities of the movement of the user, i.e., how the user moves will change the avatar, e.g., slow and calm then the "gentle" avatar will appear, whereas if the user moves quick and violently then the "hostile" avatar will occur.
The above example could be also implemented using the EEG biosensor e.g., Neurosky Mindwave, in order to morph the avatars in real time. In this experiment, we plan to implement an BCI (Brain Computer Interaction) interface, where the user will be able to change the from an avatar to another by measuring his/her brain activity. In other words, if the User relaxes, her/his brain activity is in low level and its avatar morphs to a more "gentle" or "friendly" avatar, if the User is in alert, his/her brain activity is in high level and its avatar morphs to a more "intimidating" or "hostile" avatar. This feature can be applied in sessions where the different states of mind play an important role. For example, ballet students and professionals turn to alternative training methods [11] such as yoga and Chi Kung to practice physically and mentally. Thus, in order to practice their mind to change to a state of relaxation they can use the above feature as diagnostic tool to "measure" their concentration and finally achieve to have full control over their state of mind.
Finally, our future work includes the analysis of the recordings of the experiment we have described using both the .bvh files and the video recordings of the users.

ACKNOWLEDGMENTS
This paper includes part of the research which is conducted in the framework of the WhoLoDancE 2 H2020 EU funded project. The implementation has been conducted with the equipment and support of members of HUBIC lab 3 at Athena Research Center. The free pre-defined avatars come from Unity's asset store. We also thank Maya Lara for the design of the custom 3D prototypes.