A Web-based system for annotation of dance multimodal recordings by dance practitioners and experts

Recent advances in technologies for capturing, analyzing and visualizing movement can revolutionize the way we create, practice, learn dance, and transmit bodily knowledge. The need for creating meaningful, searchable and re-usable libraries of motion capture and video movement segments can only be fulfilled through the collaboration of both technologists and dance practitioners. Towards this direction, manual annotations of these segments by dance experts can play a four-fold role: a) enrich movement libraries with expert knowledge, b) create "ground-truth" datasets for comparing the results of automated algorithms, c) fertilize a dialogue across dance genres and disciplines on movement analysis and conceptualization, and d) raise questions on the subjectivity and diversity of characterizing movement segments using verbal descriptions. The web-based application presented in this work, is an archival system with, browsing, searching, visualization, personalization and textual annotation functionalities. Its main objective is to provide access to a repository of multimodal dance recordings including motion capture data, video, and audio, with the aim to also support dance education. The tool has been designed and developed within an interdisciplinary project, following a user-centered, iterative design approach involving dance researchers and practitioners of four different dance genres.


INTRODUCTION
Motion capture can significantly transform the way digital dance archives are created, analyzed and preserved. In parallel, the production of new types of dance recordings, creates new requirements for dance data archiving in various formats, in order to make these data accessible, findable and re-usable within a variety of research and practical contexts. Nevertheless, the diversity of what are the types, and formats, of data which are considered as "dance data", combined with the open issue of movement representation [7] and the lack of standard conceptual models for managing movement archives, emerges as a complex interdisciplinary [15] and technical issue. As a result, the role of dance experts and movement practitioners in the process of creating, managing and enriching movement libraries is crucial. Experts' annotations can play a great role in both enhancing such libraries and drafting further specifications and research questions.
The process of manual annotation in particular can play a fourfold role towards this direction: a) To enrich the repository of recordings with metadata through tag annotations, b) to select "ground-truth" data, based on observation of experts to compare with automated annotation and feature extraction, c) To offer an analytical tool for the experts in the context of education, choreography and composition, movement research and analysis, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. MOCO, June 28-30, 2018 through movement analysis and conceptualization and d) highlight the subjectivity and diversity of characterizing movement using text-based tags, depending on dance practice background and purpose. The Web-based Movement Library (WML) application, which we describe in this work, has been designed and developed within the framework of an interdisciplinary EU funded project [28], aiming at investigating and creating tools for dance education. In Section 2 we present the background and context of designing, developing and using the WML tool, while in Section 3 we discuss the relevant research work and annotation tools. Section 4 presents the movement descriptors and vocabularies we have used for the textual annotations. In Section 5, we describe the functionality and architecture of the application, while in Section 6 we present the methodology and results of the iterative evaluation process with the dance experts. In Section 7, we discuss the potential of the WML and the process of annotation, as it has emerged through the co-design, use and evaluation process with the dance practitioners and experts.

CONTEXT
One of the first outcomes of our work has been the creation of a large library of movements through multimodal recordings, including motion capture, video and audio. The process of capturing and recording movement sequences from four different dance genres, contemporary dance, ballet, Greek folk and flamenco, aimed at selecting the main building blocks to become the learning material within different educational scenarios. One of the first challenges of the project was to organize these data so that they can be easily accessible, searchable and subject to enrichment through manual and automatic processes. The main goals of the WML is to provide an intuitive interface for accessing over 750 recordings and gather "ground truth" data on how the dance experts characterize different segments of the recordings regarding their movement aspects. More specifically, the user can browse the recordings by dance genre, and search by using keywords that are included as metadata or in the annotations of the recordings. The keywords refer to different aspects of movement (principles, qualities, actions, etc.). A viewer has been integrated so as to allow the synchronized playback of a video, as well as its corresponding motion capture file. The synchronization of the two formats [5], as well as the development of the 3D visualizer, falls out of the scope of this paper and is not reported here.
The main objective of WML ( Fig.1) is to provide the end-users, including dance educators, choreographers, movement and dance practitioners and experts, a tool to browse, search, observe and reflect on particular movement descriptors and characterize them using a variety of semantic descriptors referring to general movement principles, movement qualities and actions. The first step was to enrich the repository with "ground-truth" data regarding the movement aspects of the various segments. The process of co-designing, evaluating and using the tool with the dance experts, brought to the surface a number of research issues on text-based annotations in movement analysis and highlighted its potential within educational, creative and research domains and other contexts.

DANCE AND ANNOTATION TOOLS
Annotation tools have been proposed the last decades as tools for manually annotating or transcribing non-verbal communication, with Elan [29] and Anvil [20] being two of the most well-known and used tools in the area. Memo Rekall [22], is an open source video annotation tool as a part of a larger project on the documentation, preservation and analysis of performing arts. Similarly, eClap, was a project focused on creating a platform and managing metadata for performing arts [3], while i-treasures has focused on capturing, managing and automatically analyzing two dance genres (Greek folk and contemporary). Nixon et. al. [23] have developed an open source movement database, while Alemi et. al. [2], have presented Mova, an interactive analytics platform.
Annotation for enriching performing arts repositories and supporting findability of content. Some efforts are using both manual and automated annotation, focused on particular dance genres in the context of enhancing search in dance archives. El Raheb et. al. [16] have presented an archival system with a manual annotation interface and search functionalities, framed by an ontology describing the ballet movement vocabulary. Ramadoss et. al. [24] have presented a semi-automated system for retrieving dance performances of pop-Indian dance. Chauldry et. al. [10] have applied classification algorithms on motion captured data aiming at automatically labeling specific segments of Malaisian folk dance sequences, while Ma-Thi et. al. has applied a similar approach for folk Vietnamese dance [21].
Annotation as a tool for collaboration and part of the creative process The project Transmedia Knowledge Base for contemporary dance has investigated the process of annotation as part of the creative process using multimodal annotations on the Creation Tool [6], including 2D "digital ink" annotations, while Ribeiro et. al. [25] have explored the use of 3D annotations. The Choreographic Notebook, by Singh et. al. [26], is another example of a multimodal annotation tool, which was designed to support the collaborative, creation process of contemporary dance choreography. DancePro, is designed to support the creative and compositional processes for accessing dance content and creating extensive metadata [27].
Annotation as a way to produce a new dance score Manual annotations in dance can be seen as both a way of generating a score [4], and "assisting choreographic reflection" [1]. In the description of the Piecemaker2 software, which was created A Web-based system for annotation of dance multimodal recordings by dance practitioners and experts MOCO2018, June 2018, Genoa, Italy 3 within the Motion Bank Project, the process of annotation has been described as a process of "scoring video recordings and sharing and this information with others" [12].
Annotation as part of the educational process One aspect of the annotation process, which has not yet been extensively explored, is the use of such tools within an educational context. Although, this is not an issue we cover extensively in this work, the process of co-design, using and evaluating the WML tool has bought forth the potential of using the process of annotating, searching and observing other annotations within a learning process. As Jennet states: "Labels can help remember complex things by giving them a simple name. In addition, things with names can be more easily discussed, shared and related to other things with names." [19]. In this sense, verbal annotations can become a starting point for not only learning and memorizing specific sequences but also for deepening the knowledge around specific movement aspects, studying differences across a variety of performances and rethinking the embodiment of these concepts in performance. In the case of WML where these verbal descriptions are connecting different dance genres, there is also the added value of an across dance genres fertilization.

THE MOVEMENT LIBRARY AND ANNOTATOR
The WML is a web-based application, compatible with any browser that supports WebGL, designed and developed as a means for searching, browsing, editing, viewing and annotating dance recordings, which may include a motion capture segment and/or video. The user is able to explore the rich library of recordings through an intuitive interface that requires minimum training even by the less experienced users.
The WML allows exploration of the recordings, grouped into four dance genres, as well as searching that takes into account the rich metadata associated with each recording.
More specifically, the recording metadata description includes the dance genre, the motion capture venue and time, the performers, the dance company, as well as information on the related movement principle(s). Recordings of Greek dance motion capture segments have additional information, such as the name of the dance in the local dialect, the region where it has been developed, the gender of the dancer and the type of segment. Furthermore, WML serves both as a viewer and an annotation tool and a custom player has been developed, in order to support the simultaneous playback of the video and the 3D skeleton (or "stick figure") of the motion capture that are related to the selected recording. During the process of viewing the recordings, users are able to track the recording annotations, through a Timeline structure as well as through a Table structure, both located beneath the player and enhanced with annotation functionalities.

WML Functionality
The WML offers a variety of functionality to the user, which presented briefly in this section.

Search by keywords or browse by dance genre
The WML home page combines a search bar with the possibility to browse recordings organized by dance genre. The search functionality uses the recording metadata and annotations to locate recordings (Fig.2). After searching or browsing users are redirected to the "results" page where they can sort or further filter the results and view detailed characteristics for each. Search includes the option of using movement descriptors or free text as keywords, offering the option of searching the repository using terms related to movement.  Table view of annotations View recordings A specialized, custom player is used in order to offer the possibility of viewing the recordings. The player's structure allows the synchronized playback of a video alongside with the corresponding motion capture file (Fig.3).

Figure 3: WML Table view of annotations
Manage and view annotations Annotations on the recordings describe and analyze the dancer's motion. As Jennet indicates "Annotating should be an iterative process: add, more annotations than needed, remove and refine later." [19]. The need for managing annotations was covered by developing a specialized table structure. The "Annotation Table" allows users to quickly add or alter annotations. Furthermore, WML offers a timeline structure which is synchronised with the media player (Fig. 4). During the playback of the recording, a vertical, red line moves through the timeline segments, in order to present the annotations that correspond to each time point. The tool allows the editing of the annotations directly on the Timeline. Since manual annotation can be a time-consuming and complex process, it is important that the users can re-visit the annotations at a later time to edit and refine them. Create a personal channel and playlists WML has integrated personalization functionality for the users to select recordings and create playlists. More specifically, users can create their own personal playlists with specific name and settings, as well as save playlists with public rights. Through their channels, users can view and edit their personal info and also locate created and saved playlists. By selecting one of those playlists, users can either begin the recordings playback or view the playlist's details.

Architecture
The WML storage layer is based on a server where the motion capture recording files are stored along with a database used for the user generated recording annotations. The backend is implemented through a CKAN data and metadata management system that provides tools for sharing and finding the data that are stored in the archival server as well as a search engine that is used in the WML. The WML also features a User Management system to support, among others, the personalization features of the tool, including user annotations and playlists. The WML tool is a web-based application, compatible with every browser, based on the principles of a Model-View-Controller (MVC) software design pattern, implemented with the Spring Web MVC framework. The view component has been developed by utilizing the JSP script-based templating system, Bootstrap and AJAX to send and retrieve data from the server asynchronously (in the background). Fig.5 provides an overview of the WML architecture.

MOVEMENT DESCRIPTORS AND VOCABULARIES
In WML, the type of annotations allowed is text-based, providing a controlled vocabulary which is based on the conceptual framework adopted by the project. The conceptual framework provides a number of specific movement descriptors which are independent of the dance genres and can be used as labels or tags, to characterize a particular segment. These descriptors are categorized in three main types: movement qualities descriptors, movement principles descriptors and actions.
Movement principles descriptors are related to Movement Principles, which are high-level concepts that all dance practitioners deal with regardless of the genre and practice, such as: Symmetry, Directionality, Rhythmicality, Coordination [8].
For more details, see Table 1.
Movement quality descriptors are related to movement qualities such as fluid, rigid, light et., derived from a subset proposed by Camurri et. al. [9]. Movement qualities have been adapted to the needs of the project through the co-design focus groups and interviews sessions with internal and external dance experts.
Finally, the Actions descriptors consist of a list of basic, generic actions such as jump, turn, step, arm gesture etc., which are also aligned with the basic actions in Laban Movement Analysis [18] and used in other works [14] [16]. It is true that finding and applying one and only conceptual model or ontology for describing human movement is a challenging issue. While, Laban Movement Analysis and notation systems such as Labanotation [14] and Benesh [11] provide a theoretical basis for doing so, they are not part of everyday language of all dance practitioners, independent of the dance background and genre.
At this point, we should stress that the vocabulary which is provided is more indexical than exhaustive and serves to limit down the options into specific descriptors to study as use-cases juxtaposing the manual annotations with the automated algorithms within the project. As Jennet indicates, "Finding the appropriate labels or names for each annotation is important. An 'inside out' process where the material teaches the names and 'outside in' when one arrives with a set of names are both valid approaches and have their own strengths." [19].
In this sense, within the tool we provide the option of choosing one of the proposed descriptors and also adding free text to reflect additional annotations the user might find relevant. In addition, during the annotation process the user has the option of indicating if the annotation is based on observing the 3D stickfigure of the motion captured segment, the video, or both, and also provide a value for the descriptor on a range of 1-10. Finally, the user can indicate the performer (there are maximum two performers in the recordings currently in the library) and body part to which the annotation refers.
A Web-based system for annotation of dance multimodal recordings by dance practitioners and experts MOCO2018, June 2018, Genoa, Italy 5 The conceptual framework used, which is not reported in this work, highlights yet another benefit of creating tools for manual annotations: Distant collaboration and observation of movement recordings. Even when the framework used is an established system like Laban Movement Analysis and the observation is held by trained experts in the same system, coming to "consensus doesn't mean that every person will see the same thing" but rather opening a discussion towards a "more reliable and valid observation" [17].

EVALUATION
This section focuses on the formative and summative evaluation activities organized for the beta version of the tool, which implemented the functionality described in Section 4.1. These included on-going co-design and formative evaluation activities with dance practitioners as well as an organized usability evaluation event which involved usability and user experience (UX) design experts and a user workshop where external dance practitioners were invited to interact with the tool.

A user-centered iterative design approach
The WML tool development has followed, since the early requirements and design phase, a user-centered and iterative design approach. The development team has been in constant interaction with the internal user group of the project, which consists of 12 dance professional practitioners with expertise in different genres and in some cases in the more theoretical, research and also cultural heritage aspects of dance. At the early stages, the dance experts participated in regular co-design and formative evaluation sessions, individualy or in small groups, where they were able to provide their insight and ideas on the 1 www.ueq-online.org/ direction the development should take. When the first working prototypes were available, the user group was asked to interact with the tool and offer feedback to the development team in the form of brief reports and through interviews and focus group sessions. Th is process has informed all aspects of the tool, including the offered functionality, the design and also the underpinning conceptual framework.

Usability and user experience evaluation
For the usability and UX evaluation of the WML tool, a task-based user evaluation in a laboratory setting was organized. Seven usability and UX experts were invited to participate in the evaluation. After a brief introduction to the project and signing of consent forms, without initially making clear the objective of the platform, the users were asked to freely navigate in the platform and to express their opinion as to what the objective of the platform is. Our goal was to verify whether the interface makes clear the objective of the tool. The evaluators to this end asked the participants questions like: "What do you think this tool is used for?", "What would you expect to be able to do with it?" After the discussion that followed, the users were introduced with more details about the platform and presented with its main functionality. They were then given a series of tasks to perform following a think aloud protocol, while they were observed by the evaluators and their reactions video-recorded. The users were given a series of multiple step tasks in order to be able to interact and experiment with all the aspects of the main functionality of the platform. These included: 1. Search and browse by dance genre and specific metadata. 2. Search for specific recordings and annotations. 3. Work with the timeline.

Add new annotations.
After the evaluation, the users were briefly interviewed and then asked to fill in a questionnaire, which is based on the User Experience Questionnaire (UEQ) 1 . The UEQ evaluates user feedback in 5 dimensions, which result from 26 questions Likerttype questions in a scale from -3 to 3: • Stimulation: Is it exciting and motivating to use the tool?

Results
The results of the evaluation, including user observation and notes during the testing, interviews and questionnaires have been analyzed leading to qualitative and quantitative results. The dimensions of the UEQ can be grouped into pragmatic quality (Perspicuity, Efficiency, Dependability) and hedonic quality (Stimulation, Originality). Results confirm the success of the tool in all dimensions except perspicuity. This is to be expected as the tool is complex, and it can be initially daunting to the inexperienced user, requiring some time to get familiar with it. However, even with this initial impression, in general, users were very positive about the usability of the tool. Taking into account the complexity of the functionality offered by the platform, users were able with instructions to navigate and complete simple tasks.  They felt that the design is aesthetically attractive, and they were all able to understand the objective of the tool. The users proposed several improvements, both in terms of enhancing usability and extending the offered functionality. The analysis of the evaluation results has informed the re-design of specific features of the tool towards a new, improved version.

WML workshop with dance practitioners
The WML has been evaluated in a targeted Workshop organized in September 2017. The WML Workshop lasted two hours and was addressed to dance experts and practitioners. Nine participants attended with the following profiles: 2 choreographers and dance teachers, specialized in contemporary dance and improvisation, 3 dance researchers focusing on digital technologies related to dance, 1 dance anthropology researcher with extended knowledge in Labanotation [18], 1 director of dance theatre and, 2 professional level dancers (Greek folk dance).
After a presentation of the WML overall approach and the objectives, the participants were asked to freely experiment with the tool individually or in pairs for 45 minutes, with the support of the organizers when requested. During this session, comments and questions of the participants were recorded and interesting discussions were issued between them. After this session, the participants were asked to fill in a questionnaire and another 45 minutes were dedicated to focus group discussion about the platform. Table 3 summarizes the participant outlook for WML. The participants felt it was usable, clear and useful and proposed extensions and improvements. For example, they proposed extending the filters in the search results page with additional metadata of the recording, including body parts, joints, and additional movement qualities, etc.

ANNOTATIONS IN DANCE PRACTICE AND RESEARCH
Dance practitioners both during the ongoing co-design sessions and the evaluation activities proposed different uses of the WML tool based on their expertise and needs in relation to the domain of dance and offered different perspectives as to how the WML functionality could accommodate these needs.

A tool for dance movement analysis and expert collaboration
The WML offers a complete framework for dance movement viewing, studying and annotation. Dance experts with an interest in movement analysis recognized the potential of the tool in the context of their work. Firstly, WML gives the option of advanced search based on specific movement characteristics within a rich library of content, and, additionally, allows the user to focus on the details of selected movement segments by viewing at the same time the movement itself through different perspectives through the interaction with the avatar and also to observe existing annotation in particular parts of the movement segment. The expert is also able to use this tool to communicate her approach for the analysis of a specific segment, by adding her own annotations.
In this way, the practitioners felt that WML has strong potential as a tool for academic research in dance movement and for communicating the expert's perspective to other colleagues in the field. A suggestion by the experts was to use the full vocabulary developed within the project conceptual framework to support the search and filtering of annotations.

A tool for advanced dance movement visualization
WML offers at the moment the possibility to view and interact with the motion capture stick figure avatar and at the same time view the video of the performers. The practitioners found the combination of these two perspectives very interesting as a means to study movement and proposed the use of additional avatars for the presentation of the motion capture recording. They felt that there is strong potential in understanding how these different avatars could be employed to highlight and reflect specific movement qualities and they would like to see this direction investigated in order to be incorporated in a next version of the tool.

A tool for dance education and choreography
Those practitioners who are involved with dance education felt that the tool has educational potential and the possibility to be used to prepare and share choreographies with colleagues. The WML offers a rich library of movement segments that can be used to: • Prepare a list of movements to be used during dance class or a performance • Share these movements with students and colleagues to comment on or rehearse them The aforementioned user needs led to the inclusion to the WML of the playlist feature, which was not initially foreseen for the tool.
The playlist can be used for the users to save and share their own playlists.
Some of the practitioners felt that it would be useful for them to upload their own motion capture segments and videos.Some of the dance educators also noted that the tool could be used by the students themselves to practice on more theoretical assignments, like studying specific movements in depth as to their qualities and using the annotation tools to share their perspective with their teacher. Lastly, as the tool offers material from different genres under a unified framework for search and annotation, it can promote cross-fertilization between different genres.

A tool for preserving and disseminating intangible cultural heritage
Practitioners from traditional dance genres like Greek folk dance highlighted the strong potential of the tool in terms of preserving, researching and disseminating intangible cultural heritage. The tool allows access to a library of different folk dances that allows the presentation of the dance to different audiences in high quality along with other genres. The more experienced practitioners are also able to study the movements and compare with different variations of the same dance, keeping annotations for their own benefit or for colleagues to review.

The annotation process within a multidisciplinary research context
The Annotation tool has been used for the duration of one year, as a tool to select "ground truth" data to be compared with the results of automatic feature extraction annotation algorithms. The idea is to select data concerning the textual characterizations of the segments in relation to higher level descriptors such as movement qualities. Interestingly, the difficulty for experts to reach a common agreement, even amongst experts of the same dance genre, as well as the difficulty of providing one and only mathematical definition for these movement aspects, lead to the investigation of other methodologies for automatically analyzing the data, apart from until now used rule-based techniques. The results of these methodologies are out of the scope of this work, however, we would like to highlight the importance of providing tools for gathering the dance experts' views, in the form of data, and analyzing whatever these data imply, before choosing the best techniques for movement computing methodologies.