WhoLoDancE: Deliverable 2.6 - Motion capture sequences and skeleton avatar

The conceptualization development of generic inverse kinematic (1) (IK) avatar skeletons for the WhoLoDance project was derived from the guidelines and requirements that emerged from discussions with the consortium dance partners from the project’s start and through several dance partners and technical partners meetings in the last 14 month.  

The creation of a unified performer IK Skeleton Fitting (retargeting) and visualization was based on several principles manifested in those guidelines and pertained to the type of functionality an IK avatar skeleton should have in the project’s context.  

For the avatar skeletons creation stage to be executed properly, there was first the need to analyse the motion capture sequences from the perspectives of: 

• Global scale deviations between performers 

• Range of motions rotational scope per performer 

• Unified mean deviation across all performers 

Once this stage was complete, the data was used to determine the dimensions and internal scale of an empiric unified avatar skeleton that would fit the scaling of all the performers that were captured for the project.    

The stages of the skeleton creation pipeline involved 3D modelling of skeletal hierarchies, setting up correct biomechanical human limits rotations constraints for the skeleton and coding of transformation parameters so that motion capture data could be propagated correctly into the avatar skeletons.  Furthermore, work was carried out in enabling the use of 3D geometry on top of the avatar skeletons, whether in a form of parent-child relation (2), or as a flexible envelope (3) 

This deliverable describes the processes of creation and implementation of inverse kinematic skeletons for the avatars modelled in T2.4, next to the creation of materials, textures, lighting setup and shaders for the models. This deliverable also includes preliminary results of real-time testing of those assets. 

In the upcoming WP6, the data will also undergo a stage of 3D modelling optimizations of avatar for volumetric / holographic projection and optimization for alternative projection methods and systems. The last part of this task involves the skeleton fitting (retargeting to fit the anatomy and morphology of the 3D avatars with the human performers that will be captured in the future), optimization of the inverse kinematic skeleton, and, finally, optimization of materials, textures, lighting setup and shaders for real-time interactive display. 

(1): Inverse kinematics is the Mathematical process of recovering the movements of an object in the world from some other data, such as a film of those movements, or a film of the world as seen by a camera which is itself making those movements. This is useful in robotics and in film animation. In robotics, inverse kinematics makes use of the kinematics equations to determine the joint parameters that provide the desired position for each of the robot's end-effectors. Specification of the movement of a robot so that its end-effectors achieve the desired tasks is known as motion planning. Inverse kinematics transforms the motion plan into joint actuator trajectories for the robot. Similar formulae determine the positions of the skeleton of an animated character that is to move in a particular way in a film, or of a vehicle such as a car or boat containing the camera which is shooting a scene of a film. Once a vehicle's motions are known, they can be used to determine the constantly-changing viewpoint for computer-generated imagery of objects in the landscape such as buildings, so that these objects change in perspective while not themselves appearing to move as the vehicle-borne camera goes past them. The movement of a kinematic chain, whether it is a robot or an animated character is modelled by the kinematics equations of the chain. These equations define the configuration of the chain in terms of its joint parameters. Forward kinematics uses the joint parameters to compute the configuration of the chain, and inverse kinematics reverses this calculation to determine the joint parameters that achieve the desired configuration.

 (2): Parent-child relational modelling In the world of 3D, users are able to organize their scenes by creating a hierarchy. The hierarchy is created through the process of parenting objects to one another from inside the program. When an object becomes a Child of another object (Parent), it will follow all transformations applied to the Parent. This is useful in the case where a character or 3D object has multiple parts and needs to move around in the scene. That way you only need to animate the Parent model and the Child objects will follow automatically. 

 (3): Flexible envelope Avatars can be enveloped (skinned) to a skeletal rig. The skeletal rig is then animated with keyframes, or in the case of Wholodance, driven by motion capture data. This animation in turn, deforms the envelope.