Biophysics-based simulation of virtual human model interactions in 3D virtual scenes

Virtual simulations of operating tasks provide knowledge on a variety of parameters, useful for preventive ergonomic analysis, that helps to improve safety, quality and promote human-centered design. This paper presents the development of a biophysics-based simulation tool that is used for the evaluation of virtual interactions in different synthetic 3D scenes. Evaluation is based on the simulation of the motion of a digital human skeletal model interacting with virtual 3D objects, followed by inverse dynamics simulation of multi-body systems. This tool can be used for the estimation of body joints loads and energy expenditure during tasks’ operation in different environments, thereby allowing to detect potential risks in repetitive movement patterns and to adjust accordingly the object’s arrangement. Comparative results of spatiotemporal energy distribution support the validity of the simulation framework.


INTRODUCTION
The assessment of workload, cognitive strain and risk of developing task-related musculoskeletal diseases in workplace environments, demands for ergonomic evaluation, in order to implement enough preventive and corrective measures [1]. Results of pilot studies in industrial operations often indicate the need to reorganize the working environment layout or introduce equipment aid to the implementation of tasks [2]. As tasks and physical processes become more generalized, ergonomics analysis in the real world becomes laborious and infeasible due to the large parameter space. The use of 3D virtual environments composed of synthetic scenes [3] [4], CAD models [5], or fully scanned scenes [6] [7], instead of realworld scenes, can be a solution to this problem [8]. By developing 3D scene models or subparts and combining pose tracking technology (Kinect [9], Leap Motion [10]) with virtual reality (VR), engineers can assess the dynamic interaction of a human user with the 3D scene, providing information on fine-grained articulated human poses at each frame and trajectory [11]. The 3D scene models are based on data from field studies on movement behavior in pre-defined spaces, such as in manufacturing lines or the interior of vehicles, and on the operation of elements inside them (manipulation of a machine, adjusting the car seat, etc).
Already more than a decade ago it was suggested [12] to include in digital human models (DHMs) valid posture and motion prediction models based on real motion data in order to assure validity for complex dynamic task simulations and also to enrich them with biomechanical models to provide better predictions. Nevertheless, such computational approaches are still sparse. A few DHMs have introduced physics-based models that account for human limitations, external loads, and environmental factors [13]; yet, there is limited evidence on the fidelity of their output, especially within the * e-mail:moustakas@upatras.gr † http://www.vvr.ece.upatras.gr healthcare sector where small deviations might be critical. Experiments [14] have shown that the kinematics and kinetics computed using the posture prediction and motion capture driven approaches to modeling a patient repositioning task, using both Siemens Jack (V 8.4) and Santos Pro were not in direct agreement with lab-based outputs. This may have been a result of differences in kinematic modeling assumptions related to the structure of skeletal linkage models, joint decompositions, degrees of freedom (DoF) in each model and anthropometrics used in DHM software.
To bypass the necessity for a human-in-the-loop optimization process, in this paper we present the development of a system that evaluates the design of a workspace based on a virtual reality physicsbased simulation. A digital human model interacts with the virtual environment and motion data are extracted. The same model is imported in a biomedical analysis platform that is equipped with all tools necessary to solve the physics equations that govern the interactions of the model with the environment, using the previously extracted data as input. Inverse kinematics is employed to assist posture estimation, followed by inverse dynamics for the calculation of torques and forces. Alternative motion scenarios are tested, and the loads exerted in the model joints are calculated. The biomechanical analysis results can be used for human-centered product and workspace design. Upon ergonomic analysis and parameter optimization, the obtained virtual design can be validated by means of immersive reality technologies applied in an identical way to the physical world.

RELATED WORK
Many recent studies found in the literature, focus on performing ergonomic evaluation based on the interactions between a DHM and products or workspaces that are represented using virtual 3D environments. These studies that combine DHM and virtual ergonomics are performed across a broad range of science domains such as autonomous driving, aerospace, health-care and industrial and office equipment design manufacturing. The motivation behind this line of research is to optimize working environments in a way that promotes human comfort, safety and productivity and simultaneously minimizes impairments, disorders and physical and cognitive strain. Many of these works aim to proactively assess ergonomics during the early design phase, targeting to efficiently evaluate the features and characteristics of the product or workspace even before the development of the physical prototype.
A study by Aromaa et al. [15] explored the suitability of virtual prototypes developed with augmented reality (AR) and a virtual environment to support human ergonomics evaluation during simple maintenance tasks for a rock crushing machine. The authors stated that virtual prototypes could assist designers to make good decisions.
A comparison between VR and DHM was the objective of a study performed by Ahmed et al. [16]. While DHM can provide access to biomechanical and ergonomics evaluation tools in the early stages of the product design, the limitations regarding assessment of the cognitive user performance could be circumvented by integrating the user in the analysis loop using virtual reality. The performances of a computational prototype and a mixed prototype (CAD design of the workplace combined with VR projection and actual user assessment) were compared with respect to the ability to appropriately identify the pilot's posture during a fire break out inside an airplane's cockpit. Also, the cockpit environment design was evaluated. The mixed prototype demonstrated superior performance in capturing the changes in the user's posture. On the other hand, the computational prototype strategy is more suitable for exploring design ideas, since numerous design compounds can be developed.
In a study published by Jun et al. [17] an automated approach for modeling worker's movements was followed. Movement of the user was captured by multiple Kinect devices and the motion data were first calibrated and validated and then converted to initiate a human engineering-simulation using Jack. The authors report that the proposed modeling method minimizes time, cost and increases the fidelity of the DHM. A study by Maurya et al [18] explored the applications of DHM for ergonomic assessment and improvement of working environment for the specially-abled and elderly. Modeling of the human and the environment can assist in office design and furniture layout, in the evaluation of the accessibility of public places and also, to the design of prosthetic implants and exoskeletons.
The potential of DHM could be maximized by the development of anthropometric and biomechanical databases in order to achieve an effective and validated assessment of the proposed products and working environments. In a paper published by Mao et al. [19], scanned human geometries were used to ergonomically assess chair design, by estimating and evaluating the pose of the human models. The pose is defined by the interaction geometry between the 3D chair objects and the human models, which can have a wide range of anthropometric characteristics providing a realistic and user-tailored evaluation approach. The objectives of that work were to provide a VR-based tool that predicts the fit of human body models on selected chair designs (pose prediction) and to provide an ergonomic rating. Then, after pose extraction, the physical stresses acting on and generated within the body are calculated, which is helpful for modifying and controlling the mechanical environment, making decisions about treatment and rehabilitation, or evaluating their effects.

METHODOLOGY
The workflow followed throughout this paper is illustrated in Fig. 1. A skeletal representation is selected as a model of the human body allowing to control motion at a given number of joints. This rigid multi-body representation is used for biomechanical analysis during interaction with the virtual environment, as described later in section 3.2.
It represents the skeleton of the upper human body and consists of 9 bodies, 9 joints and 54 degrees of freedom (DoFs), i.e. 3 rotational and 3 translational for each joint. The bodies represent the geometry and properties of the different segments of the system. The joints are the points of articulation of the different segments and they enable their relative motion.

Virtual environment design and interaction
In the present work two 3D environments were designed representing an office (Fig. 2b) workspace and a vehicle cockpit (Fig. 2a), in which the OpenSim model described above is imported. The design of the virtual workspaces is achieved, using the Unity Real-Time development platform. The office environment is parametrized with respect to the modifiable variables of constituting objects, such as the location in which the telephone and the printer has been placed on the desk or the positioning of the chair. The car environment is not parametrized and only the movements of the model inside the environment can be assessed.  Experimental markers are attached to the model bodies in order to obtain translation and rotation coordinates of the joints, during specific virtual interactions of the DHM with the VE. The hand motion kinematics are calculated using a FABRIK Inverse Kinematic algorithm implementation [20] and the motion is tracked through the experimental markers positions serving as input to the biomechanical analysis. In the scope of this work interactions of the DHM with the printer and telephone virtual 3D objects within the two VEs are evaluated.

Biomechanical analysis
The biomechanical analysis is performed using the open source software platform OpenSim [21], that enables modeling and simulation of the musculoskeletal system of humans, animals and robots and provides tools for the analysis of the interactions of this system with the environment, regarding movement [22]. It is commonly used for  In order to perform an analysis that can be validated with data collected during real world experiments, the design of a virtual 3D model that represents the subject's musculoskeletal system with high fidelity and accuracy, is of critical importance. The main functionality of OpenSim is the solution of the Inverse Dynamics (ID) problem, which produces the biomechanical forces and joint moments acting on a musculoskeletal system during movement using experimental kinematic and kinetic data. For this purpose, the skeletal model is imported into OpenSim and Inverse Kinematics (IK) is applied on a marker set that corresponds to the experimental Unity markers (illustrated in Fig. 4).

Inverse Kinematics Tool
Kinematics refers to the description of motion in terms of velocity, acceleration, displacement without taking force into account. The IK Tool computes the generalized coordinate (joint angles and/or translations) trajectories by positioning the model in the appropriate posture at each time step. This posture is decided based on the experimental data by minimizing a weighted least square distance described by Eq. 1. In the scope of this work, marker data that are exported by the Unity platform IK tool, are considered as the experimental data and the marker error that the IK solver tries to minimize, is the distance between each experimental marker and the corresponding marker in the OpenSim model.
g j = q exp j , for all prescribed coordinates j. (1) where: i is the marker index,q are the generalized coordinates being solved expressed as a vector, x exp i is the 3D position of the i-th experimental marker, x i (q) is the position of the corresponding marker of the OpenSim musculoskeletal model, q exp j is the experimental value of coordinate j, w i and ω i are the marker and coordinates weights respectively.

Inverse Dynamics Tool
Dynamics refer to the classical mechanics science that deals with the analysis of forces that are related to motion. The OpenSim's Inverse Dynamics Tool receives as input the known or predicted by the IK Tool motion data and calculates the unknown generalized forces by solving the classical equations of motion: where: q,q andq are the generalized positions, velocities and accelerations respectively, M(q) ∈ R N×N is the system mass matrix, C(q,q) ∈ R N is the vector of Coriolis and centrifugal forces, G(d) ∈ R N is the vector of gravitational forces,τ ∈ R N is the vector of generalized torques and N is the number of the DoF.
The left-hand side of Eq. 2 contains the known terms and on the right hand-side are the unknown generalized torques and forces. The motion data that were obtained after the implementation of IK are streamlined to the ID Tool. In order to obtain an estimation about the load that each joint sustains when the model performs each activity, we follow an energy estimation approach. Generally,  the relationship between energy, torque and angular displacement is described by: where: E is the energy expressed in Joules, τ is the torque in Newtons × meters, and θ is the angular displacement in radians.
Eq. 3 states that in SI units, a torque of 1N × m applied through a full rotation of a joint, will require 2 π joules of energy. Thus, for each joint the angular displacement at each time step is calculated from the IK output angles and is multiplied by the corresponding torque value. The energy values are scaled logarithmically and used to create a heat map that illustrates the energy distribution on each joint during the simulated task. The heat map color range has been normalized and the minimum value is mapped to green color and the maximum value is mapped to red, across all joints and during the whole motion trajectory.

RESULTS AND DISCUSSION
The skeleton movement and the corresponding instances of the heat map that illustrate the energy distribution across the upper body joints for the interactions within the office workplace and the car environment are presented in Fig. 3 and Fig. 5. In the case of the car VE a model interaction with the instrument panel is simulated and in the case of the office VE the interaction is extended by changing the parameters of the environment objects. These parameters are the position of the objects on the desk surface and the distance between the model and the desk. In Table 1 the peak and total energy values of three key joints are displayed, while the distance between the torso and the desk border is 40 cm long. By observing the lower back joint energy total values, it is clear that the load is larger during the interaction with the telephone. This is justified by the fact that the object is located further from the user's position than the printer object, thus an increased rotation of the lower back is necessary.
These results provide a useful insight about the proposed setup. For example, when the user interacts frequently with an entity, this object should be placed in a more ergonomic position, that provides comfort and applies less accumulated load to the user.
Next, the distance between the torso and the desk is reduced to 21 cm and the values that correspond to the same parameters as in Table 1 are calculated and displayed in Table 2. It is clear that by positioning the model closer to the desk, less load is exerted to the joints and especially to the lower back.  Fig. 6 the distribution of energy in left shoulder and lower back joints before and after reducing the distance parameter between the model and the desk are displayed. It is evident that the energy is reduced for both joints.

CONCLUSION
In this work a tool for assessing the design of different virtual environments and the interactions within is developed. Two virtual environments were designed with different context, demonstrating  a potential for the application of the proposed methods to different design concepts. The results of the biomechanical analysis could be used to provide an assessment of a proposed workspace design. Factors that were evaluated are the distance between the user and the workspace objects, not only in terms of euclidean distance, but also taking into consideration the kinematic features of the interaction between the user and the environment (joint angles, accelerations, and torques). Furthermore, the proposed workflow can be easily applied on all segments and joints of the human body (ankle, knee, spinal cord etc.), if a valid musculoskeletal digital replica is available. A limitation of the suggested approach is that the muscle activations during the biomechanical analysis are not taken into account and only joint moments are considered. Adding muscle forces could provide a more robust and realistic analysis, maximizing the performance of a biomedical based ergonomic evaluation.