A Framework for Real-time and Personalisable Human Ergonomics Monitoring

The objective of this paper is to present a personalisable human ergonomics framework that integrates a method for real-time identification of a human model and an ergonomics monitoring function. The human model is based on a floating base structure and on a Statically Equivalent Serial Chain (SESC) model used for the estimation of the whole-body centre of Mass (CoM). A recursive linear regression algorithm (i.e., Kalman filter) is developed to achieve the online identification of the SESC parameters. A visual feedback provides a minimum set of suggested human poses to speed up the identification process, while enhancing the model accuracy based on a convergence value. The online ergonomics monitoring function computes and displays the overloading effects on body joints in heavy lifting tasks. The overloading joint torques are calculated based on the displacement of the Center of Pressure (CoP) between the measured one and the estimated one. Unlike our previous work, the entire process, from the model identification (personalisation) to ergonomics monitoring, is performed in real-time. We evaluated the efficacy of the proposed method through human experiments during model identification and load lifting tasks. Results demonstrate the high exploitation potential of the framework in industrial settings, due to its fast personalisation and ergonomics monitoring capacity.


I. INTRODUCTION
Today's manufacturing industries are faced with an increasingly competitive market, which brings productions lines closer to the customers [1]. To address the mass customisation needs of the new market, production processes must constantly adapt to new workflows and manual operations. As a result, industrial workers may be exposed to greater danger in terms of posture and load, increasing the already dramatic statistics on Musculoskeletal Disorders (MSDs) [2]. A continuous assessment of the workers' ergonomics using the traditional pen-and-paper based approaches is impractical and costly in such dynamically changing environments [3], [4]. On the other hand, in the past few years, the development of collaborative robots (cobots) has become an important trend both in academia and in the industry worldwide. This aroused the interest of the European Commission which published a scientific report emphasizing the potential and the challenges, from a regulatory point of view, of human-robot collaboration (HRC) in industrial scenarios. This technology finds its application in the so-called Industry 5.0, which will bring the spotlight back on humans, acknowledging their superiority over robots in the accomplishment of complex and skilldemanding industrial tasks [5]. Factories of the future will have to adapt their work organisation and processes so to guarantee stimulating and healthy working conditions. In this perspective, the robot is no more the substituting agent but is a "smart tool" flanking the workers in their duties, empowering their abilities and enhancing their working conditions [6].
In light of the above, it is of primary importance to improve the online monitoring of human body postures and loads to enhance both self-awareness employing real-time feedback [7] and to trigger the reaction of a responsive environment (e.g. cobots) [3]. As of today, the only method available for measuring neuromuscular loads in actual work is based on surface electromyography (EMG) [8]. However, this approach presents several issues: first of all, a wholebody EMG interface is evidently bulky in a workplace; second, a human or robot pose optimisation based on these biosignals is cumbersome and costly from a computational point of view, undermining the real-time functionality.
An attempt towards online ergonomic monitoring has been made trying to automate the filling of the traditional pen-andpaper worksheets [10], [11], [12]. However, most of these methods lack of generality: they address a specific body district engaged in a limited number of activities. The job may be the result of a much wider succession of atomic stages. This arises multiple problems. First of all the choice of a method to evaluate an elementary task is not always straightforward. Secondly, the non-homogeneity of the scoring scales makes the comparison among methods impossible. Therefore, if different methods are used to evaluate different phases of the activity, no global score (i.e. for the whole activity) can be computed [4].
To respond to these challenges, in this paper, we propose Fig. 2: Flow diagram that summarizes the overall procedure for estimating the ergonomics indices to reduce the workload employing human-robot collaboration. The whole method consists of two parts, as graphically suggested by the colours of the scheme. The left part displays the identification procedure of the subject-specific BSIP parameters through an online SESC parameters estimation. A recursive approach enables accessing the filter's variables at each iteration suggesting the best pose using visual feedback to enhance convergence. The recursive linear regression algorithm needs to be fed with the configuration coming from a motion capture system and the centre of pressure position collected with a force plate or any other pressure sensor. The second part deals with ergonomic monitoring during task execution. The algorithm calculates the difference between the SESC driven CoP estimation calculated starting from the configuration data and it compares it with the actual CoP measured using pressure sensors. An intuitive visual feedback illustrates to the subject the ergonomic status [9]. A robotic agent may be triggered by the ergonomic evaluation anticipating dangerous behaviours.
a comprehensive framework to fill in the gap between simplistic ergonomics monitoring models of the industry, which are insufficient and non-scalable, and the complex laboratory models of humans, which are hardly personalisable and impractical for industrial settings. We propose a framework which enables real-time identification (personalisation) of a dynamic human model, which is then used for the estimation of the loading effect of external forces on body joints during movement. Unlike our previous work [13], the entire process of human detection, model identification, and overloading torque estimation is executed in real-time, with no need for any offline processing or calibration stages. The method is based on the difference between the on-line estimated CoP of the human body with no external load or contacts other than those at the feet using SESC technique [14], and the measured one using an external pressure sensor system. We believe that the real-time functionality and fast personalisation features of the proposed framework can substantially increase its potential in realistic industrial settings, and contribute to a coherent reduction of MSDs. The rest of the paper is organized as follows. In Sec. II, Sec. III and Sec. IV the proposed methodology is discussed. Sec. V shows an experimental validation of the method. In Sec. VI, we discuss the results and highlight future improvements.

II. METHOD OVERVIEW
This study aims to demonstrate the strength of a completely online and subject-specific ergonomics assessment tool that fits in a wider ergonomic human-robot collaboration framework [15], [16]. The overall procedure can be divided into two main parts: a calibration phase (left side of Fig. 2) and an ergonomics assessment phase during a robotic collaboration task (right side of Fig. 2). The purpose of the first phase is to identify the unknown body segment inertial parameters (BSIPs) of the human body through an online interface. Combining the equation of motion with the sensors' data is possible to obtain the subject-specific BSIP parameters from the statically equivalent serial chain (SESC) technique [14]. Subsequently, the real-time ergonomics assessment comes in through a reduced-complexity model for the estimation of human joint load variations due to the external interaction forces [15]. By means of a screen, intuitive warning messages are provided to the subject using colour-coded spheres on the body joints, whose colours are proportional to the level of estimated danger. For the sake of completeness, in Fig. 2 a robotic agent whose action may be triggered by the ergonomics monitoring tool is shown. This chance has already been addressed in our previous work [3] and it will not be discussed in detail in this paper.
As shown in Fig. 3, the human model is represented by a sequence of rigid links connected by revolute joints. The pelvis frame is set as the base frame Σ 0 attached to the inertial frame Σ W through six virtual degrees of freedom (DoFs). Each link has a mass, m i , (with link index i ∈ [1 · · · n i ]) and the total mass M is represented by the sum of the link masses, i.e., M = m i . The position of the base frame 0 is defined as x 0 ∈ R 3 , and its orientation as θ 0 ∈ R 3 . The position vector of the centre of mass (CoM) for each link, c i ∈ R 3 , is represented with respect to the local reference frame Σ i . d i ∈ R 3 is the link length vector with respect to the previously connected link local frame Σ i−1 . The rigid links are articulated through n j revolute joints, and the angular position of those joints is denoted as q h = [q 1 . . . q nj ] T ∈ R nj . The generalized coordinate of the system is defined by Assuming n k contact forces vector, f 1 . . . f n k , applied at locations vector, p 1 . . . p n k , the Lagrangian based equation of motion can be written with respect to the inertial frame as: where M, C and G represent the inertia matrix, the vector of centrifugal and Coriolis forces, and the vector of the gravity force, respectively. S = [0 nj ×6 I nj ×nj ] ∈ R nj ×(nj +6) is the actuation matrix, τ ∈ R nj is the vector of applied joint torques, and J p k is the contact Jacobian at the point p k with respect to Σ W . By employing Eq. 1, given J p k and f k , it is possible to address the overloading effect induced on the joints of a human which is performing a manipulation task interacting with a tool or an object [13].

III. ONLINE SESC PARAMETERS ESTIMATION
Even though we are tackling the modelisation problem using robotic means we cannot change the fact that we are still dealing with a human body. Especially for the estimation of the CoM, we cannot rely on detailed project blueprints that uniquely identify the position of local CoM in every link. Traditionally, the CoM of the human body is calculated by means of anthropometric heuristic tables [17] which not only lack in specificity but require cumbersome measurements. Therefore we make use of the so-called SESC method that allows us to provide subject-specific CoM estimation. Previous work on this method used a batch identification approach to find the relevant SESC parameters for a subject [14]. As suggested by [18], the use of a Kalman Filter for a recursive identification strategy allows to have real-time access to the parameters and their associated errors. This not only speeds up the process but makes it much more applicable in real scenarios.

A. SESC model
In [14], it has been demonstrated that the CoM position of a generic linked chain is equivalent to the end-effector's position of a virtual chain (see Fig. 3). The structure of the virtual chain, known as the statically equivalent serial chain (SESC), can be build from the static and geometric parameters of the original whole body structure as: where n i is the number of local reference frame, with L being the last local reference frame in each branch. As a result, the CoM, denoted as C M , of the original chain is represented at the end-effector location through a forward kinematics operation.
where matrix B = A 0 · · · A ni ∈ R 3×3(ni+1) contains the orientation of the original chain, and Φ = φ T 0 · · · φ T ni T ∈ R 3(ni+1) includes the SESC parameters. To simplify the identification procedure is possibile to rewrite Eq. 3 in the following way: where 0 C M is the CoM represented in the base frame Σ 0 .

B. Kalman Filter
The identification of the parameter vector Φ in a form such as the one in Eq.4 can be recognised as a classical least-squares problem. The classical least squares estimator exists in two equivalent forms, "batch" and "sequential". The Kalman filter may be considered as a generalization of the least-squares technique to dynamical systems. The equations of the sequential least squares estimator are the same as of the Kalman filter, except that the system dynamics matrix is an identity matrix and the process noise covariance matrix is equal to zero. Therefore, no prior information regarding the system dynamics is used for the estimation. The problem can be reformulated as where the index k denotes the time step. To estimate the vector Φ k a certain number of 0 C M,k measurements are necessary. Each measurement is a linear combination of the states and a zero-mean measurement noise v k with covariance R k as shown in Eq. 6. The Kalman filter provides an optimal linear solution to the problem where the noisy system is defined by Eq. 5 and Eq. 6. For accurate identification, a certain number of measurements should be acquired. However, we cannot access objectively to the subject's CoM position. In [14], the authors explain how an estimation of Φ k is possible with partial data. On the other hand, a good solution is offered by the possibility to measure the CoP which, under certain circumstances, specifically during static conditions, is a good approximation of the horizontal projection of the CoM [17]. The criteria to determine if a pose is stable enough to be considered static consists in a double-check both on the CoP position and on the standard deviation of the joint angles in a window of 20 samples. The Kalman filter equations can then be written as [19]: where Q k is the noise covariance matrix, I is the identity matrix and K k is the filter's gain. P − k and P + k are, respectively, the estimation covariance matrices before and after the state update. These matrices gives as a first-hand knowledge of the progress of the estimation because the trace of P k is the sum of the mean squared errors.

C. Boost Filter's Convergence
As shown in Fig. 2, to accelerate the filter's convergence, hence to reduce the discomfort of the subject in the model identification/personalisation phase, a pose-suggesting interface is developed. At each iteration, the values of the diagonal elements of the covariance matrix P k , namely P jj , are used as weights to maximise the cosine distance of the poses performed up to the m-th update of the filter. It is likely that a poor estimation of one of the links of the chain structure is associated with a high value of the relative covariance element in the covariance matrix diagonal. The choice of the next poseŷ = [x T 0 θ T 0 q h ] ∈ R (nj +7) is made among a space of feasible random poses y in the operational space Ψ collected in a previous one-off stage. The optimisation problem can be formulated as: where the previous m poses x are collected in the space χ and w are the elements of the covariance matrix normalised by the max of their values. Moreover, a set of linear constraints may be set based on the model knowledge. Bilateral symmetry of the model, as well as the legit assumption of the CoM position for all limb on the longitudinal axis connecting two joints, generate a great drop in the number of parameters to be estimated as explained in [18]. Finally, feeding the filter with a trustworthy initial state coming from a subject with a similar body structure can further speed up the convergence of the filter. To provide the subjects with real-time feedback about the suggested body postures, we adapted and used a graphical interface that we recently proposed in [9].

IV. OVERLOADING JOINT TORQUE
In this section, we introduce a dynamic model of the human body to define the relation between the CoP and the interaction forces (applied from the support plane and the external object/robot) to estimate the overloading human joint torque. Such a model was first proposed in [13] and it is based on the displacement of the CoP, computed as the difference between the estimated one,Ĉ Pwo , and a measured one, C Pwt , collected using an external sensor system. In fact, it is possible to achieveĈ Pwo with the ground-projected CoM (using the procedure explained in Sec. III). WhileĈ Pwo and C Pwt are almost equal if no external forces are applied on the human body, if an interaction of the human with the environment (or with an object) occurs,Ĉ Pwo and C Pwt differ and the overloading joint torque can be computed accordingly. Let us consider f wo and f wt as the vGRF vectors applied at the CoP without and with the effect of external forces and f h as the interaction forces that are applied at the contact points a h . By usingĈ Pwo and the vGRF vector f wo , the overloading torque vector τ wo , considering any external force except the ground reaction force (i.e. the body weight), can be computed as Similarly, by using C Pwt and the vGRF vector f wt the overlaoding torque vector τ wt considering, on the contrary, the effect of any external force, can be computed as where τ b ∈ R n+6 is equal to M(q)q + C (q,q)q + G(q) ∈ R n+6 , n f is the number of contact forces exchanged with the ground and n h is the number of contact points where the external forces are applied. The relationship between the interaction forces f h and vGRF variation ∆f w,i = f wt,i − f wo,i can be defined as Deriving from (12) and (13) and (14) the overloading joint torques can be defined as with the Jacobian of the CoP displacement defined as J ∆C P i = J C P wt i − JĈ Pwoi . Since the external force effect is included in ∆τ s , τ b does not affect the overloading torque vector ∆τ s in any body configuration. By disregarding τ b , the number of human model parameters to be identified can be considerably reduced [20]. 0 ≤ ζ i , η j ≤ 1 are the distribution gains for vGRF and interaction forces, respectively, which can be calculated from the body configuration [21], [22] (N.B. i ζ i = 1, j η j = 1 is a necessary condition).

V. EXPERIMENTAL DEMONSTRATION
This section presents the results to validate the proposed real-time ergonomics framework that demonstrates the sequential and autonomous integration of the identification and ergonomics monitoring module. We performed an experiment on one healthy male adult subject (age: 25 years; mass: 72 kg; height: 177 cm). The whole experimental procedure was approved by the ethics committee Azienda Sanitaria Locale Genovese (ASL) N.3 (Protocol IIT HRII 001).

A. Experimental setup
The human body configuration is monitored using a wearable lightweight IMU-based motion-capture suit, the MVN Biomech suit by Xsens Technologies [23]. This solution offers indeed a better performance in workplaces as noted in [11] where the limitations of optical marker-based systems are enhanced. The CoP and the ground reaction forces measurements are collected using a Kistler force plate instead of wireless insoles sensors for sake of simplicity. Nevertheless, as highlighted in Fig. 1, the possibility and the importance of using wireless, wearable insoles sensors to assure mobility is crucial when shifting to real task execution scenarios. The acquisition and synchronization of all the sensor data is managed using Robot Operating System (ROS) environment.

B. Experimental flow
As explained in Sec. II the procedure is divided into two phases: the SESC identification phase and the overloading monitoring phase. During the SESC identification phase, the subject was asked to perform a sequence of different body configurations following the guidance of a graphical interface, depicted in the upper part of Fig. 5. Both the current human body configuration and the suggested one were shown at the same time on the screen (see the left side of Fig. 5). Once a match between the current posture and the suggested one was achieved, the volunteer was asked to remain still (typically a couple of seconds were enough) in order to reduce the accelerations of his CoM enabling the update of the Kalman filter. A threshold equal to 0.002 was set for the values on the diagonal of the covariance matrix P k , namely P jj , to assume the convergence of such a filter and thus the fulfilment of the SESC identification. As soon as the convergence criterion was reached and thus the overloading monitoring phase started, the visualisation mode changed highlighting the overloading joint torque (see the right side of Fig. 5). The latter torques were displayed in the form of spheres superimposed on the human model and color-coded to denote a high (red), medium (yellow) or (green) level of overloading. The 3 levels of overloading are determined as explained in Table I. The tuning and personalisation of the maximum joint torque values ∆τ i were established experimentally. Increasing joint torques profiles were applied on each joint until uncomfort was experienced by the subjects. The results where then compared with those present in the literature [24].
To show the effectiveness of the ergonomics assessment, the subject was asked to lift a 4kg box and to move in the operational space. Hence, the overloading joint torques varied consequently.

C. Results
The results of the aforementioned experimental demonstration are summarized in Fig. 6 and Fig. 7. In accordance with what has been covered in Sec II, the procedure was divided in two phases: SESC identification and overloading

Overloading level
Control threshold monitoring. This subdivision is clearly highlighted in Fig. 6 by a black dashed line in the graphs. In the upper part of the plot a 2D stick-model of the human is displayed showing some peculiar body configurations that the subject adopts throughout the experiment. We can clearly distinguish when the ergonomic assessment comes into play, namely when the color-coded feedback of the overloading torques appears. A second dashed line in red divides the unloaded stage from the box lifting task. The overloading joint torques of the human most significant body joints, hip (H), knee (K), ankle (A), shoulder (S) and elbow (E) are illustrated in the middle graph of Fig. 6 while the bottom graph represents the estimated and measured CoP,Ĉ Pwo and C Pwt , respectively. In the first phase, the estimated CoP,Ĉ Pwo initially presents considerably erroneous values since the SESC identification is still at an early stage and the same behaviour occurs for the overloading joint torques ∆τ S . Afterwords, with the progress of the SESC identification procedure,Ĉ Pwo and ∆τ S are estimated more and more accurately due to the Kalman filter's convergence till the end of the first phase, when their estimation can be finally considered reliable. In the second phase then, it can be noted that ∆τ S is almost equal to zero in all the body joints during the unloaded stage (before the red dashed line) while it shows significant variations during the lifting task (after the red dashed line). Accordingly,Ĉ Pwo and C Pwt are very similar during the unloaded stage (percentage error equal to 16.17 %) while they tend to differ during the lifting task. In Fig.7 the linear regression of the SESC parameters Φ k and the behaviour of the values on the diagonal of the covariance matrix P k , namely P jj , throughout the experiment are depicted in the upper and in the lower graphs, respectively. It can be noted that Φ k varies significantly for all the correspondent body links in the beginning of the process and then stabilises. Accordingly, P jj values initially decrease rapidly and then the decrement steps are considerably reduced. Only when all the P jj values go underneath the convergence criterion value of 0.002 highlighted in Fig.7 with a black dashed line, the SESC estimation phase can be terminated triggering the second phase. In our trial this occured in just 25 poses. The accuracy of the CoP estimated through the SESC technique and the capability of the overloading joint torques as a method to assess human ergonomics and potentially reduce the worker's physical load have been extensively validated in [14] and [13], [15], respectively. Accordingly, a further validation is not presented in this paper.

VI. CONCLUSION
In this paper, we integrated and showed the potentiality of a comprehensive method for estimating ergonomic quantities based on personalised features. The whole procedure has been designed to satisfy the needs of a realistic industrial scenario where there is no time for dull calibration routines. The improvement is clear when comparing the results obtained above with those of our previous work [13]. In the latter 150 static poses where necessary to feed the "batch" linear regression estimation together with a great post processing effort. We do furthermore understand the limits of IMUbased sensorised suits in a workplace, however at this stage they offer a better performance compared to other visionbased motion capture system who suffer of occlusions and privacy problems. In future works, we will investigate the possibility of reducing the number of IMUs as well as embedding the sensors in the working clothes.