Motion analysis of Parkinson diseased patients using a video game approach

Parkinson's disease (PD) is a progressive neurological disorder and the second most common age-related neurodegenerative disease after Alzheimer's disease. The primary symptoms of the disease are associated with the loss of motor skills affecting patients' movement and coordination and disrupting their daily life. Unfortunately, such motor symptoms cannot be fully relieved by therapeutic options. On the other hand, studies have shown that regular training and exercising can prove neuroprotective in PD patients helping them maintain independent longer. Based on recent studies stating that computer-based physical therapy games can be used as an option for facilitating PD rehabilitation exercise programs, we present the development of a body motion based videogame, using the Kinect sensor, targeted for PD patients. We tested twelve patients with advanced forms of PD motor symptoms (UPDRS motor score>20) and six initial stage PD patients (UPDRS motor score<20). All participants underwent an (UPDRS) motor skills pretest and afterwards performed three training sessions. In this paper, we will present part of our research aiming to analyze the movement patterns of PD patients in order to detect statistical significant differences between groups of different impairment level based on their UPDRS motor score and their performance. Consequently, we adopt a deep learning approach by analyzing the recorded human skeleton sequences for predicting the players' level of motor skills decline. Such methods and data can serve as preliminary evidence for further larger and controlled studies to propose such an exergame that can independently detect and adapt its difficulty level to better match players' ability providing a more targeted and personalized rehabilitation option.


INTRODUCTION
Parkinson's disease (PD) is an age related, non-reversible, progressive neurodegenerative condition that affects movement. PD is characterized by diversity since the related clinical symptoms and their progression rate can be different from one person to another. It is incurable and its symptoms tend to worsen over the passage of time. Specifically, its clinical symptoms may include tremor (mainly at rest), bradykinesia, loss of ability of movement, limb rigidity and problems with balance and falling [1]. The main way we measure PD progression in clinical studies and trials today is using subjective rating scales, such Unified PD Rating Scale, UPDRS [2], the Hoehn and Yahr [3], Schwab and England [4] and other scales, with the first one being the one most popular and often used especially in the clinical context. The scale itself has four components, the third one, which interest us more concerns motion and provides us a point score, which gives a general estimate of patient's motor disability. However, a number of limitations nonetheless exist since such scales are susceptible to problems of sensitivity and reliability [2]. In addition, while its exact cause has not yet been addressed, treatment options, with the potential to slow, stop or reverse PD symptoms vary and include medications and deep brain stimulation [5], but usually signs become unresponsive to them over time. Besides the medication, physiotherapy plays the most important role in treating PD [6] as it can lead neuroprotective and even neurorestorative effects [7]. A number of methods have been proposed within physiotherapy such as general physiotherapy, aerobic, treadmill walking, cueing (with auditory, visual or somatosensory cues), dance or martial arts, with short-term improvements of gait speed, freezing of gait, balance and falls, global motor function, activities of daily living, and quality of life [7,8], However, physiotherapy is only effective when training is done on an intensive daily basis [5,6,9] since beneficial gains weaken within weeks or months after halt of exercising [10]. In addition, several barriers for exercise persistence have been identified in people with PD with low perceived benefit of exercise, boredom, lack of time, as well as fear of falling [11,12]. Therefore, new therapeutic strategies have been explored as a promising rehabilitation option for PD such as the use of exergamesmotioncontrolled gaming systems that allow players to interact with the game through body movements and are intended to be used as an exercise tool. Such games use audio and visual cues in loose, virtual reality (VR) environments and offer an enormous motivating potential because they engage users in immersive, goal-oriented scenarios. Studies have shown that body-motionbased games can have a variety of benefits for the players and are increasingly applied in physical therapy, motor skills teaching, rehabilitation and prevention for older adults [13]. During the last decade, the availability of affordable motion capture sensors (Kinect, Wii, leap motion, etc.) have contributed in the growing interest in designing and developing such games that stimulate movement, optimize motor learning, potentially compensate defective internal mechanisms using external cueing and allow augmented feedback [14]. In addition to a challenging and motivating virtual environment, gaming also allows training for specific motor and non-motor signs with an amusing and enjoyable approach that can be performed at home with increased training frequency and/or on demand [14]. Specifically, one study showed that treadmill walking coupled with virtual reality also promotes fall reduction 6 months after the cessation of training [15]. One customized game has recently been tested for arm movement training with encouraging results [16]. Another study proved that long term rehabilitation with a customized videogame to treat gait and balance disorders is feasible, well accepted, and effective in parkinsonian patients [17]. Technology-supported physiotherapy can also offer additional benefits, such as motion analysis, providing a description of human motion features which can be measured through the use of wearable or not sensors and enhance the quality of motor symptoms assessment. Data acquired can be pre-processed by advanced knowledge processing methods to allow continuous monitoring, management and personalized treatment. Specifically, PERFORM, is such an intelligent closed-loop proposed system offering continuous remote monitoring and management of Parkinson's disease (PD) patients by integrating a wide range of wearable sensors constantly monitoring several motor signals of the PD patients allowing health professionals to remotely monitor the overall status of the patients, adjust medication schedules and personalize treatment [18]. Another study proposed a method of Bayesian gait recognition using Microsoft (MS) Kinect image and depth sensors and skeleton tracking in three-dimensional space. The results included the estimation of the characteristics of selected gait features for patients with Parkinson's disease and for individuals from the reference sets, presentation of decision boundaries, and comparison of classification efficiency for different features [19]. Synnot et al. [20] proposed WiiPD, a game based approach to home-based objective assessment of Parkinson's disease that uses the Nintendo Wii Remote in combination with a number of bespoke data gathering methods providing an engaging and capturing a wide range of motor and non-motor metrics. Results of testing showed that the system is able to differentiate between normal and abnormal motor performance associated with PD. Liu et al. proposed a PD predictive analytics through pad game. The implemented game aims at detecting possible PD symptoms at an early stage by data collection, real-time data analysis and monitoring [21]. In this paper, we present part of our research work proposing the use of an engaging Kinect v2 body motion based game, created with Unity 3d, that will collect and store various players' data (skeletal sequences, rgb, performance parameters, etc.). Such an approach intents to analyze the movement patterns of PD patients in order to detect statistical significant differences between groups of different impairment level based on their UPDRS motor score and their performance. Specifically, we adopted a deep learning approach by analyzing the recorded human skeleton sequences for predicting the players' level of motor skills impairment. Such methods and data can serve as preliminary evidence for further larger and controlled studies to propose such an exergame that can independently detect and adapt its difficulty level to better match players' ability providing a more targeted and personalized rehabilitation option. our proposed approach was implemented within the context of i-Prognosis (Intelligent Parkinson eaRly detectiOn Guiding NOvel Supportive InterventionS), which aims to develop early and unobtrusive Parkinson's disease detection tests based on the interaction of users with their everyday technological devices [22].

THE EXERGAME
The videogame was built specifically for this study. The videogame was controlled by a laptop placed 1 m in front of the patient. The patient played by moving in front of Kinect2 motion sensor placed 1.5 m in front of them. The game scenario consists of an avatar controlled by the players' body movements that moves around its hands to collect apples. For each appearing apple players have a time interval of 5 seconds to successfully catch them. Each game session consists of a static sequence of nineteen positions where the apples appear one after the other ( Figure 2). The difference between the green and red apples is that the first ones are placed in front of the player while the second ones in parallel to his body. Apples positioned at places that demand by the players to achieve greater hand range of motion are characterized as more difficult to complete in comparison with the ones placed near him. For each successful try, players' score is boosted and positive auditory feedback is provided. After the players' performance is completed, an overall evaluation score is offered along with an appropriate message. During the game-play a variety of players' performance parameters is collected and saved online. Specifically, in order to estimate the players' velocity and accuracy we record the required time necessary to catch an apple for each position, the overall game duration, the failures and their score. In addition, during each game the Kinect v2 RGB stream is captured and saved at 16fps in order to be used for further motion data analysis.

UPDRS PREDICTION MODEL
In this section, we propose an accurate and robust deep learningbased post analysis methodology for detection of the motor impairment of PD patients from video sequences, recorded using the Kinect v2 sensor during players' gameplay sessions. Our method, based on the work of [23], relies on body skeletal features extracted from RGB videos and, therefore, it acquires discriminative for upper body movement recognition skeletal data without the need for any additional equipment, such as data gloves, that may restrict player's movements. Specifically, we performed body pose estimation using the OpenPose library [24] [25]. The outputs of the body detection networks are 18 body joints, out of which we used only the six related to the users' arms ( Fig. 2.). Before introducing the skeletal features to the proposed skeleton classification network, a preprocessing was implemented. All 2D joint coordinates were initially transformed from the image to a local coordinate system by placing the neck of the body skeleton at the origin. The purpose of this preprocessing is to make skeleton data invariant to the absolute location of the human in the scene. The selected raw joint coordinate features along with their calculated joint line distances form a two-stream deep neural network that consists of stacked LSTM (Figure 3) layers, having as a task to produce descriptive temporal information from the spatial features The optimization of the hyper-parameters that affect the performance of the proposed motion analysis system is performed after experimentation on the dataset. These hyper-parameters define the size and number of LSTM layers, dropout percentage, batch size and learning rate. More specifically, two layer LSTMs are considered, consisting of 128, 256, 512 or 1024 neurons, while the dropout percentage is in the range [0.0-0.5]. The streams fed with the skeleton joint coordinates employ two-layer LSTMs with optimized number of neurons and dropout percentage. Finally, the network is implemented in Keras-Tensorflow framework and trained using the Adam optimizer with batch size of 32 and learning rate equal to 0.0001.

EXPERIMENTAL RESULTS
Eighteen people with PD (table 1)were recruited at the Department of Neurology, Aristotle University of Thessaloniki and chosen according to the following criteria: 1) diagnosis of PD, 2) age below 75 years, 3) not suffering from dementia, inability to maintain upright posture or walk independently or from other pathologies affecting gait and balance. Procedure of the pilot experimental study consisted of 3 training sessions with our proposed body motion-based videogame. Specifically, we tested twelve patients (Group_Ad) with advanced forms of PD motor symptoms (UPDRS motor score>20) and six initial stage PD patients (Group_In) with UPDRS motor score<20. All participants underwent an (UPDRS) motor skills pretest to detect their motor skills impairment level. The analysis of the gathered experimental results consists of two parts. The first part, described in section 4.1, deals with the statistical analysis based on users' performance (score) and time data. Following, at section 4.2, we focus on analyzing the collected skeletal data of the users. We used deep-learning motion analysis approach for PD motor impairment prediction and association.

Statistical Analysis of Experimental Results
This section presents part of our research aiming to analyze the movement patterns of PD patients in order to detect statistical significant differences between groups of different impairment level based on their UPDRS motor score and their performance. To assess the difference between the two PD patients group we analyzed three different performance parameters of the game: the average value of the duration time, achieved score and number of failures for each groups. We first tested the data distribution using the ShapiroWilk test and the variance homogeneity using Levene's test. If data were found to be normally distributed, we performed t-Test: two samples assuming equal variance or assuming unequal variance depending on Levene's test result. If not the Mann-Whitney U-Test was used instead. Results were considered significant at p < 0.05. Specifically, the statistical analysis revealed statistical significant difference (Table 2) regarding the average game's time duration parameter between the two groups of PD patients. Therefore, we considered it useful to investigate further the average game's time duration parameter in order to discover if and which apples' positons within the game's coordination system were more difficult to achieve for one of the two groups. Following the same analysis methodology, results revealed significant difference (table 3) for two out of the nineteen positions.

Deep-learning Analysis
The dataset was again divided into two classes: the Group_In (UPDRS motor sub-score<20) consisted of six PD patients and the Group_Ad (UPDRS motor sub-score>20) consisted of twelve PD patients. We performed a k fold cross-validation technique to assess the predictive performance of the model presented at section 3. Specifically, we set as a validation data nine out of the fifty-four video sequences (each participant performed three game sessions). At first, we implemented a classifier accepting as an input the upper body joints data and afterwards we implemented a second classifier with an input the calculated joint-line distances of the upper body joints [26]. Consequently, we combined the two classifiers creating a two-layer LSTM network. The fusion of the two approaches improved the percentage of the achieved accuracy reaching the level of 77.7%.

CONCLUSIONS
In this paper, we presented part of our research aiming to analyze the movement patterns of PD patients in order to detect statistical significant differences between groups of different impairment Group_In Group_Ad level based on their UPDRS motor score and their performance. Specifically, the statistical analysis of various recorded game performance parameters showed significant time difference between the games' overall duration between the two groups of PD patients with initial and advanced motor disability. Consequently, we proposed a deep learning approach by analyzing the recorded human skeleton sequences for classification and prediction of the PD patients' level of motor skills decline. The proposed two-layer LSTM network system achieved a 77.7% accuracy. However, it should be taken into consideration that the motion analysis dataset of the PD patients is quite limited and in the future it should be expanded further in order the achievement of better results of accuracy to become possible. In addition, the generation and testing of different kind of features should also be considered. Future work will also include larger and controlled studies to propose such an exergame of many difficult levels that can independently detect and adapt online its difficulty level to better match players' ability providing a more targeted and personalized rehabilitation option.