Detecting hypomimia symptoms by selfie photo analysis: for early Parkinson disease detection

Hypomimia, i.e. reduction in the expressiveness of the face, is a cardinal sign of the PD, often present at its early stages. Within the EU-funded i-Prognosis project (http://www.i-prognosis.eu), early and unobtrusive Parkinson's disease detection tests are developed, based on the interaction of users with everyday technological devices. The selfie analysis module translates facial expression features into an index reflecting the severity of PD hypomimia symptoms that affect the variability of patients' facial expressions. Monitoring of such an index over time holds the promise to detect the onset of hypomimia symptoms in an unobtrusive way. Our approach proposes a methodology for detecting and quantifying the progressive decrease of variability of facial expressions in early PD patients by analysing patterns emerging from photos (selfies) during daily life. Promising results are presented from both a) a small development set of 36 users (both PD patients and healthy controls) and b) a large set of selfie photos obtained from 1292 users that were analysed by the iPrognosis cloud analysis module.


INTRODUCTION
Parkinson's Disease (PD) is one of the commonest neurodegenerative diseases, affecting approximately 1% of individuals older than 60 years and 2-4% over than 75 years, causing progressive disability that exhibits the greatest loss of Quality-Adjusted Life Years (QALYs) among 29 major chronic conditions [1] [2].
The EU i-PROGNOSIS proposes an intelligent ICT-based approach for a) early PD symptoms detection by unobtrusive behavioural analysis using large volumes of data, acquired from their natural use of smart devices ( mobile smartphone and smartwatch) and b) early interventions to those identified as early stage PD patients, promoting active and healthy ageing, such as personalized gaming (exercise, mental or dietary), nocturnal and assistive interventions. PD early detection is based on features extracted from multiple cues (modalities), including voice, movement and handling (of phone), typing, selfie photos, physical activity, sleep stage, food intake and others. Early Detection of PD Parkinson signals for each modality is performed in two consecutive stages as follows: a) First, the modalities are captured and features are extracted and/or processed by the i-Prognosis smartphone application [3] and transmitted to the cloud at a daily basis when the phone is charging and connected to the WiFi. User privacy and ethical constraints are fully respected, as no sensitive personal data are transmitted to the cloud, but just a set of features that cannot reveal a person's identity or data. b) In the second stage, the extracted features are further analyzed by a cloud component and are finally provided to a user-specific behavioural modelling module. which can provide an early detection signal if certain deviations are observed. Hypomimia (or masked facies) is a cardinal sign of PD often presented in its early stages [4]. The syndrome is characterised by a marked diminution of expressive gestures of the face, including brow movements that accompany speech and emotional facial expressions [5]. Punctuation brow movements -very brief (~50 ms) contractions of the muscles of the upper face that occur during speech and appear to add semantic emphasis, are often absent. Additionally, hypomimia is commonly manifested in only one side of the face [6]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request   In this paper, the hypomimia (selfies) analysis module of i-Prognosis is presented and results are presented using both a test development set as well as data collected from the cloud during the project. The goal is to detect and quantify the progressive decrease of variability of the user's facial expressions in a) the left or b) the right part or c) both parts of the face. Furthermore, we utilize, as a guide for this "quantification" task, the Unified Parkinson's Disease Rating Scale (UPDRS) [7], which is the most commonly used assessment method for clinical and scientific purposes.
The UPDRS Part III scale (sub-item 19) includes a classification of symptoms of hypomimia noticed from facial expressions into five classes: 0 = Normal. 1=minimal, 2=Slight, 3=moderate (lips parted, sometimes), 4=severe (lips parted 1/4 inch or more). Ultimately, the hypomimia (selfies) analysis module attempts to estimate the facial expression-related rankings of the UPDRS Part III scale.
The rest of the paper is organized as follows: Section 2 examines related work, while Section 3 presents the proposed i-Prognosis approach for hypomimia symptom detection. Section 4, presents a study regarding additional features that can be used as indicators of hypomimia, while Section 5 provides some experimental results. Finally, Section 6 draws conclusions drawn and describes ideas for future work.

RELATED WORK
The effect of hypomimia (masked face) is a common secondary motor symptom of Parkinson's disease [8]. Hypomimia is a reduced degree of facial expression that can be caused by motor impairment (for example, weakness or paralysis of the facial muscles). As reported in [9], experiments revealed that the PD participants showed reduced spontaneous facial expressivity across experimental situations and had more difficulty than controls posing emotional expressions and imitating nonemotional facial movements.
Therefore, the detection of hypomimia may be used for the diagnosis of Parkinson. In [10] methods are developed to automatically detect and assess the severity of hypomimia by monitoring facial expressions that patients make, while watching video clips. Specifically, machine learning tools are employed using measurements from a depth sensor that allows for fairly accurate facial movements tracking. Encouraging results are obtained, providing proof of concept that automatic evaluation of hypomimia can be sufficiently reliable to be useful for clinical early detection of PD-related hypomimia. In [11], hypomimia is investigated in patients suffering from depression using ultrasonic markers placed on participants' faces. New wearable technology could enable home monitoring of patients, but since the number of sensors that can be put on a patient's face is limited, the quality of assessing hypomimia severity is reduced.
According to [12], little attention has been paid to this relevant clinical aspect of the disease and there is still a lack of validated assessment methods and therapeutic approaches. As far as we know, the approach presented in this paper is the first attempt to estimate hypomimia symptoms for Parkinson Disease by simply processing selfie photos stored on a user's phone.

PROPOSED APPROACH
The proposed processing pipeline is illustrated in Figure 1. Specifically, the smartphone application periodically examines each of the images that are stored on the mobile phone and processes only those that have dimensions compatible to those of the front (selfie) camera. It relies on the Face API offered by Google Play Services (previously part of Mobile Vision API and currently part of the new ML Kit). Face API is able to a) identify the available faces in the image and return their bounding boxes, b) provide, for each of these bounding boxes, a set of features, namely locations of eight (8) specific face landmarks (eyes, nose, cheeks, left/right/bottom mouth corners), smiling probability and right/left eye open probabilities. The smartphone app provides output for selfie photos for which exactly one face was detected, since in this case, we can assume that this is most likely the face of the user. In this case, it generates a string containing the detected face features and associated parameters that are transmitted to the cloud for further processing.
The cloud processing pipeline attempts to detect and quantify the progressive decrease of variability of the user's facial expressions in: a) the left or b) the right part or c) both parts of the face. It consists of three consecutive steps: a) Face normalization, where the detected face features are normalized to a common reference frame. Specifically, a common reference frame is defined by rotating and scaling all face landmarks so that the detected eye and nose locations, which are considered as fixed points, coincide for all photos b) Conditional filtering, which discards entire data corresponding to a selfie photo if certain constraints are not met (e.g. if the face is not frontal or if some of the landmarks are not detected) and c) Hypomimia feature detection that performs the final computation of hypomimiarelated characteristics from the normalized face features. Specifically, the horizontal and vertical variances of the five "moving" features (LEFT/RIGHT CHEEK, LEFT/RIGHT/BOTTOM MOUTH) are measured for each user, resulting to a vector of 10 measurements. Furthermore, the variances of three additional available features (SmilingProbability, LeftEyeOpenProbability and RightEyeOpenProbability) are also measured, resulting to a final vector of 13 measurements. Intuitively, when a significant decrease is noticed in the values of these variances within a specific time period, this could be considered as an early hypomimia symptom.

USE OF ADDITIONAL HYPOMIMIA SYMPTOMS
Furthermore, Bandini et al. [13] and Sanchez et al. [14] propose the following list of symptoms that are common in PD patients: • Widened Palpebral Fissures (PF) on the side where symptoms occur / PF asymmetry • Unintentional lips separation (mouth opening) • Reduced blink rate • Flattened nasolabial Considering that our hypomimia analysis module processes only photo data of the users and not video data, the estimation of blink rate is not possible. Furthermore, flattened nasolabial is very hard to detect as a feature. Therefore, we further studied the first two symptoms, i.e., the Palpebral Fissure (PF) asymmetry and the unintentional lips separation. For this reason, we had to detect additional landmarks in the photos, as the eight basic face landmarks provided by the i-i-PROGNOSIS mobile application using the Google Face API were not enough. Therefore, we decided to use the Microsoft Azure Face API which can identify 27 predefined landmark points of each user (Figure 2). In addition, the Microsoft Face API offers an emotion recognition service that provides probability of 8 emotions: anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise.
The new features were evaluated using same pipeline described in Section 3, and similar variances were computed that could indicate hypomimia symptoms. However, it was not possible to use the Microsoft Face API within the iPrognosis app, as the API can run only in the cloud, so it is not allowed to process photos from a user's phone, which consist sensitive personal data. Therefore, some indicative results were obtained only for the development set and are presented in the Experimental Results section.
However, it was possible to include the Palpebral Fissure (PF) asymmetry feature, by using some additional image processing to estimate the eye opening based on the estimated eye locations provided by (Google) Face API that runs locally at the phone. Specifically, we extended the technique described in [15] to estimate the top and bottom points of each detected eye in the common reference frame (see Section 3). Specifically, after defining a window around each eye, calculation of three proposed projection functions IPF (Integral), VPF (Variance), GPF (weighted average of the above) is performed and the top and bottom positions are estimated as local maxima of the derivative of these functions. In Figure 3, an indicative result showing the IPF function, which provided the best results in our experiments, as well as the estimated top and bottom eye positions, is illustrated.

EXPERIMENTAL RESULTS
In order to estimate the relationship of the extracted features with hypomimia symptoms we initially used a development dataset composed of 23 Parkinson patients (Hoehn-Yahr stages I, II or III with confirmed PD diagnosis on average 10.59 years) and 11 healthy controls, between features aggregated on a photo/selfie collection session level and inherited-from-subject UPDRS Part III.19 (Facial Expression) sub scores of facial expressions for each participant.
The i-PROGNOSIS hypomimia (selfies) analysis module attempts to estimate the facial expression-related rankings of the UPDRS Part III scale by evaluating the variability of the user's facial expressions in specific time windows. The aim of the photograph/selfie analysis is to transform facial expression features into an explainable index reflecting the severity of PD hypomimia symptoms that may affect the variability of patients' facial expressions. Observing such an index over time, as part of the i-PROGNOSIS behavioural vector, holds the promise to detect the progress of relevant hypomimia symptoms onset.
UPDRS Part III Facial expression subscores are available for the study cohort that generated the development dataset (Table 1). In order to estimate the relationship of the extracted features with hypomimia symptoms, we performed the estimation of the bivariate (Pearson) correlation coefficient ! ∈ [−1,1] and its corresponding P-value on the development dataset.  These three features, were then used to train a linear regression model in order to estimate the accuracy of estimated values and trend of a Hypomimia Severity index (HSi1) with respect to Item 19 -'Facial expression' sub scores (target value range: 0-3), as seen in Figure 4. For this reason, we performed a Leave-One-Subject-Out (LOSO) experiment on the development data yielding an average (over LOSO iterations) RMSE of 0.85. While this value may seem high, hence indicating a bad fit, we should also take into consideration that it could be misleading, since UPDRS scores are discrete, quantized and of low resolution, meaning that the true level of severity may well lie between UPDRS levels. The resulting regression model granularizes the outputs and offers an index of higher resolution. Therefore, the fact of a seemingly high RMSE, with respect to the strictly quantized targets, does not necessarily prove that the estimated severity index does not approximate the true level of participants' hypomimia stage. By selecting an appropriate threshold, we can also classify each individual to one of the two classes (PD patient or healthy control). The corresponding boxplot and confusion matrix are presented in Figure 5 and Figure 6 respectively. The sensitivity/specificity was 0.79/0.82 respectively, showing that the HSi1 feature has good potential to categorize healthy vs PD individuals. Note that four (out of the total of twenty three) PD patients were omitted from the results as pre-filtering of their captured selfie photos resulted in a very small (less than 3 photos) dataset.

Figure 4 Target values and trend of the Hypomimia Severity index (HSi1) by training the model with development data, with respect to the actual UPDRS Part III Face Expression score.
After implementing the detection of PF Assymetry feature, as described in Section 4, a new Hypomimia Severity index (HSi2) using four features was defined, and, similarly the corresponding result illustrated in Figure 7, Figure 8 and Figure 9 were obtained. The sensitivity/specificity was 0.89/0.73, respectively. Finally, some results were also obtained using data from the data collected in the cloud via the i-Prognosis app. In this case, the actual condition of the users (or their UPDRS score) is unknown, so we can only compare against the self-reported assessment of individuals that is recorded by the app. During approximately 12 months of operation in five countries (Greece, UK, Germany, Portugal and Spain), 1436 selfie photos were processed by 221 distinct self-declared PD patients and 4827 photos from 1071 distinct self-declared healthy individuals. Approximately 40% of these data recordings were found to be valid for analysis and the estimated HSi1 index achieved a 0.58/0.54 sensitivity/specificity.

CONCLUSIONS AND FUTURE WORK
Hypomimia (or maked facies) is a common symptom of PD, often from its early stages. This paper presents a methodology, developed in the framework of EU i-Prognosis project, for detecting and quantifying the progressive decrease of variability of facial expressions in early PD patients, by analyzing patterns emerging from photos (selfies) they take during their daily life. Two indices were defined, using three or four features respectively, to quantify such hypomimia symptoms, based on the movement of specific face landmarks. Promising results were obtained, but performance can further improved by exploiting additional hypomimia features in the future. Furthermore, this information can be combined with information provided by other modalities developed by i-Prognosis project detecting other Parkinson symptoms (e.g. tremore, bradykinesia, voice degradation, etc.), so as to increase the accuracy of detection and decrease false alarms.