On capturing older adults’ smartphone keyboard interaction as a means for behavioral change under emotional stimuli within i-PROGNOSIS framework

. The unobtrusive use of smartphone technology, as a facilitator and as a means of capturing the daily activities, can be seen as a great challenge in routine monitoring and in promoting behavioural change in older adults. In the pre-sent study, a protocol of a sequence of emotional stimuli database was combined with a sequence of emotion-free text typing using a dedicated keyboard of a smartphone and used for capturing the users’ patterns of typing, in terms of hold time (HT), alteration time (AT) and pressure (PR) of each key. Six older adults (three male/female) were employed in the study and sequences of images with facial expressions of Ekman’s six basic emotions (with the addition of the neutral one) were used as stimuli in a three-trial fashion. Statistical analysis of HT, AT and PR data revealed differences in the typing due to emotions alteration, setting a new domain for the analysis and behavioural modeling of older adults’ typing patterns under specific emotional stimuli. This combinatory approach amongst emotional and physical status could be adopted in the field of intelligent monitoring of the healthy ageing and could be extended to elderlies’ pathology cases, such as Parkinson’s disease, as approached by the i-PROGNOSIS initiative.


Introduction
Based on the WHO, Active and Healthy Aging (AHA), in a broad sense, is defined as the process of optimizing opportunities for health to enhance quality of life as people age [1]. Although current healthy ageing discourse places responsibility on individuals for achieving good physical health it would be interesting to approach the physical/emotional changes of ageing and the social environment by focusing on what older people themselves value in regards to healthy ageing [2]. In this perspective, the use of ICT, as a facilitator and as a means of capturing the daily activities, places the challenge in routine monitoring via a clearly unobtrusive way. The latter is the focus of the Horizon 2020 project i-PROGNOSIS (www.i-prognosis.eu) that tries to capture the behavioral change of older adults (>50+ years) towards the Parkinson's Disease (PD) early detection via the use of smart devices (e.g., a smartphone), in the framework of which the proposed study is developed. In particular, here, the analysis of the interaction of the older adults with their smartphone keyboard is proposed as a means to examine any effect in their typing behavior under different emotional stimuli. In the last 30 years, keystroke dynamics has been studied by various research groups and commercially employed as a biometric [3]. Nevertheless, this approach was seldom applied to the medical field, with the rare example of [4], who used the typing speed in login sessions to evaluate sensory-motor speed in healthy subjects [4]. Moreover, Giancardo et al. [5], tried to identify a pattern from keystroke dynamics that could detect a state of psychomotor impairment in healthy subjects and the ability to distinguish PD patients at the early stage of the disease from comparable healthy controls [6]. However, none of the previous approaches involved any emotional factors in their studies. In the present study, a protocol of a sequence of emotional stimuli based on the Pictures of Facial Affect (POFA) database [7], [8], was combined with a sequence of text typing using the keyboard of a smartphone and used for capturing the users' patterns of typing, in terms of hold time (HT), occurring between pressing and releasing a key, alteration time (AT), occurring between releasing a key and pressing another key, and pressure (PR) applied on each key (initial pressing value). These keystroke dynamics were explored as a means that could reflect the emotion influence to the older adults' typing patterns, revealing a combinatory approach amongst emotional and physical status that could be adopted in the field of intelligent and unobtrusive monitoring of the healthy ageing. The rest of the paper is constructed as follows: first, a literature review related with smart technologies, healthy ageing and emotion recognition, and the main experimental procedures is presented, followed by data characteristics and a description of the protocol used. Next, a description of the implementation issues, analysis of the results, along with discussion and interpretation of the findings are provided. Finally, conclusions and future work conclude the paper.

Smart technologies & Healthy Ageing
Health technology interventions have been used, to help individuals to monitor their own health [9], to provide information and social support [10], and for homecare monitoring [11]. With the rapid growth of mobile phone technologies, paralleled with the rapid increases of the elderly population, there is a golden opportunity to use mobile phones to help manage older adults' health, in order to affect, in a positive and sustainable way, their quality of life and well-being [11], [12]. Mobile phones, especially smartphones, increasingly play an important role in the homecare of older adults, covering an increasing variety of clinical areas; for instance, in order to study the gradual loss of autobiographical memory of Alzheimer's disease patients, De Leo et al. [13] used the smartphone technology to automatically take photos for helping to improve the memory recall of the patients. Moreover, recent advances in smartphone and sensor technologies have digitized a range of medically relevant biometric data, giving the potentiality to detect disease patterns, such as risk of falls and mood assessment, and providing a window to diagnose and tailor treatments remotely [11].

Basic emotion categories
In the present study, the emotion recognition refers to the identification of emotional states. In fact, there are many techniques and modalities used to detect affect, namely: physiologic sensors, facial expression and speech recognition, and pressure sensors. Affect sensors are often coupled with algorithms that are designed to distinguish and classify patterns associated with emotional states [14]. In particular, from the study of facial expression of emotions, Ekman [15] defined 6 emotions, namely: joy, anger, fear, disgust, surprise and sadness, as basic emotions which has been largely used in the field of psychology and robotics. In Ekman's theory, the basic emotions were considered to be the building blocks of more complex feeling states [16]; although in other studies he is skeptical about the possibility of two basic emotions occurring simultaneously [17]. Moreover, Ekman and Friesen [8] developed the Facial Action Coding System (FACS), a method for quantifying facial movement in terms of component muscle actions.
Recently automated, the FACS remains the one of the most comprehensive and commonly accepted methods for quantifying and identifying emotion from visual observation of faces.

Emotion recognition based on keystroke dynamics & pressure sensor keyboard
Monitoring the dynamics of keyboard computer use has been studied in different fields, such as biometric authentication [18] or personality characterization [19]; however, with mobile computing area gaining popularity through the use of smartphones, more recent studies have been done in mobile environments. In fact, since the smartphone is embedded with the accelerometer and gyroscope sensor, more information can be used on the pattern analysis [11], [20], [21]. Some of the main keyboard dynamics are based on latencies of the keystrokes (e.g., time between keystrokes or the length of time that each keystroke is pressed), revealing that the typing patterns of the same individuals vary over time and are affected by other factors, such as stress or gradual changes in cognitive or physical function [22]; thus, keyboard dynamics can provide relevant behavioral information about the affective/cognitive state of the user. Khanna and Sasikumar [23], used the keyboard dynamics to differentiate between neutral/positive and negative emotions. The results revealed that the negative emotional state was associated with more typing mistakes and slower speeds in comparison with the more neutral affective condition. On the other hand, Epp et al. [24], measured the keyboard dynamics of 12 participants in a naturalistic experiment to discriminate between 15 emotional states. The results shown that although some emotions, such as anger and excitement, produced a classification performance of 84%, the recognition results for stress were not reported. Several surveys have shown that monitoring the keystroke pressure feature may be relevant in the context of affect measurement. In a survey with 100 respondents [25], 65% of the participants reported an increase in the typing pressure when angry. Analyzing the opinion of 769 undergraduate students, Karunaratne et al. [26], found that 118 of the students reported hitting the keyboard harder when under stress. On the other hand, Lv et al. [27], used a pressure-sensitive keyboard to recognize 6 emotions of 50 individuals (3000 samples in total); the results shown that although they obtained an average classification accuracy of 93.4%, the stress was not considered as one of their emotions, however, their work provided very limited data about how typing pressure varied for each emotion. More recently, Hernandez et al. [28], revealed that stress influences keystroke pressure in a controlled laboratory setting; they found that during stressful conditions, the majority of the participants (>79%) showed significantly increased typing pressure. While there is some work using pressure keyboards in the context of emotion recognition, the present study seems to be the first to use them in the context of smart technologies and healthy ageing, to unobtrusively capture the older adults' patterns of typing interaction, as well as to monitor behaviors that are influenced by emotional stimuli and detect when and how these behaviors change.

Experimental Procedures & Protocol
In the experiments of this study, six healthy Portuguese older adults (60-75 yrs, mean value 67±5.3 yrs, 3 male, 3 female) have participated after meeting the inclusion criteria as evaluated via a questionnaire regarding their level of education, experience in using a smartphone, non-existence of any typing difficulties and vision problems, along with sufficient mental capability to use the smartphone keyboard for typing and the nonexistence of mental disorder, like depression (see Table 1). A voluntary informed consent was obtained from all participants in this study, describing the purpose, procedures, risks and benefits involved in the study. In their home environment, after providing written consent, all participants were seated in front of a computer screen displaying the task, and requested to provide some demographic information. The whole procedure complied with the guidelines of the Ethics Council of the Faculdade de Motricidade Humana, Lisbon, Portugal.
The experimental protocol consisted of a sequence of emotional stimuli based on the Pictures of Facial Affect (POFA) database [7], [8], including all the six basic emotions (i.e., happy (E1), sad (E2), fear (E3), anger (E4), surprise (E5), disgust (E6)) and neutral (E7). This was combined with a sequence of text typing using the keyboard of a  smartphone (LG G5 H850) and used for capturing the users' patterns of typing, in terms of HT, AT and PR of keys. Knowing that the posture is a factor that significantly affects the keystroke patterns [20], participants were also informed to hold the LG smartphone in a hand (being more natural for the participant) while typing the text (78 characters with spaces) that appeared on the screen of the computer. The emotional stimuli were shown on a computer screen; before each emotional stimulus, a countdown from 5 to 1 followed by a cross (+) took place, so to neutralize the echo from the previous emotional stimuli and prepare the focus for the next one. The typing text was neutral in character (i.e., "A square has for equal sides and two equal and perpendicular diagonals", presented in Portuguese) and kept the same across all sessions ( Fig. 1(a)). Three trials (with a small break between them <30'') per subject were used and a randomized selection of the seven emotional stimuli was followed across each trial and across each subject. At the end of each session, the user self-categorized the previewed images, by selecting with the mouse the emotional content s/he perceived by each one (Fig. 1(b)). The total time duration of the test was around 40 minutes.

Implementation Issues
The first three authors developed a smartphone keyboard application (i-PROGNOSIS keyboard, available at https://tinyurl.com/hcn5sfj) for the typing test and it was installed in LG G5 H850 smartphone with Android OS. The data acquired by the i-PROGNOSIS keyboard were saved as .txt files and exported to the Matlab 2015a (The Mathworks, Inc., Natick, USA) environment. The statistical analysis adopted was Wilcoxon signed rank non-parametric test (level of statistical significance < 0.05), due to the limited number of participants and the focus on the within subject analysis. The visualization of the sequence of the emotional stimuli based on the POFA database along with the data analysis were carried out using Matlab custom-made programming code.

The emotion effect on keystroke dynamics
Comparative analysis of the results from the user's self-assessment of the visual stimuli with the norms provided by the Ekman's POFA database [7], [8], has shown an agreement greater than 90%. This shows that, almost in all cases, the intended emotion elicitation level was achieved. Figure 2 depicts an example of the HT, AT and PR data captured from the six users across the three trials after being exposed to the visual stimuli of emotion "happy" (E1). From Fig. 2 it is noticeable that most of the users have presented a similar keystroke behavior under the effect of E1, presenting a low variance in the acquired data. This observation was also noticed in the case of the rest of emotions (E2-E7). This allowed for the within-subjects analysis across the different emotions.
The significance in the keystroke dynamics change due to the transition from the emotion stimulus of to ( ≠ , = 1,2, … ,6, = 1,2, … ,7) was explored, in terms of exhibited statistically significant difference in the acquired data. Figures 3(a)-(c) present the valid values (<0.05) distributed in the combinations (by two) of the examined emotions for the cases of HT ( Fig. 3(a)), AT ( Fig. 3(b)) and PR (Fig. 3(c)), columnized in female-male groups (each row corresponds to a subject). From the latter, it seems that HT and PR data capture the effect of the emotion to the keystroke dynamic change better than the AT ones, as most of their upper triangles have valid values (Figs. 3(a), (c)), compared to the one of AT case (Fig. 3(b)). From the latter, it is clear that the transitions from ( = 1 − 4) to E6, from E2 to E5 and from E6 to E7 are only captured by the AT data, in both sex groups. The independence from the sex was also noticed in all cases, as the derived results show similar consistency in both male and female groups. Finally, a complementary character is noticed between the results from HT and PR data (see the distribution of the white cells in Figs. 3(a) and 3(b)), implying that features based on both HT and PR data could be used for the emotion categorization from keystroke dynamics.

3.2
Probing further within the i-PROGNOSIS context i-PROGNOSIS project supports the concept of unobtrusive capturing of the behavioral data from a smartphone, towards the early identification of PD, setting a placeholder for the results of the current study, despite of their intrinsic limitations (i.e., sample size, cultural issues, education level and smartphone use experience). The latter comply with the findings of other works, such as [6], who introduced a PD motor index related computer-based keyboard interaction. However, in the case of PD, the detection of typing patterns combined with the behavioral change analysis under emotional stimuli may help to identify early onset of PD, in a more holistic and intelligent way.

Conclusion
The proposed approach sets the natural interaction at the center of the data capturing mechanism, as they can be captured at home with a frequency much higher of the current standard of care, simultaneously addressing the problem of the artificial circumstances created during a consultation with a physician. The work described here provides a first step towards enabling a future perspective, by showing in the limited context of a typing test done in a home environment, the potential to track the interconnection of the emotional and the physical expression, showing great potential in transferring the notion from the healthy to the pathological cases, such as PD, as it is holistically explored in the H2020 i-PROGNOSIS project.