Automatic Scoring of Cognition Drawings

This project contributes to the standardisation of cognitive assesments in large-scale surveys by developing
an automated process of scoring cognition drawings using machine learning.

The correct drawing of figures, like cubes or clocks, is frequently used as an indicator in cognitive assessment
protocols (e.g. the Montreal Cognitive Assessment; MoCA). Generally these tests are administered in a
clinical setting as a screening tool and used in conjunction with additional individual tests for diagnosing
cognitive deficiencies, like dementia and related diseases. Scoring of the drawings is done by trained
clinicians.

More recently cognitive drawing tests have been applied in large scale survey studies (e.g. SHARE, HCAP,
HRS, ELSA). While these tests have been adapted from clinical cognition assessments, in this setting
they serve the purpose of population based estimates, like e.g. prevalence of dementia symptoms, rather
than individual diagnoses. Collecting these data at a large scale in an interviewer adminstered survey
context (in the case of SHARE this means 10,000s of interviews conducted by 1,000s of interviewers
repeatedly for multiple waves), calls for a high degree of standardisation in order to measure consistently.
This seems particularly challenging when considering that survey interviewers have generally no clinical
training regarding the diagnosis of cognitive problems.

Against this background we collected the actual physical drawings from SHARE wave 8 in several countries
together with the interviewers’ scorings. We want to use these to achieve two aims:
1. Assess the quality of scoring done by the survey interviewers: Do they score consistently? Are there
interviewer effects? Do they adhere to the scoring rules? Do the scoring rules work?
2. Use the collected drawings and scorings to train a machine learning model automatically scoring
cognition drawings that can be applied in future waves of data collection. Ideally this will do
away with the burden for the interviewers of having to score the drawings manually, even if some
uncertain cases might have to be scored manually back at the office. At the same time we hope to
improve data quality by having a more standardized and consistent approach.

The current paper is a proof of concept based on the first 2,000 drawings that were already scanned
and prepared for analysis. We actually start with the second bullet point, as a fairly well performing
prediction model is an essential tool for assessing the interviewers’ scoring performance. We will therefore
present the first results of our modelling approach using convolutional neural networks and verify to what
extend we can reproduce the original scorings using a fully automatic approach