Aasis Nonverbal rating scale 2024

General remarks on using the nonverbal scale 

• The scale was developed for research purposes of the Aasis project (read more).

• Research interests of the Aasis project include automated speaking assessment, verbal and non-verbal features of spoken interaction, L2 Finnish automated feedback. 

• The scales were developed for assessing adult L2 Finnish learners’ dialogue performances, targeting A2-B2 level speakers, tasks available: von Zansen, A. 2024. Aasis speaking tasks for Finnish L2 learners (2024). Zenodo. https://doi.org/10.5281/zenodo.11046535  

• The scale is not intended to be used separately, but as a supplement to the linguistic scales (see e.g. Council of Europe 2020, appendix 3 p. 183–185). 

• In addition to human ratings, the Aasis project investigates L2 learners’ interaction e.g. with the help of automatic tools, the computer-joystick method and ELAN transcripts. 

Documentation on the development of the scale (previous versions attached) 

Version1  

File attached: version1_aasis_nonverbal_rating_checklist_2023_11_06 

• This version was drafted based on nonverbal communication theories (Giri 2009), the CEFR (Council of Europe 2001) and previous research available investigating nonverbal behaviour e.g. in clinical interactions (Blanch-Hartigan et al. 2018).  

• The nonverbal scale was intended to accompany the holistic and analytical linguistic scales available in the CEFR, see e.g. appendix 3 of the Companion volume) 

• The idea was that the human raters, experts in language assessment, would assess whether the listed nonverbal behaviours support or reduces interaction or are missing from the dialogue performance. They also would have possibility for commenting on the speaker’s behaviour. 

• This version was submitted for an ethical review before starting the research, resulting in a favourable statement from the ethics committee.  

• This was also the basis for the design of the study's privacy notice and the information to be given to the participants. 

• However, at the planning stage, it was not yet known which nonverbal features the project would prioritise.  

• Moreover, it was not known which features would be analysed with algorithm-based tools, or transcribed with ELAN software. 

Version2 on paper and online 

Files attached: version2_aasis_nonverbal_rating_scale_2024_09_16 (scale on paper), version2_Webropol_implementation_aasis_nonverbal_rating_scale_2024_09_16 (scale online) 

• This version was drafted in autumn 2024 for the piloting stage involving eight external raters in addition to the language assessment experts of the Aasis project team. 

• At this stage, the project already knew to some extent, what human transcribers using ELAN as well as joystick coders are focusing on, and which features could eventually be calculated automatically (e.g. facial expressions, action units). It was taken into account, what the phonetic research of the project could focus on. 

• Based on information listed above, the project team prioritized nonverbal features that were selected for human raters to assess from dialogue performances, in addition to the CEFR-based linguistic dimensions (i.e. proficiency level, range, accuracy, pronunciation, fluency, interaction). 

• The idea was that the raters would assess both the frequency and significance of each observed nonverbal feature, based on each speaker’s performance in a dialogue speaking task. In the table, the first column lists the features (facial expressions, eye-contact with interlocutor, head movements, verbal backchannelling, hand movements, body movements, laughing, self-regulation), the second column is the “observed amount” (on a scale: “none” – “a lot”), and the third column is for assessing the “significance for interaction” (on a scale "made interaction difficult” – “supported interaction”, "had no affect"). 

• It should be noted that, regarding this version, the project team discussed the number of the scale steps (i.e. 1-4) and if the steps should be numbered / labelled or just the end points.  

• However, as the ratings are collected using online tools (Webropol online survey for the pilot, final implementation as a Moodle course) which do not always allow for all desired question types. As can be seen from the online implementation, questions 10 – 11 and 16 – 17  on the Webropol survey, the scale points had to be numbered due to technical limitations. 

Version 3 (on paper and online) 

Files attached: version3_aasis_nonverbal_scale_2024_11_05 (on paper), version3_Moodle_implementation_aasis_nonverbal_scale_2024_11_05 (online), version3_aasis_nonverbal_scale_2024_11_05_translatedIL 

• This version was drafted based on experiences from the pilot rating round in autumn 2024, after which the raters responded to a post-test survey and were interviewed and their rating performances were analysed.  

• Results posed challenges in terms of rater consistency, the scale was perceived as unclear and the raters perceived the workload and the assessment task requested as too cumbersome. 

• Therefore, the version 2 scale was significantly changed and returned closer to the version 1 checklist-type scale.  

• The main changes included: 
  • stating the purpose of the scale (research) and defining how the project sees the concept of language proficiency
  • the inclusion of assessing both speakers on the same scale, since the assessment concerns nonverbal aspects of dialogue performance
  • removing the scale for observed frequency (not asking how much)
  • asking the raters to focus on assessing if the features support or hinder the interaction
  • reducing the nonverbal feature list by removing self-adaptors and body movements, combined sounds
  • adding the aspect of a nonverbal feature missing
  • two questions were added in the end of the scale concerning the whole dialogue, i.e. both speakers and their interaction
  • one question was added asking if the speaker’s observed nonverbal behavior affected the (linguistic) proficiency rating given by the rater 

• This version had several revisions, involving iterations based on internal piloting among the project team members. Finally, it is to be noted, that for the technical implementation (online version) in Moodle, wordings of the paper-based scale had to be shortened and rephrased. Therefore, the paper and online versions of the scales are not entirely similar. 

Acknowledgements

This scale was developed for the Aasis project research (Automatic assessment of spoken interaction in a second language, Research Council of Finland 2023–2027) at the University of Helsinki (grant number 355586). In addition to my colleagues at the University of Helsinki, I thank the researchers of the consortium from Aalto University (grant number 355587) and the University of Jyväskylä (grant number 355588) for their insightful comments:

• During the piloting stage, the Webropol interface was drafted with Maria Kautonen, who integrated the non-verbal scale as part of the dialogue assessment view in Webropol.
• The Moodle interface was drafted with Ekaterina Voskoboinik, who was in charge of the technical implementation of the scales in Moodle.
• Research assistant Ilona Lähteenmäki translated the rating scale (version 3) into English for publishing purposes.
• I thank professors Ari Huhta and Raili Hilden for their insights and guidance in creating the scale.