JUSThink Alignment Analysis

1. Description

This repository contains tools to automatically analyse how participants align their use of task-specific referents in their dialogue and actions for a collaborative learning activity, and how it relates to the task success (i.e. their learning outcomes and task performance).

As a use case, it processes data from a collaborative problem solving activity named JUSThink [1, 2], i.e. JUSThink Dialogue and Actions Corpus data set that is available from the Zenodo Repository, DOI: 10.5281/zenodo.4627104, and reproduces the results and figures in [3].

In brief: 

1. JUSThink Dialogue and Actions Corpus contains transcripts, event logs, and test responses of children aged 9 through 12, as they participate in the JUSThink activity [1, 2] in pairs of two, to solve a problem on graphs together. 
2. The JUSThink activity and its study is first described in [1], and elaborated with findings concerning the link between children's learning, performance in the activity, and perception of self, the other and the robot in [2]. 
3. Alignment analysis in our work [3] studies the participants' use of expressions that are related to the task at hand, their follow up actions of these expressions, and how it links to task success.

2. Publications

If you use this work in an academic context, please cite the following publications:

• Norman*, U., Dinkar*, T., Bruno, B., & Clavel, C. (2022). Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do. Dialogue & Discourse, 13(2), 1–48. *Contributed equally to this work. https://doi.org/10.5210/dad.2022.201

• Norman, U., Dinkar, T., Bruno, B., & Clavel, C. (2021). JUSThink Alignment Analysis. In Dialogue & Discourse (v1.0.0, Vol. 13, Number 2, pp. 1–48). Zenodo. https://doi.org/10.5281/zenodo.4675070

3. Content

The tools provided in this repository consists of 7 Jupyter Notebooks written in Python 3, and two additional external tools utilised by the notebooks.

3.1. Jupyter Notebooks

We highlight that the notebooks up until the last (i.e. to test the hypotheses (tools/7_test_the_hypotheses.ipynb)) present a general pipeline to process event logs, test responses and transcripts to extract measures of task performance, learning outcomes, and measures of alignment.

1. Extract task performance (and other features) from the logs (tools/1_extract_performance_and_other_features_from_logs.ipynb): Extracts various measures of task behaviour from the logs, at varying granularities of the activity (i.e. the whole corpus, task, attempt, and turn levels). In later notebooks, we focus on one of the features to estimate the task performance of a team: (minimum) error.
2. Extract learning outcomes from the test responses (tools/2_extract_learning_gain_from_test_responses.ipynb): Extracts measures of learning outcomes from the responses to the pre-test and the post-test. In later notebooks, we focus on one of the features to estimate the learning outcome of a team: relative learning gain [4]
3. Select and visualise a subset of teams for transcription (tools/3_visualise_transcribed_teams.ipynb): Visualises the transcribed teams among the other teams in the feature space spanned by task performance and learning outcome, as well as the distribution of their number of attempts and turns.
4. Extract routines from transcripts (tools/4_extract_routines_from_transcripts.ipynb) (uses dialign to extract routines): Extracts routines of referring expressions that are "fixed", i.e. become shared or established amongst interlocutors.
5. Combine transcripts with logs (tools/5_construct_the_corpus_by_combining_transcripts_with_logs.ipynb): Merges transcripts with event logs to have a combined dialogue and actions corpus, to be processed e.g. to detect follow-up actions.
6. Recognise instructions and detect follow-up actions (tools/6_recognise_instructions_detect_follow-up_actions.ipynb):  Extracts verbalised instruction such as "connect Mount Basel to Montreux", and pairs them with the follow-up action that may match (e.g. if the other connects Basel to Montreux) or mismatch (e.g. if the other connects Basel to Neuchatel) with the instruction.
7. Test the hypotheses in [3] (tools/7_test_the_hypotheses.ipynb) (uses effsize to estimate effect size, specifically Cliff's Delta): Considers each research questions and hypotheses studied in [3] and generates the results in [3].

3.2. External Tools

1. dialign tool to extract routines, specifically Release 1.0 from dialign-1.0.zip:
   It extracts routine expressions that are "shared" among the participants from transcripts. 
   It is used as an external module (in accordance with its CeCILL-B License, see License).
2. effsize tool to compute estimators of effect size.
   We specifically use it to compute Cliff's Delta, which quantifies the amount difference between two groups of observations, by computing the Cliff's Delta statistic.
   It is taken from project DABEST (see License).

4. Research Questions and Hypotheses in [3]

• RQ1 Lexical alignment: How do the interlocutors use expressions related to the task? Is this associated with task success? 
  • H1.1:  Task-specific referents become routine early for more successful teams.
  • H1.2: Hesitation phenomena are more likely to occur in the vicinity of priming and establishment of task-specific referents for more successful teams.
• RQ2 Behavioural alignment: How do the interlocutors follow up these expressions with actions? Is this associated with task success?
  • H2.1: Instructions are more likely to be followed by a corresponding action early in the dialogue for more successful teams.
  • H2.2: When instructions are followed by a corresponding or a different action, the action is more likely to be in the vicinity of information management phenomena for more successful teams.

The RQs and Hs are addressed in the notebook for testing the hypotheses (i.e. tools/7_test_the_hypotheses.ipynb).

Acknowledgements

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 765955. Namely, the ANIMATAS Project.

License

The whole package is under MIT License, see the LICENSE file.

Classes under the tools/effsize package were taken from project DABEST, Copyright 2016-2020 Joses W. Ho. These classes are licensed under the BSD 3-Clause Clear License. See tools/effsize/LICENSE file for additional details.

Classes under the tools/dialign-1.0 package were taken from project dialign. These classes are licensed under the CeCILL-B License. This package is used as an "external module", see tools/dialign-1.0/LICENSE.txt for additional details.