Reflection Analytics in Educational Chatbot Rebo: A Field Study with Apprentices

We investigate a conversational agent for reflection for apprentices (Rebo). To develop Rebo’s adaptivity mechanism, we performed a feature analysis on 153 interactions with a non-adaptive version of the agent from previous work. We identified which linguistic features correlate with manual reflectivity coding in said dataset. Rooted in these reflection analytics, we implemented rules within Rebo to adapt its responses to learner statements, as well as provide feedback to the learner at the end of the interaction. We then conducted a 12-week field study with 19 apprentices who reflected with Rebo 130 times (1-14 interactions per apprentice). Rebo’s feedback and our manual coding correspond (classifications "reflective", "partly reflective", and "non-reflective" correspond to manually coded reflectivity scores of 9.8, 8.6, and 6.0 out of 10 points), and reflectivity is improved by adaptive follow-up questions in 93 interactions. Throughout the field study, engagement of apprentices did not decline, evidenced by the stable levels of reflectivity reached in conversations with Rebo. Finally, reflection competence increased statistically significantly (pre/post comparison Wilcoxon test: r=0.7537, z=-2.820, p<0.01), which indicates that apprentices learned to reflect through interaction with Rebo.