Comparing The Impact of Generative AI Chatbots on Students' Reflective Thinking in Learning Programming

This full paper explores how a specifically tuned chatbot can support students' reflective thinking. Research in reflective thinking is relatively uncommon in computer science education. In addition, implementing reflective thinking in teaching can be challenging for academics due to the need for student personalized support. Students also often view reflection as an additional assignment rather than a way to improve their learning. Despite these challenges, programming courses can benefit from supporting student learning through specific reflective activities to foster students' reflective thinking. Supporting students' reflective thinking in programming can be achieved by triggering the process through reflective dialogue. A generative AI-based chatbot can facilitate individual and contextual conversations. This study investigates the potential of tuning large language models to trigger reflective dialogue as a learning tool. This study examined the interaction between eleven participants and the use of generative AI learning tools in a controlled laboratory setting, with one-on-one interactions between the researcher, participant, and chatbots. An experimental research design was conducted, including a control group (ChatGPT) and an experiment group (our proposed chatbot). A mixed-methods approach was employed, combining both quantitative data from pre-test and posttest questionnaires using Kember's Reflective Thinking Scale with qualitative data from observations and interviews to assess participants' reflective thinking. The results did not show any statistically significant differences in scores on the Kember's Reflective Thinking Scale. However, thematic analysis of the qualitative data revealed that participants in the experimental group demonstrated more reflective thinking behaviours.