Enhancing Teaching and Learning with Conversational AI Agents: Pilot Testing and Impact Assessment in Real Classroom Settings

This paper presents the evaluation of the educational impact of two conversational AI applications, one designed to support student selfstudy (Study Buddy) and the other to assist teachers in instructional planning (Teacher Mate) within the context of the AI4EDU project.
The evaluation was conducted during the second piloting phase across four European countries in real classroom settings. In Greece,
the evaluation involved 148 participants, 3 teachers and 145 students. The evaluation protocol included training workshops, pre- and
post-intervention surveys, system interaction data analysis, and classroom-based implementation. Greek participants were assigned to
experimental and control groups for comparative analysis. During a multi-week period, students used Study Buddy during self-study both
at school and at home, while teachers integrated Teacher Mate in lesson planning and student monitoring tasks. Quantitative data from
motivation and engagement questionnaires and interaction logs were triangulated with qualitative data from semi-structured interviews
and classroom observations. The analysis provided insights into the impact of AI tools on users’ educational performance, engagement
and motivation and helped identify possible drawbacks and key aspects of AI implementation that can be exploited to demonstrate the
issues of ethics, transparency and inclusive use of AI in real educational settings. In general, participants from both groups expressed
favorable opinions of the AI-driven applications. The combination of our study’s empirical results emphasizes the need to align AI tools
with user needs, deliver customized training, and cultivate trust and openness of AI applications. These initiatives are critical for fostering
inclusive, stimulating, and impactful AI-supported educational settings.