SANN: A Subtree-based Attention Neural Network Model for Student Success Prediction Through Source Code Analysis

Program code analysis is an important component in several kinds of intelligent educational systems for CS education. The ability to analyze and understand student-written code enables these systems to assess students learning and understanding of the essential programming constructs. The central issue of code analysis is a concise code representation, which enables machine learning (ML) approaches to automatically process and analyze students’ code. Not surprisingly, research on developing embedding approaches that map programming code into vector representations has surged in recent years. While embedding techniques are quite well-established in the field of Natural Language Processing, the structural nature of programming codes requires special adaption of such techniques to preserve crucial semantic information. In this paper, we propose a Subtree-based Attention Neural Network (SANN) to represent novice programmer’s code. Unlike existing models working on entire Abstract Syntax Trees or context paths, SANN splits each tree into a sequence of subtrees and uses an Attention-based Neural Network to encode important information from student codes in vectors while giving more attention to the important subtrees. To compare our model’s effectiveness in analyzing novices’ programming codes, we performed two program classification tasks: detecting correct student submissions from a single assignment and across multiple assignments. We compared SANN with Code2Vec and some well-established traditional ML models including Support Vector Machines, K-Nearest Neighbors, and Extreme Gradient Boosting. Experimental results suggest that SANN outperforms other ML models, including the state-of-the-art Code2Vec model in both student program classification tasks.