Optimizing Representations and Policies for Question Sequencing using Reinforcement Learning

This paper studies the use of Reinforcement Learning (RL) policies for optimizing the sequencing of learning materials to maximize learning as measured by expected future student performance. We conduct the training completely offline based on the publicly available dataset EdNet, a large-scale hierarchical dataset of diverse student activities collected by an existing online learning platform used by tens of thousands of students. We confront the challenges of offline RL through the construction of a student model in the form of a Markov Decision Process derived from observations in the dataset. A feature pool is created from the raw student logs, from which, different state representations are formed by sampling greedily using an iterative augmentation procedure to optimize the state-space. The paper explores the influence of the state representations on the performance of the RL agent and its robustness towards perturbations on the environment dynamics. We show that a larger, more complex representation constituting of more features, yields better policy performance. We also show that the policies derived from the larger representations are more robust towards perturbations in the expected environment induced by stronger and weaker learners. This work is a first step towards optimizing representations when designing policies for sequencing educational content that can be used in the real world.