Using a Randomized Experiment to Compare Mastery Learning Thresholds

Many modern adaptive learning and intelligent tutoring systems implement the principles of mastery learning, where a student must demonstrate mastery of core prerequisite material before working on subsequent content within the system.  Typically in such cases, a set of rules or algorithms is used to determine if a student has sufficiently mastered the concepts in a topic.  In a previous work, we used a quasi-experimental design to investigate the relationship between two different mastery learning thresholds and the forgetting of the learned material.  As a follow-up to this initial study, in the present work, we analyze the results from a randomized experiment---or A\B test---directly comparing these two mastery learning thresholds.  These latest results seemingly agree with those from our initial study, giving evidence for the validity of the conclusions from our original quasi-experiment.  In particular, we find that although students who learn with the higher mastery threshold are less likely to forget the learned knowledge, over time this difference decreases.  Additionally, we build on these analyses by looking at how the relationships between the mastery thresholds change based on other factors, such as the amount of struggle students experience while learning or the subject matter being covered.