Latent program modeling: Inferring latent problem-solving strategies from a PISA problem- solving task

Response process data have the potential to provide a rich description of test-takers’ thinking processes.
However, retrieving insights from these data presents a challenge for educational assessments and educational
data mining as they are complex and not well annotated. The present study addresses this
challenge by developing a computational model that simulates how different problem-solving strategies
would behave while searching for a solution to a Program for International Student Assessment (PISA)
2012 problem-solving item, and uses n-gram processing of data together with a naive Bayesian classifier
to infer latent problem-solving strategies from the test-takers’ response process data. The retrieval of
simulated strategies improved with increased n-gram length, reaching an accuracy of 0.72 on the original
PISA task. Applying the model to generalized versions of the task showed that classification accuracy increased
with problem size and the mean number of actions, reaching a classification accuracy of 0.90 for
certain task versions. The strategy that was most efficient and effective in the PISA Traffic task evaluated
paths based on the labeled travel time. However, in generalized versions of the task, a straight line strategy
was more effective. When applying the classifier to empirical data, most test-takers were classified as
using a random path strategy (46%). Test-takers classified as using the travel time strategy had the highest
probability of solving the task. The test-takers classified as using the random actions strategy
had the lowest probability of solving the task. The effect of (classified) strategy on general
PISA problem-solving performance was overall weak, except for a negative effect for the random actions
strategy (β ≈ −65, CI95% ≈ [−96,−36]). The study contributes with a novel approach to inferring
latent problem-solving strategies from action sequences. The study also illustrates how simulations can
provide valuable information about item design by exploring how changing item properties could affect
the accuracy of inferences about unobserved problem-solving strategies.