Understanding Why: Statistical Techniques to Infer Causality are Underused in Computing Education Research

A common thread through education research is asking questions about how
treatments applied to students affect their education, career, and other
outcomes. For example: Will being taught in a certain way increase students'
learning? Will taking a computer science course lead to higher job satisfaction
in the future? Are remedial programs serving their intended purpose?

The most robust way to establish the causal effects of treatments is
to perform randomized controlled trials. However, in the context of
education, it would frequently be unethical or logistically
impossible to simply assign students to take a certain class or
participate in a certain program for the purpose of research.

As a result, we often take advantage of natural experiments or
quasi-experiments. In such situations, the traditional method of
analysis is to look at the correlation between the treatment and
outcome variables. However, this doesn't tell us whether the outcome
was caused by the treatment, as there are almost always substantial
selection biases or confounding variables.

In the past few decades, advanced statistical methods have been
developed to analyze the assignment of subjects to treatments as if it
was random, allowing us to deduce the causal effect of the
treatment. Such methods include difference-in-differences,
instrumental variables, and regression discontinuity design.

In this paper we argue that these methods have been underused in computing
education research. To encourage their increased use, we describe the methods
and present selected examples of education studies where they have allowed
researchers to bridge the gap from correlation to causation.