Conditioning on and controlling for variates via cumulative differences: measuring calibration, reliability, biases, and other treatment effects

tutorial.pdf contains a slide deck.

interactive.mov contains a QuickTime movie referenced by the slide deck in its penultimate subsection (for viewers who wish to see the video).

tutorial.mp4 contains an MPEG-4 recording of the presentation.

Abstract:

Conditioning on or controlling for variates helps compare only those who are comparable. That often means matching up people by their age or income, for example, and then looking at differences in results between people with similar ages or similar incomes. Yet that raises the question: how many people with exactly the same age or exactly the same income are in the data? If there are too few, then they will be unrepresentative. When there are too few, the randomness in the results fails to average away. This would seem to call for matching up people whose ages or incomes are only close, but not exactly the same. How close is “close”? Does it matter?

Choosing how close is “close” turns out to make all the difference. In many cases, the data can be made to support any arbitrary conclusion simply by manipulating how close is considered “close.” Conventionally, adjusting data for covariates such as age and income often ends up fudging the numbers, spinning facts or figures. Even the well-intentioned are susceptible to confirmation bias, cherry-picking, or otherwise making the data merely confirm expectations.

This tutorial shows how to avoid setting how close is “close.” With no parameters to tune, the presented graphical methods and scalar summary statistics cannot mislead, not even in principle. These methods are thus well-suited for the assessment of bias, fairness, reliability, the calibration of predicted probabilities, and other treatment effects. The analysis applies to observational studies as well as to randomized controlled trials.