The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality

We introduce the Manokhin Probability Matrix, a two-dimensional diagnostic framework that classifies machine learning classifiers into four archetypes — Eagle (Type I), Bull (Type II), Sloth (Type III), and Mole (Type IV) — based on their calibration (Spiegelhalter Z-statistic) and discrimination (AUC-ROC) properties.

The framework separates two quantities that the Brier score conflates: reliability (calibration error) and resolution (discriminatory ability). Using a large-scale empirical study of 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 benchmark suite (150 dataset-fold combinations), we assign each classifier to a quadrant and derive actionable prescriptions:

- Eagles (CatBoost, TabICL, EBM, TabPFN, GBC, RF): well-calibrated, strong discriminators — ship as-is
- Bulls (XGBoost, LightGBM, HGB, NCA, TabM): strong discriminators, poorly calibrated — fix with Venn-Abers
- Sloths (LDA, TabTransformer, LR, SVM, AVG): well-calibrated but weak discriminators — retrain
- Moles (ExtraTrees, RealMLP, NB, MLP, KNN): poor on both axes — start over

Key contributions: (1) a formal proof that monotone calibrators cannot improve AUC (Proposition 1); (2) robustness analysis showing quadrant assignments are stable under absolute |Z|=1.96 thresholds; (3) empirical verification that AUC and Brier-resolution rankings yield 90.5% quadrant agreement (Spearman rho=0.948); (4) head-to-head Venn-Abers vs beta calibration comparison across all Bull models.

Code, data, and experimental logs: https://github.com/valeman/classifier_calibration