The Paradox of Overfitting

Measuring the Self-Consistency Index: A Preliminary Protocol Validation Study (Version 3)

Companion paper to The Paradox of Overfitting, Version 2 (Revised).

The Self-Consistency Index (SCI) quantifies the degree to which a neural system prioritizes internal coherence over external perturbation. This study closes a recognized gap in the parent framework — the absence of an operational measurement protocol — by specifying a concrete, reproducible procedure: length-normalized last-token attention entropy as the internal consistency proxy (C_int), and next-token KL divergence under query perturbation as the external sensitivity proxy (S_ext), with SCI defined as their ratio.

The full pipeline is validated end-to-end on a 357K-parameter character-level transformer trained from scratch, using a 2×2 within-subject design analogous to the Role-Oriented versus Task-Oriented prompt (RoP/ToP) asymmetry reported by Wang et al. (2026, arXiv:2602.04294). The framework's distinctive interaction prediction was confirmed: few-shot demonstrations significantly increased SCI under an identity-style prefix (p = 0.003) and significantly decreased it under a task-style prefix (p = 0.020), with a significant interaction (p = 0.004) carried almost entirely by external sensitivity. The main-effect prediction SCI(RoP) > SCI(ToP) was significantly reversed (p = 0.008), sharpening the framework's empirical content: the RoP/ToP differential, if real in large models, must be semantic and must overcome an opposing structural baseline.

Version 3 changes. This version supersedes the initial release following an independent code review that reproduced the original results bit-identically and then identified two pipeline defects (a silent context-window truncation affecting one cell of the design, and perturbations not matched across conditions) and one reporting gap (the paired statistical procedure was not included in the released code). All three are corrected, all statistics recomputed with paired sign-flip permutation tests (50,000 resamples), and the changes documented in Section 4.3 of the paper. The corrections strengthened the confirmed prediction rather than weakening it.

Contents: paper (PDF), model implementation (model.py), training script (train.py), corrected measurement pipeline including all statistical tests (experiment_v2.py), and raw per-query results (results_v2.json). All results are reproducible from fixed seeds on commodity CPU hardware in under five minutes.

Keywords: self-consistency index · attention entropy · perturbation robustness · prompt-based defenses · protocol validation · transformer interpretability