Calibration over Accuracy: A Pre-Registered Live Benchmark of Three AI Reasoning Architectures on the 2026 FIFA World Cup
Description
When an AI system reports 70 percent confidence, is the number worth anything? We present inocta-bench, a pre-registered live benchmark that compares three AI reasoning architectures on probabilistic calibration, not accuracy: a single-call generalist (SOLO), a sequential specialist pipeline (PIPELINE), and a multi-model council (COUNCIL). All three forecast the 104 matches of the 2026 FIFA World Cup in public and in real time, with the pre-registered protocol, prompts, and scoring rules hash-locked before the first scored match. We compare reasoning architectures on calibration, using the strongest published baseline (Elo with a home dummy) as the comparison line, alongside the betting market's closing odds and a human forecaster. Across the full tournament, the council, an ensemble of three independent model families with a step that weighs their disagreement, posts the lowest expected calibration error of every machine tested (0.100, the human reference aside), below the single-call and pipeline architectures, the deterministic baseline, and the market's own closing line. It is the architecture whose stated confidence tracks reality most closely, and for an organization choosing an AI shape to trust with a forecast, it is the one to reach for. We report the margin honestly: on a chance-dominated event the bootstrap intervals overlap, so the lead is directional rather than resolved, and a second tournament would sharpen it. Football has a deep public betting market already pricing every match. A business forecasting its own pipeline has no such reference, and the calibrated AI probability is the closest thing to one it can build, which is what makes the calibration axis and the council on it the operational result of the study. Every prediction, cost, and the full dataset is published with the paper at the pre-registered repository (OSF 10.17605/OSF.IO/R5SBJ).
Files
inocta-bench-scientific-en.pdf
Files
(193.5 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:f6b1f7aff8bfd0cf56c5509b188ca752
|
193.5 kB | Preview Download |
Additional details
Related works
- Is supplemented by
- Dataset: 10.17605/OSF.IO/R5SBJ (DOI)