Benchmark record: per-replicate predictions and goodness-of-fit results with the true risk known (Tabular foundation models are not a shortcut to calibrated clinical risk prediction)

The per-replicate predictions and test results behind the article "Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk" by Ebrahim Khaled Ebrahim. The study was pre-registered on OSF Registries (doi:10.17605/OSF.IO/8HMTZ); this record is the open benchmark planned in Section 14 of the protocol, so that a new calibration method or test can be run on exactly the same predictions, with the true risk known.

Simulation. For 58 scenarios (four true-risk shapes at event rates of 3, 10 and 30 % and development sizes of 200 to 3,000, and five stress families) and 1,000 random designs: for every evaluation record, the true risk, the outcome, the covariates, and the prediction and exact-null outcome of every arm (logistic regression, Firth logistic regression with intercept correction, ridge, GAM, LightGBM with and without Platt scaling, TabPFN-3.5, TabICLv2, Mitra-v2, and side arms), with the recalibration ladder; every row of the goodness-of-fit layer (ebrahim.gof 2.10.0) on the full evaluation sets and on nested subsamples of about 50 and 200 events; the oracle size study; and the true-risk and empirical calibration estimands of every fit.

Real data. Per-replicate predictions, test results and estimands for the Sepsis Survival Minimal Clinical Records and the PhysioNet/CinC Challenge 2012, whose licences allow it. For the International Stroke Trial and CRASH-2, only aggregate results are included; no row-level, per-patient, per-subsample or per-hospital result is deposited. The CRASH-2 results are published with the permission of the trial's Chief Investigators.

Also included: the tidy analysis tables, every summary table and figure of the article and its supplement, the computing-cost tables, the design tables, the registered protocol with its deviations log, a loader (benchmark_loader.py) and a data dictionary. The code that produced the record is archived separately (doi:10.5281/zenodo.23737378). Licence: CC BY 4.0.