TabSets: calibration and test probabilities for set-valued evaluation of tabular classifiers

The calibration and test probabilities of every run of a tabular classification benchmark, released so that prediction sets and their metrics can be recomputed without refitting a model. A conformal set is a function of four arrays: the class probabilities and labels on a calibration split, and the same on a test split. This deposit is those arrays, together with the software that reads them.

The benchmark comprises 218 classification datasets and 22 models from four families, among them 13 tabular foundation models, each evaluated with the same ten seeds. Of the 218, 178 are real datasets and 40 are a synthetic block generated for this benchmark; three of the real ones are Bayesian-network resamplings of small classical datasets and are much larger than their seeds; of those, only one has its own seed in the collection, so that pair is not independent of itself while a rank test treats datasets as independent. Dataset size does not indicate provenance: the largest datasets in the collection are real. Coverage is not uniform and the per-model dataset count is reported for every cell. Eleven datasets carry more than ten classes, and the in-context models measured on them cannot enter them: one refuses and names the limit, another builds a ten-output head and fails inside a kernel. That is a capability limit observed on the models that were run rather than a property claimed of the family, and it is not missing work. One foundation model is measured on the original datasets only. Anything computed across models should be computed on a complete submatrix: the datasets on which every model being compared has all ten seeds.

The calibration rows are held out of fitting, of tuning, of early stopping and of the in-context data, and each cell records which rows they were. The code that does it is included, with the line of each step documented.

Probabilities are float64 and are not rounded: a prediction set is a comparison against a threshold, and a point sitting on that threshold moves if the value it is compared to is shortened.

The data are released under CC BY 4.0. The software included in the source archive is released under the MIT licence, as stated by the LICENSE file it contains. The datasets themselves are not redistributed: each cell carries the row indices into its source dataset, and the synthetic datasets regenerate from their seeds with an included script. Model weights are not redistributed; every checkpoint is named in the model table.