Published November 12, 2026 | Version 1.0.0

Trust-Calibration Benchmark (TCB): dataset for "Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks"

  • 1. Clemson University
  • 2. Metacognition Institute

Description

The Trust-Calibration Benchmark (TCB): 60 yes/no items across five domains with gold labels and a per-item label audit, per-trial responses of nine models under three administration protocols, repeated-sampling, confidence-elicitation manipulation and transfer-study data, cached LLM responses, analyzers that reproduce every number in the paper offline, and the camera-ready paper materials. Adds the 54-item computable-gold battery used in the transfer study and the full 5,700-entry response cache from the project folder.

Dataset accompanying the paper Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks, 2026 ACM Conference on Human-AI Complementarity and Alignment (HCOMP 2026) (https://doi.org/10.1145/3834580.3838735).

Paper abstract. When several language models are wired together into a human-computation pipeline, the system routes, arbitrates, and escalates work according to how confident each model says it is, so a team builder needs a way to screen candidate models the way crowdsourcing has long screened human contributors: with a small, inspectable qualification test. We present such an instrument, a compact battery of yes-or-no questions across everyday domains on which a model reports an answer and a confidence, with every gold label independently audited. Our central contribution, however, is the audited harness and the interface properties it measures, not accuracy discrimination: qualification verdicts for model workers are only as valid as the harness that administers them. Evaluating a diverse panel of contemporary models under three administration protocols, we show that a naive harness, with a fixed token budget and an unaudited parser, manufactures failing workers out of competent ones, misreading a model that answers essentially every item correctly as badly inaccurate, answer-biased, and overconfident. Properly administered, every model in the panel proves admissible on accuracy for these common-knowledge items, and the differentiators that remain, and that transfer to disjoint downstream tasks, are interface properties: calibration of stated confidence, format discipline, and availability under budget. A controlled manipulation further shows the calibration axis is dissociable from accuracy. We release the items, the audited protocol, all per-trial responses under every protocol, and the scoring code, so the check, and the audit of the check, are each a single command.

Known limitations.

  • Small by design (60 items); intended as a readable qualification screen, not a leaderboard.

License: MIT License (https://opensource.org/licenses/MIT).

Files

croissant.json

Files (7.9 MB)

Name Size
md5:d0ed4106d786e8600fdfdaf0e828e1d7
1.7 kB Download
md5:982ac0ffceaedfd4b73017e67fb42a00
2.1 MB Preview Download
md5:c6565484a170be2d6d97904b9881ea23
1.1 kB Download
md5:88b2abbf78059baeab2be11c67b48129
5.7 kB Preview Download
md5:7c328637ac109b8ed1f536c5e75c708b
5.8 MB Preview Download

Additional details

Related works

Is identical to
Dataset: https://huggingface.co/datasets/ai4he/trust-calibration-benchmark (URL)
Is supplement to
Conference paper: 10.1145/3834580.3838735 (DOI)
Is supplemented by
Software: https://github.com/ai4he/qualification-by-calibration (URL)