Trust-Calibration Benchmark (TCB): dataset for "Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks"
Description
The Trust-Calibration Benchmark (TCB): 60 yes/no items across five domains with gold labels and a per-item label audit, per-trial responses of nine models under three administration protocols, repeated-sampling, confidence-elicitation manipulation and transfer-study data, cached LLM responses, analyzers that reproduce every number in the paper offline, and the camera-ready paper materials. Adds the 54-item computable-gold battery used in the transfer study and the full 5,700-entry response cache from the project folder.
Dataset accompanying the paper Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks, 2026 ACM Conference on Human-AI Complementarity and Alignment (HCOMP 2026) (https://doi.org/10.1145/3834580.3838735).
Paper abstract. When several language models are wired together into a human-computation pipeline, the system routes, arbitrates, and escalates work according to how confident each model says it is, so a team builder needs a way to screen candidate models the way crowdsourcing has long screened human contributors: with a small, inspectable qualification test. We present such an instrument, a compact battery of yes-or-no questions across everyday domains on which a model reports an answer and a confidence, with every gold label independently audited. Our central contribution, however, is the audited harness and the interface properties it measures, not accuracy discrimination: qualification verdicts for model workers are only as valid as the harness that administers them. Evaluating a diverse panel of contemporary models under three administration protocols, we show that a naive harness, with a fixed token budget and an unaudited parser, manufactures failing workers out of competent ones, misreading a model that answers essentially every item correctly as badly inaccurate, answer-biased, and overconfident. Properly administered, every model in the panel proves admissible on accuracy for these common-knowledge items, and the differentiators that remain, and that transfer to disjoint downstream tasks, are interface properties: calibration of stated confidence, format discipline, and availability under budget. A controlled manipulation further shows the calibration axis is dissociable from accuracy. We release the items, the audited protocol, all per-trial responses under every protocol, and the scoring code, so the check, and the audit of the check, are each a single command.
Known limitations.
- Small by design (60 items); intended as a readable qualification screen, not a leaderboard.
License: MIT License (https://opensource.org/licenses/MIT).
Files
croissant.json
Files
(7.9 MB)
Additional details
Related works
- Is identical to
- Dataset: https://huggingface.co/datasets/ai4he/trust-calibration-benchmark (URL)
- Is supplement to
- Conference paper: 10.1145/3834580.3838735 (DOI)
- Is supplemented by
- Software: https://github.com/ai4he/qualification-by-calibration (URL)