Headroom Decision Procedure Examples: dataset for "Headroom Before Protocol Effects: A Decision Procedure for Human-Agent Evaluation"
Description
Synthetic worked example for the headroom decision procedure: worked-headroom, planning-headroom, severity-scaled and guard-sensitivity result tables, and the scripts that generate them.
Dataset accompanying the paper Headroom Before Protocol Effects: A Decision Procedure for Human-Agent Evaluation, Human-AI Coevolution Workshop (HAIC) at NeurIPS 2026 (https://openreview.net/forum?id=odOgsQYleZ).
Paper abstract. A team protocol has no interpretable benefit without a matched no-protocol baseline and decision-relevant headroom. We present a reporting method that tests admissibility, identifies the marginal contrast, evaluates opportunity, and maps paired uncertainty to an operational action. It distinguishes exact headroom on a fixed battery from uncertain population headroom used in planning. A published meta-analysis of human-AI experiments illustrates why the estimand matters: the same evidence supports augmentation relative to humans alone but contradicts synergy relative to the better constituent. Neither comparison identifies a coordination protocol without a matched no-protocol team. An executable synthetic application further shows that a promising point estimate can remain operationally unresolved once paired uncertainty is considered, and that near-perfect pilot performance does not by itself make a modest benefit impossible. Several inherited multi-model summaries fail the method's artifact gates, so they do not validate it. This is a bounded position and reporting contribution rather than an adoption-ready standard. Human-facing claims still require direct human outcomes, and we introduce no new human experiment.
Known limitations.
- All tables are synthetic illustrations; the five earlier pilots audited in the paper have no retained raw data.
License: Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Source code files are released under the MIT License.
Files
croissant.json
Files
(44.8 kB)
| Name | Size | |
|---|---|---|
|
md5:f66e8cfcbe26c7be32b917f83bcf29c3
|
1.2 kB | Download |
|
md5:1e00c423c77ead6383bcc5093c122091
|
20.4 kB | Preview Download |
|
md5:2abe9b6f418c6ee6036a095e8e8352ac
|
16.1 kB | Preview Download |
|
md5:dce724e3e39e198fe91dc9c2046a1c5d
|
1.6 kB | Download |
|
md5:ad05064be0c2e4e93dcf60d256139738
|
5.5 kB | Preview Download |
Additional details
Related works
- Is identical to
- Dataset: https://huggingface.co/datasets/ai4he/headroom-decision-procedure (URL)
- Is supplement to
- Conference paper: https://openreview.net/forum?id=odOgsQYleZ (URL)