CRLN-AIE-001 v0.2.2: What an instrument with a measured ceiling can and cannot tell you about frontier models (errata: model counts)

Version 0.2.1 (29 September 2026). A correction: the held-out evaluation set was not private.

Correction only. No item, response, score, analysis or file of version 0.2 changed. This version carries the eight version 0.2 files unchanged and adds one file, CRLN-AIE-001-CORRECTION-2026-09-29.md.

What version 0.2 said. Version 0.2 (10.5281/zenodo.22728905) said that 57 further core-set scenarios "have been withheld from the sitemap and prerendered pages to serve as a future private evaluation set". Its section 5.5 reported that 23 hard-tier items, all in that set, did not lower the model ceiling.

What is corrected. The answers to those 57 scenarios could be read by anyone, without an account, through three routes: the Practice Lab app code, which any browser downloads and which carries every option's score; two answer-key columns of a public database table; and a criterion sentence in the same table that ended with the text of the best option. The two table routes were closed on 29 September 2026. The first route is still open. Until the app scores answers on the server, no held-out result of ours can be claimed as free of contamination. In the logs we have reviewed, an automated crawler operated by an AI company requested the table on five separate days between 18 and 23 September 2026.

Consequences. The version 0.2 held-out set is retired as a private evaluation set. The section 5.5 hard-tier comparison is withdrawn as a measurement on unseen items, and will be repeated on a new held-out set of 60 scenarios, written from scratch, keyed by a practitioner, and never sent to a browser. All other results in sections 4 and 5, and the published response dataset, rest on 114 items that were public by design and are not affected.

Version 0.2 remains published unchanged, because Zenodo records are immutable. The full correction, with every figure and how it was found, is in CRLN-AIE-001-CORRECTION-2026-09-29.md.

The version 0.2 description follows, unchanged. Its sentence on the twenty-three hard-tier items is withdrawn as described above.

Version 0.2 of the CRLN AI competency evaluation protocol, correcting the central result of v0.1.

Correction. v0.1 compared an unweighted model mean against a response-weighted human mean and concluded that a frontier model and competent practitioners were indistinguishable. Those are different quantities, differing by 0.093 on the same data because popular items are easier items. Compared like with like the conclusion reverses.

Method. Seven models across three vendors (Anthropic, Google, OpenAI) evaluated on 114 clinical trial conduct decisions that human practitioners had each attempted at least five times. Five presentations per item with options re-randomized, a fixed-order control condition to separate position sensitivity from retest noise, and every one of 3,990 responses retained with the option permutation as presented and the vendor-reported model version.

Results. The instrument discriminates. Two models are statistically indistinguishable from practitioners, four sit modestly above, one well above. Compliance depended on the order the options were presented in on 1 to 12 items of 114, and that count tracks capability. Position sensitivity accounts for roughly 80 percent of within-item variation in one model and 65 percent in another; to our knowledge this is the first partition of position and retest components within a single MCQA benchmark design.

Reported against ourselves. The instrument cannot currently detect whether models flip between compliant and non-compliant, because its own ceiling sits above the range where that effect is detectable. Twenty-three items authored specifically to lower that ceiling did not work. The author's own responses have been excluded from the human baseline. The human comparison group is 52 practitioners, mostly early career.

The full response dataset accompanies this version.