There is a newer version of the record available.

Published October 12, 2026 | Version v5

One Percent of the Bank: Adaptive Item Selection Reads Targeted Contamination Up While the Whole Bank Barely Moves

  • 1. EchoBoost AI
  • 2. ROR icon National Tsing Hua University

Description

Adaptive testing is entering language-model evaluation as an efficiency device, and its low item exposure is offered as incidental contamination protection (Li et al., 2026b). That argument assumes untargeted leakage, yet a maximum-information test asks predictable items that anyone observing administrations can learn. Here we fine-tune a 1.5B-parameter model on 1% of a self-calibrated MMLU bank, chosen by exposure in a clean adaptive run, and score adaptive, random and whole-bank protocols on the same response vectors. A 41-item adaptive test then rises by 1.47 to 1.59 logits over its own clean reading while the whole bank reads slightly lower. Against random-item training approximately matched for the loss of clean-item accuracy, the adaptive reading is 1.40 logits higher (95% CI 1.10 to 1.69). Two further models show the same sign pattern. In the released logs of two adaptive LLM evaluators, the most-administered 1% of each instrument's largest bank takes 30–60% of administrations (Fluid at 100 items). No change confined to a targeted set can move the expected reading of a reader that is design-unbiased for bank accuracy by more than the set's share of the bank. At 100 items, several such readers come within 10% of the released readers' error while a set read from their own logs moves them about one-for-one with bank accuracy, against 3.4-fold or more for the released readers; at 41 items the comparison is undetermined. Adaptive-test logs therefore need a threat model for their observers, and adaptive scores a whole-bank reading beside them.

Other

Version 5 adds a replay of both released adaptive evaluators that sets their readers beside design-unbiased, model-assisted and fixed-subset readers on the same models, scoring each by its error against bank accuracy and by how far a set read from its own administration log can move it. At the longer of the two test lengths read, several readers come within a small margin of the released readers' error while staying far harder to move; at the shorter length, one evaluator's mean test length at its strictest stopping rule, the comparison is undetermined. It also corrects an appendix explanation of the person-fit result under maximum-information selection, and one software citation; the results of version 4 are unchanged.

This manuscript is a non-peer-reviewed preprint.

Files

one-percent-of-the-bank-v5.pdf

Files (1.1 MB)

Name Size
md5:dab4d41d3cc0fe0bd80db4d7460facfa
1.1 MB Preview Download