Who Wins the Window? Agentic Vulnerability-Response Benchmarks Measure Model Competence, Not Vulnerability Difficulty

A measurement-validity study of agentic CVE benchmarks. Given a public advisory and a patch diff, we measure a 'defender window': how fast a language-model agent gets a deployable defense in place versus a working reproducer, over nine patched N-day web vulnerabilities. Across four models the deployable-defense rate spans 0.15 to 0.76; the ordering is a property of the model, not of the CVE. The gap survives two scorer relaxations and a gate-threshold sweep, so it is not a scoring artifact; the sole model that favors the defender does so only on CVEs it can recall from memory; and laboratory reproduction cost does not track the real-world delay to a first public exploit. We argue that a defensively framed window metric of this kind confounds model competence and memorization with vulnerability difficulty. Pilot scale (n=9 CVEs); all vulnerabilities are patched N-days exercised in local isolated containers, and no working exploit is released.