Replication package for "A Memorization Floor for LLM Refinement of Decompiled Code"
Authors/Creators
Description
We introduce a memorization floor: a within-item control separating what LLM refinement of decompiler output recovers from its input from what it recovers from its prior. Refine a function, then refine it again from an input whose identifiers have been destroyed, and measure what survives. Because the comparison is within-item, corpus difficulty cannot contribute; it costs twenty API calls. Applied to functions written after our analysis plan was committed, so no released model could have memorized them, it reports two things. Recovery is real: refined output sits +0.072 to +0.137 above an arm-matched permutation null built from its own output vocabulary. But it does not depend on the input we ablate: destroying the input's dataflow changes the naming gain by +0.001 (95% CI [-0.026, +0.026]), and removing type prefixes or permuting names changes it by no more. A second refiner from another vendor, registered in advance and given byte-identical inputs, reproduces this — twelve contrasts, two models, twelve nulls. Readability stays at ceiling throughout, so a reader is given no signal. The null is bounded, not absolute: contributions under 0.056 are invisible, and the ablation leaves operations intact, so naming from those alone remains a competing reading. No registered hypothesis was confirmed, and we report the five instrument failures behind that in full, including a reassembly harness biased against the treated arm and an equivalence checker we registered without checking it worked on our inputs.
This deposit is the replication package: corpus (both tiers), full pipeline, pre-registered analysis plan, the complete dated research log including every correction and deviation, per-run configuration and raw API responses for all runs including the 270-call cross-vendor replication, and a per-number provenance map (paper/NUMBERS.md) tying every figure in the paper to a committed producer. The manuscript itself is not included; see LICENSE-DATA.
Changes in v1.1.0: corrects the paper's account of why the registered equivalence checker did not discriminate (a control-dependence cascade, established in correspondence with the tool's authors) and reattributes that finding from the tool to our own procedure; adds an exploratory control-dependence sensitivity analysis; and reflects a restructure and retitle of the manuscript. No previously reported number changed.