Executing the Output: An Execution-Scored Benchmark of LLM-Generated Neutron Shielding Inputs Across Three Vendors

We benchmark large language models on a task whose ground truth cannot be argued with: write a complete OpenMC input script for a neutron shielding problem, run it, and compare the answer with a published reference. Five natural-language specifications, ordered by physical complexity, were given single-shot, with no tools and no error feedback, to eight frontier models from three vendors and one open-weight model: 135 samples, and every complete script was executed (four samples were refusals; three were cut off by our own token cap and not run).

Four observations stand out. At three samples per model per rung they are signals, not rates. First, whether code runs and whether it is right order models differently. GPT-6 Astra and Claude Opus 5.5 tie for the most scripts that run on the three non-trivial rungs (7 of 9 each), but Opus 5.5 converts four of them into physically correct answers and Astra two. Second, the failures that stopped scripts were API details, and some of them are durable. One pair of individually valid library calls, a thermal-scattering table attached to water and then water passed to a mixing routine, defeated 6 of 6 samples in the August matrix and 8 of 21 in the September one, in models from all three vendors and both Claude generations. Third, a newer model is not monotonically more usable. Claude Opus 5.5 refused all three samples of the conversationally worded rung, flagged by the provider's filter as "cyber" content, where Opus 5 and Sonnet 5.5 answered the same prompt. Fourth, most of these failures surfaced only because the scripts were run, and the same was true of our own instrument: we log 29 defects in it, and the most consequential were found only because a number came out wrong.

A one-turn repair arm, in which each failed script is shown its own error once, passes 11 of 21 frontier-model failures and clears the mixing defect in 7 of 7 decidable cases. An open-weight model passed nothing, single-shot or repaired.