Published November 6, 2026 | Version 1

Reproducibility package for: Download the Right Quantized Code LLM by Budgeting Gigabytes and Joules

Authors/Creators

  • 1. Independent Researcher

Description

Reproducibility package for the paper "Download the Right Quantized Code LLM by Budgeting Gigabytes and Joules". It regenerates every measured number in the paper from the logs of the measurement runs. It runs no model and holds no model files.

What it holds:
- The main grid: for each of 24 Qwen2.5-Coder model files on HumanEval+ and MBPP+, the raw and sanitized samples, evaluation results, per-task metrics, power log and run metadata, with the idle baselines.
- The paired energy repeat: five alternating pairs of sessions, with the logs of each block.
- The server log of the main grid, the source of file digests, bits per weight and load buffers.
- The serial correctness vectors and the four registry manifests the analysis reads.
- The grid harness, the runner of the paired repeat, the analysis scripts, a file with one macro per printed number, and the two figures.
- SHA-256 lists, notes on the files normalized for release, and the record of a clean-room test.

How to regenerate: unpack the archive, then run these from the package root.

python3 analysis/p1_numbers.py
python3 paper_greens/figures/make_figures.py --check

The first command prints 131 pass, 0 fail, 5 explained deviations of 136, and the second prints 0 mismatches. Tested with Python 3.9.6, numpy 2.0.2 and matplotlib 3.9.4 on macOS 26.5.1 (arm64). README.md also gives the commands that rebuild the paired repeat's summaries from its raw logs, and it says what the hash checks prove and what they do not.

Companion study: "Broken on Arrival" (arXiv:2609.05881). The two papers share the correctness observations of these 24 configurations.

Licences: scripts carry MIT. Data and documentation carry CC BY 4.0. Model-generated solutions come from Qwen2.5-Coder models and benchmark task text comes from EvalPlus, and those stay under their own terms.

Files

Files (138.1 MB)

Name Size
md5:bcf6e94c77d636e3d70886337cf31256
138.1 MB Download

Additional details

Related works

References
Preprint: 10.48550/arXiv.2609.05881 (DOI)