EcoCompute: LLM Inference Energy on NVIDIA RTX 4090 (Ada) — FP16/NF4/INT8, 0.5B–7B, direct NVML, with paired perplexity and INT8 repeatability

Direct NVML energy measurements of decode-phase LLM inference on a single NVIDIA GeForce RTX 4090 (Ada, 24 GB), for five open models from 0.5B to 7B parameters in FP16, NF4 and INT8, now with a measured run-to-run coefficient of variation.

Version 3 adds a third session (2026-08-20) in which all five INT8 configurations were run three times each on the same instance as the 2026-08-19 session. This answers the question the earlier single-trial files could not: how much of the disagreement between our two sessions is measurement noise?

1. Repeatability. The coefficient of variation of the FP16-normalised energy delta is 0.6% to 3.9% across the five sizes (energy CV 0.3% to 2.0%, FP16-baseline CV 0.1% to 2.5%), and the n=3 means land on the single trials published in v2 (largest deviation 2.3 standard deviations). Fifteen raw reports and their logs are included.
2. The cross-session gap is 30x to 50x the run-to-run noise. The same five INT8 configurations measured 100 to 140 percentage points higher in the 2026-08-19 session than in the 2026-07-24 session (for example TinyLlama-1.1B: +146.1% against +307.1%). That cannot be attributed to sampling. The decomposition constrains any explanation: INT8 energy per token rose 1.41x to 1.58x at unchanged GPU package power (74 W to 91 W against roughly 76 W), so it is entirely a throughput effect, and at the small sizes the later FP16 baseline was also lower (0.5B: 0.78x), which is what stretches that size's ratio to 2.0x. The leading hypothesis is the LLM.int8() kernel path, which changes with the torch build - and no two sessions in this record ran the same one: session A (2026-07-24) is recorded with torch 2.4.1+cu121 on Python 3.8, sessions B and C (2026-08-19 and 2026-08-20) with torch 2.5.1+cu121 on Python 3.10, and the EcoCompute container image pins torch 2.13.0, which none of these sessions used. The hypothesis is untested. The cheapest control is one 1.1B INT8 run on the later instance with torch pinned back to 2.4.1+cu121; running the image itself as a control is not possible on that rented instance, which cannot nest Docker. This also corrects v2, which listed the CUDA build as an excluded cause.
3. Absolute joules are the least stable quantity in this record. Between two consecutive days on the same host, absolute energy per token drifted 12% to 17% lower for INT8 and FP16 alike, while the normalised delta held. Compare deltas measured against a baseline from the same run; do not compare absolute J per 1k tokens across days, let alone across machines.

Retained from version 2: the erratum on v1's derived vs_fp16_pct column (no measurement was changed), and the 2026-08-19 session pairing each of ten measured configurations with a same-run teacher-forcing perplexity for both the quantized model and its FP16 baseline, scored on a fixed vendored public-domain text (SHA-256 22ac091a6383740d30f8e41ae144032c873c772db5e1c901112c1c330fdc5504) after the power sampler stopped. The two axes disagree: INT8 costs almost no perplexity (+0.5% to +1.2%, except Qwen2.5-3B at +3.8%) but 106% to 595% more energy, while NF4 saves energy above roughly 1.4B parameters yet damages the language model (+9.5% to +27.6% perplexity). They are reported side by side and deliberately not combined into a single "quality-adjusted energy" index, which would hide an arbitrary weighting.

Scope and limitations. Every value is a real hardware measurement (basis = measured, measurement_source = direct-nvml). GPU-package power only, sampled by NVML at 10 Hz and integrated over the decode loop: this is not wall-plug power and excludes CPU, DRAM, PSU losses, PUE and CO2e. n = 1 per configuration in the 2026-07-24 and 2026-08-19 sessions; n = 3 for the five INT8 configurations of the 2026-08-20 session, which is still a small sample and well short of the 10 repetitions the main protocol asks for. The 10 decode iterations inside a single run are integrated into one energy total and are not independent trials. Batch size 1, 256 generated tokens, greedy decoding. The perplexity column of the repeated session is bit-identical across replicates because teacher forcing is deterministic: the quality axis has CV = 0 by construction and is not independently replicated. Perplexity is a proxy for language-model damage, not a downstream-task quality guarantee, and its absolute value is specific to this text and to each model's own tokenizer. The sessions must not be pooled; what reproduces across them is the shape (the NF4 penalty falls monotonically with model size and crosses over into savings; INT8 never saves energy), not the magnitudes. Two known metadata gaps: the CPU section of environment.txt in the 2026-08-20 raw archive is empty because the sweep script aborted that command, and the driver version of the 2026-07-24 session is only in that session's run_metadata/gpu_summary.csv, not in the summary tables.

Erratum to the README file shipped in this version: section 2 of README.md states that session A ran the container image and therefore torch 2.13.0. That is wrong, and it contradicts the environment table in section 4 of the same file, which correctly records Python 3.8 and torch 2.4.1+cu121 for that session. The description above supersedes the README on this point; the file itself is left as published rather than issuing a new version for a text fix.

Measured with the open EcoCompute container (https://github.com/hongping-zh/ecocompute-mlcube), report schema ecocompute-energy/1.1. Interactive curves: https://quantenergy.tech. Supplements, and does not replace, the main EcoCompute dataset (10.5281/zenodo.19647290). Not a certified MLPerf/MLCommons benchmark; energy is reported in an MLCommons-style format, and MLCOMMONS, MLPERF and MLCUBE are trademarks of MLCommons Association, referenced nominatively.