# Reproduction package

Measurement harness and results for the paper:

**Open or Frontier? A Cost- and Energy-Aware Benchmark of Large Language Models for Software Vulnerability Detection**
Patrick Deininger and Wolfgang Slany. Submitted to MDPI *Computers*.

Archived at Zenodo: https://doi.org/10.5281/zenodo.21391074

## Contents

| Path | What it is |
|---|---|
| `harness/` | The measurement harness: backends, energy meters, scoring, analysis, 82 tests |
| `harness/runs/` | Every run's raw per-task records, resolved config, seed and price snapshot |
| `harness/runs/primevul_combined/` | The main N=1549 run behind Table 2 and both figures |
| `harness/configs/` | Run configurations |
| `harness/scripts/` | Analysis entry points (see below) |

**Not included:** `harness/data/primevul/primevul_test.jsonl` (66 MB). The PrimeVul corpus
belongs to its original authors and is re-downloaded rather than redistributed — see
`harness/README.md`. Everything needed to re-run the *analysis* is present; only re-running
inference against the models requires the corpus.

## Setup

```bash
python -m venv .venv && . .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r harness/requirements.txt
python -m pytest harness/tests -q                # expect 82 passed
```

## Reproduce the analysis (no API keys, no GPU, no spend)

Every number in the paper is derived from the committed run records:

```bash
python -m harness.scripts.build_final_analysis   # Table 2, both Pareto figures, enriched_metrics.csv
python -m harness.scripts.revision_stats         # bootstrap CIs + Holm-corrected paired tests
python -m harness.scripts.reasoning_vs_direct    # Section 4.5
python -m harness.scripts.contamination_check    # CVE-year memorisation gradient
python -m harness.scripts.flawfinder_baseline    # static-analysis reference point
python -m harness.scripts.learned_baseline       # CodeBERT-Devign reference point
```

Bootstrap results are seeded (seed 12345, n_boot 20000) and reproduce exactly.

## Re-run inference (costs money)

Requires the PrimeVul test split placed at `harness/data/primevul/primevul_test.jsonl` and
API credentials. Copy `.env.example` to `.env` and fill in your own keys — no credentials
are included in this archive.

```bash
python -m harness.run --config harness/configs/<config>.yaml
```

Generation is deterministic (temperature 0, fixed maximum output length) and results are
written incrementally, so long runs are crash-safe and resumable.

## Measured energy

On-GPU energy for the two locally servable open models was measured on a dedicated NVIDIA
H200 via the NVML cumulative energy counter at concurrency 1, gross and idle-subtracted.
See `harness/README.md` for the cloud-GPU procedure.