The Deception Delta: Artifact for Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics
Description
Artifact for the CCS 2026 paper “The Deception Delta: Adversarial Evaluation of LLM-Based Smart Contract Bytecode Forensics.” The release contains frozen LLM responses and judge outputs, canonical smart-contract bytecode and source inputs, gold-standard labels, analysis scripts, cached outputs, and local EVM tests. Its principal offline workflow reproduces 112 baseline checks, 67 checks for the shepherding extension, and every paper table (Tables 1–17) plus the Figure 2 heatmap data in the paper's own layout, without external API credentials, blockchain access, or real assets.
ERRATA.md lists the statements of the paper that disagree with the released data, and a script recomputes each corrected value. The archive also includes the accepted paper and an evaluator guide with a claim-to-evidence map.
Security note: the archive contains deliberately deceptive and drain-capable research contracts. They must not be deployed to a public chain or used with real assets.
Notes (English)
- New `reproduce_tables.py`: regenerates Tables 1–17 and the Figure 2 heatmap data with the paper's rows, rounding and model names, and compares each of 667 result cells with the printed value.
- New `ERRATA.md` and `reproduce_errata.py`: statements of the paper that disagree with the released data, with the corrected values recomputed from the data. None changes the headline result or a value in Tables 1–4; several weaken secondary statements.
- New `analysis_scripts/crossed_glmm_ml.py`: maximum-likelihood fits of the crossed model/contract and the model-intercept specifications (ERRATA.md, E3 and E4).
- New `tests/frozen-bytecode-drain-tests/` (15 Foundry tests): runs the drains on the frozen runtime bytecode itself; all drain vectors except HV-1 move all deposits to the owner, HV-1 fails against a standard ERC-20 (ERRATA.md, E7).
- New `gold_standard/blinded_recoding/`: original blinded inputs of the LLM re-coding and the merged labels.
- New `reference_values/figure2.json` (printed Figure 2 cells) and `requirements-ae-full.txt` (optional dependencies).
- `analysis_scripts/prompt_injection_failures.py`: uses a marker set fixed in advance; the earlier marker set, which had been adjusted to the printed values, is reported for comparison only (ERRATA.md, E25).
- `reproduce.py`: section headers now name the paper tables; computations unchanged. `trust_signals.py`: fixed definitions instead of a selection among candidate definitions (identical output); SciPy imported only where needed.
- `effect_size.py`: compares with the printed values (ERRATA.md, E6).
- `analysis_outputs/formal_tests_report.md`: translated to English, numbers unchanged.
- `contracts/MANIFEST.md`: C2 note corrected, HV-1 moved to the derived source files. `README.md`: several numbers and pointers corrected.
- Response and judge data are unchanged from 1.0.1.
Files
DeceptionDeltaArtifacts-v1.0.2.zip
Additional details
Related works
- Is supplement to
- Conference paper: 10.1145/3830454.3846626 (DOI)