Benchmarking large language models for ACMG/AMP variant interpretation and variant calling
Description
Agentic large language models are increasingly used across the genomic workflow, from variant calling to clinical interpretation, yet they are evaluated by accuracy alone, a single figure that cannot say whether a system is safe or where in the workflow a failure originates. We present ClawBench, a framework that attributes each outcome to the architectural layer that produced it across both halves of the canonical pipeline. Two design choices remove the confounds that make agentic genomics hard to evaluate: a temporally blinded truth set, in which every scored ClinVar label first became available only after the training cutoff of every model tested, and a fail-closed evidence contract that blocks evidence circular with the truth label. We score validity, safety, provenance and reproducibility, not accuracy alone, under a constraint gradient that relocates correctness from a model's prior into executed, validated code.
We show three things. First, dangerous misclassification is uniformly rare, a controlled precondition of the executed architecture rather than a frontier, while truth-circular evidence is measurable and is neutralised by execution. Second, different variant classes are rate-limited by different layers: loss-of-function variants by the deterministic combiner threshold, and rare missense by evidence formation, where evidence acquisition is asymmetric and capped and strength assignment is a recoverable layer that naive strength-licensing prompts confound. Third, for variant calling the arms separate not on whether a model can plan a pipeline, which all do, but on trust properties, pinning, provenance, auditability and reproducibility, which climb monotonically toward validated execution; and a local open-weight model reproduces the safety result on a single-replicate pass yet meets the structured-output and provenance contract far less often than frontier models, a conformance gap rather than a capability or safety gap. An end-to-end join defines a single attribution label for each variant that is both called and interpreted, distinguishing a missed call from a propagated genotype error from a correctly called but misinterpreted variant; on the benign-dominated reference overlap available here we demonstrate the join as a validated mechanism rather than measure these failures directly.
ClawBench shows that apparently identical outcomes arise from distinct, independently measurable failure modes, and that trustworthiness in agentic genomics is a property of the pipeline architecture rather than of the model, providing a portable, contamination-resistant unit of attribution for the field.
This record is the reproduction archive for the paper above (preprint; analysis code at https://github.com/ClawBio/ClawBench, MIT licence). It contains: heldout_manifest.json, the immutable temporally blinded held-out truth set of 6929 two-star ClinVar variants (build date 2026-06-15, effective cutoff 2025-11-29, content_hash 8d745d32ad8a377f05022174f80041d0f237aa24fc5c1dcb87fcebe16c242c66); clinvar_GRCh38_2026-06-15.vcf.gz, the frozen 2026-06-15 ClinVar snapshot used as the builder input so the released builder reproduces the truth set bit-for-bit from a clean clone; exp1_pilot_v2_runs.jsonl, the Tier-A interpretation pilot agent emissions; acquisition_probe_runs_sonnet45.jsonl and acquisition_probe_runs_gpt-5.2.jsonl, the acquisition-arm emissions; and ancestry_table_chr20.tsv, the per-ancestry GIAB chromosome-20 calling table.
Files
heldout_manifest.json
Files
(209.0 MB)
| Name | Size | |
|---|---|---|
|
md5:1e0cffa656988e884c230e2c9f0cc89a
|
399.2 kB | Download |
|
md5:43bd40214c033abb8df9db2c090fb999
|
424.0 kB | Download |
|
md5:7e17e418294aa190504192e4fbef783b
|
308 Bytes | Download |
|
md5:7064f3281a9f6e90df384b9b039aa6c4
|
196.6 MB | Download |
|
md5:689d7282c8397430395effb53b926336
|
5.2 MB | Download |
|
md5:f6e03bc93ccae6d8d3855b5a894b913f
|
6.4 MB | Preview Download |
Additional details
Related works
- Is supplement to
- Preprint: 10.64898/2026.06.11.731523 (DOI)
- Is supplemented by
- Software: https://github.com/ClawBio/ClawBench (URL)