Published July 2, 2026 | Version v1

Benchmarking large language models for ACMG/AMP variant interpretation and variant calling

Authors/Creators

  • 1. Faculty of Life Sciences, University of Westminster, London, UK

Description

Agentic large language models are increasingly used across the genomic workflow, from variant calling to clinical interpretation, yet they are evaluated by accuracy alone, a single figure that cannot say whether a system is safe or where in the workflow a failure originates. We present ClawBench, a framework that attributes each outcome to the architectural layer that produced it across both halves of the canonical pipeline. Two design choices remove the confounds that make agentic genomics hard to evaluate: a temporally blinded truth set, in which every scored ClinVar label first became available only after the training cutoff of every model tested, and a fail-closed evidence contract that blocks evidence circular with the truth label. We score validity, safety, provenance and reproducibility, not accuracy alone, under a constraint gradient that relocates correctness from a model's prior into executed, validated code.

We show three things. First, dangerous misclassification is uniformly rare, a controlled precondition of the executed architecture rather than a frontier, while truth-circular evidence is measurable and is neutralised by execution. Second, different variant classes are rate-limited by different layers: loss-of-function variants by the deterministic combiner threshold, and rare missense by evidence formation, where evidence acquisition is asymmetric and capped and strength assignment is a recoverable layer that naive strength-licensing prompts confound. Third, for variant calling the arms separate not on whether a model can plan a pipeline, which all do, but on trust properties, pinning, provenance, auditability and reproducibility, which climb monotonically toward validated execution; and a local open-weight model reproduces the safety result on a single-replicate pass yet meets the structured-output and provenance contract far less often than frontier models, a conformance gap rather than a capability or safety gap. An end-to-end join defines a single attribution label for each variant that is both called and interpreted, distinguishing a missed call from a propagated genotype error from a correctly called but misinterpreted variant; on the benign-dominated reference overlap available here we demonstrate the join as a validated mechanism rather than measure these failures directly.

ClawBench shows that apparently identical outcomes arise from distinct, independently measurable failure modes, and that trustworthiness in agentic genomics is a property of the pipeline architecture rather than of the model, providing a portable, contamination-resistant unit of attribution for the field.

This record is the reproduction archive for the paper above (preprint; analysis code at https://github.com/ClawBio/ClawBench, MIT licence). It contains: heldout_manifest.json, the immutable temporally blinded held-out truth set of 6929 two-star ClinVar variants (build date 2026-06-15, effective cutoff 2025-11-29, content_hash 8d745d32ad8a377f05022174f80041d0f237aa24fc5c1dcb87fcebe16c242c66); clinvar_GRCh38_2026-06-15.vcf.gz, the frozen 2026-06-15 ClinVar snapshot used as the builder input so the released builder reproduces the truth set bit-for-bit from a clean clone; exp1_pilot_v2_runs.jsonl, the Tier-A interpretation pilot agent emissions; acquisition_probe_runs_sonnet45.jsonl and acquisition_probe_runs_gpt-5.2.jsonl, the acquisition-arm emissions; and ancestry_table_chr20.tsv, the per-ancestry GIAB chromosome-20 calling table.

Files

heldout_manifest.json

Files (209.0 MB)

Name Size
md5:1e0cffa656988e884c230e2c9f0cc89a
399.2 kB Download
md5:43bd40214c033abb8df9db2c090fb999
424.0 kB Download
md5:7e17e418294aa190504192e4fbef783b
308 Bytes Download
md5:7064f3281a9f6e90df384b9b039aa6c4
196.6 MB Download
md5:689d7282c8397430395effb53b926336
5.2 MB Download
md5:f6e03bc93ccae6d8d3855b5a894b913f
6.4 MB Preview Download

Additional details

Related works

Is supplement to
Preprint: 10.64898/2026.06.11.731523 (DOI)
Is supplemented by
Software: https://github.com/ClawBio/ClawBench (URL)