Published July 25, 2026 | Version v1
Preprint Open

REFUTE: Reasoning Over Evidence — A Benchmark for Scientific Critique and Epistemic Calibration in Language Models

Authors/Creators

  • 1. BGPT

Description

REFUTE (Reasoning Over Evidence) asks a simple question: can an AI model read science honestly? It tests whether models know what a study found, what would overturn it, what the evidence licenses, which summary is faithful, how sure they should be, and whether they can critique open-ended.

Latest public release: 320 auto-graded questions, 20 models shown, 15 complete Truth Scores (46.3 to 74.5). Main finding: critique skill and calibration dissociate. Grok-4.2 leads by point estimate at 74.5; the top three are within 1.3 points.

Dataset: Hugging Face. Leaderboard: Space. Site: bgpt.pro/refute.

Files

refute_preprint.pdf

Files (374.9 kB)

Name Size
md5:20f22fa0fea28a587011a472ce5e8b02
134.2 kB Download
md5:db4834e8322d637e8a2560bd0358f0cd
240.7 kB Preview Download

Additional details

Is described by
Other: https://bio.tools/bgpt-refute (URL)
Other: https://ascl.net/code/v/4795 (URL)
Other: https://bgpt.pro/refute (URL)
Other: https://www.wikidata.org/wiki/Q140690703 (URL)
Other: https://osf.io/qxyfe/ (URL)
Is identical to
Dataset: 10.7910/DVN/IGAJWO (DOI)
Software: https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/connerlambden/refute-inspect (URL)
Is supplement to
Dataset: https://huggingface.co/datasets/BGPT-OFFICIAL/refute (URL)
Other: https://huggingface.co/spaces/BGPT-OFFICIAL/refute-leaderboard (URL)
Dataset: https://www.openml.org/d/47266 (URL)
Dataset: https://www.openml.org/d/47268 (URL)
Software: https://github.com/connerlambden/refute-inspect (URL)
Is variant form of
Preprint: 10.5281/zenodo.21540200 (DOI)