Published November 2, 2026 | Version 2

Treat Every Claim as a Hypothesis: Measurements from a Multi-Model Agent Fleet

Authors/Creators

  • 1. Alliance Research Group

Description

A field report from operating a small fleet of LLM agents from several model families on real engineering and research work. Negative results first. Topics: a command-classification gate for a code-executing agent and the bypass classes that review found (and that a later revision left open); measurements in an agent loop, including a benchmark that leaked its answers through comments the agents could read, and an episode in which a model wrote an autouse conftest.py so that a flaky test would pass; cross-family blind evaluation of a document pipeline, including an injected OCR fault that was detected and ignored, a repaired version that passed the known variant, and a first blind variant (commit-reveal) that failed because the pseudonymisation layer removed a negation; and a verified case base of agent claims contradicted by artifacts, in which the most frequent source is the orchestrator itself.

All case data used in the evaluations is synthetic. Section 7 states what the report does not claim.

Version 2: typeset in LaTeX and adds figures (system and gate diagram, review timeline, evaluation chart, document pipeline with the fault-injection and commit-reveal points, case-base chart) and fixes a list-formatting error in Section 3.3 of version 1. No numbers were changed; every number in the figures appears in the text.

Files

treat_every_claim_as_a_hypothesis.md

Files (225.6 kB)

Name Size
md5:d0c946d39659082077093b41e28224c5
36.3 kB Preview Download
md5:ca1d3cdf3bce4b39d9a3f827c89c6f9a
120.4 kB Preview Download
md5:940d5a2daf03bd7de957aca30e74feda
69.0 kB Download