Consensus Drift: How Reward and Peer Approval Change What Verifier Agents Check

Multi-agent systems increasingly rely on panels of verifier agents: several models vote on whether a task was completed, and the majority is recorded as the outcome. This paper asks what moves such a panel when the evidence stays fixed. We ran four pre-registered experiments on an LLM verifier (Claude Haiku 5.5; 9,504 decisions; 2.92 USD) using 48 fault-injected completion claims with known ground truth. We varied how the verifier was scored (for accuracy against an audit, for agreement with the panel majority, or for accuracy with a bonus for correct dissent), how many peers had already approved, whether those peers had seen independent evidence, and whether the evidence log was in front of the verifier or had to be opened at a small cost.

When the evidence was in the prompt, nothing moved the vote: false acceptance stayed at 2 to 4% in all 15 cells. When the evidence had to be opened, the scoring rule and the peers decided whether the verifier looked. Paid for agreement and facing seven prior approvals, verifiers opened the log 6 to 9% of the time and passed 76 to 85% of faulty completions. Across the whole experiment a verifier that opened the log accepted 3% of faulty work and one that did not accepted 59%. One pre-registered hypothesis was refuted: verifiers paid for accuracy did discount peers they were told had read only the executor's message (2% false acceptance, against 40% when the same seven peers were described as independent checkers). A consensus that was correct when formed and then went stale was accepted 98% of the time under agreement scoring and 0 to 5% under the other two rules. Peer pressure shifted a judged question (which code is higher quality) by 29 points under agreement scoring and shifted a measured one (which code is faster, with the benchmark shown) by zero.

Under selection across six generations, accuracy-scored and bounty-scored panels kept opening logs 86 to 98% of the time. Agreement-scored panels opened them 15 to 41% of the time, their majority followed the first voter in 195 of 216 cases, and they reached the wrong outcome in 33 of 216 cases, most of them rejections of correct work. One panel converged on rejecting 43 of 48 correct completions in a round and recorded the round as a perfect score. The results suggest that the reward attached to a verifier acts mainly on its decision to look, that provenance labels on peer votes are an effective and cheap control, and that agreement is a poor objective for a verification panel in either direction.

Files: the manuscript (PDF) and an artifact archive containing the pre-registration, harness, every cached model response, result rows and analysis code (code under MIT).