Tribunal: Two Complementary Methods for Measuring Frontier LLM Bias on Contested Figures, with Judge Meta-Evaluation in No-Ground-Truth Domains

Frontier large language models are widely used both as raters of other models (LLM-as-judge) and as substantive interlocutors on contested questions of politics, history, and character. Both uses are downstream of the same unanswered question: how reliable are LLMs as evaluators in domains without verifiable ground truth?

Existing work on LLM-as-judge has documented biases — position, length, style, self-preference — almost exclusively in domains with verifiable ground truth. The classes of question that scalable oversight cares most about, however, are largely unverifiable: judgments of helpfulness, alignment, character, intent, and reasoning quality. In these domains, judge biases compound invisibly because there is no answer key against which to detect them.

This working paper introduces Tribunal, a benchmark methodology with two complementary tracks: (A) a political bias comparator that scores frontier LLMs against each other on the character of public figures, with inter-model variance and refusal patterns as the primary signal; and (B) an adversarial debate-and-judge meta-evaluation that drills into whether bias-comparator disagreements survive scrutiny, including a partial-ground-truth anchor via verifiable factual sub-claim extraction. The paper describes the architecture, governance, jurisdictional posture, adversarial red-team discipline, and phased release roadmap.

This is a position paper. Empirical results from a Phase 1 prototype will follow in companion papers as the methodology is exercised. Project repository forthcoming on GitHub.