Published November 12, 2026 | Version 1.0.0

LLM-Judge Teammate Coding Scheme: dataset for "LLM-Judge Behavioral Coding Scheme for Agent Teammate Quality"

  • 1. Clemson University
  • 2. Metacognition Institute

Description

The full stimulus set (single-turn, matched-register and subtle-stress partner lines; multi-turn dialogues are defined in the runner script), the exact judge system prompt, 1,338 judge rating vectors from five LLM judges across three stimulus families, the human pilot (ratings, results, blinding key and the blank rating instrument), the camera-ready summary statistics, and the stimulus, runner and analysis code.

Dataset accompanying the paper LLM-Judge Behavioral Coding Scheme for Agent Teammate Quality, 14th International Conference on Human-Agent Interaction (HAI '26), 2026 (https://hai-conference.net/hai2026/program-schedule/).

Paper abstract. Evaluating whether an AI agent behaves like a good teammate is becoming a bottleneck as multi-agent language-model systems proliferate, because human ratings do not scale to the volume of transcripts these systems generate. We ask whether a language model can stand in for a human rater of teammate quality, and we present a controlled proof-of-concept validation of a measurement instrument for that purpose rather than a large-scale study. Drawing on the autonomous-agent teammate-likeness construct from the human-autonomy-teaming literature, we define a six-dimension behavioral coding scheme, covering altruistic, benevolent, interdependent, emotive, communicative, and synchronized conduct, and adapt it for use by a language-model judge. We validate the adaptation with experiments in which the partner's lines in team-task transcripts are rewritten as cooperative, neutral, or defective while the rest of the conversation is held fixed. Five judges from five model families (GLM, GPT-OSS, Qwen, Gemma, and Mistral) rate single-turn scenarios and multi-turn dialogues, and a surface-feature audit with length- and politeness-matched registers tests whether the judges read teammate conduct rather than verbosity. Every judge orders the manipulated behavior as intended, most with fully separated confidence intervals between adjacent conditions, and every pair of judges agrees strongly on relative ordering, though absolute calibration differs across judges by up to about a scale point in the middle of the range. On matched registers the judges still separate cooperative from degraded conduct, but resolution at the subtle-defective boundary is limited. A blinded single-rater human pilot on all stimuli corroborates both findings: the rater closely reproduces the judges' orderings and shows the same resolution limit at the subtle-defective boundary. The scheme is therefore valid for relative comparisons, such as ranking systems or versions, but not yet for absolute teammate-quality thresholds applied across judges, and not yet for fine distinctions among subtly poor teammates. We give design implications for building an LLM-judge teaming-evaluation pipeline, and release the scheme, stimuli, judge prompts, and all individual ratings from this version's experiments.

Known limitations.

  • The original 108-rating run behind Table 2 was overwritten by the August 2026 re-run, which is the version included.
  • Only the last 200 characters of judge reasoning were kept for judges C-E.
  • Scripts expect data/ and outputs/ next to them; adjust paths before re-running.

License: Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Source code files are released under the MIT License.

Files

croissant.json

Files (153.9 kB)

Name Size
md5:f931b6e3a6222c28eab26b8e04e52a8f
1.5 kB Download
md5:54040595dbe2b6595a364eb829478881
20.6 kB Preview Download
md5:88ffdf926e8c860fe6996b55583fef02
1.6 kB Download
md5:00c7a3239355edf802e95a3b55d72b5d
121.4 kB Preview Download
md5:19da0468d4984cc8894268cbba741a8a
8.8 kB Preview Download

Additional details

Related works