All in the Family? Lineage, Size, and Agreement in LLM Content Coding

Researchers increasingly use large language models (LLMs) as “independent” content coders. We examined agreement and repeatability using 65 frozen institutional documents, a 91-code instrument and 16 models from six vendors. Fifteen models were retested; each model’s agreement with itself provided a comparison reference.
First, observed retest agreement ranged from .90 to .71 under the historical pooled nominal method. Second, among selected non-small tiers, raw same-vendor agreement averaged .783 and cross-vendor agreement .694. Dividing pair agreement by the geometric mean of the two retests gave descriptive means of .926 (95% document-bootstrap interval .911–.940) within vendors and .842 (.822–.860) across vendors. These ratios are not validated attenuation corrections. Third, an early Haiku run whose prompt was not saved agreed with two later runs at about .59, while the later runs agreed at .826. Fourth, API-versus-tool point differences were small on 25 documents, but intervals did not establish equivalence. Fifth, ethical-principle agreement patterns resembled our earlier study of the same corpus; selected cross-family pairs diverged more on interpretive principles. Sixth, output adherence did not vary monotonically with tier.
We recommend reporting retests for all LLM coders, saving exact prompts and settings, building panels across vendors, and avoiding treating same-vendor models as independent replications. This exploratory study measures agreement, not accuracy; family, route and incompletely recorded settings remain partly confounded.