There is a newer version of the record available.

Published August 19, 2026 | Version v1

Do AI Detectors Agree? An Inter-Rater Reliability Study of Commercial AI Text Detectors on Academic Writing

Authors/Creators

Description

Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid human-AI texts agreement falls to chance level (mean pairwise kappa 0.02). In total, 28 of 30 hybrid texts received both a human verdict and an AI verdict from different tools, and the median per-text score range was 100 points. On mixed human-AI writing, the classification a text receives relies more on the choice of detector than on the text itself.

This record contains the working paper (PDF), the full 90-text corpus with per-text provenance, the 90 x 9 detector score matrix, the generation protocol, and the deterministic analysis and figure scripts. Canonical page: textpulse.ai/research/ai-detector-agreement

Files

textpulse-detector-agreement-2026-data.zip

Files (1.9 MB)

Name Size
md5:b366388d3a828fa33ee09c06e1873ad3
1.2 MB Preview Download
md5:74ae1675721f8c6f64e8e90c17f1d107
711.3 kB Preview Download

Additional details

Related works