Do AI Detectors Agree? An Inter-Rater Reliability Study of Commercial AI Text Detectors on Academic Writing
Authors/Creators
Description
Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid human-AI texts agreement falls to chance level (mean pairwise kappa 0.02). In total, 28 of 30 hybrid texts received both a human verdict and an AI verdict from different tools, and the median per-text score range was 100 points. On mixed human-AI writing, the classification a text receives relies more on the choice of detector than on the text itself.
This record contains the working paper (PDF), the full 90-text corpus with per-text provenance, the 90 x 9 detector score matrix, the generation protocol, and the deterministic analysis and figure scripts. Canonical page: textpulse.ai/research/ai-detector-agreement
Files
textpulse-detector-agreement-2026-data.zip
Additional details
Related works
- Is identical to
- https://textpulse.ai/research/ai-detector-agreement (URL)