Published August 19, 2026 | Version 1.2

Do AI Detectors Agree? An Inter-Rater Reliability Study of Commercial AI Text Detectors on Academic Writing

Authors/Creators

Description

Nine commercial AI text detectors (Turnitin, GPTZero, Originality.ai, Pangram, Copyleaks, ZeroGPT, Winston, Sapling, and QuillBot) were treated as independent raters of 90 academic texts: 30 purely human-written before 2022, 30 AI-generated by five model families, and 30 hybrid (human + AI) splices. Overall agreement is substantial (Krippendorff's alpha 0.71, Fleiss' kappa 0.78), but on hybrid human-AI texts agreement falls to chance level (mean pairwise kappa 0.02). In total, 28 of 30 hybrid texts received both a human verdict and an AI verdict from different tools, and the median per-text score range was 100 points. On mixed human-AI writing, the classification a text receives relies more on the choice of detector than on the text itself.

This record contains the working paper (PDF), the full 90-text corpus with per-text provenance, the 90 x 9 detector score matrix, the generation protocol, and the deterministic analysis and figure scripts. Canonical page: textpulse.ai/research/ai-detector-agreement

Version 1.1 adds nine screenshots of corpus hybrid text SS1 scored in each detector's own web interface (account details blurred), matching the case study shown on the paper page, and updates the paper PDF to the final published byline.

Version 1.2 retakes the Originality.ai screenshot on the same hybrid text (SS1) as the other eight detectors.

Files

screenshot-copyleaks.png

Files (4.2 MB)

Name Size
md5:b5c3cae4717c9a3ccb6c14e3e7fbc840
269.2 kB Preview Download
md5:ac8b578a54a2976ac5919d27bc651bba
340.3 kB Preview Download
md5:98e17bfb786647201a6e9b4cbc795ebb
295.9 kB Preview Download
md5:ec7c1acd58171cee886787e38703a3ec
206.2 kB Preview Download
md5:aa230092ef2b7fde73cf558f10bb7ebd
265.3 kB Preview Download
md5:d3e3d5d12e7652481cf67251f0821c54
163.8 kB Preview Download
md5:7c2c2e1530b6243244559ae6fcab6706
189.7 kB Preview Download
md5:7255e5f1b23deab03eaf2a1651a4e3e4
381.9 kB Preview Download
md5:10ecc49915ccf94e66f161d826b56fe0
170.6 kB Preview Download
md5:b366388d3a828fa33ee09c06e1873ad3
1.2 MB Preview Download
md5:487d8916f95bceff534fe8cfe2ab9322
711.2 kB Preview Download

Additional details

Related works