Published October 24, 2026 | Version v1

Human inter-rater reliability of retrospective neuro-type classification: a Registered Report Stage 1 protocol (stratified n=200, blinded multiple raters)

Authors/Creators

  • 1. Polytechnic Institute, Zhejiang University, Hangzhou, China

Description

Stage 1 Registered Report for a blinded human inter-rater reliability study. A stratified random sample of 200 historical figures plus 30 gold-standard items is independently coded by at least three blinded raters with the six-dimensional rubric. Primary endpoint: six-category Fleiss kappa of at least 0.70 with bootstrap 95% lower bound at least 0.60 (three-category ND-subtype kappa at least 0.80); Gwet AC1 reported as a robustness check. A first-round kappa below 0.50 triggers exactly one rubric revision; persistent failure below 0.50 constitutes person-level falsification of the coding scheme. LLM-based ratings (observed agreement up to ~0.79) are treated as auxiliary and cannot replace human reliability. The protocol is registered before raters are recruited or any results are seen; requires IRB review.

Notes

Non-diagnosis Goldwater Rule; AI-assisted writing declared; labels are most-plausible attributions not diagnoses.

Files

M02_human_blind_reliability_RR.pdf

Files (84.2 kB)

Name Size
md5:744f1b4ef4a65da1000bc2d51ec82810
84.2 kB Preview Download