Two Kinds of Hallucination, One Positive Class A Concentrated Conflation in RAGTruth

Every RAGTruth (Niu et al., 2024) evaluation whose preprocessing is publicly inspectable scores
its gold hallucination class as a single, undifferentiated positive: a span is hallucinated or
it is not. The benchmark’s own annotators mark a distinction that both such evaluations
discard. The implicit_true field records spans that are unsupported by the retrieved context
but happen to be true — 13.5% of gold spans by count, 14.56% by character mass — and
marks them, in 90.6% of cases, with the annotators’ own low-intensity severity prefix, against
4.4% for other gold spans. These two subclasses are marked as differing in severity by the
benchmark’s own annotators, and are nonetheless scored identically. The conflation is not a
scattering of annotator disagreement: it concentrates in one span type — 73.64% of spans
typed “Subtle Baseless Info” are themselves the low-intensity, ungrounded-but-true subclass —
and is unevenly distributed by task type, the structure of a real, recognized subclass rather
than of noise. We quantify what this structured conflation costs under RAGTruth’s official
scoring: because the protocol counts every flagged character directly, any prediction set that
excludes the low-intensity subclass is capped at 90.5% char-overlap span recall against gold,
independent of any model — a direct consequence of the conflation, not a second, independent
result. We also tested whether the conflation could be exploited at training time, by down
weighting the low-intensity subclass in the loss under a pre-registered decision rule; across three
lambda values it produced a monotonic, dose-dependent decline in clean-span and response F1
rather than an improvement, ruling out both “the model is insensitive to this distinction” and
“the intervention was too weak to register” as explanations — a negative result we report as one
rather than repackage as a further finding. We report this as a benchmark-quality contribution,
not a detection method, and show that the metric needed to distinguish these subclasses already
exists in data the benchmark ships but no published evaluation uses.