Spanda: Zero-Cost Lexical Entropy Matches Neural Semantic Uncertainty—Until Frontier Models Break It

Detecting hallucinations in Large Language Models (LLMs) requires estimating uncertainty. Neural Semantic Entropy accurately detects hallucinations by clustering generated paths using an auxiliary Natural Language Inference (NLI) model, but incurs massive computational overhead. In this paper, we propose Spanda, introducing a zero-cost normalized lexical metric ($R_{sc}$). We empirically evaluate models across four scales (1.5B to 120B). We demonstrate that for mid-sized models (7B–27B), $R_{sc}$ achieves an AUROC of 0.889, matching neural semantic entropy without the NLI overhead. However, on frontier models (120B+), we discover that intense RLHF alignment induces 'Confident Mode Collapse'—the model hallucinates the exact same incorrect answer across all paths, causing AUROC to invert to 0.091 and bypassing self-consistency assumptions.