Script Sensitivity in Multilingual Language Models: A Paired Pilot Study on Devanagari and Romanised Hindi

We report two preliminary studies on whether writing system affects large language model performance on Hindi. First, we measured tokeniser fertility across ten public tokenisers and find that the historical Devanagari penalty has been eliminated and in several cases reversed: OpenAI’s own encodings move from 3.81× to 2.68× to 0.90× across three generations. Second, we ran a paired behavioural comparison in which five hard Hindi tasks were presented in Devanagari and in romanised Latin script with wording held identical, across three open-weight models. We observe a mean gap of 2.3 points out of 25 favouring Devanagari, but the gap is not uniform: it is +4.0 for GPT-OSS-20B, +3.4 for Llama-3.1-8B, and −0.6 for Sarvam-M, an Indic-specialised model. Decomposing by criterion, fluency shows no degradation (+0.07) while adherence to the requested script shows the largest (+0.93). Script defection was entirely one-directional: in 4 of 15 romanised trials the model replied in Devanagari, and in 0 of 15 Devanagari trials did the reverse occur. We interpret this as consistent with the hypothesis that romanisation does not impair the model’s linguistic competence but does impair something governing output-form selection.

Preliminary pilot study; not peer-reviewed. Includes tokeniser script, fertility results, scored data and raw model outputs.