Cross-Dataset Generalization in Urdu Fake News Detection: An Empirical Study with XLM-RoBERTa and a Length Confound Analysis

Urdu fake news detection remains an under-resourced problem despite the language being spoken by over 231 million people worldwide. While prior work has demonstrated strong in-domain performance on individual Urdu datasets, whether models trained on one corpus generalise to another has received little systematic attention.

This paper presents the first cross-dataset generalisation study for Urdu fake news detection, using two publicly available balanced datasets: the Ax-to-Grind Urdu corpus, comprising 10,083 articles across 15 domains, and the Notri-Fact Urdu dataset, containing 13,388 articles. We fine-tune xlm-roberta-base under four experimental conditions, namely in-domain on each dataset and two zero-shot cross-domain transfer directions, and compare against TF-IDF baselines using Logistic Regression and Support Vector Machines.

Our experiments reveal a striking asymmetry in cross-domain performance. While transfer from Notri-Fact to Ax-to-Grind achieves a macro F1 of 0.771, the reverse direction collapses entirely to an F1 of 0.005, with the model predicting fake for 99.7% of all test articles. Through class-conditional length analysis and predicted label distribution inspection, we demonstrate that this collapse is attributable to a systematic length confound present in the Ax-to-Grind dataset, where fake articles average 117 words compared to just 35 words for real articles, a 3.4× asymmetry that induces shortcut learning. Since Notri-Fact articles are uniformly long across both classes, the model trained on Ax-to-Grind has no valid length signal to apply at test time, causing it to fail catastrophically.

These findings carry direct implications for dataset construction standards and evaluation practices in low-resource NLP. We further provide a diagnostic methodology for identifying confound-driven model behaviour that is reusable across multilingual fake news detection settings.