Scaling Multilingual Pre-training and Performance Gaps in XTREME-R Tasks

Multilingual language models are widely used to extend NLP systems to low-resource languages. However, concrete evidence for the effects of multilinguality on language modeling performance in individual languages remains scarce. Here, we pre-train over 10,000 monolingual and multilingual language models for over 250 languages, including multiple language families that are under-studied in NLP. We assess how language modeling performance in each language varies as a function of (1) monolingual dataset size, (2) added multilingual dataset size, (3) linguistic similarity of the added languages, a

Research goal: To what extent does scaling the number of languages in multilingual pre-training affect the performance gap between high-resource and low-resource languages on downstream XTREME-R tasks?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 9.5/10.