Comparative Analysis of Cross-Lingual Transfer in Multilingual Versus Monolingual Models on Domain-Specific Benchmarks

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6\% average accuracy on XNLI, +13\% average F1 score on MLQA, and +2.4\% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7\% in XNL

Research goal: How does the cross-lingual transfer performance of multilingual models compare to monolingual models when evaluated on domain-specific benchmarks beyond MKQA, such as XNLI or PAWS-X, focusing on F1 score and accuracy metrics?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.7/10.