Performance of mBERT on XTREME with Intermediate Task Fine-Tuning in Low- vs. High-Resource Languages

Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (e.g., BERT). These models are pre-trained in a self-supervised fashion with a large English text corpus and further fine-tuned with a massive English QA dataset (e.g., SQuAD). However, QA datasets on such a scale are not available for most of the other languages. Multi-lingual BERT-based models (mBERT) are often used to transfer knowledge from high-resource languages to low-resource languages. Since these models are pre-trained with huge text corp

Research goal: How does the performance of mBERT on the XTREME benchmark change when fine-tuned on intermediate tasks in low-resource languages versus high-resource languages?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.6/10.