Fine-tuning multilingual models on task-specific intermediate data for cross-lingual generalization

Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (e.g., BERT). These models are pre-trained in a self-supervised fashion with a large English text corpus and further fine-tuned with a massive English QA dataset (e.g., SQuAD). However, QA datasets on such a scale are not available for most of the other languages. Multi-lingual BERT-based models (mBERT) are often used to transfer knowledge from high-resource languages to low-resource languages. Since these models are pre-trained with huge text corp

Research goal: What is the effect of fine-tuning multilingual models on task-specific intermediate data from multiple high-resource languages (e.g., English, Spanish, French) on XTREME-R benchmarks, and does sequential versus joint training improve cross-lingual generalization to low-resource languages?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.6/10.