Scaling of English Intermediate-Task Training Effectiveness with Model Size for Zero-Shot Cross-Lingual Transfer on XTREME-R

Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tas

Research goal: Does the effectiveness of English intermediate-task training for zero-shot cross-lingual transfer on XTREME-R benchmarks scale with model size (e.g., comparing small, medium, and large models), as measured by the relative improvement in average F1 score across languages?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.7/10.