Task Diversity in Intermediate-Task Training for mT5 Cross-Lingual Zero-Shot Generalization

Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tas

Research goal: What is the impact of task diversity (e.g., mixing reasoning, question answering, and natural language inference tasks) in intermediate-task training on the zero-shot cross-lingual generalization of mT5 models in the XTREME benchmark, and how does it compare to homogeneous task sets in terms of F1 score and inference throughput?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 9.2/10.