Scaling Intermediate-Task Fine-Tuning Datasets for Zero-Shot Cross-Lingual Performance

An exciting advancement in the field of multilingual models is the emergence of autoregressive models with zero- and few-shot capabilities, a phenomenon widely reported in large-scale language models. To further improve model adaptation to cross-lingual tasks, another trend is to further fine-tune the language models with either full fine-tuning or parameter-efficient tuning. However, the interaction between parameter-efficient fine-tuning (PEFT) and cross-lingual tasks in multilingual autoregressive models has yet to be studied. Specifically, we lack an understanding of the role of linguistic

Research goal: How does the scaling of intermediate-task fine-tuning datasets influence the zero-shot cross-lingual performance of multilingual models on XNLI and PAWS-X benchmarks, measured by accuracy and inference time?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.7/10.