Pre-training on Intermediate English Tasks for Zero-shot Cross-lingual Performance with PEFT

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing expert modules trained separately on language or task data. Our method composes textit\language\ and textit\task\ PEFT adapters via element-wise arithmetic operations to leverage unlabeled data and English labeled data.

Research goal: Does pre-training on intermediate English tasks improve zero-shot cross-lingual performance on the XTREME benchmark when using models with parameter-efficient fine-tuning (PEFT) methods like LoRA or adapter modules, compared to full fine-tuning?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.