XLM-R Adapter Scaling and Zero-Shot Cross-Lingual Transfer Performance

Adapter modules, additional trainable parameters that enable efficient fine-tuning of pretrained transformers, have recently been used for language specialization of multilingual transformers, improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained transformer's parameters. Our zero-shot cross-lingual transfer experiments, invo

Research goal: What is the impact of scaling the number of language-specific adapters in XLM-R on zero-shot cross-lingual transfer accuracy in multilingual GLUE benchmarks compared to full parameter fine-tuning?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.5/10.