Multimodal Intermediate-Task Training Effects on Zero-Shot Cross-Lingual Transfer Performance in Language Models

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single lang

Research goal: What is the impact of multimodal intermediate-task training on zero-shot cross-lingual transfer performance for language models when evaluated on XNAT, compared to monolingual text-only intermediate training?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.5/10.