Effectiveness of Intermediate-Task Training for Zero-Shot Cross-Lingual Transfer on XNLI Across Transformer and Decoder-Only

Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of different languages in the models may not be aligned very well. In this paper, we aim to improve the zero-shot cross-lingual transfer performance by proposing a pre-training task named Word-Exchange Aligning Model (WEAM), which uses the statistical alignment information as the prior knowledge to guide cross-lingual word prediction. We evaluate our model on multilingual machine rea

Research goal: How does the effectiveness of intermediate-task training for zero-shot cross-lingual transfer on XNLI compare between transformer-based models (e.g., BERT, RoBERTa) and decoder-only models (e.g., GPT-3, PaLM) when controlled for model size?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.