Multimodal Intermediate-Task Training for Zero-Shot Cross-Lingual Transfer in Vision-Language Models

This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (MultiHowTo100M) for pre-training. Experimen

Research goal: Does intermediate-task training with multimodal models (e.g., BLIP-2) improve zero-shot cross-lingual transfer on vision-language tasks like MMMU, and how does this scale with model size (e.g., 1B vs. 10B parameters)?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.