Multimodal Training with Non-English Video-Text Pairs and Zero-Shot English Captioning Performance

This report synthesises findings from 14 peer-reviewed papers addressing the following research question: How does increasing the proportion of non-English video-text pairs in multimodal training datasets impact zero-shot video captioning accuracy on English benchmarks like Charades-Ego for models such. We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM (Lample and Conneau 2019) and Unicoder (Huang et al. 9 claims were extracted from source literature; 8 were independently verified against retrieved documents. An automated multi-reviewer quality assessment produced a score of 7.5/10. This report is a machine-generated literature synthesis and does not constitute original research.

Research goal: How does increasing the proportion of non-English video-text pairs in multimodal training datasets impact zero-shot video captioning accuracy on English benchmarks like Charades-Ego for models such as Flamingo?

Autonomous literature synthesis. Automated review score: 7.5/10. Full text and citation available at Assignee Research.