Alignment of Multimodal Representations in Cross-View Language Models and Robustness to Noisy Image-Text Pairs in Zero-Shot

In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the s

Research goal: How does the alignment of multimodal representations in Cross-View Language Models affect their robustness to noisy or low-quality image-text pairs during zero-shot cross-lingual transfer on tasks like VisualNLI or Multilingual Visual Entailment?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.5/10.