Performance of Multimodal Models in Zero-Shot Cross-Lingual Text-Image Retrieval on MMMU Benchmark

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-c

Research goal: How do multimodal models like CLIP perform on zero-shot cross-lingual text-image retrieval tasks compared to text-only models like mBERT when evaluated on the MMMU benchmark?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.5/10.