Jina CLIP's Unified Embeddings for Robust Text-Image Retrieval on Adversarial Inputs

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-c

Research goal: Can Jina CLIP's unified embedding approach improve robustness in text-image retrieval tasks when evaluated on adversarial or noisy inputs compared to separate text and multimodal models, as measured by Top-1 accuracy on perturbed MMMU test sets?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.8/10.