CLIP and Multilingual Language Models for Zero-Shot Cross-Lingual Visual Question Answering on xGQA

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based mode

Research goal: How does the integration of contrastive language-image pretraining (CLIP) and multilingual language models affect zero-shot cross-lingual visual question answering accuracy on the xGQA benchmark compared to models trained without CLIP?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.