Impact of Typologically Diverse Training Data on Cross-Lingual Zero-Shot Accuracy in Vision-Language Models

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based mode

Research goal: What is the impact of adding typologically diverse languages to the training data on the cross-lingual zero-shot accuracy of vision-language models like BLIP-2 on the xGQA benchmark?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.6/10.