What is the impact of multimodal observation diversity (e.g., vision+text vs. vision+audio vs. vision+tactile)

Abstract With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as bidirectional encoder representations (BERT), vision transformer (ViT), generative pre-trained transformers (GPT), etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutti

Research goal: What is the impact of multimodal observation diversity (e.g., vision+text vs. vision+audio vs. vision+tactile) on the zero-shot generalization performance of transformer-based models, measured by accuracy on cross-modal retrieval benchmarks like COCO-Text and AVS?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 9.0/10.