What is the performance gap trend in visual reasoning accuracy between SMoES-based MoE-VLMs and dense models a

With the increasing data volume, there is a trend of using large-scale pre-trained models to store the knowledge into an enormous number of model parameters. The training of these models is composed of lots of dense algebras, requiring a huge amount of hardware resources. Recently, sparsely-gated Mixture-of-Experts (MoEs) are becoming more popular and have demonstrated impressive pretraining scalability in various downstream tasks. However, such a sparse conditional computation may not be effective as expected in practical systems due to the routing imbalance and fluctuation problems. Generall

Research goal: What is the performance gap trend in visual reasoning accuracy between SMoES-based MoE-VLMs and dense models across 7B, 13B, and 34B parameter scales on the MMMU benchmark under varying expert activation ratios?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 7.7/10.