Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization

Mixture-of-Experts (MoE) architectures enable conditional computation by routing inputs to multiple expert subnetworks and are often motivated as a mechanism for scaling large language models. In this project, we instead study MoE behavior in an image classification setting, focusing on predictive performance, expert utilization, and generalization. We compare dense, SoftMoE, and SparseMoE classifier heads on the CIFAR10 dataset under comparable model capacity. Both MoE variants achieve slightly higher validation accuracy than the dense baseline while maintaining balanced expert utilization th

Research goal: How does the inference latency and throughput of SMoES-based 7B VLMs compare against dense VLMs and hard-routing MoE baselines on MMBench and SEED-Bench at varying batch sizes and sequence lengths?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 7.8/10.