Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization

Mixture-of-Experts (MoE) architectures enable conditional computation by routing inputs to multiple expert subnetworks and are often motivated as a mechanism for scaling large language models. In this project, we instead study MoE behavior in an image classification setting, focusing on predictive performance, expert utilization, and generalization. We compare dense, SoftMoE, and SparseMoE classifier heads on the CIFAR10 dataset under comparable model capacity. Both MoE variants achieve slightly higher validation accuracy than the dense baseline while maintaining balanced expert utilization th

Research goal: Does AnyExperts' dynamic expert specialization improve compositional generalization on multi-step reasoning tasks compared to fixed routing, as measured by accuracy on the GQA and NLVR2 benchmarks?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.7/10.