Meta-Mixture-of-Experts: A Higher-Order Framework for Generating, Reconfiguring, and Optimizing Mixture-of-Experts Systems

Mixture-of-Experts (MoE) architectures improve the capacity and computational efficiency of neural networks by conditionally activating selected experts for different inputs. However, conventional MoE systems primarily operate within a predefined expert architecture, expert pool, and routing mechanism. This paper introduces Meta-Mixture-of-Experts (Meta-MoE), a conceptual framework in which an MoE system becomes the object of higher-order modeling, adaptation, generation, and optimization. Rather than merely deciding which existing experts should process an input, a Meta-MoE system can determine which experts should exist, how they should be specialized, how routing mechanisms should be constructed, how expert groups should be organized, and when the overall architecture should be modified in response to task requirements, observed performance, and resource constraints.

The proposed framework integrates task-conditioned MoE construction, adaptive expert-pool management, meta-routing, expert specialization, structural reconfiguration, and cost-aware evaluation. It represents an MoE system as a configurable object and formulates meta-level decisions as a constrained sequential optimization problem. The framework distinguishes five levels of intervention: routing, expert configuration, expert-pool evolution, architecture construction, and meta-mechanism optimization. A bounded recursive extension is also proposed, allowing a higher-level controller to evaluate and modify the mechanism responsible for constructing and managing MoE systems.

This paper establishes a conceptual foundation, formalizes its principal components, proposes an optimization objective and algorithm, and outlines an empirical evaluation protocol. It does not claim that the framework has demonstrated empirical superiority. Its potential contribution lies in integrating expert selection with the construction and evolution of the expert system itself, subject to explicit performance, computational, reliability, and operational constraints. Experimental comparisons with established MoE, adaptive-routing, expert-expansion, and model-combination methods are necessary to determine whether the proposed framework provides distinct methodological value.