MoE^2: A Mixture-of-Mixtures of Experts for Ensemble-Free Domain Generalization

Abstract

Domain Generalization (DG) requires models to generalize across unseen data distributions. Kernel-based theory reveals a No-Free-Lunch problem: any model with a fixed representation is fundamentally sub-optimal for all possible shifts. While large ensembles mitigate this, they are computationally expensive and remain static once trained, inheriting the same theoretical limitation. We introduce MoE² (Mixture-of-Mixtures of Experts), a framework that uses a single frozen backbone to dynamically synthesize a bespoke adapter for each input, allowing it to continuously adapt its effective kernel. We provide a theoretical grounding for this process, proving our routing mechanism is a principled non-parametric estimator for the optimal Bayes mixture of experts. We derive a generalization bound that cleanly separates the router's estimation error from the reduction in a kernel-mismatch penalty achieved via synthesis. MoE² matches or exceeds state-of-the-art ensemble baselines on major DG benchmarks while using only a single, compact model. MoE² thus provides a theoretically-grounded and lightweight alternative to large-scale ensembles for robust domain generalization.