Overview
- MetaNet is a support-set controller that predicts, for each MoE layer, how many expert sub-networks to keep active and a small routing bias to nudge token assignment.
- The method leaves the backbone, the expert networks, and the base router frozen and trains only the controller on a small support set.
- On DeepSeek-MoE-16B-Chat the paper reports a conservative MetaNet setting averages 3.61 active experts per token versus fixed k=6 with similar MMLU accuracy and an aggressive setting averages 2.28 experts with a roughly 3.7 percentage-point accuracy drop.
- The authors say a controller trained on MMLU transfers to C-Eval without retraining, activating about 2.90 experts on average with lower absolute accuracy in their tests.
- Results are from a newly posted arXiv preprint and remain preliminary, so real-world benefits for latency, integration with production routers, broader tasks, and robustness need independent validation before adoption.