Mixture of Experts (MoE)
A model architecture that contains many specialized sub-networks (experts) but activates only a few per input, delivering large-model capability at lower compute cost.
What Is Mixture of Experts (MoE)?
Mixture of Experts (MoE) is a neural network design that splits the model into many parallel sub-networks called experts, plus a lightweight router (or gating network) that decides which experts to use for each input token. Crucially, only a small subset of experts is activated per token — for example, 2 out of 8 — so the model has a very large total parameter count but a much smaller number of active parameters during any single forward pass.
This sparsity is the key advantage: an MoE model can hold the knowledge capacity of a huge dense model while costing far less to run, because most experts stay dormant for any given token. Models such as Mixtral 8x7B, and reportedly several frontier models, use MoE to scale capability efficiently. The trade-off is added complexity in training (balancing expert usage so no expert is overloaded or starved) and higher memory requirements, since all experts must be loaded even though few are active.
Why It Matters
Mixture of Experts is one of the main techniques making frontier-scale AI economically viable. By decoupling total model size from per-token compute, MoE lets labs build models with enormous capacity that remain affordable to serve. Understanding MoE explains why some very large models are surprisingly fast and cheap at inference, and why "total parameters" and "active parameters" are now reported separately.
Real-World Examples
Mixtral 8x7B
Mistral's Mixtral has 8 experts of ~7B parameters each but activates only 2 per token, giving roughly 47B total parameters while running at the speed of a ~13B active model.