What is a Mixture of Experts?
A Mixture of Experts (MoE) model replaces each dense feed-forward layer in a transformer with multiple parallel expert networks and a learned router that selects which experts process each token. Only the top-K experts activate for any given token — the rest receive no compute.
This creates a powerful separation: total parameters (which determine the model’s capacity to store knowledge) and active parameters per token (which determine inference cost) can be scaled independently. Mixtral 8×7B has 47B total parameters but activates roughly 13B per forward pass — similar compute cost to a dense 13B model, but with the representational capacity of a 47B model.
Architecture: router + experts
A standard transformer feed-forward layer maps each token through a single large MLP. An MoE layer replaces this with:
- N expert networks — each a standard MLP with identical architecture but different learned weights.
- A router — a small linear layer that maps each token’s representation to a probability distribution over experts, then selects the top-K by score.
The router computes gate scores g_i = softmax(Wx) for each expert i, selects the top-K indices, renormalizes those K scores to sum to 1, and returns a weighted sum of the top-K expert outputs:
y = ∑i ∈ TopK(x) g_i ⋅ Expert_i(x)
In practice, MoE layers are interleaved with regular attention layers (not every transformer layer is replaced). Mixtral uses MoE for all feed-forward layers but keeps standard multi-head attention.
Google’s Switch Transformer (Fedus et al., 2021) simplified MoE to K=1: each token routes to exactly one expert. This maximizes sparsity and communication efficiency.
Switch Transformer achieved a 7× speedup over a T5 dense baseline with matched FLOPs while scaling to 1.6T parameters across 2048 experts. The paper introduced the key insight that routing to a single expert is sufficient — the model learns complementary specializations without explicit supervision.
The challenge: with K=1, all tokens could collapse to the same popular expert (expert collapse). Switch Transformer addresses this with an auxiliary load balancing loss that penalizes uneven routing across experts.
Load balancing: the core training challenge
Without intervention, routers tend to create rich-get-richer dynamics: once an expert becomes slightly better at some token type, it receives more training signal on those tokens, improving further and attracting more tokens. Other experts starve, collapse to zero utility, and the effective model degrades to a small number of active experts.
Solutions include:
- Auxiliary load balancing loss (Switch, ST-MoE) — adds a differentiable loss that penalizes the coefficient of variation in expert load. This loss is weighted by a small hyperparameter (α ≈ 0.01) so it guides but does not dominate the main task loss.
- Expert capacity buffer — each expert has a maximum number of tokens it can process per batch (capacity factor × batch_size / num_experts). Tokens routed to a full expert are dropped (or passed through unmodified via a residual).
- Expert choice routing (Zhou et al., 2022) — inverts the routing direction: each expert selects its top-K tokens rather than each token selecting its top-K experts. This guarantees perfect load balance by construction, at the cost of variable computation per token.
Mixtral 8×7B: practical MoE
Mixtral 8×7B (Mistral AI, 2023) is the first widely adopted open MoE model. Its architecture:
- 32 transformer layers, each with 8 feed-forward experts
- Top-2 routing: each token activates exactly 2 of 8 experts
- Each expert is a ~1.75B parameter MLP (7B / 4 active params ≈ 1.75B per expert × 8 = 14B, plus shared attention weights)
- Total parameters: ~47B; active per token: ~13B
On most benchmarks Mixtral 8×7B matches or exceeds LLaMA 2 70B and GPT-3.5 while using roughly half the inference compute. The model is open-weight (Apache 2.0), making it a common baseline for MoE research.
Mixtral 8×22B followed with 141B total / 39B active parameters, competing with 70B dense models.
Expert specialization
A natural question: do experts specialize? Analysis of Mixtral’s routing behavior shows partial but not absolute specialization:
- Experts do develop topic preferences (some tokens related to code route disproportionately to certain experts) but the effect is soft rather than sharp.
- Adjacent tokens in the same sentence often route to different expert pairs, suggesting the model uses expert diversity to capture different aspects of local context.
- Specialization is largely learned without explicit supervision — the model discovers useful partitions of its representational space through gradient descent alone.
Distributed training and inference
MoE models introduce communication overhead in distributed settings that dense models do not have. In a data-parallel setup, each GPU holds all experts. In expert-parallel training, each GPU holds a subset of experts — but every forward pass requires an all-to-all communication step to route tokens to the correct expert GPU and collect results.
This all-to-all is the main scalability bottleneck for large MoE systems. Its cost scales with number of tokens, number of experts, and the size of expert representations. Techniques to reduce it include grouping tokens by expert before communication and using high-bandwidth interconnects (NVLink, InfiniBand) between GPU nodes.
Inference is simpler: with fixed batch sizes, the all-to-all can be fused with matrix multiplications, and modern serving frameworks (vLLM, TGI) handle expert dispatch efficiently.
MoE vs dense: practical trade-offs
| Property | Dense | MoE |
|---|
| Params vs compute | Coupled | Decoupled |
| Training stability | High | Lower (load balancing) |
| Memory per GPU | Lower | Higher (all experts) |
| Inference FLOP/token | High | Lower (sparse) |
| Communication | None (single GPU) | All-to-all needed |
Recap
- MoE replaces dense feed-forward layers with N expert networks and a learned router that activates only top-K per token.
- Total parameters (capacity) and active parameters (compute) are decoupled — Mixtral 47B activates 13B per token.
- Switch Transformer introduced K=1 routing at scale; load balancing loss prevents expert collapse.
- Expert specialization emerges through training; experts develop soft topic preferences without explicit labels.
- All-to-all communication is the distributed training bottleneck for large MoE systems.