Mixture of Experts (MoE) Routing & Expert Parallelism Simulator
An interactive high-performance compute laboratory modeling sparse Mixture of Experts (MoE): token dispatch routers, top-k expert selection, load balancing loss, and NVLink vs InfiniBand all-to-all communication bottlenecks.
The Mechanics of Sparse Mixture of Experts & Expert Parallelism
Dense transformer models compute every parameter for every token. Sparse Mixture of Experts (MoE) decouple total parameter capacity from per-token FLOPS by routing each token to a specialized subset of feed-forward network (FFN) experts.
1. Top-k Softmax Gating
A linear router projects hidden representations into expert logits: H(x) = x * W_gate. The top-k highest logits are activated via normalized softmax, with tokens multiplied by their gating weights upon combining expert outputs.
2. Auxiliary-Loss-Free Balancing
Historical MoE suffered from routing collapse (all tokens routing to 2-3 popular experts). Modern architectures introduce expert bias terms dynamically adjusted during training to ensure uniform capacity factor utilization without degrading gradient quality.
3. The All-to-All Bottleneck
In distributed setups with Expert Parallelism (EP), tokens must be transferred across GPUs to reach their assigned expert. If interconnect bandwidth (e.g. InfiniBand vs NVLink) is saturated, the GPU compute cores stall awaiting network packets.