Mixture of Experts (MoE) is a neural network architecture where multiple specialized subnetworks, called experts, process different parts of the input. A gating mechanism (router) decides which experts to activate for each token. Unlike dense models where every parameter contributes to every forward pass, MoE models are sparse: only a fraction of total parameters are used per token. This lets builders increase total model capacity (more parameters) without multiplying inference cost proportionally.

In a transformer MoE layer, each token passes through a router that scores all available experts. The top-K experts (often 2 of 8, or 8 of 128) are selected to process that token. Their outputs are weighted and combined. Example: Mixtral 8x7B has eight expert feed-forward networks per layer but activates two per token, giving large total capacity with roughly the inference cost of a smaller dense model. Routing is learned during training. Experts specialize implicitly: some handle syntax, others factual recall, others reasoning patterns, though specialization is not guaranteed or interpretable without analysis.

MoE addresses a core scaling tension: bigger models are more capable but more expensive to run. Benefits: More parameters without linear compute growth per token. Lower inference cost per capability level compared to equivalent dense models. Room for specialization across expert subnetworks. MoE has become the default architecture choice for many open-weight frontier releases because it balances capability and serving cost.

MoE is not free complexity: Memory: All experts must be loaded in GPU memory even though only some activate per token. Total VRAM requirements can exceed dense models of similar active parameter count. Load balancing: If routing collapses so one expert handles most tokens, capacity is wasted and training becomes unstable. Serving complexity: Inference engines need MoE-aware kernels for efficient expert routing and batched execution. Fine-tuning: Adapting MoE models (LoRA, full fine-tune) requires tooling that handles sparse activation patterns. Quantization: MoE models can be harder to quantize efficiently due to expert diversity.

For teams evaluating open-weight models, MoE affects infrastructure planning: VRAM sizing must account for all experts, not just active parameters. Inference platforms (vLLM, Fireworks, TensorRT-LLM) need MoE support for production serving. Domain fine-tuning may require MoE-aware LoRA targeting specific experts. MoE models excel at high-throughput general workloads. Domain-specific enterprise tasks may still benefit from smaller dense models or specialized architectures tuned for professional reasoning depth.