Why MoE models are huge yet cheap — inside DeepSeek's innovations

"Enormous in size, yet cheap to run." That, in a nutshell, describes MoE (Mixture of Experts) models such as DeepSeek and Kimi. Although their total parameter counts run into the hundreds of billions or even trillions, each token is routed through only a small handful of experts, so the number of parameters actually active at any given moment is just a tiny fraction of the model's overall capacity. In this slide deck, we walk through the fundamentals of how this mechanism works, the load-balancing challenges that routers face, and the auxiliary-loss-free load balancing that DeepSeek introduced—along with the approaches that have since built on it—all illustrated with diagrams.