Mixture of Experts
Sparse activation and the DeepSeekMoE architecture
36.1 What You Will Learn
In this chapter you’ll learn a concept that moves models from being just large language models into efficient large language models. To give you an idea of how much of an impact being efficient can have on memory, DeepSeek-V3 carries 671 billion parameters and puts only 37 billion of them to work on any given token. The mechanism responsible for this is Mixture of Experts or MoE, which replaces the single feed forward network in each transformer block with a large collection of small ones. Now what I want you to keep in mind moving forward is that MoE does not reduce the total amount of memory needed, but reduces the amount of fast memory we need (like from RAM or GPU compute) while increasing throughput resulting in a very efficient way to compute. How it works internally is that a router examines each token, scores every expert against it, and sends the token to a handful of the highest scoring. The remaining experts do no arithmetic at all for that token, so the model pays for the capacity it stores but only for the compute it uses. Understanding how a router decides, and what stops every token from crowding onto the same few experts, is the substance of this chapter.
To get some insight into these concepts, in this chapter we will build five programs that illustrate the architecture from the bottom upward, and each of them of course measures a property rather than asserting it. A single expert comes first, followed by a router that scores every expert and selects a handful for each token. A complete layer then combines the routed experts with a shared one, that every token visits regardless of what the router decided. Load balancing follows, since a router left alone will send almost everything to whichever experts it happened to prefer at initialization. The chapter closes by reconstructing the DeepSeek-V3 parameter budget from the published configuration, so you get some insight into how things work in a well known production model. With that being said, let’s get into the subject matter of the chapter.
36.2 The Idea
A standard transformer block sends every token through the same feed forward weights. What that means is that the cost per token is fixed by the size of that network. Mixture of Experts breaks the network into N separate experts and routes each token to K of them. What doing this accomplishes is that it decouples the parameter count from the cost. The router itself is a small linear layer that produces one affinity score per expert, and the top K of those scores decide where the token goes. Nothing else about the block changes, and the experts themselves are ordinary feed forward networks of the kind built in earlier chapters. It’s a really simple mechanism if you think about it, but the results it produces can’t be argued with. Attention, normalization and the residual stream are left untouched, which is part of why the idea spread so quickly through architectures that already worked. Take a look at these equations:
Each Expert_i is an ordinary feed forward network of exactly the kind the dense line uses, and g_i is the weight the router assigns to it. The sum runs over the K experts with the highest scores rather than over all N of them, so the other experts contribute nothing and are never evaluated. Setting K to N and every g_i to one recovers an average of all the experts, and setting N to one recovers the dense line above.
With 256 experts and 8 activated per token, each token touches roughly three percent of the feed forward parameters in that layer. The model still stores all 256 of them, so capacity grows with the expert count while the arithmetic per token stays where it was. Memory is not free in the way idle compute is, since every expert has to live somewhere even when no token in the batch selects it. MoE models are therefore cheap to run and expensive to hold, and they are served differently from a dense model of the same active size. The rest of the chapter is the machinery required to make that trade hold up in practice.
Figure 36-1 traces one token through the layer at the size the programs use, eight routed experts with two chosen. The router scores all eight and the two with the highest gates are the only ones that run, drawn solid while the six that stay idle are faint. The gate values on the two live experts are the ones the program prints, and they sum to one because they come from a softmax.
Two paths in that picture are absent from the classic mixture and present in this one. The shared expert on the left is never routed and never skipped, so whatever it learns is available to every token, and the residual runs around the entire layer so a token can pass through changed very little if that is what the weights decide.
36.3 A Single Expert
So that brings us into what exactly an expert is. An expert is an ordinary feed forward network, so it projects up into a wider hidden layer, applies a nonlinearity, and projects back down. The listing initializes both weight matrices from a uniform distribution scaled by the inverse square root of the fan in. That scaling keeps the output roughly the same magnitude as the input instead of collapsing it toward zero over successive layers. The detail matters more here than in a dense network, because an expert whose output is a hundred times smaller than the residual it joins is indistinguishable from one that is switched off. The activation is SiLU, matching the DeepSeek-V3 configuration, which passes positive values almost unchanged and lets small negative values through with a smooth taper.
/* 173_Expert.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
static float silu(float x)
{
return x / (1.0f + expf(-x));
}
#define DM 6
#define D_EXPERT 8
typedef struct {
float W_up[D_EXPERT][DM];
float W_down[DM][D_EXPERT];
}
Expert;
/* Uniform init scaled by fan in, so the expert
neither amplifies nor erases its input */
static void expert_init(Expert *e)
{
float a_up = sqrtf(3.0f / DM);
float a_dn = sqrtf(3.0f / D_EXPERT);
int i, j;
for (i = 0; i < D_EXPERT; i++)
for (j = 0; j < DM; j++)
e->W_up[i][j] = (randf()*2-1) * a_up;
for (i = 0; i < DM; i++)
for (j = 0; j < D_EXPERT; j++)
e->W_down[i][j] = (randf()*2-1) * a_dn;
}
static void expert_fwd(const Expert *e,
const float x[DM],
float hid[D_EXPERT],
float out[DM])
{
int i, j;
/* Up project, then SiLU */
for (i = 0; i < D_EXPERT; i++) {
hid[i] = 0;
for (j = 0; j < DM; j++)
hid[i] += e->W_up[i][j] * x[j];
hid[i] = silu(hid[i]);
}
/* Down project */
for (i = 0; i < DM; i++) {
out[i] = 0;
for (j = 0; j < D_EXPERT; j++)
out[i] += e->W_down[i][j] * hid[j];
}
}
static float norm(const float *v, int n)
{
float s = 0;
int i;
for (i = 0; i < n; i++) s += v[i] * v[i];
return sqrtf(s);
}
int main(void)
{
Expert e;
float x[DM] = { 0.5f, -0.2f, 0.8f, 0.1f,
-0.3f, 0.6f };
float hid[D_EXPERT], out[DM];
int i, suppressed = 0;
srand(42);
expert_init(&e);
expert_fwd(&e, x, hid, out);
printf("Single expert FFN (d_model=%d, "
"d_exp=%d):\n\n",
DM, D_EXPERT);
printf(" Input: [");
for (i = 0; i < DM; i++)
printf("%+.3f%s", x[i], i<DM-1 ? ", " : "");
printf("]\n");
printf(" Hidden: [");
for (i = 0; i < D_EXPERT; i++)
printf("%+.3f%s", hid[i],
i<D_EXPERT-1 ? ", " : "");
printf("]\n");
printf(" Output: [");
for (i = 0; i < DM; i++)
printf("%+.3f%s", out[i], i<DM-1 ? ", " : "");
printf("]\n\n");
for (i = 0; i < D_EXPERT; i++)
if (hid[i] < 0) suppressed++;
printf(" SiLU pushed %d of %d hidden units "
"negative\n",
suppressed, D_EXPERT);
printf(" Input norm %.3f, "
"output %.3f, gain %.2f\n",
norm(x, DM), norm(out, DM),
norm(out, DM) / norm(x, DM));
int params = D_EXPERT * DM + DM * D_EXPERT;
printf("\n Parameters per expert: %d\n", params);
printf(" With 256 experts: %d stored, %d active\n",
params * 256, params);
return 0;
}
Figure 36-2 shows the hidden activations, the output, and the gain from input to output. The hidden layer carries eight values and SiLU pushed six of them negative, which is the usual picture for a randomly initialized layer receiving a mixed sign input. SiLU does not zero those units the way a rectifier would, passing them through at small negative magnitude so they still carry gradient. The gain from input to output is 0.87, so the expert neither amplifies what it receives nor erases it, which is what the fan in scaling was chosen to produce. Each expert holds 96 parameters at these dimensions, the product of the model width and the expert width counted twice for the two projections. Scaling that to 256 experts gives 24576 parameters stored against 96 active, and the ratio is the whole argument for the architecture in miniature. Once you understand how that works at this small scale you’ll have no problem understanding the larger architectures.
36.4 The Router
The router holds one centroid vector per expert and scores a token by taking the dot product of the token against each centroid. DeepSeek-V3 passes that dot product through a sigmoid rather than a softmax, then selects the top K and normalizes only the selected scores. Sigmoid scoring gives every expert an affinity that does not compete with the others until selection time. That is a different gradient path from a softmax taken across all the experts at once. The listing runs one set of tokens through two routers. The first with random centroids and the second with centroids placed on the domain prototypes that training would eventually find. Placing them by hand is a shortcut, but it isolates the question of what a trained router buys you from the separate question of how it gets trained. Take a look at this equation:
The vector e_i is the centroid the router stores for expert i, so the dot product measures how well the token lines up with what that expert has come to represent. Passing it through a sigmoid gives an independent score per expert in the range zero to one, which is the departure from the softmax most routers use. The second expression renormalizes the chosen K scores so the gates sum to one. Because only the selected experts appear in the denominator, a token that scores poorly everywhere still produces gates that add up.
/* 174_Router.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
static float sigmoid(float x)
{
return 1.0f / (1.0f + expf(-x));
}
#define DM 6
#define N_EXPERTS 8
#define TOP_K 2
#define N_DOMAIN 3
#define PER_DOMAIN 10
#define N_TOKENS (N_DOMAIN * PER_DOMAIN)
/* Which domain each expert has specialised in */
static const int owner[N_EXPERTS] = { 0, 0, 0, 1,
1, 2, 2, 2 };
/* Three unit directions standing in for code, math
and prose after the router has been trained */
static const float proto[N_DOMAIN][DM] = {
{ 0.707f, 0.707f, 0, 0, 0, 0 },
{ 0, 0, 0.707f, 0.707f, 0, 0 },
{ 0, 0, 0, 0, 0.707f, 0.707f },
};
/* Score every expert, select top K, normalise the
gates, and report how far apart the scores were */
static float route(const float c[N_EXPERTS][DM],
const float x[DM], int sel[TOP_K],
float gate[TOP_K], float *raw)
{
float s[N_EXPERTS], lo = 2, hi = -2, sum = 0;
int i, j, k;
for (i = 0; i < N_EXPERTS; i++) {
float dot = 0;
for (j = 0; j < DM; j++)
dot += x[j] * c[i][j];
s[i] = sigmoid(dot);
if (s[i] < lo) lo = s[i];
if (s[i] > hi) hi = s[i];
}
for (k = 0; k < TOP_K; k++) {
int best = -1;
float bs = -1;
for (i = 0; i < N_EXPERTS; i++) {
int used = 0;
for (j = 0; j < k; j++)
if (sel[j] == i) used = 1;
if (!used && s[i] > bs) {
best = i;
bs = s[i];
}
}
sel[k] = best;
gate[k] = s[best];
}
*raw = gate[0];
for (k = 0; k < TOP_K; k++) sum += gate[k];
for (k = 0; k < TOP_K; k++) gate[k] /= sum;
return hi - lo;
}
/* Run every token and count how many land on an
expert that owns the token's own domain */
static void evaluate(const char *label,
const float c[N_EXPERTS][DM],
const float tok[N_TOKENS][DM],
const int dom[N_TOKENS],
int verbose)
{
float spread = 0, top_gate = 0, top_raw = 0;
int hits = 0, t;
printf(" %s\n", label);
for (t = 0; t < N_TOKENS; t++) {
int sel[TOP_K];
float gate[TOP_K], raw;
spread += route(c, tok[t], sel, gate, &raw);
top_gate += gate[0];
top_raw += raw;
if (owner[sel[0]] == dom[t]) hits++;
if (verbose && t % PER_DOMAIN < 2)
printf(" dom %d -> [%d, %d] score %.3f"
" gates %.3f %.3f\n",
dom[t], sel[0], sel[1], raw,
gate[0], gate[1]);
}
printf(" mean score spread %.3f\n",
spread / N_TOKENS);
printf(" mean winning score %.3f\n",
top_raw / N_TOKENS);
printf(" mean winning gate %.3f\n",
top_gate / N_TOKENS);
printf(" routed to own domain %d of %d\n\n",
hits, N_TOKENS);
}
int main(void)
{
float rnd[N_EXPERTS][DM], trained[N_EXPERTS][DM];
float tok[N_TOKENS][DM];
int dom[N_TOKENS];
int i, j, t;
srand(42);
/* An untrained router: centroids are noise */
for (i = 0; i < N_EXPERTS; i++)
for (j = 0; j < DM; j++)
rnd[i][j] = (randf()*2-1) * 0.3f;
/* A trained router: each centroid sits on the
prototype of the domain that expert owns */
for (i = 0; i < N_EXPERTS; i++)
for (j = 0; j < DM; j++)
trained[i][j] = 2.5f * proto[owner[i]][j]
+ (randf()*2-1) * 0.4f;
/* Tokens are prototypes plus noise */
for (t = 0; t < N_TOKENS; t++) {
dom[t] = t / PER_DOMAIN;
for (j = 0; j < DM; j++)
tok[t][j] = proto[dom[t]][j]
+ (randf()*2-1) * 0.25f;
}
printf("Router: %d experts, top-%d, %d domains\n\n",
N_EXPERTS, TOP_K, N_DOMAIN);
evaluate("Untrained router", rnd, tok, dom, 0);
evaluate("Trained router", trained, tok, dom, 1);
printf(" Training moves the winning score, not "
"the\n");
printf(" gate. Normalising two close scores "
"always\n");
printf(" returns something near one half, so "
"the\n");
printf(" routing decision carries the "
"information\n");
printf(" and the gate barely varies at all.\n");
return 0;
}
Figure 36-3 puts the same tokens through both routers, comparing score spread, winning score, and whether each token reaches an expert of its own domain. The untrained router produces a mean score spread of 0.151 across eight experts, so every expert scores within a whisker of one half. What I want you to keep in mind is that selection under those conditions is decided by noise rather than by anything about the token. The trained router spreads its scores by 0.530 and lifts the winning score from 0.564 to 0.914, and all thirty tokens land on an expert that owns their domain. What does not change is the gate itself, sitting at 0.509 before training and 0.502 after. This works out because normalizing two close scores always returns something near one half. The routing decision carries essentially all of the information, while the gate carries almost none, which is worth knowing before you go looking for meaning in gate magnitudes.
The untrained router placed one of the thirty tokens on an expert owning its domain, which is worse than the ten that blind chance would have delivered. That figure is a lottery rather than a result, because random centroids point in fixed directions and an entire domain either falls near the right expert or it does not. Change the seed and the count can climb toward twenty without anything about the router improving. This can all be while the spread and the winning score barely move at all!
A distinction that will make it easy for you to understand routing is that consistency of routing is free and arrives at initialization, whereas correctness of routing has to be learned. This is important as any demonstration reporting only which experts a token visited, without checking whether those experts were the right ones, cannot distinguish the two situations.
36.5 The Complete Layer
Finally we get to the good part which is actually implementing the complete layer. A DeepSeekMoE layer combines three separate contributions into its output vector. The residual carries the token forward unchanged and one shared expert processes every token that arrives. The top K routed experts add their gated outputs on top of both. The shared expert exists so that behavior every token requires is not replicated across all 256 routed experts, which frees those experts to specialize rather than each relearning the same general function. Without it, the routed experts spend part of their capacity duplicating one another, which is capacity the architecture was supposed to save. The listing reports the norm of all three contributions separately, since the output vector on its own as you may have realized by now, cannot tell you whether the experts did anything. Look at our C implementation:
/* 175_Moe_Layer.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
static float sigmoid(float x)
{
return 1.0f / (1.0f + expf(-x));
}
static float silu(float x)
{
return x / (1.0f + expf(-x));
}
#define DM 6
#define DE 8
#define N_SHARED 1
#define N_ROUTED 8
#define TOP_K 2
typedef struct {
float W_up[DE][DM];
float W_down[DM][DE];
}
Expert;
typedef struct {
Expert shared[N_SHARED];
Expert routed[N_ROUTED];
float centroids[N_ROUTED][DM];
}
MoELayer;
static void expert_init(Expert *e)
{
float a_up = sqrtf(3.0f / DM);
float a_dn = sqrtf(3.0f / DE);
int i, j;
for (i = 0; i < DE; i++)
for (j = 0; j < DM; j++)
e->W_up[i][j] = (randf()*2-1) * a_up;
for (i = 0; i < DM; i++)
for (j = 0; j < DE; j++)
e->W_down[i][j] = (randf()*2-1) * a_dn;
}
static void expert_fwd(const Expert *e,
const float x[DM],
float out[DM])
{
float hid[DE];
int i, j;
for (i = 0; i < DE; i++) {
hid[i] = 0;
for (j = 0; j < DM; j++)
hid[i] += e->W_up[i][j] * x[j];
hid[i] = silu(hid[i]);
}
for (i = 0; i < DM; i++) {
out[i] = 0;
for (j = 0; j < DE; j++)
out[i] += e->W_down[i][j] * hid[j];
}
}
static void moe_init(MoELayer *m)
{
int k, j;
for (k = 0; k < N_SHARED; k++)
expert_init(&m->shared[k]);
for (k = 0; k < N_ROUTED; k++)
expert_init(&m->routed[k]);
for (k = 0; k < N_ROUTED; k++)
for (j = 0; j < DM; j++)
m->centroids[k][j] = (randf()*2-1) * 0.3f;
}
/* Returns residual plus shared plus gated routed, and
reports the norm each of the three contributed */
static void moe_fwd(const MoELayer *m,
const float u[DM],
float out[DM], float part[3],
int sel[TOP_K], float gate[TOP_K])
{
float sh[DM] = {0}, rt[DM] = {0};
float s[N_ROUTED], sum = 0;
int i, j, k, t;
for (k = 0; k < N_SHARED; k++) {
float e[DM];
expert_fwd(&m->shared[k], u, e);
for (i = 0; i < DM; i++) sh[i] += e[i];
}
for (k = 0; k < N_ROUTED; k++) {
float dot = 0;
for (j = 0; j < DM; j++)
dot += u[j] * m->centroids[k][j];
s[k] = sigmoid(dot);
}
for (t = 0; t < TOP_K; t++) {
int best = -1;
float bs = -1;
for (k = 0; k < N_ROUTED; k++) {
int used = 0;
for (j = 0; j < t; j++)
if (sel[j] == k) used = 1;
if (!used && s[k] > bs) {
best = k;
bs = s[k];
}
}
sel[t] = best;
gate[t] = s[best];
}
for (t = 0; t < TOP_K; t++) sum += gate[t];
for (t = 0; t < TOP_K; t++) gate[t] /= sum;
for (t = 0; t < TOP_K; t++) {
float e[DM];
expert_fwd(&m->routed[sel[t]], u, e);
for (i = 0; i < DM; i++)
rt[i] += gate[t] * e[i];
}
for (i = 0; i < DM; i++)
out[i] = u[i] + sh[i] + rt[i];
for (i = 0; i < 3; i++) part[i] = 0;
for (i = 0; i < DM; i++) {
part[0] += u[i] * u[i];
part[1] += sh[i] * sh[i];
part[2] += rt[i] * rt[i];
}
for (i = 0; i < 3; i++) part[i] = sqrtf(part[i]);
}
int main(void)
{
MoELayer moe;
float x[DM] = { 0.5f, -0.2f, 0.8f, 0.1f,
-0.3f, 0.6f };
float out[DM], part[3], gate[TOP_K];
int sel[TOP_K], i;
srand(42);
moe_init(&moe);
moe_fwd(&moe, x, out, part, sel, gate);
printf("DeepSeekMoE layer:\n");
printf(" %d shared expert always "
"active\n", N_SHARED);
printf(" %d routed experts, top-%d per token\n\n",
N_ROUTED, TOP_K);
printf(" Input: [");
for (i = 0; i < DM; i++)
printf("%+.3f%s", x[i], i<DM-1 ? ", " : "");
printf("]\n Output: [");
for (i = 0; i < DM; i++)
printf("%+.3f%s", out[i], i<DM-1 ? ", " : "");
printf("]\n\n");
printf(" Routed to %d and %d, gates %.3f %.3f\n",
sel[0], sel[1], gate[0], gate[1]);
printf(" Contribution by norm:\n");
printf(" residual %.3f\n", part[0]);
printf(" shared %.3f\n", part[1]);
printf(" routed %.3f\n\n", part[2]);
int per_expert = DE * DM + DM * DE;
int router = N_ROUTED * DM;
int total = per_expert
* (N_ROUTED + N_SHARED) + router;
int active = per_expert
* (TOP_K + N_SHARED) + router;
printf(" Parameters:\n");
printf(" Per expert %d\n", per_expert);
printf(" Router %d\n", router);
printf(" Stored %d\n", total);
printf(" Active/token %d\n", active);
printf(" Ratio %.2fx "
"stored per active\n",
(float)total / active);
printf(" Experts only %.2fx\n",
(float)(per_expert * (N_ROUTED + N_SHARED))
/ (per_expert * (TOP_K + N_SHARED)));
return 0;
}
Figure 36-4 gives the norm each of the three paths contributed to the output. When we run our layer, we see that the residual contributes a norm of 1.179, the shared expert 1.031, and the two gated routed experts 0.263 between them. The shared expert very nearly matches the residual while the routed pair adds about a quarter as much. That is what two half weighted outputs pointing in unrelated directions will tend to do. Parameter accounting is where the layer earns its keep, with 912 stored against 336 active per token for a ratio of 2.71. Counting only the experts and ignoring the router raises that figure to exactly 3.00, and the difference between the two is router overhead, negligible at 256 experts and conspicuous at eight. The ratio grows with the expert count while the active cost holds still, which is why production models reach for hundreds rather than the eight used here.
36.6 Load Balancing
Something we need to talk about is load balancing. A router with no counterpressure will send most tokens to whichever experts it happened to favor at initialization, leaving the remainder idle. Idle experts are wasted parameters, and overloaded ones become the bottleneck that determines how fast the whole layer runs on distributed hardware. The conventional solution or “remedy” in this case rather, adds an auxiliary loss term penalizing imbalance, which works but degrades the objective the model is actually trying to optimize. DeepSeek-V3 came up with an ingenious way around this, introducing an alternative that adds a bias to each expert score for the top K comparison and nowhere else, steering tokens toward underloaded experts without touching the loss. The gate applied to the expert output still comes from the unbiased score, and that separation is the whole trick. Look at this equation:
Two different quantities are doing two different jobs here. The bias b_i is added only when deciding which experts to select, so an underused expert can be nudged into the top K by raising its bias. The gate that scales the expert output is computed from the unbiased score s_i, so the balancing pressure changes who gets chosen without distorting how much their output counts once they are.
/* 176_Load_Balance.c */
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
static float randf(void)
{
return (float)rand() / RAND_MAX;
}
static float sigmoid(float x)
{
return 1.0f / (1.0f + expf(-x));
}
#define DM 4
#define N_EXPERTS 6
#define TOP_K 2
#define N_TOKENS 600
#define ROUNDS 200
static float centroids[N_EXPERTS][DM];
/* Route one fresh batch, filling load[] */
static void batch(const float bias[N_EXPERTS],
int load[N_EXPERTS])
{
int t, i, j, k;
for (i = 0; i < N_EXPERTS; i++) load[i] = 0;
for (t = 0; t < N_TOKENS; t++) {
float x[DM], s[N_EXPERTS];
int sel[TOP_K];
for (j = 0; j < DM; j++)
x[j] = (randf()*2-1) * 0.5f;
for (k = 0; k < N_EXPERTS; k++) {
float dot = 0;
for (j = 0; j < DM; j++)
dot += x[j] * centroids[k][j];
/* Bias enters the routing decision only */
s[k] = sigmoid(dot) + bias[k];
}
for (i = 0; i < TOP_K; i++) {
int best = -1;
float bs = -1e9f;
for (k = 0; k < N_EXPERTS; k++) {
int used = 0;
for (j = 0; j < i; j++)
if (sel[j] == k) used = 1;
if (!used && s[k] > bs) {
best = k;
bs = s[k];
}
}
sel[i] = best;
}
for (i = 0; i < TOP_K; i++) load[sel[i]]++;
}
}
static float imbalance(const int load[N_EXPERTS])
{
int i, lo = load[0], hi = load[0];
for (i = 1; i < N_EXPERTS; i++) {
if (load[i] < lo) lo = load[i];
if (load[i] > hi) hi = load[i];
}
return lo ? (float)hi / lo : 999.0f;
}
/* Imbalance of an evenly random assignment, which is
the best any balancing scheme could reach */
static float floor_imbalance(void)
{
int load[N_EXPERTS], i, t, r;
float acc = 0;
for (r = 0; r < 50; r++) {
for (i = 0; i < N_EXPERTS; i++) load[i] = 0;
for (t = 0; t < N_TOKENS * TOP_K; t++)
load[rand() % N_EXPERTS]++;
acc += imbalance(load);
}
return acc / 50;
}
int main(void)
{
float bias[N_EXPERTS] = {0};
float gamma = 0.001f;
int load[N_EXPERTS];
int i, j, r;
float off = 0, on = 0;
srand(42);
for (i = 0; i < N_EXPERTS; i++)
for (j = 0; j < DM; j++)
centroids[i][j] = (randf()*2-1) * 0.3f;
printf("Auxiliary-loss-free load balancing\n");
printf(" %d tokens per batch, top-%d of %d "
"experts\n",
N_TOKENS, TOP_K, N_EXPERTS);
printf(" expected load %d, bias step %.3f\n\n",
N_TOKENS * TOP_K / N_EXPERTS, gamma);
/* Control: the bias stays at zero throughout */
for (r = 0; r < ROUNDS; r++) {
batch(bias, load);
if (r >= ROUNDS - 50) off
+= imbalance(load) / 50;
if (r == 0) {
printf(" bias off, round 0: ");
for (i = 0; i < N_EXPERTS; i++)
printf("%4d", load[i]);
printf(" %.2f\n", imbalance(load));
}
}
/* Treatment: the same router, bias now updating */
for (r = 0; r < ROUNDS; r++) {
batch(bias, load);
if (r >= ROUNDS - 50) on
+= imbalance(load) / 50;
if (r == 0 || r == ROUNDS - 1) {
printf(" bias on, round %-3d: ", r);
for (i = 0; i < N_EXPERTS; i++)
printf("%4d", load[i]);
printf(" %.2f\n", imbalance(load));
}
for (i = 0; i < N_EXPERTS; i++) {
if (load[i] > N_TOKENS * TOP_K / N_EXPERTS)
bias[i] -= gamma;
else
bias[i] += gamma;
}
}
printf("\n mean max/min over the last 50 "
"rounds\n");
printf(" bias held at zero %.3f\n", off);
printf(" bias updating %.3f\n", on);
printf(" even random routing %.3f\n",
floor_imbalance());
printf("\n final bias ");
for (i = 0; i < N_EXPERTS; i++)
printf("%+.3f ", bias[i]);
printf("\n\n");
printf(" The bias enters the top-K comparison "
"and\n");
printf(" nothing else, so gate values still "
"come\n");
printf(" from the unbiased score and no loss "
"term\n");
printf(" is involved. That is equation 16.\n");
return 0;
}
Figure 36-5 measures load balancing against two controls, the same router with the bias held at zero and the imbalance an evenly random assignment would produce. Measuring this properly requires those two controls rather than a before and after picture. A batch of six hundred tokens spread across six experts is never perfectly even, so some imbalance survives no matter how good the routing is. Holding the bias at zero gives a mean imbalance of 1.765 over the final fifty rounds, letting the bias update brings it to 1.220, and an evenly random assignment sits at 1.213. The bias therefore recovers almost the whole distance to the statistical floor, finishing within one percent of a target that no balancing scheme can beat. Essentially, four of the six final bias values have returned to zero, while the two experts the router underused keep small positive offsets. In practice this means a few thousandths is all it takes, because only the differences between experts affect any comparison.
The step size deserves attention here because the update is sign based, rather than proportional to the size of the error. A seemingly small step of 0.1 (which is actually large) exceeds the entire range across which sigmoid scores differ, so each round overcorrects and the loads swing between extremes without ever settling down. The 0.001 used here is small enough that the bias accumulates gradually toward the offset each expert actually needs. DeepSeek-V3 anneals its own step downward over the course of training for exactly the same reason, starting larger to move quickly and shrinking to stop the oscillation. Load balancing of this kind is a control problem before it is a machine learning problem, and control problems are sensitive to gain which if you ever had the pleasure of tuning a PID loop you can relate to.
36.7 DeepSeek-V3 at Scale
I want to use this section to kinda link what we’re doing here to deepseek internals. The published configuration supplies everything needed to reconstruct the parameter budget by hand. Remember there are 61 layers at a model width of 7168. The way it’s broken down is that the first three keep a dense feed forward network with an intermediate width of 18432 and the remaining 58 are replaced by MoE layers. Each MoE layer holds one shared, and 256 routed experts with an intermediate width of 2048, of which 8 routed experts are activated per token. The feed forward networks are gated, so every expert holds three weight matrices rather than two, and getting that single detail wrong misplaces more than two hundred billion parameters. Attention is Multihead Latent Attention, with the compression dimensions given in the paper they published, and it is dense in the sense that all of it runs on every token. Now that you have some insight you can appreciate the density of the model. Take a look at this listing in C:
/* 177_Deepseek.c */
#include <stdio.h>
#define LAYERS 61
/* first layers keep a dense FFN */
#define DENSE 3
#define D_MODEL 7168
#define D_FFN 18432 /* dense layer intermediate */
#define D_EXPERT 2048 /* MoE expert intermediate */
#define N_ROUTED 256
#define N_SHARED 1
#define TOP_K 8
#define VOCAB 129280
/* SwiGLU FFN carries gate, up and down projections */
static long long ffn_params(int d_model, int d_hidden)
{
return 3LL * d_model * d_hidden;
}
/* MLA projections for one layer */
static long long mla_params(void)
{
long long q_a = 7168LL * 1536;
long long q_b = 1536LL * 128 * (128 + 64);
long long kv_a = 7168LL * (512 + 64);
long long kv_b = 512LL * 128 * (128 + 128);
long long o = 128LL * 128 * 7168;
return q_a + q_b + kv_a + kv_b + o;
}
int main(void)
{
int moe_layers = LAYERS - DENSE;
long long per_expert =
ffn_params(D_MODEL, D_EXPERT);
long long per_dense = ffn_params(D_MODEL, D_FFN);
long long per_moe = per_expert
* (N_ROUTED + N_SHARED);
long long ffn_all = DENSE * per_dense
+ (long long)moe_layers * per_moe;
long long ffn_act = DENSE * per_dense
+ (long long)moe_layers * per_expert
* (TOP_K + N_SHARED);
long long attn = mla_params() * LAYERS;
long long embed = 2LL * VOCAB * D_MODEL;
long long all = ffn_all + attn + embed;
long long act = ffn_act + attn
+ (long long)VOCAB * D_MODEL;
printf("DeepSeek-V3 parameter budget:\n\n");
printf(" Per routed expert: %lld (%.1fM)\n",
per_expert, per_expert / 1e6);
printf(" Per MoE layer: %.2fB (1 + %d "
"experts)\n",
per_moe / 1e9, N_ROUTED);
printf(" Per dense FFN: %.2fB\n",
per_dense / 1e9);
printf(" MLA per layer: %.1fM\n",
mla_params() / 1e6);
printf("\n Component Total Active\n");
printf(" --------- ----- ------\n");
printf(" FFN %6.1fB %6.1fB\n",
ffn_all / 1e9, ffn_act / 1e9);
printf(" Attention %6.1fB %6.1fB\n",
attn / 1e9, attn / 1e9);
printf(" Embeddings %6.1fB %6.1fB\n",
embed / 1e9, VOCAB * (double)D_MODEL / 1e9);
printf(" --------- ----- ------\n");
printf(" Model %6.1fB %6.1fB\n",
all / 1e9, act / 1e9);
printf("\n Published: 671.0B 37.0B\n");
printf(" Error: %5.1f%% %5.1f%%\n",
(all / 1e9 - 671.0) / 671.0 * 100,
(act / 1e9 - 37.0) / 37.0 * 100);
printf(" Ratio: %.1fx capacity per unit of "
"compute\n",
(double)all / act);
printf("\n The FFN holds %.0f%% of the weights "
"and\n",
100.0 * ffn_all / all);
printf(" contributes %.0f%% of the active ones.\n",
100.0 * ffn_act / act);
return 0;
}
Figure 36-6 checks the reconstruction against the reported totals. Based on what we know now, the reconstruction we can calculate arrives at 670.9 billion parameters which is spot on with the 671 billion the paper reports, and 36.5 billion active against the 37 billion reported in the MoE architecture. The residual gap covers the normalization layers, the biases, and the multi token prediction module, none of which this calculation attempts to include. Feed forward weights account for 98 percent of the stored parameters and only 66 percent of the active ones, which is exactly the asymmetry the architecture was built to create. Attention costs 11.4 billion parameters and every one of them runs on every token, so it contributes almost nothing to the total, but almost a third of the cost. The sparse portion of the model is the part that grew, and the dense portion is the part that stayed where it was.
36.8 Key Takeaways
Mixture of Experts replaces one feed forward network with many and routes each token to a few of them, which decouples stored capacity from compute per token.
The router scores each expert with a sigmoid over the dot product between the token and a learned centroid, selects the top K, and normalizes the selected scores into gates.
Training moves the winning score, not the gate. An untrained router scored 0.564 and a trained one 0.914, while the normalized gate stayed near 0.50 in both cases because two close scores always normalize to roughly one half.
A shared expert that every token visits captures the behavior all tokens need, which frees the routed experts to specialize rather than each relearning the same general function.
Auxiliary-loss-free balancing adds a bias to the routing comparison only. Measured against two controls it cut imbalance from 1.765 to 1.220 where an evenly random assignment sits at 1.213, so it recovered almost the whole available gap.
The bias is sign updated and therefore sensitive to step size. Too large a step overcorrects every round and the loads oscillate rather than settle.
The DeepSeek-V3 budget reconstructs to 670.9B stored and 36.5B active from the published configuration, provided the experts are counted as gated networks holding three matrices apiece.
36.9 Exercises
Raise the expert count to 32 and the top-K to 4 in the router program. How does the load distribution change, and do any experts never get selected at all?
Implement the complementary sequence-wise balance loss from equations 17 through 20 of the paper. Compare the resulting imbalance against the bias only scheme measured here.
Remove the shared expert and raise the top-K by one, which leaves the active compute unchanged. Measure whether the output changes and reason about what the shared expert was contributing.
Extend the router program so the centroids are learned by gradient descent from the domain labeled tokens rather than placed by hand. How many steps does it take before every token reaches its own domain?
Implement SwiGLU for the expert, which uses three projections so that the output is the elementwise product of a gate branch and an up branch before the down projection. Count the parameter increase and confirm it matches the factor used in the budget calculation.
DeepSeek-V3 restricts each token to at most 4 nodes during training, which with 256 experts across 8 nodes leaves at most 128 candidates per token. Implement the constraint and measure what it costs in routing quality.