Everyone on the payroll, three at the table

Five words this page leans on. A model's weights are the billions of numbers that make it up — the previous page hauled all of them through the card for every word; this page will picture them as a staff on a payroll. A dense model brings every employee to every meeting: all of the weights read every token. A mixture of experts (MoE) organizes most of its weights into teams called experts, and a small learned dispatcher called the router picks a handful of teams per token. The weights that actually work on a given token — the chosen experts plus the staff who attend everything — are the active weights. Everything else stays on the payroll, in memory, at its desk.

The tag with two numbers

Walk the model library today and you will meet names like 30b-a3b. The first number is the payroll: the parameters hired, trained, and taking up space — thirty billion by the tag, 32.9 billion by the model file itself, which is your first hint that a tag is not a measurement. The second is the meeting: the parameters that actually show up for any given token — about three billion by the vendor's bookkeeping, 3.7 billion by ours; we compute both below. Two numbers, because the model's two costs come apart: what it knows is priced by the payroll, and what each token pays is priced by the meeting.

A dense model has one number because its two costs are the same cost. Our previous page showed the mechanism directly: a dense model must haul all of its weights through the card on every trip — and, in the same breath, caught its own dense 27-billion-parameter model lawfully cheating that rule with a draft head, 2.95 tokens riding each trip. Hold both halves; they come back at the end. If that page's single-file line is fresh in your mind, this page is one sentence long: a mixture of experts shrinks the trip, not the staff. The rest is receipts.

The payroll and the meeting

Inside a transformer, most of the weights live in blocks called feed-forward networks — the heavy lifting between the attention layers. A dense model has one wide feed-forward network per block, and every token flows through all of it.

An MoE replaces that one wide network with many narrower ones — the experts. Our live model carries 128 experts per block. In front of them sits the router: a small piece of the model, itself learned during training, that reads each token and scores which experts should handle it. The top few — 8 of 128 on our live seat, 6 of 128 on the other mixture in this page's table — get the token. The other hundred-twenty stay idle for that token, and may be chosen for the next one.

Two honest corrections to the picture that phrase "expert" paints:

The experts are not experts in anything you could name. They are not "the biology expert" and "the poetry expert" — training carved them out on its own, unlabeled, and what published attempts to read them back find is statistical and partial, never a tidy subject. On our machines the question is unanswerable anyway: you cannot ask which expert answered you, and neither can we. The router's choices are per-token and per-block — roughly one decision per token per mixture block, and our live seat has 30 of those blocks — so a single sentence passes through hundreds of routing decisions, not one.

The meeting is bigger than the invitation list. Some staff attend every meeting no matter what the router says: the attention layers (the part that decides which earlier words matter), the token embeddings and output head (the model's dictionary going in and coming out), and — on both mixtures in this page's table — an always-on feed-forward branch: a declared shared expert on one, a per-block dense network working alongside the experts on the other. We computed the real per-token traffic from the model files' own inventories — every block of numbers each file contains, listed with its dimensions (its tensor shapes) and its storage width — and on both models it comes to roughly double what the expert fraction alone suggests. Take our live seat: 8 of 128 experts is 6.25% of the expert weight, but the true active share is 14.4% of the model's bytes, once the always-on staff are counted. On the other benched mixture: 6 of 128 is 4.7%, true share 11.1%. If you ever estimate an MoE's cost by dividing experts-used by experts-total and stopping, you will be flattering it by about half. This is probably the most useful correction on this page.

The part nobody warns you about: the whole payroll needs an office

Here is the asymmetry that makes MoE strange to buy hardware for. The router picks different experts for every token — so we keep all 128 in the card's memory, ready, at all times, which is how every residency figure below was measured. (Runtimes can instead stream experts in from system RAM per token; that trades the memory bill for a latency one, and nothing on this page measured it.) Memory is priced by the payroll. Speed is priced by the meeting.

Our own residency census, published in the August arrivals, shows exactly this shape. The 32.9-billion MoE holds 23.64 GiB resident at 32k context — more than any 27-billion dense model on the same card — while touching about 2.6 GiB (2.83 decimal GB) of weights per token. An MoE is a big model that works in small bites: it rents the whole building and holds its meetings in one room.

That cuts both ways, and the honest sentence for a buyer is this: an MoE gives you a bigger model's knowledge at a smaller model's speed, provided you can afford a bigger model's memory. If your card is small, a dense model of the size that fits may serve you better than an MoE that doesn't.

Measured on our own machines

Our bench published a stopwatch leg in the chair trials: decode speed as eval_count / eval_duration from the runtime's own counters, ten runs per cell, num_ctx 32768 throughout, thinking off, one card, one day, one rule, with prompts at three sizes — the 1k, 8k and 32k tiers. The kit's own discipline says these rows are descriptive — compared, never ranked — and we honour that here. Read down the 32k tier (a roughly 32,000-token prompt, the hardest of the three), with the architecture of every arm verified first-hand from its model file (more on how below):

* the census marked this row NOT-RUN (the model was serving live work); 7.55 GiB was read as-found from the runtime at the same 32k context, and the roster publishes it with exactly that caveat.

Two sentences fall straight out of that column, and they are the whole thesis. Both mixtures out-write the dense 11.9 B while holding more weight on the card (16.27 and 23.64 GiB against its 7.55). And both out-write the dense 27.8 Bs by a wide margin — 2.0× and 1.8× — while their meetings run a seventh and a ninth the size of their own staff by bytes (the 14.4% and 11.1% figures above).

The cleanest pair in the table is the gemma family: same vendor, same quantization, same runtime, same card, same counting rule — one sparse, one dense. The 25.8-billion mixture writes about a third again as fast as its own 11.9-billion dense sibling — 124.975 against 95.195. Bigger payroll, smaller meeting, faster word.

The controlled rows landed, 2026-08-21 — the same stopwatch, on an empty card. What this note adds, for a reader landing here cold: the table above was measured beside live product traffic and said its controlled twin would land here when it existed. It ran 2026-08-21, 07:22–07:29 UTC, in a declared maintenance window — zero contention, proven request-by-request in the kit. Decode, median of ten (ranges in the kit): the 25.8 B mixture 208.0 tok/s at the ~1k tier and 201.7 at ~8k; its 11.9 B dense sibling 126.9 and 122.7 — the mixture's lead holds at 1.64× on the quiet card. The dense 27.3 B with its guesser runs 107.7 / 102.8. One registered prediction failed and prints as failed: we expected the 32.9 B a3b to be the fastest of the four; it ran second at both tiers (127.9 at ~1k, tied with the 12 B under the range rule). And the quiet window's real purchase surprised us: against the beside-traffic rows above, the same-tag medians moved only 1.10–1.20× — contention mostly widens readings (a 23.87% range collapsed to 0.18%), it barely slows them. One column does not publish: the run's fixed prompt let the runtime's prompt cache serve the scored prefills (up to 21× inflation) — the defect, its evidence, and the cure are named in the kit instead of the number.

Three disclosures, because a table this tidy earns suspicion. The gemma4:26b row is the thinnest on the page: it was serving production during the census (it is our live world-narrator seat — more on that in a moment), so it was never reloaded for measurement; its 32k row is ten runs with a wider spread than any other arm (103.2–125.4), and its 1k and 8k cells returned no usable counters at all. The a3b's speed is confounded: its model file carries a small draft head of its own — the guesser mechanism the previous page caught — so its 107.7 may be sparsity, sparsity plus speculation, or either of those plus the fact that most of its blocks are not attention at all but Mamba-style state-space layers — effects we cannot separate from the published rows. Of our mixtures, only gemma4:26b has no draft head anywhere in its file, which makes it the cleanest sparsity receipt we own. And one arm of the published table, a model no longer installed here, could not be re-verified either way — we re-read eight dense models tensor-by-tensor (the list is in the kit); that one is a label we could not re-read.

The arithmetic that catches a mixture red-handed

The previous page taught a trick, and it works even better here. Take the weights a token must haul through the card, multiply by tokens per second, and you get required memory traffic. Our two tensor-verified dense arms with no draft head — the 27.8 B qwens, sibling versions of one architecture — land within a few percent of each other: about 1.07 terabytes per second of achieved traffic at the 32k tier. (Our dense 11.9 B lands well below that, at 0.70 — the band is a floor being approached, not a wall.) A measured floor on what the card actually does; no spec sheet required.

Now do gemma4:26b naively, as if it were dense: 18 decimal gigabytes of weights — the whole file, vision tower included — × 125 tokens per second = 2.25 TB/s — roughly double what our own dense rows are measured achieving on the same card. The number argues back, exactly the way the previous page's did. And the resolution is the same kind of move with a different mechanism: count what a token actually touches. On the active path — eight experts plus the always-on staff, 2.6 decimal GB per token, the vision tower excluded for the same reason the residency arithmetic excludes it — the demand falls to 0.32 TB/s. Comfortable. The arithmetic closes only if the model is sparse, and the model file's own tensor shapes say it is.

Step back and the workshop now owns a lovely symmetry. Two of our models write "faster than their weights allow." One (the previous page's dense 27 B) cheats by carrying more tokens per trip — a draft head, 2.95 tokens riding each pass. The other (our MoE seat) cheats by carrying fewer bytes per trip — a seventh of its payroll per meeting. Same symptom, two different lawful mechanisms, one shared lesson: stop assuming every token pays for every weight. When a model's speed looks impossible, its tag is trying to tell you something.

What the label doesn't tell you

Here is the part we did not expect to be writing. When we went to verify the "3.8 B active" our own pages carry for the live seat, we found there was nothing to verify it against. The runtime's metadata names the expert counts — you can read expert_count 128, expert_used_count 8 off any MoE with one command — but no field anywhere says how many parameters are active. The vendors' labels are conventions, not measurements, and the conventions disagree: computing the a3b's meeting from its tensor shapes gives 3.70 billion — unless you exclude the token embeddings and output head, which gives 2.99 billion, which is exactly where "a3b" comes from. The gemma label ("a4b" in its original tag) includes what the nemotron label excludes. Two vendors, two bookkeeping rules, same word.

So we computed ours. Every tensor in each model file, its shape and its bytes, summed by hand: the totals match the runtime's own parameter counts exactly, and the byte sums match the files on disk to within a tenth of a percent (0.03% and 0.09%). The meeting sizes in the table above are computed from the models' own files, not transcribed from a marketing page — as far as we know, the first time our own labels have been checked against anything.

And the check cut both ways, in the house tradition: it caught us. A third model on our shelf — a chair-holder two of our published pages listed as a plain "33.0 B" — turned out to carry expert_count 128, expert_used_count 6 in its own file. It is a mixture of experts, and our pages never said so. (The correction it needed is not the one you might guess, either: sparsity does not make a model lighter — all 128 experts are resident regardless — its 23.21 GiB at 32k context is its weights minus a vision-and-audio tower that text serving apparently never loads.) Both pages and both backing kits were corrected, dated, before this article published. An explainer that teaches "here is how to tell" had better run the test on its own shelf first. We did; we were wrong once; it says so on the pages now.

The number that looks like sparsity and isn't

One trap, named so you don't fall in it later. Elsewhere in our published residency rows there is a genuine open anomaly: a model variant reporting 2.34 GiB resident where the same weights served without their draft companion report 15.60 GiB — a ratio that looks temptingly like an active-weights fraction. It is not. The same byte count appears for two different quantizations, which sparsity cannot produce (a reporting glitch can), and the finding is registered as an open re-measure — not an explanation — on the August arrivals and the chair trials. A reader who has just learned about active weights will want that number to be sparsity. It very probably isn't, and the honest sentences are "we don't know yet" and "the recount is owed." Not every strange ratio is a mixture; some are just bugs wearing a costume.

What sparsity does not buy

Now the other half of the honesty, because nothing on this page says MoE is free. Speed and smarts are different columns. On the same day, on the same card, our published chair trials scored these same arms on real work. The a3b — quick by any reading of the stopwatch column — was the only arm here that produced verdicts at the judge exam and still missed both of its floors (9 of 12 kills against a floor of 11; 6 of 9 preservations against 8), while the two dense 27.8 Bs cleared both, clean. At narration it posted the lowest pairwise win-rate of the arms in this page's table — 0.360, against 0.539 for the live seat, 0.685 for the dense 11.9 B, and 0.711 for a dense 27.8 B. Tokens per second measure how fast the words arrive, not whether they are worth reading. Every seat in this workshop is chosen by exam, never by stopwatch — and this paragraph is why.

Three smaller fences, cheap to state. Quantization is a different lever: the most memory-hungry resident on our whole census is a dense 28.9 B at q8_0 (33.3 GiB at 32k context) — twice the memory of dense siblings its size — because precision, not architecture, is what's expensive there; never read a residency table without checking the quant column. The notes are their own story: how much memory a model's context costs (the KV cache, the previous page's "notes in the margin") is set by attention geometry, not sparsity — one of our dense 27.8 Bs grows 620 MiB from 16k to 32k context, another dense model (a 32.2 B) grows just 32 MiB, and the 32.9 B mixture grows 48 MiB — the small ones are dense and sparse alike, so don't let "MoE" absorb credit for a small KV cache. And active weights set the ceiling, not the speed: our two benched mixtures touch nearly the same bytes per token yet differ 14% on the clock — routing overhead, kernel maturity, and architecture mix all take their cut. Sparsity is a speed limit raised, not a speed guaranteed.

The model that was proving it all along

The best part of this story is where the receipt was hiding. gemma4:26b — the 3.8-billion-meeting mixture in the table — is not a lab specimen. It is the live seat that narrates our game world and answers as our products' heavyweight voice, serving strangers while this page was written. Its speed was never a benchmark curiosity here — a world that answers on one card answers under the same counting rule as the table above, where the active-path arithmetic is the only reading that closes.

And the workshop's own notebooks, it turns out, called this in advance. In July — three weeks before the stopwatch leg existed — a planning session wrote: "gemma4:26b is actually a mixture-of-experts… this means it would actually decode faster than a dense 12b model." On 12 August the published kit printed 124.975 against 95.195. The prediction is in our archive with its date; the receipt is in a public kit with its counting rule. We like this ending better than a triumphant one, though, so here is the other half: on 18 August, an audit lane hit the previous page's impossible arithmetic and guessed "either [that model] is actually a mixture-of-experts model where only a fraction of the weights get read per token… or the stated numbers are simply wrong" — and the guess was wrong (that model is dense; the answer was the draft head), and the lane was right to refuse to assert it, and wrote "present the math and ask" instead. Same arithmetic, two mysteries, two different answers, one discipline. That discipline is the product; the models just pass through it.

The robber, asked a third time

Regular readers will recognize the guest star. When Three librarians walked a question's journey to a cited page, the question was "can the robber go back to the desert?" — Catan's favorite argument, settled live as ruling 4615 with the rulebook page attached. When the previous page measured the physics of writing an answer, it was this machinery writing answers like that one. So we put the same question, raw, to both of this page's architectures — no retrieval, no rulebook, just the models and their memories — under a small pre-registered rule (five scored calls per arm, temperature 0, one warm-up, beside live traffic; the posture and receipts are in this page's kit):

Three things worth carrying out of that little table. The speeds tell this page's whole story in one glance — the mixture writes about three-quarters again as fast on the identical prompt at the identical moment (213 against 124 tokens a second), exactly as the meeting arithmetic predicts (these are short-prompt runs on today's runtime, so the numbers sit above the published 32k-tier rows and are not comparable to them; the rule travels in the kit). At temperature zero the machines are clockwork — each arm's five replies came back byte-identical, which is why one verbatim answer per cell carries all five. And the answers disagree about the rule — one talks itself into a muddle, one matches the cited ruling's own answer — and nothing about payroll size or meeting size explains which; ruling 4615, with the actual page of the actual rulebook, is what referees. Models answering from memory are a bar argument; the product asks the book. That is the whole reason the machinery of Three librarians exists — and it is why the seminar keeps this one question around: every page teaches a different organ of the same animal answering it.

One thread left deliberately hanging, now paid: we also put the robber to the same model at two different compressions — and what that did (and pointedly did not do) to the answer is the opening of The compressed photograph, the next page in this series — what those Q4_K_M tags actually mean, walked down a four-build ladder.

What this page does not say

It does not say MoE causes the speeds above — our arms differ by vendor, training, attention geometry, and in one case a draft head; this is association under one counting rule on one card, not a controlled experiment. (The controlled version ran 2026-08-21 in a declared maintenance window and its rows landed above — one of its own predictions failed and prints as failed.) It does not say our computed meeting sizes are runtime measurements — they are derived from the model files' own tensor shapes, checkable by anyone with the same command. It does not rank the stopwatch column into a winner — the kit that published it forbids that, and rightly. And it does not say a word about the big cloud models — several open-weight vendors publish total/active pairs for their largest mixtures (284 B total / 13 B active on one, 118 B / 8 B on another — their labels, not our measurements), and nothing on this page measured any of them, so nothing on this page vouches for any of them.

What to take with you

Five things, each with its number from this page:

• The two-number tag is the whole story: payroll and meeting. 30b-a3b means ~33 billion parameters resident, ~3 billion working per token — memory priced by the first, speed by the second.
• Our mixtures out-write dense models a fraction of their size. On one card under one published rule at the 32k tier: a 25.8 B MoE at 124.975 tokens/s against its own 11.9 B dense sibling at 95.195 — while holding 16.27 GiB resident against roughly 7.55.
• The expert fraction understates the real meeting by about half. 8 of 128 experts sounds like 6%; the true active share is 14.4% of bytes once the always-on staff are counted. Divide-and-stop flatters every MoE.
• "Active parameters" is a convention, not a field. No runtime metadata reports it; vendors count it differently (3.70 B vs 2.99 B for the same model, depending on bookkeeping); ours are computed from the tensor shapes and the arithmetic is in the kit.
• Fast is not smart. An arm near the top of our stopwatch column missed both judge-exam floors and posted the lowest narration win-rate in this page's table — which is why seats here are won by exam, and why a stopwatch page like this one will never hand out a chair.

How to check our work — and see it live

The one-command version, on your own machine, if you run a local model under ollama:

curl -s http://localhost:11434/api/show -d '{"model": "<your-model>"}' \
  | grep -o '"expert[^,]*'

A mixture answers with its expert count and its experts-used count; a dense model answers with nothing. That one line is this page's entire detection method, yours in ten seconds — the same fields our kit read — and if the model is sparse, you now also know its expert fraction understates the meeting. Our derivation — every weight bucket, every byte, the tensor-name patterns each bucket was assigned by, and the active-path arithmetic for both mixtures — ships in this page's kit, beside the published stopwatch and residency kits it reads from (the chair trials' and the August arrivals'). Check our sums, or run them on a mixture we've never met.

And the living receipt: the mixture in this page's table is narrating RealKeep's world right now, its meeting a seventh of its payroll by bytes, under the same counting rule as everything above. The machinery shows itself if you ask.

The rest of the seminar

This is the machines' journey, part two. Reading is fast, writing is slow is part one and this page's whole foundation — the group photo, the single-file line, and the trip through the weights that mixtures learned to shrink. Three librarians and a careful reader walks what happens before any of this: how a question finds its page. The stopwatch rows this page leans on live in the chair trials, and the memory prices in the August arrivals — benches with their own stories. Still ahead on this road, tracked in a register we keep of owed work: what those Q4_K_M tags actually mean (the quantization lever this page kept fencing off), what a token costs and why, and the controlled dense-versus-sparse stopwatch this page's honesty section promises. The whole shelf holds every bench behind every claim, failures included. If one of those is the piece you want next, say so — the suggestion box is read.

Licence: CC BY 4.0, the whole page — name the source and link to it.