Mixture of experts, in plain words

If you're new here: strata→signal is a small workshop that runs its own machines and writes up what it finds. This page follows Everyone on the payroll, three at the table, which taught the two numbers on models that fit a card whole, and it is the primer for a comparison still to come between DeepSeek V4.1 Flash and Qwen3.8-Flash-Next. Nothing was measured for this page: every figure is printed first on a page of ours, or read from a maker's own card, config or paper on the date beside it; the one exception, a line from an engine's log in our own records, is dated where it stands.

The words this page leans on

• A token is a word or a piece of one, the unit a model reads and writes.
• A parameter, or a weight, is one of the learned numbers a model is made of; "125B" means about 125 billion of them.
• A layer is one of the stacked stages a token passes through; Qwen3.8-Flash-Next has 48, DeepSeek V4.1 Flash 40.
• An expert is one small sub-network inside a layer. A router is the small learned part of the layer that scores the experts for each token and picks a few.
• Active parameters are the ones a single token meets: the picked experts plus everything that runs for every token.
• The card's memory is on the graphics card. System memory is the computer's own, on the motherboard, which a card reaches far more slowly than its own.
• A spill is any part of the weights kept in system memory because the card has no room.
• Bits per weight is how compactly a file stores each parameter; a tag such as IQ3_S or IQ2_XXS names the compression a file was built at. The quantization ladder measures one model at five such levels, Q3_K_M to Q8_0, and Inside a GGUF file shows what a level is inside the file.
• GB and GiB. A GiB is 1,024 × 1,024 × 1,024 bytes, the unit a computer reports about itself; a GB on a file listing is a billion bytes; a card's label counts in powers of two.

Two numbers on every model card

Qwen's card, read 2026-10-29 (UTC): "Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP"; "Number of Experts: 512"; "Number of Activated Experts: 10 Routed + 1 Shared". DeepSeek's, the same day: "552B backbone parameters"; "1 shared expert and 384 routed experts per MoE layer, activating 6 routed experts per token"; "8B parameters per token during prefill" and "16B during decode". Mixtral 8x7B, an open-weights mixture released in December 2023, stands beside them as the familiar reference.

Qwen and DeepSeek from their model cards and config.json files (num_experts, num_experts_per_tok, moe_intermediate_size, hidden_size; n_routed_experts, n_shared_experts, quantization_config); Mixtral from its paper's abstract (arXiv 2401.04088) and its config.json. DeepSeek's two active counts are its makers' own; the abstract explains the difference only by naming a "Causal Encoder-Decoder (CED) architecture", and this page goes no further.

A tag like 125b-a6b or a suffix like A3B puts the small number after the big one, "A" for activated. Everyone on the payroll found that active counts are a vendor's convention, not a field in the file, and that two vendors can count the same thing two ways.

What an expert is, and who picks

Most of a language model's weights sit in the block of each layer the trade calls the feed-forward network, a wide stretch of arithmetic every token passes through after attention. A dense model has one such block per layer and every token uses all of it. A mixture of experts replaces it with many narrow ones: 512 in each of Qwen3.8-Flash-Next's 48 layers, every one 640 wide on a 2,560-wide model; 384 in each of DeepSeek V4.1 Flash's 40 layers, each 2,304 wide on a 5,120-wide model.

In front of the experts sits the router. For every token it scores each expert and keeps the best few, 10 of 512 in Qwen's case, 6 of 384 in DeepSeek's. Both models also keep one shared expert that is on for every token, which DeepSeek's 2024 paper describes as "aiming at capturing common knowledge and mitigating redundancy in routed experts" (arXiv 2401.06066). The picked experts each read the token's vector, their outputs are weighted by the router's scores and summed, and the token moves to the next layer, where a fresh router picks again: a token through Qwen3.8-Flash-Next meets 48 routers, 480 routed experts and 48 shared ones. No expert is the biology expert or the poetry expert; training carved them out unlabelled, as Everyone on the payroll says.

One expert is three matrices (a gate, an up and a down projection), which is how the expert tensors stand in the Flash-Next file on Inside a GGUF file: 2,560 × 640 × 512 for the gate and up projections and 640 × 2,560 × 512 for the down, three tensors a layer. On that assumption, our arithmetic from the two config files:

• Qwen3.8-Flash-Next: one expert is 3 × 2,560 × 640 = 4,915,200 parameters; 512 in each of 48 layers is 120,795,955,200, about 121 billion of the card's 125B. The 11 a token meets in each layer, over 48 layers, come to 2,595,225,600, about 2.6 billion of the card's "6B activated"; the rest would be attention, routers and the input and output layers, which the card does not break down.
• DeepSeek V4.1 Flash: one expert is 3 × 5,120 × 2,304 = 35,389,440; 384 in each of 40 layers is 543,581,798,400, about 544 billion of the 552B backbone. The 7 a token meets, over 40 layers, are about 9.9 billion of the makers' 16B.

That 544 is a number you may meet in an engine's log. When the Strata engine checked the 2-bit DeepSeek files on one of our machines on 2026-10-29 (UTC), its pack-info line read "all experts 132.1 GB at 1.94 bpw (544 G parameters)": the routed experts' count, never the model's size, which its makers put at 552B for the backbone and ISTA-DASLab, for the whole model with its lookup tables, at "763.2B parameters". Despite the similar name, Strata (github.com/Niko1221/Strata) is an unrelated project; strata2signal has no affiliation with it.

Why the router does not send everything to one favourite is a training matter: the Switch Transformer balanced the load with an extra loss, and the DeepSeek-V3 report "pioneers an auxiliary-loss-free strategy for load balancing" (arXiv 2412.19437); V4.1 Flash's config sets topk_method to noaux_tc, which we read, by its name, as that method.

Where the idea came from, in four lines

• 1991. "Adaptive Mixtures of Local Experts", Jacobs, Jordan, Nowlan and Hinton, Neural Computation 3(1), 79–87: the phrase, and, in its abstract, "many separate networks, each of which learns to handle a subset of the complete set of training cases". We checked the citation and the abstract against Crossref's record and did not read the paper.
• 2017. Shazeer and colleagues put "a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks" inside a language model, with "A trainable gating network" choosing among them (arXiv 1701.06538).
• 2021. The Switch Transformer "selects different parameters for each incoming example": "a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost" (arXiv 2101.03961).
• 2023 to 2024. Mixtral 8x7B, where "each layer is composed of 8 feedforward blocks (i.e. experts)" and "a router network selects two experts" (arXiv 2401.04088), released under Apache 2.0 for anybody to download and run; then DeepSeekMoE, which cut the experts finer and added the shared ones both models on this page use.

Why the big number decides whether it fits

The router picks different experts for different tokens, and nothing says in advance which. So every expert must be within reach when a token arrives, on the card first of all; Hugging Face's 2023 explainer puts it in one line, "all parameters need to be loaded in RAM, so memory requirements are high". Beside the weights the card holds the conversation's working memory, the KV cache, and the engine's own buffers.

The two lookup tables are the exception that proves the rule. Qwen's card says embeddings are "more amenable to offloading than Mixture-of-Experts (MoE)"; DeepSeek's lists "Engram conditional memory (196B parameters, sparsely accessed via token-based lookup)"; ISTA-DASLab's card says the tables "are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM". A table is read a few rows per token, never hauled whole, so it lives in system memory or on disk on every engine we have run, and that is not a spill. Our comparison will label placement the same way: SPILL x % on card, with the lookup table named beside it and never counted as a spill.

Why the small number decides how fast it writes, once it fits

Reading is fast, writing is slow taught the trip: writing one token means hauling through the card every weight that token needs, and a dense model needs all of them, so its writing speed is set by how fast the card can move its memory. A mixture hauls its active slice and leaves the rest at rest. A short history of Mistral has the cleanest receipt we own: on one NVIDIA GeForce RTX 3090 (24 GB), at one 250 W cap and with one prompt, gemma4:26b, a mixture with about 4 billion active parameters, wrote 126.12 tokens a second where the dense mistral-small3.2:24b wrote 34.13, 3.7 times as fast on a model larger on disk. With its cap lifted, the same page finds the dense model reading its weights at about four fifths of the card's published bandwidth, close to the ceiling: the road to faster writing is less to haul. One caution from Everyone on the payroll: on the two mixtures it measured, the always-on parts roughly doubled the bytes a token touches over what the expert fraction alone suggests, so "6 of 384" is a floor on the trip, not its size.

A draft head is a separate lever, with its own piece planned: a small extra network that guesses several tokens for the model to check in one trip, so one haul yields more than one token, which the Strata engine's rows below have and our llama.cpp rows do not.

Fits, or spills, on one 96 GB card

The card is an NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB). The arithmetic below uses its label: a card's label counts in powers of two, so 96 GB is 96 GiB, and no engine gets more room than that, so each shortfall below is a floor. The files are ISTA-DASLab's GGUF builds, compressed copies of each model's weights in the format llama.cpp reads; the sizes are the repositories' own listings.

The Flash-Next row is measured, on the full bench: llama.cpp at commit 3cf03257 logged "offloaded 49/49 layers to GPU"; the Strata engine 0.1.39 logged "expert cache 24576 slots, 46.84 GiB of VRAM" and "100% VRAM resident". The DeepSeek rows subtract published file sizes from the card's label, and nothing more: no DeepSeek file has been measured on any card of ours.

One more ISTA-DASLab build of DeepSeek V4.1 Flash, published on 2026-11-02 (UTC), moves the big number itself rather than the bits a weight. Called the Coder, it removes 192 of the 384 routed experts in every layer, keeping the ones a search picked for code, tool use, long context and vision, and stores the backbone in 72.12 GB, 67.17 GiB. Arithmetic only, as in the rows above: 96 − 67.17 = 28.83 GiB of the card's label would be left for the KV cache and the engine's buffers. It is a smaller model, not the same one squeezed; in its makers' words, "Removing half of a model's experts necessarily reduces its capabilities". Nor has it been measured on any card of ours.

So one model's big number fits a 96 GB card with room to spare at about 3.5 bits a weight, and the other's, with every expert kept, does not at 2, or at 1.75. That is the big number doing its job. What happens next is the engine's choice.

Two ways to spill

By layer. llama.cpp's server README (read at commit 0893ec98, 2026-10-29) lists -ncmoe, --n-cpu-moe N, "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU", and -cmoe, --cpu-moe, "keep all Mixture of Experts (MoE) weights in the CPU". Our rows ran 3cf03257 with --n-cpu-moe 34: on the 24 GB cards llama.cpp kept "the experts of 34 of the 48 layers in system memory", computed them on the processor while writing, and put the rest of every layer on the card; the lookup table stayed in system memory.

By expert. The Strata engine (github.com/Niko1221/Strata) pins all 24,576 expert slots in system memory, 46.84 GiB for the IQ3_S file, and copies the ones its shipped profile ranks hottest to the card, "ranked by routing frequency, no eviction" in its own words: 8,120 of 24,576 on the laptop card, 8,144 on the 24 GB desktop card, with a hit rate of 0.921 at 4K, the share of expert lookups it served from the card. For its DeepSeek files the project's README adds a third tier: "What does not fit in memory is read from your SSD, so it needs a lot of free disk space."

What a spill costs is set by the memory the expert now lives in, or by the processor that computes it there.

By these ratings an expert in the desktop's system memory is reached about 9 times slower than one on its 96 GB card (204.8 against 1,792 GB/s), or computed by the processor instead: a bound within one computer, not a measurement; the laptop's memory is rated at under half the desktop's. The measurement is what the processor did: on the 24 GB desktop card, "those eight threads, not the card, set llama.cpp's writing speed". The same-card rows, each with its engine and draft head named:

All from Qwen3.8-Flash-Next on three cards: the IQ3_S file, a 4,038-token prompt, the median of three. No ratio is drawn between cards: the computers, builds and placements differ. The Strata engine's lead is not placement alone; on the 96 GB card, where both engines had every expert on the card, that page reads its lead as the draft head.

One receipt moves the share on the card and nothing else. On A short history of Mistral, qwen3.5:122b, a mixture of 122 billion parameters with 10 billion active, was split across an NVIDIA GeForce RTX 3090 (24 GB) and the 96 GB card in one computer: with 39.5 % of it on the cards it wrote 38.22 tokens a second; with the workshop's services stopped and 89.6 % on the cards, 71.48. Same model, same box, same minute.

The spill also moves the memory bill off the card. "For the Strata engine, system memory is the gate, not the card": the 64 GB laptop had 2.66 GiB left at a 128K prompt, and even with every expert on the 96 GB card the engine held 49.60 GiB of the desktop's memory.

What it means for your own card

Three habits travel with the table. Check where it sat by the card's own reading, not the server's account of itself: our RTX 3090 against RTX 3090 Ti page corrected itself on 2026-09-21 for exactly that. Expect the quantizers to cut the experts deepest. ISTA-DASLab's DeepSeek card: "the routed experts sit at a much lower bit-width than the dense path at any given budget"; DeepSeek itself ships the routed experts in FP4 and the rest in FP8 (expert_dtype "fp4", quant_method "fp8"); the Flash-Next file keeps its output layer at Q6_K and its expert down-projections at IQ4_NL in 39 layers and Q2_0 in 9. And count system memory before the card. For the Strata engine, on every card we ran it on, it was the gate.

What this cannot say

• Anything measured on DeepSeek V4.1 Flash. Its rows are file sizes subtracted from the card's label, labelled as arithmetic; no speed, placement or hit rate of it has been read on any card of ours.
• A ratio between cards. The rows sit side by side with their conditions; the computers, builds, placements and nights differ.
• How the makers' active counts are made. Qwen's 6B and DeepSeek's 8B and 16B are theirs; our 2.6 and 9.9 billion are the experts alone, under a three-matrix assumption the Flash-Next file supports and the DeepSeek file we did not open.
• Anything about quality, of a mixture against a dense model or of 2 bits a weight against 3.
• A placement ladder of our own. Only the Mistral page's two-point split moves the share on the card with everything else held still.

Who ran this, and thanks

Thanks to Qwen (the model, its card and config, under the Qwen Community License 1.0); DeepSeek (the model, its card, config and papers, MIT); ISTA-DASLab (the GGUF builds of both models and their cards); Georgi Gerganov and the ggml authors (llama.cpp, MIT); the Strata engine's author (github.com/Niko1221/Strata, MIT); Mistral AI (Mixtral, its paper and its config); the authors of the papers quoted; Hugging Face (the cards, the file listings and the 2023 explainer); and NVIDIA (the published bandwidth). None of them owed us anything. No model was run and no card was touched for this page. A small (human) team asked the question and will approve this page; a fleet of AI agents read the sources, re-ran the arithmetic and drafted the prose under that team's rulings.

Sources

Read 2026-10-29 (UTC) and again 2026-11-04 (UTC), with every quote and figure unchanged, unless dated otherwise.

• Our own pages: Why the first word comes sooner, the full bench and Qwen3.8-Flash-Next on three cards (the placement lines, the writing rows, the memory figures); Inside a GGUF file (the expert keys, tensor shapes and file sizes); A short history of Mistral (the dense-against-mixture rows and the split table, whose values are its q1-split-resident-seatsup.json and q1-split-resident-seatsoff.json); Everyone on the payroll, three at the table and Reading is fast, writing is slow (the mechanism); One 3090 against one 3090 Ti (the 2026-09-21 correction).
• Qwen3.8-Flash-Next, model card and config.json: https://huggingface.co/Qwen/Qwen3.8-Flash-Next (weights under the Qwen Community License 1.0; short factual quotes).
• DeepSeek-V4.1-Flash, model card and config.json: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash (MIT); the paper, arXiv 2609.19969: https://arxiv.org/abs/2609.19969.
• ISTA-DASLab, the GGUF builds and their cards: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (revision ed59f920) and https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF (sizes from the repository listing; built from DeepSeek's DeepSeek-V4.1-Flash-1017 upload, whose config.json matches the one quoted here; the quantized weights inherit MIT, per the card); and the Coder, published 2026-11-02 and read 2026-11-04 (UTC): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF (the backbone's size from the repository listing, the 192 kept experts a layer from its kept-experts.json; MIT, per the card).
• Papers: Jacobs, Jordan, Nowlan and Hinton, "Adaptive Mixtures of Local Experts", Neural Computation 3(1), 1991, https://doi.org/10.1162/neco.1991.3.1.79 (citation and abstract checked against Crossref's record; the paper not read); Shazeer et al., arXiv 1701.06538, https://arxiv.org/abs/1701.06538; Fedus, Zoph and Shazeer, arXiv 2101.03961, https://arxiv.org/abs/2101.03961; Jiang et al., "Mixtral of Experts", arXiv 2401.04088, https://arxiv.org/abs/2401.04088 (CC BY 4.0); Dai et al., "DeepSeekMoE", arXiv 2401.06066, https://arxiv.org/abs/2401.06066; DeepSeek-AI, "DeepSeek-V3 Technical Report", arXiv 2412.19437, https://arxiv.org/abs/2412.19437. Abstracts quoted under arXiv's non-exclusive licence.
• Mixtral 8x7B, config.json: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 (Apache 2.0).
• Hugging Face, "Mixture of Experts Explained", 2023-12-11: https://huggingface.co/blog/moe (one short quote; further reading).
• llama.cpp, the server README at commit 0893ec98 (2026-10-29): https://github.com/ggml-org/llama.cpp/blob/0893ec98f5eafad36a43a72cd1c3723d72b0d03a/tools/server/README.md (MIT). Our rows ran commit 3cf03257.
• The Strata engine, README (MIT): https://github.com/Niko1221/Strata. An unrelated project; strata2signal has no affiliation with it.
• NVIDIA, RTX PRO 6000 Blackwell Workstation Edition product page: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/ (a published specification, not a measurement).

Two models read to the byte, no model run, no card touched: a guide, not a bench. Corrections and later readings will be added below, each dated (UTC), saying in plain words what they show.