The bench · six models on three CPUs, no GPU in the path

Two Hours at 12 tok/s — On Battery

exhibit thirty-two The bench
Published 2026-08-31 (UTC)
a small (human) team and a fleet of AI agents — the humans ran the harness and signed the numbers
main measurement window 2026-08-25, 14:09–17:18Z (UTC)
battery test 2026-08-31, 02:38–03:05Z (UTC), same mini PC

We pulled the plug on this bench's own mini PC mid-generation, and it kept answering at twelve tokens a second — for a projected two hours. That test now names the page; the body is what it was built on: the CPU-only bench where the biggest model turned out to be the fast one — a memory-bandwidth story (up to 2.3× for the 30B mixture-of-experts), what it trades in judgment, and the one thread setting that moves any of it by half.

A few words this page leans on

  • Token — the chunk a model reads and writes in, a little under a word (roughly ¾ of one). Everything on this page is counted in tokens; a 256-token answer is about one solid paragraph.
  • Parameter — one of the learned numbers baked into a model. "9B" means nine billion of them. More parameters means more to move for every token — which turns out to be the whole plot here.
  • Prefill — before answering, a model reads your whole prompt in. On a CPU, this is the wait.
  • Decode — writing the answer, one token at a time, in tokens per second (commonly cited: ~4–6 tok/s reads comfortably, ~10+ feels fluid — a rule of thumb, not our measurement).
  • Dense / MoE — a dense model uses all its parameters for every token. A mixture-of-experts (MoE) model wakes only a fraction per token — a 26B model doing ~4B of work per word.
  • Q4 — weights stored in ~4 bits each (Q4_K_M throughout this page): the CPU-realistic packaging.
  • Thread — one CPU worker. A modern chip has many; how many the software actually uses is a setting, and getting it wrong is the most common way to leave CPU speed on the floor. It gets its own section below, because on two of our machines the default was wrong.

Two hours at 12 tok/s — the test that named the page

Last night, with a two-hour provider outage fresh in mind, we plugged this bench's mini PC into a consumer 1500 VA UPS, set a 26-billion-parameter model generating, and pulled the plug mid-sentence. Nothing happened — that is the story. Twelve tokens a second before, twelve after, the battery good for about two hours flat-out and seven and a half idle. The measured tables are directly below; beneath them, the bench this page was already about. (This section was measured 2026-08-31, 02:38:31–03:05:02Z — six days after the bench window, on the same mini PC the bench scored. The bench's own not-measured list, further down, names battery-vs-plugged behavior; this answers the mini-node half of it. The GPU-server half stays unmeasured — by design: its outage machine sheds the GPU within 30 seconds rather than carrying it.)

The headline: a thoroughly mid-range mini PC on a consumer 1500 VA UPS will run a 26 B model at ~12 tokens/second for about two hours on battery — and sit ready to answer for seven and a half. Not "stays on": it generates full ~1,400-token answers, continuously, at full speed, on battery.

The hardware, because you'll ask: this bench's mini rig — a Minisforum MS-01 mini PC, 12th-gen Intel Core i5-12600H (12 cores / 16 threads), 32 GB RAM, no GPU, CPU-only inference. The model: 26 B parameters at 4-bit quantization, ~18 GB resident, 32 K context pinned, kept warm (per-request load times 338–358 ms all night). The battery: a CyberPower PR1500 — 1500 VA on the label, 900 W by its own firmware's nominal, a consumer rack/tower UPS. Whole-box wall draw by the UPS's own meter: 0 W shown idle (real draw ~5 W, below the meter's floor) · ~76 W generating on mains · ~78 W generating on battery.

And the speed did not flinch. Across 12 back-to-back generates spanning the entire test — mains, battery, and back, sampled every 5 seconds (318 samples) — throughput ran 11.62–12.21 tok/s (mean 11.94). The four answers completed on battery averaged 11.87 tok/s against 12.0 on mains: noise. The plug came out mid-generate and the model never noticed.

phasewindow (UTC)UPS statusloaddrawbatteryruntime estimate
idle, on mains02:38:31–02:41OL0%0 W shown (the box's ~5 W real draw is below the UPS meter's floor)100%27,000 s = 450 min
generating, on mains02:41–02:56:56OL8%60–82 W, mean 75.6 W100%~7,575 s
generating, ON BATTERY02:56:56–03:03:00OB DISCHRG8–9%74–81 W, mean 78.2 W100→96%6,186–7,573 s (the estimator hunting between generates)
generating, recovery03:03–03:05:02OL CHRG8%75–76 W96%, chargingrebuilding: 7,301→7,314 s

Two independent readings of "how long would it last": the UPS's own estimator settled around ~7,300 s ≈ 122 minutes under continuous generation, and the linear charge-burn arithmetic (4 percentage points in 6 min 04 s, run to the 10% low-battery floor) says ~130 minutes. Different instruments, same answer: call it two hours. Also visible in the table, and honest to note: on-battery draw ran ~3 W above on-mains for the same work — the inverter's own tax, measured.

The load behind those rows: 12 sequential ~300-word generations against the resident 26 B model, streaming off, model kept warm.

#done (UTC)tokenstok/s#done (UTC)tokenstok/s
102:42:491,52311.85702:54:091,48911.84
202:44:331,25912.21802:56:161,50011.90
302:46:341,43511.949 †02:58:121,38211.96
402:48:271,34011.9910 †03:00:201,50711.85
502:50:141,29812.1711 †03:02:251,47511.85
602:52:031,30712.0712 ‡03:04:531,71311.62

† completed entirely on battery · ‡ started on battery, finished on restored mains.

Between two hours and seven — the real number tracks the traffic. The two measurements are the endpoints. Idle, the mini draws so little the UPS meter reads zero and estimates seven and a half hours; answering flat-out, two hours. A real outage sits between: this box only speaks when asked, so its battery life tracks how busy the night is — a quiet night lives near the seven-hour end, a busy one near two. The relationship is not linear (the UPS's own electronics tax the battery proportionally more at light loads), so we publish the measured endpoints rather than an interpolated table; a half-busy point is measurable the same way if we ever want the curve.

Why a battery test belongs on a CPU bench. The GPU boxes this page deliberately excluded cannot do this and are not asked to. A server carrying a ~500 W-class accelerator under inference would eat even a large consumer UPS in minutes — which is exactly why such a box's outage machine sheds its GPU within 30 seconds of losing mains (measured 2026-08-28: held, it draws 114 W and buys about an hour). The mini node is the inversion of that design: it cannot answer fast, but it can answer for hours on a battery the size of a shoebox, at a speed — ~12 tok/s — that reads faster than most people do. Storm answering, solved for the price of a mini PC and a UPS.

The question our GPU benches never ask

Not every machine we can reach has a usable GPU. So we picked three that don't — for this bench, no GPU was allowed in the path at all — and asked the question our planning kept circling: if every GPU were busy or gone, could a language model still answer, and on what?

Six models chosen to span the CPU-realistic space, one frozen prompt from our rules assistant's live content (505–1,023 tokens depending on each model's tokenizer — the same file, counted differently), five scored runs per cell (three for the 70B), the serving layer everyone actually uses (Ollama, 0.32.15). And because speed without judgment is noise, the models also sat sealed exams — the same instruments our production models take.

Every one of the nineteen scored decode-and-prefill arms proves it never touched a GPU¹ — all nineteen with byte-identical device snapshots before and after. The twentieth file carrying the full receipt is the discarded arm itself: the only snapshot in the pack that changed, which is how it was caught. (The side arms — the concurrency pair, the embedder, the memory-fit run, the thread sweeps and the cold-path probes — ran through the same CPU-only servers, between arms that carry the full receipt on either side, but their own records keep the frozen prompt's hash and the timings rather than the GPU snapshot.)

The three machines, named the way this house names hardware:

  • the server — 128G of ECC RAM, a server-class processor.
  • Rig B — the laptop: 24G-VRAM class, GPU deliberately unused here.
  • the mini — a mini PC with 32G RAM and a 12-core Intel processor.

They are not three of a kind, and the dek's "RAM-heavy server" earns the caveat: one of these is a big machine. But the gap between them is smaller than it looks, and the next section is why.

Read this before the table: CPU speed is a memory-bandwidth story

Measured 2026-08-25 (UTC), 17:57–19:46Z — after the main bench closed, we went back and measured each machine's raw memory bandwidth, so the claim underneath every number on this page stops being an assertion and becomes something you can check.

On a CPU, decoding a token means streaming the model's working weights out of RAM and past the processor — once per token. So decode speed is capped by how fast the machine can move bytes, not by how clever the processor is. We measured that ceiling directly with a STREAM triad (a[i] = b[i] + scalar*c[i] over a 2,289 MiB working set, far larger than any cache on any of these boxes; gcc -O3 -march=native -fopenmp; best of 9 timed reps):

boxbest triadworstspread
the server (128G ECC)100.7 GB/s100.4 GB/s0.3%
Rig B laptop59.3 GB/s49.2 GB/s20.5%
the mini (32G)51.1 GB/s50.9 GB/s0.5%

The server is 1.70× the laptop's bandwidth and 1.97× the mini's; the laptop is 1.16× the mini. That is the whole spread of raw memory speed across a server and two much smaller boxes — under 2×. It is also the answer to "is this portable": the expensive machine is not in a different league here, because this workload rides the memory bus, and memory buses do not scale the way core counts do. (Rig B's 20% spread is real and left un-smoothed: its 24 threads split across 8 fast and 16 small cores with a static schedule, so each rep depends on how the halves interleave. Read it as "roughly 50–59 GB/s." The triad receipts are in the kit.)

The rule to keep in your head: tok/s ≈ bandwidth ÷ bytes streamed per token. A dense model streams (almost) all its weights every token; an MoE streams only its active fraction. Hold that, and the table below stops being surprising and starts being arithmetic. We cash the rule out with a worked check right after it.

The one table to remember

Decode speed, median of five scored runs (three for the 70B), model loaded and warm, 2026-08-25 (UTC), no GPU in the path:

model (all Q4_K_M)classserverRig B †mini †
qwen3.5:9b9B dense13.6 tok/s7.5 tok/s9.1 tok/s
gemma4:12b12B dense11.0 tok/s5.0 tok/s6.0 tok/s
phi4:14b14B dense11.1 tok/s4.4 tok/s5.2 tok/s
mistral-small3.2:24b24B dense7.1 tok/s2.7 tok/s3.3 tok/s
gemma4:26b26B MoE (~4B active)24.7 tok/s13.3 tok/s12.7 tok/s
nemotron-3.5-lightning:30b-a3b30B MoE (~3B active)31.5 tok/s15.4 tok/s14.6 tok/s
llama3.3:70b70B dense2.7 tok/s

Both Intel boxes' columns are Ollama's own default of 4 threads — not their core count. That default understates the laptop badly and the mini mildly, and, oddly, is near-optimal for the two MoEs. The full story, with the cause and a tuned column, is in "Set the thread count" below. The cross-machine ratios on this page are therefore configuration-inclusive, not silicon-pure.

A feel-legend for the tok/s above: ~4–6 tok/s reads at a comfortable pace, ~10+ feels fluid; a 256-token answer (about a paragraph) is ~25 s of decode at 10 tok/s, ~50 s at 5.

The spread behind these medians is tight — across the 19 measured cells the median cell varies just 1.2% min-to-max, the tightest 0.1% (mistral on the laptop) and the widest 6.0% (phi4 on the server); 18 of 19 cells sit within 5%. The full per-cell min–max table is in the kit.

Read it twice, because it inverts the intuition that bigger is slower:

The 30-billion-parameter MoE decodes faster than the 9-billion dense model. On every machine. By 1.6–2.3×. And the two MoEs beat the nearest dense model we tested (the 24B) by these multiples:

machine26B MoE ÷ 24B dense30B MoE ÷ 24B dense
server3.5×4.4×
Rig B laptop4.9×5.7×
mini3.8×4.4×

Now the promised worked check. CPU decode is a memory-bandwidth game — the machine streams the working weights past the processor for every token — and an MoE streams only its active fraction. If that is true, a model's tok/s should track each box's measured bandwidth. It does: gemma4:26b decodes at 24.7 / 13.3 / 12.7 tok/s on the server / laptop / mini, and dividing by each box's bandwidth (100.7 / 59.3 / 51.1 GB/s) gives 0.245 / 0.224 / 0.248 — the same number to within a tenth, i.e. the same ~4 GB of weights-and-cache move per token on all three boxes, and speed is just that constant times the bus. A 26B model that wakes ~4B per token pays a ~4B toll with a big model's judgment. On CPU decode, sparsity is the whole strategy — and the moderation exam below prices what it costs.

The Rig B row in that ratio table is the tell, and the reason it says 4.9× and 5.7× rather than something smaller: on the laptop the 24B dense ran thread-starved (2.7 tok/s at Ollama's 4-thread default), which inflates its denominators. Give that dense model its fair thread count and both of those range-tops fall by roughly half — the recompute is in "Set the thread count" below, and the inversion still survives it on every machine.

The 70B row answers "can it even": yes — 2.7 tok/s and 41 GiB resident; ~95 s of pure decode for a 256-token answer, ~132 s from a fresh prompt. A last resort that exists, which is precisely what a last resort is for.

The wait: prefill is the real CPU tax

Decode is only half the feel. Before the first word, the model reads your prompt — and on a CPU that reading is slow in a way GPU users never see. Time to the answer's first token, nothing cached, model already loaded, median of five cold-prefix runs, same date and no-GPU verification as above:

model (all Q4_K_M)prompt tokensserverRig B †mini †
qwen3.5:9b5374.9 s14.1 s16.0 s
gemma4:12b5366.9 s22.2 s25.6 s
phi4:14b5058.1 s26.4 s35.4 s
gemma4:26b (MoE)5363.7 s8.5 s11.4 s
nemotron-3.5-lightning:30b-a3b (MoE)5324.0 s9.9 s14.1 s
mistral-small3.2:24b1,02324.6 s82.4 s97.7 s
llama3.3:70b50537.6 s

The three rows worth staring at are the extremes: the fastest (the 26B MoE), a typical dense baseline (the 9B), and the worst case (the 24B dense) — the rest fill in the gradient between them. Yes: ask the dense 24B a fresh question on the mini and you wait a minute and a half before the first word — with the honest footnote that its tokenizer read this one prompt as 1,023 tokens, roughly double the others; at its measured 10.5 prefill tok/s a 537-token prompt would cost it ~51 s. Per token, the MoE prefills ~4.5× faster than the dense control; on this prompt it answered ~8.6× sooner — and part of that last figure is simply the doubled token count. That longer prompt has a second-order cost on the other side too: 1,023 tokens make a larger KV cache to stream on every decode step, a small extra drag on the 24B's decode as well as its prefill (small here — 1,023 tokens against a 4,096-token context — but not zero, and it always points the same way).

A caution for anyone benching this themselves: repeat the same prompt and Ollama caches the prefix. On the cell that gave it away — the 9B model on the mini — a re-sent prompt's first token came back in 211 ms against 16.0 s for a fresh one with the model already loaded: 76× faster, and on the worst cell (the 24B dense, same machine) 310×. Counted from a cold start instead, with the model load included, the same cell overstates by 96×. We caught it because the implied prefill arithmetic was physically impossible for the silicon. We then decomposed the cold path and cross-checked one cell against a unique-prefix probe to within 1.4%. Cached-prefix first tokens are genuinely 99–712 ms across all three machines — lovely, and only honest if your workload actually repeats its prompts.

Two more numbers a planner needs: cold start (model not loaded: add 4.3–20.1 s of load for the six, 49.4 s for the 70B — measured warm-disk, so floors), and the finding that surprised us most: more cores buy prefill, not decode. Against the mini, the server's prefill advantage runs 3.1–4.4× across the six models, while its decode advantage runs only 1.5–2.2× — roughly twice the leverage on the reading half, model by model between 1.6× and 2.2×. Prefill parallelizes across cores; decode just waits on RAM, exactly as the bandwidth section predicts. That split is also the hinge of the next section.

Set the thread count — the one free lever, and the exception that proves the rule

Measured the same day, 2026-08-25 17:57–19:46Z (UTC), after the main bench closed: we went back and asked what these machines do when you stop letting the defaults choose the thread count. The short version — the dense models were leaving up to ~90% on the table, and the two MoEs were, against all intuition, better off with the wrong-looking default. There is no single right setting, and that is the finding.

First, why the default was 4. Ollama does not choose a thread count at all; the library beneath it, llama.cpp, does — and on these two Intel chips it chooses wrong. Both are hybrid parts: a few fast "performance" cores plus a cluster of small "efficiency" cores. llama.cpp's counter walks the performance cores and, after counting each one, unconditionally skips the next CPU as if it were that core's hyperthread sibling. On the laptop, which has no hyperthreading at all, that skip steps over a real performance core — so the count comes back 4 when the machine has 8 performance cores (of 24 logical), and Ollama runs on 4. We proved this is the cause rather than a cgroup, an affinity mask, or an environment variable — all three were checked and are clear — with a discriminating test. Both Intel boxes expose 8 performance logical CPUs — the laptop as 8 real cores with no hyperthreading, the mini as 4 real cores with hyperthreading on. The heuristic walks those eight and skips every second one as a sibling, so both land on 4: on the mini that is the correct performance-core count and the only loss is the 8 efficiency cores it declines to use, while on the laptop it is half the performance cores, before the efficiency cores are even considered. A rule that merely counted performance cores would have predicted 8 on the laptop; Ollama used 4. Only the halving rule survives both boxes. The server, not a hybrid part, never enters that path and lands on its 32 physical cores — confirming the fallback branch independently. Receipts — the launch argv with zero -t flags, the runner's own n_threads = 4 startup line, the exported common_cpu_get_num_math symbol, and a CPUID probe of every logical core — are in the kit.

What tuning buys — for the dense models. Pin num_thread to each box's measured dense optimum (the laptop plateaus at 16–20 of its 24 cores; the mini at 12 of 16) and every dense model speeds up, for free:

modelclassRig B default, t=4 (tok/s)Rig B tuned, t=20 (tok/s)gain (×)mini default, t=4 (tok/s)mini tuned, t=12 (tok/s)gain (×)
qwen3.5:9b9B dense7.513.11.759.110.11.11
gemma4:12b12B dense5.09.51.916.07.11.19
phi4:14b14B dense4.48.21.865.26.41.22
mistral-small3.2:24b24B dense2.75.11.903.34.01.20

The laptop was thread-starved, not slow: at the default it decoded the 9B slower than the smaller mini, which is what made this worth chasing; given its cores it lands level with the server. The mini gains less because at 4 threads it is already close to its bandwidth ceiling — its raw memory speed (51 GB/s) is the wall, and threads cannot buy past a wall. (Measured with the parent bench's own scoring — 1 warmup + 5 scored, median-of-5, num_predict 256 — which reads a hair below the earlier best-of-2 sweep that first flagged the laptop at 1.78×; both agree the plateau is 16–20 cores. The tuned arms and the sweeps are in the kit.)

Now the exception. Force that same dense-optimal thread count onto the two MoEs and they get slower — much slower:

MoE modelRig B default, t=4 (tok/s)Rig B at t=20 (tok/s)mini default, t=4 (tok/s)mini at t=12 (tok/s)
gemma4:26b13.38.4 (0.64×)12.77.4 (0.59×)
nemotron-3.5-lightning:30b-a3b15.48.5 (0.56×)14.67.3 (0.50×)

A dedicated thread sweep of the 30B MoE on the mini shows just how sharp the cliff is: it peaks at t=4 (14.4 tok/s) — Ollama's accidental default — sags to 7.3 at t=12, and at t=16 (all 16 logical CPUs) collapses to 1.27 tok/s, an 11× fall. An MoE that only wakes ~3B parameters has so little per-token work that piling on threads buys nothing and the coordination overhead — threads fighting over the same starved memory bus, plus the small cores dragging a static schedule — swamps it. So the machines want opposite settings: dense models want every core, MoEs want a handful, and the buggy 4-thread default happens to sit near the MoE optimum while starving the dense. (The mini's own MoE thread sweep is in the kit.)

What this does to the headline. Give each model its own best measured thread count — dense at its tuned optimum, MoE at its ~4-thread peak — and the MoE-vs-dense advantage is smaller but still decisive:

machine26B MoE ÷ 24B dense30B MoE ÷ 24B dense30B MoE ÷ 9B dense
server (as measured — 32 threads, not re-swept)3.5×4.4×2.3×
Rig B laptop2.6×3.0×1.2×
mini3.2×3.7×1.4×

The MoE-over-dense range tightens from 3.5–5.7× (default threads) to 2.6–4.4× (each model fair), and both of the old range-tops — the laptop's 5.7× and 4.9× — were exactly the cells where the dense control ran thread-starved. The headline inversion survives intact: the 30B MoE still out-decodes the 9B dense on every machine, by 1.2× on the fairest laptop reading up to 2.3× on the server. Bigger is still faster; tuning just tells you honestly by how much.

The judgment column: same sealed exams

All six models sat the field exam (60 items, three tasks: answer from a rules excerpt; refuse to answer what the excerpt doesn't say; extract strict JSON; bars at 85%/70%/85% and 80% overall): all six passed every bar — 57 to 59 out of 60. The exam saturated: at this generation, 9B+ models clear the floor a small seat needs. That is itself the finding — the small end of this table is not a toy tier.

Four of them — plus the incumbent as control — then sat the moderation exam (374 real items, three passes, scored as agreement with our production judge; the control reproduced its baseline exactly). The 12B dense model agreed 99.5%, matching the production judge's own false-positive rate (2 of 329 clean items); the 9B: 99.2%. Both MoE models scored a touch lower (98.7–98.9%). But look at the raw disagreement counts before ranking anyone: against 374 items the candidates differ from the baseline by 2 items (the 12B), 3 (the 9B), 4 (the 30B MoE), and 5 (the 26B MoE) — and every single one of those disagreements, for every model, is on clean text: a false-positive difference in one direction or the other, never a missed offense. Three of the four candidates actually fixed one of the incumbent's own two false positives. A 3-item gap on a 374-item set is not a quality ranking, and we decline to report it as one; what it supports is "these are all in the same tight band" — the same refusal to crown a winner on a handful of clean-text flags that our serving-scheduler bench reached on the GPU side (its write-up is still in the workshop). On these 374 items no model missed anything actually offensive; we cannot speak past the set. (The exams ran on a GPU box for speed — quality rides the weights, not the silicon; the serving-lane caveat from our GPU bench is why we say "weights," not "model tag.")

So the CPU-speed champion (the MoE) sits at the bottom of a very shallow judgment band, and the top of that band (the 12B dense) decodes at a readable-but-leisurely 6 tok/s on the mini. Engineering is trades all the way down; at least now they're priced.

What this buys the house (and maybe yours)

  • A real emergency tier exists. A 26–30B MoE on the 32G mini — resident, zero swap, the 30B leaving 5.4 GiB of the machine's 30.6 GiB usable RAM free at its tightest — answers at 12.7–14.6 tok/s with an 11–14 s fresh-prompt wait. A genuinely usable assistant with no GPU anywhere, and (from the section above) one you should leave on ~4 threads, not "all of them."
  • The embedder never needed the GPU. 12–26 embeddings per second across the three machines (100 sequential short texts, 768-dim) — retrieval and memory pipelines survive a GPU outage untouched.
  • Check your thread count — but know your model. On Rig B, ~1.8× of free decode speed for the dense models sat behind one setting (num_thread); on the mini the same lever bought only ~1.2× because that box is bandwidth-bound. And for an MoE the lever runs the other way — leave it low.
  • Concurrency barely helps: two simultaneous streams of the 26B MoE on the server bought 1.38× total throughput — aggregate 23.8 → 32.9 tok/s — while each stream slowed from 24.5 to 17.4 tok/s (median reps). CPU serving is a one-reader world; plan for a queue, not a crowd.

What we did NOT measure

The load-bearing caveat first: every cross-machine ratio on this page is configuration-inclusive, not silicon-pure. Thread defaults differed — and we now show both the default and the fair-thread numbers rather than leaving that implicit — but the cpufreq governors also differed and were root-gated (the mini is measured in its least aggressive power state, so its figures are a floor). Beyond that: battery-vs-plugged behavior (the mini node's half is answered in the headline section above, measured 2026-08-31), non-Q4 quantizations on CPU, sustained thermals past each arm's window, fully-cold-disk loads (no root on two machines; load times are floors), and Rig B's GPU — deliberately parked.

What to take with you

  • On CPU decode, sparsity is the strategy: the 30B MoE out-decodes the 9B dense everywhere, and the 24B dense by 4.4–5.7× at stock settings — 3.0–4.4× once every model gets a fair thread count. Either way, the big sparse model wins.
  • Prefill is the CPU tax nobody quotes: fresh-prompt first tokens run seconds-to-minutes, and repeated-prompt benches flatter themselves 76–310× via the prefix cache.
  • Quality cleared the floor everywhere we could test it: all six passed the field exam's every bar; the 12B dense matched the production moderation judge to 99.5% — and the whole scored field lands within a 5-item band on 374 items, too tight to rank. Speed and judgment are near independent here, not opposed.
  • A 32G mini PC is a real fallback inference node: the 30B MoE resident at 14.6 tok/s scored (15.2 in the memory-fit run), zero swap.
  • Decode is bandwidth; prefill is cores. More cores buy prefill, not decode — memory bandwidth across a server and two small boxes spans under 2× (100.7 / 59.3 / 51.1 GB/s), and decode speed tracks it. One default thread setting understated a machine's dense models by up to 1.8×, and the same setting cranked up would have halved its MoEs. Set it per model, not per machine.

How to check our work

Every model here is a public tag (ollama pull gemma4:26b and kin, all Q4_K_M); every counting rule is stated where its number stands; the frozen prompt is hashed (90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c) and its per-model token counts printed above. If you compare those counts against our serving-scheduler bench when it lands, they will not match — that is expected: this page reports Ollama's prompt_eval_count, which is taken after the chat template is wrapped around the prompt (gemma 536, mistral 1,023), while that bench reports vLLM's /tokenize on the raw text (gemma 524, mistral 517). Same prompt, two counters, one templated and one not; mistral's tokenizer also expands this particular rules body far more than gemma's, which is why its row reads ~1,023.

The prompt itself came from RuleSage's own live content — you can ask it the same question the bench froze, right now, and watch a warm GPU answer in under a second what the mini takes eleven to start.

The kit is beside this page: 31 scored raw JSON files plus one arm we discarded as contaminated — kept in the pack and named as such, because losses ride beside wins — plus the supplement: tuned-thread arms, thread sweeps, the STREAM-triad sources and receipts, the CPUID core-type probe, and the per-cell dispersion table. If you have a spare machine and an afternoon, the whole thing reproduces: ollama pull the six tags, send one prompt with num_predict 256, take the median of five, and before you believe any first-token number, send a different prompt — that one step is the difference between our 76× and the truth. If a number here does not reproduce from the kit, say so at the contact desk, where a person reads every message.

Who ran this, and thanks

The measurements were run by a small human team with a fleet of AI agents doing the fetching, scripting and cross-checking; the humans set the method, ran the harness, and signed every number. The thing that surprised us most was not the headline — sparse-beats-dense on a bandwidth-bound machine is, in hindsight, arithmetic — but the MoE's thread cliff: the same lever that makes a dense model 1.8× faster makes a mixture-of-experts collapse to a tenth of its speed. We went in expecting one right answer per box and found one right answer per model.

Thanks are owed to the people who made the pieces we merely measured: the teams behind qwen3.5, gemma4, phi4, mistral-small, nemotron, and llama3.3, whose weights are the whole subject here; and to the authors of Ollama and llama.cpp — including for the thread-count quirk we spent a section chasing, which is a two-line heuristic doing its honest best on hardware it did not expect. Open weights and an open runtime are what let a bench like this exist at all.

The rest of the seminar

The two ideas this page stands on were taken apart earlier in the series: Everyone on the payroll, three at the table is where dense models, mixtures of experts and "active weights" get explained properly — this page is what happens when you run that argument on a machine with no graphics card at all. Reading is fast, writing is slow is prefill and decode in plain English, on a card, which is the same asymmetry the prefill table here measures at a hundredth the speed. And The compressed photograph is what the Q4_K_M on every row of every table actually means.

Forward: What 150 Watts Buys puts the same frozen prompt on the GPU side of the estate and turns the card's power dial down instead of its thread count — the two pages share a prompt hash, so their numbers are directly comparable. The seat trials are how a model earns a production job here in the first place.

The whole shelf holds the benches behind the claims we publish, failures included — and the serving-scheduler bench these comparison rows come from is still in the workshop; when it lands it will be here. If there is a piece of the machinery you want opened next, say so — the suggestion box is read by a person.

¹ In plain terms: no arm used a GPU. In technical terms, three receipts per arm — size_vram: 0 from Ollama's /api/ps; no bench process on any GPU in nvidia-smi --query-compute-apps; and the full nvidia-smi device snapshot byte-identical before and after each arm (the diffed artifact). On the two machines with an NVIDIA card CUDA_VISIBLE_DEVICES and every other GPU backend were blanked; the mini has no discrete GPU at all.

Released under the house licence; main measurement window 2026-08-25 14:09–17:18Z (UTC) — first scored arm to last; the tuned-thread, bandwidth and dispersion additions ran 17:57–19:46Z the same day. Three machines named by class above, Ollama 0.32.15, all models Q4_K_M GGUF.

elsewhere in the workshop

a strata→signal property · hello@strata2signal.com · say hello