# The measurement record

Every headline figure on the exhibit page, with the file it comes from and the command that
re-derives it **from this kit** — not from the page's own tables, and not from the bench's
working ledgers, which are not published here.

Run this file's commands from the kit's own root directory. Read `counting-rules.md` first:
two of the rules are traps, and a reader who applies the obvious rule instead of the stated one
gets numbers that are wrong by 96× and by 78%.

**Windows.** The main bench ran **2026-08-25, 14:09:19Z to 17:17:57Z** — first scored arm to
last file closed in `runs/`. The supplement (tuned threads, bandwidth, dispersion, the MoE
sweep) ran later the same day and its records span **17:57:19Z to 19:45:39Z**; that opening
stamp is a *finish* time — the first sweep arm does not record its own start, so the supplement
began somewhat before it.

---

## 1. Decode throughput — the headline table

Six models, three machines, plus the 70B on the server: **19 cells, all 19 reproduce exactly.**

| model | class | server | laptop (Rig B) | mini |
|---|---|---|---|---|
| `qwen3.5:9b` | 9B dense | 13.59 tok/s | 7.45 tok/s | 9.11 tok/s |
| `gemma4:12b` | 12B dense | 10.98 tok/s | 4.95 tok/s | 5.95 tok/s |
| `phi4:14b` | 14B dense | 11.09 tok/s | 4.41 tok/s | 5.20 tok/s |
| `mistral-small3.2:24b` | 24B dense — the dense control | 7.09 tok/s | 2.70 tok/s | 3.31 tok/s |
| `gemma4:26b` | 26B MoE, ~4B active | 24.68 tok/s | 13.26 tok/s | 12.66 tok/s |
| `nemotron-3.5-lightning:30b-a3b` | 30B MoE, ~3B active | 31.47 tok/s | 15.35 tok/s | 14.61 tok/s |
| `llama3.3:70b` | 70B dense (n=3) | 2.70 tok/s | not run | not run |

```
python3 -c "
import json,glob,statistics as st
for p in sorted(glob.glob('runs/cpu-*-m[1-6].json'))+['runs/cpu-server-x2.json']:
    j=json.load(open(p))
    print(f\"{p.split('/')[-1][:-5]:22s} {j['model_tag']:32s} {st.median(r['decode_tok_s'] for r in j['runs']):6.2f} tok/s\")"
```

**The headline inversion** — the *largest* model tested decodes faster than the *smallest* on
every machine: **2.31× / 2.06× / 1.60×** (server / laptop / mini). The MoE-over-dense-control
ratios are **3.5× and 4.4×** (server), **4.9× and 5.7×** (laptop), **3.8× and 4.4×** (mini).

## 2. True prefill — the cold column

Cold runs only (`counting-rules.md`, rule 2). **19 cells, all 19 reproduce exactly.**

| model | server | laptop | mini |
|---|---|---|---|
| `qwen3.5:9b` | 109.4 tok/s | 38.2 tok/s | 33.5 tok/s |
| `gemma4:12b` | 78.0 tok/s | 24.2 tok/s | 20.9 tok/s |
| `phi4:14b` | 62.3 tok/s | 19.1 tok/s | 14.3 tok/s |
| `mistral-small3.2:24b` | 41.5 tok/s | 12.4 tok/s | 10.5 tok/s |
| `gemma4:26b` | 143.7 tok/s | 63.0 tok/s | 46.9 tok/s |
| `nemotron-3.5-lightning:30b-a3b` | 132.2 tok/s | 54.0 tok/s | 37.9 tok/s |
| `llama3.3:70b` | 13.4 tok/s | — | — |

```
python3 -c "
import json,glob,statistics as st
for p in sorted(glob.glob('runs/cpu-*-m[1-6].json')):
    j=json.load(open(p))
    print(f\"{p.split('/')[-1][:-5]:22s} {st.median(c['prefill_tok_s'] for c in j['cold_load']):7.1f} tok/s\")"
```

## 3. More cores buy prefill, not decode — the per-model ratios

The server's advantage over the mini, model by model. **This is the section's whole thesis and
it is a range, not a single multiplier**: the ratio of the prefill ratio to the decode ratio
runs 1.57× to 2.19× with a median of 1.93×, and **three of the six sit below 2×.**

| model | decode ratio | prefill ratio | prefill ÷ decode |
|---|---|---|---|
| `qwen3.5:9b` | 1.49× | 3.27× | **2.19×** |
| `gemma4:12b` | 1.85× | 3.72× | **2.02×** |
| `phi4:14b` | 2.13× | 4.35× | **2.04×** |
| `mistral-small3.2:24b` | 2.14× | 3.97× | **1.85×** |
| `gemma4:26b` | 1.95× | 3.06× | **1.57×** |
| `nemotron-3.5-lightning:30b-a3b` | 2.15× | 3.49× | **1.62×** |
| | **1.49–2.15×** | **3.06–4.35×** | median **1.93×** |

Rounded for publication: prefill **3.1–4.4×**, decode **1.5–2.2×**. Note the exact bottom of
the ratio-of-ratios range is **1.57×**, which rounds to 1.6 — a page that says "between 1.6×
and 2.2×" is rounding its own floor upward by 0.03.

```
python3 -c "
import json,statistics as st
m=lambda p: st.median(r['decode_tok_s'] for r in json.load(open(p))['runs'])
c=lambda p: st.median(x['prefill_tok_s'] for x in json.load(open(p))['cold_load'])
for k in ('m1','m2','m3','m4','m5','m6'):
    d=m(f'runs/cpu-server-{k}.json')/m(f'runs/cpu-mini-{k}.json')
    p=c(f'runs/cpu-server-{k}.json')/c(f'runs/cpu-mini-{k}.json')
    print(f'{k}  decode {d:.2f}x  prefill {p:.2f}x  ->  {p/d:.2f}x')"
```

## 4. The GPU receipt — exactly which arms carry it

**Nineteen scored decode-and-prefill arms**, and all nineteen carry all three receipts with
byte-identical before/after snapshots. A twentieth file carries the same three receipts and is
**the arm the bench discarded** — its snapshot is the only one in the pack that changed, which
is how the bench knew to throw it out. Twelve further files carry no GPU snapshot at all.

| group | files | all three receipts? |
|---|---|---|
| the 18 main decode/prefill cells + the 70B | 19 | ✅ all three, snapshots identical |
| `cpu-server-m1.CONTAMINATED.json` (discarded) | 1 | ✅ all three — **snapshot NOT identical: `0, 38764 MiB` → `0, 46719 MiB`**, `size_vram` still 0 |
| the concurrency pair `cpu-server-m5-c1/-c2` | 2 | ❌ none |
| the three embedder arms `cpu-*-x1` | 3 | ❌ none |
| the memory-fit arm `cpu-mini-m6-memfit` | 1 | ❌ none |
| the two thread sweeps `threadsweep-cpu-*-m1` | 2 | ❌ none |
| the four cold-path probes `fresh-*` / `firstpass-*` | 4 | ❌ none |

**The findings that rest on a receipt-less arm, named:** the 1.38× concurrency figure; the
12–26 embeddings per second; the 15.2 tok/s memory-fit run and its swap figures; the earlier
best-of-2 sweep that first flagged the laptop at 1.78×; and the unique-prefix cross-check. All
twelve ran through the same CPU-only servers between arms that carry the full receipt on either
side, but their own records cannot prove it and the three fields cannot be re-derived after the
fact. **The evidence is fine; only an absolute quantifier would be wrong.**

```
python3 -c "
import json,glob
r=[(p.split('/')[-1], {'size_vram','nvidia_before','nvidia_after'} <= set(json.load(open(p))))
   for p in sorted(glob.glob('runs/*.json'))]
print('all three:', sum(1 for _,v in r if v), 'of', len(r))
print('without:  ', [n for n,v in r if not v])"
python3 -c "
import json,glob
for p in sorted(glob.glob('runs/*.json')):
    j=json.load(open(p))
    if j.get('nvidia_before') and j['nvidia_before']!=j.get('nvidia_after'):
        print('SNAPSHOT CHANGED:', p, j['nvidia_before'], '->', j['nvidia_after'], 'size_vram', j['size_vram'])"
```

## 5. Memory bandwidth, and the constant it predicts

STREAM triad, 2,289 MiB working set, best of 9 scored reps. Receipts in `receipts/triad-*.txt`.

| box | best triad | worst triad | spread (of the worst) |
|---|---|---|---|
| server | **100.69 GB/s** | 100.36 GB/s | 0.3% |
| laptop | **59.31 GB/s** | 49.21 GB/s | 20.5% |
| mini | **51.12 GB/s** | 50.85 GB/s | 0.5% |

Ratios: server **1.70×** the laptop, **1.97×** the mini; laptop **1.16×** the mini.

**The worked check that makes the mechanism falsifiable.** Divide each machine's 26B-MoE decode
rate by its measured bandwidth and the same constant falls out on all three:

```
24.675 ÷ 100.69 = 0.245     13.258 ÷ 59.31 = 0.224     12.657 ÷ 51.12 = 0.248  tok/s per GB/s
1 ÷ 0.245 = 4.08 GB moved per token
```

A reader with a calculator can re-derive the page's central claim from two published columns.

## 6. Latency, four ways

| quantity | six models, range | 70B |
|---|---|---|
| first token, cached prefix | **99–712 ms** (min: `phi4:14b` on the server; max: `gemma4:12b` on the laptop) | 378 ms |
| first token, fresh prompt, model loaded | 3.7 s – 97.7 s | 37.6 s |
| model load (warm disk — a floor, not a cold read) | **4.32 s – 20.11 s** | **49.40 s** |

**The cache trap:** 210.8 ms cached against 16,031 ms fresh with the model already loaded =
**76×** on the cell that gave it away; **310×** on the worst cell (24B dense, same machine);
**96×** on the same cell if you count from a cold start with the model load included. Three
numbers, one finding, three denominators — see `counting-rules.md`, rule 3.

**The cross-check:** the unique-prefix probe measured the same cell directly — a *different*
prompt on a resident model, so nothing could be cached — and got a median of **16,254 ms**
against the **16,031 ms** derived by subtracting the load time from a cold run. **1.4% apart**,
two independent routes to the same number.

## 7. The 70B, and whether it fits

`llama3.3:70b` on the server: **2.70 tok/s**, n=3, **41.16 GiB resident**. A 256-token answer
is ~95 seconds of decode; from a fresh prompt with the model already loaded, ~132 seconds
before you have the whole thing.

**The memory-fit arm** (30B MoE on the 32G mini): **15.2 tok/s**, 5.35 GiB of 30.57 GiB left
free at the low-water mark, and **no swap traffic at all** during the run — swap used sat at
16.0 MiB before and during, all four traffic deltas zero, zero major faults. The arm's own
verdict field reads *"FITS IN RAM - no swap traffic during the arm."* (Note the wording:
"no swap traffic," not "zero swap" — 16 MiB was already in use before the arm started.)

## 8. Threads — the default, the tuned column, and the reversal

| model | laptop default (t=4) | laptop tuned (t=20) | gain | mini default (t=4) | mini tuned (t=12) | gain |
|---|---|---|---|---|---|---|
| `qwen3.5:9b` | 7.45 tok/s | 13.05 tok/s | **1.75×** | 9.11 tok/s | 10.12 tok/s | **1.11×** |
| `gemma4:12b` | 4.95 tok/s | 9.46 tok/s | **1.91×** | 5.95 tok/s | 7.10 tok/s | **1.20×** |
| `phi4:14b` | 4.41 tok/s | 8.21 tok/s | **1.86×** | 5.20 tok/s | 6.36 tok/s | **1.22×** |
| `mistral-small3.2:24b` | 2.70 tok/s | 5.14 tok/s | **1.90×** | 3.31 tok/s | 3.99 tok/s | **1.20×** |

*All figures are decode tok/s, median of five scored runs. The tuned columns are the boxes'
measured optima; the mini's t=12 is a tie with t=16 broken on reproducibility, not speed
(`prereg.md`, DS5).*

**And the same lever reversed, on the two MoE models** — here t=20 / t=12 are *forced*
settings, not measured optima:

| model | laptop default | laptop forced t=20 | mini default | mini forced t=12 |
|---|---|---|---|---|
| `gemma4:26b` | 13.26 tok/s | 8.44 tok/s (**0.64×**) | 12.66 tok/s | 7.42 tok/s (**0.59×**) |
| `nemotron-3.5-lightning:30b-a3b` | 15.35 tok/s | 8.53 tok/s (**0.56×**) | 14.61 tok/s | 7.32 tok/s (**0.50×**) |

**The cliff, measured directly** — a dedicated sweep of the 30B MoE on the mini:

```
t=4  14.436 tok/s   t=6  13.604   t=8  12.524   t=12  7.280   t=16  1.273 tok/s
14.436 ÷ 1.273 = 11.3× fall
```

**What this does to the MoE-over-dense claim.** At Ollama's default threads the MoE advantage
reads **3.5–5.7×** across the three machines. Give the dense control its own tuned thread count
and leave the MoEs at their (better) default, and the same advantage reads **2.6–4.4×** — both
range-tops fall by roughly half. The deflated range is the honest one and is the published one.

```
python3 -c "
import json,statistics as st
m=lambda p: st.median(r['decode_tok_s'] for r in json.load(open(p))['runs'])
for b,t in (('laptop',20),('mini',12)):
    d=m(f'supplement-runs/tuned-cpu-{b}-m4.json')
    print(f'{b} FAIR: 26B/24B {m(f\"runs/cpu-{b}-m5.json\")/d:.1f}x  30B/24B {m(f\"runs/cpu-{b}-m6.json\")/d:.1f}x')"
```

## 9. Dispersion — the min/max behind every median

Nineteen cells. **Median cell varies 1.2% min-to-max; tightest 0.07%** (`mistral-small3.2:24b`
on the laptop); **widest 5.99%** (`phi4:14b` on the server); **18 of 19 within 5%.** Rule:
(max − min) ÷ median. Full table in `analysis/dispersion.md`.

## 10. Concurrency — one reader at a time

| | aggregate | per stream |
|---|---|---|
| one stream | 23.78 tok/s | 24.51 tok/s |
| two streams | 32.91 tok/s | 17.42 / 17.42 tok/s |
| | **1.38× total** | each stream **−29%** |

**A denominator note.** The three concurrency-1 reps produced per-stream figures of 23.869,
24.588 and 24.510 tok/s. The published aggregate, 23.782, is the **median rep** — which is the
third one, whose per-stream is **24.51**. A reader who takes 24.588 (the second rep's) is
reading a different rep from the one the aggregate comes from. This kit publishes both.

## 11. Quality — a floor check and a refusal to rank

**The field exam saturated.** All six models: 59 / 59 / 58 / 58 / 59 / 57 out of 60, every one
PASS on every bar. That is the finding — at this generation, 9B and up clear the floor a small
seat needs. It is not a ranker and this kit does not rank on it (`quality/field-exam.txt`).

**Moderation, 374 items, against the incumbent** (`quality/classifier-agreement.json`):

| model | item agreement | disagreements |
|---|---|---|
| `mistral-small3.2:24b` (incumbent) | 374/374 — exact | — |
| `gemma4:12b` | 372/374 = **99.5%** | 1 clear→hold, 1 hold→clear |
| `qwen3.5:9b` | 371/374 = **99.2%** | 2 clear→hold, 1 hold→clear |
| `nemotron-3.5-lightning:30b-a3b` | 370/374 = **98.9%** | 4 clear→hold |
| `gemma4:26b` | 369/374 = **98.7%** | 4 clear→hold, 1 hold→clear |

**Every disagreement falls in the `clean` set.** All five models scored 20/20 on the offensive
items and 25/25 on the hard negatives — *nobody missed an offense.* The incumbent's own false
positive rate is 2 of 329 clean items, and **three of the four candidates fixed one of those
two** (`hold->clear` = 1 for the 12B, the 9B and the 26B; 0 for the 30B).

A 5-item spread on a 374-item set is not a quality ranking, and this kit declines to report it
as one.

## 12. What this kit contains, in one line

**31 scored raw run records plus one arm discarded as contaminated and kept under that name**,
23 supplement records, 5 hardware receipts, the frozen prompt the printed hash belongs to, the
pre-registration with all eight deviations, and the counting rules. Every model is a public
`Q4_K_M` tag; the engine is Ollama 0.32.15 on all three boxes; the prompt's sha256 is
`90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c`. The prompt's TEXT is **not
in this kit** — it is held pending a licensing call (README, "The one file this kit does not
contain"), so the hash is what lets you check a copy you already hold, and what proves all
thirty-one scored arms sent the same bytes.

```
# sha256sum <your own copy>   # -> 90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c
python3 -c "
import json,glob
print({json.load(open(p))['prompt_sha256'] for p in glob.glob('runs/cpu-*.json') if 'prompt_sha256' in json.load(open(p))})"
```

## 13. Two records that disagree, and neither is hidden

- **The laptop's embedding rate.** This kit's artifact (`runs/cpu-laptop-x1.json`) records
  **15.868 embeddings/s**. A durable internal note of the same run records 19.8. The page
  publishes only the range "12–26 per second," which is true under either — but the artifact is
  the record, and one of the two is wrong.
- **The thread-sweep multiplier.** An earlier best-of-2 sweep put the laptop's headroom at
  **1.78×**; this kit's median-of-5 sweep puts it at **1.75×**. Different statistics, both
  correct, the conservative one published (`prereg.md`, DS3).