# Counting rules

Every number on the exhibit page and in this kit is produced by one of the rules below.
They are written out because two of them are traps: applied naively, they produce figures
that are wrong by a factor of ninety-six and by seventy-eight percent respectively, and both
of those wrong figures look perfectly reasonable on a chart.

Read this file before you compare anything in `runs/` against anything on the page.

---

## Rule 1 — Decode throughput: median of the scored runs, warmup discarded

Each arm ran **one discarded warmup plus five scored runs** (three for the 70B). The
published decode figure is the **median** of the five `decode_tok_s` values, where
`decode_tok_s = eval_count / eval_duration` as Ollama reports them.

The warmup is in the record — it is the `warmup_discarded` object in each run file — so you
can see what was thrown away and why it was thrown away. It is never in the median.

```
python3 -c "
import json,statistics
j=json.load(open('runs/cpu-server-m1.json'))
print(statistics.median(r['decode_tok_s'] for r in j['runs']))"
→ 13.594
```

## Rule 2 — True prefill: the COLD runs only. The warm prefill figure is a cache artifact.

**This is the trap.** Every scored run in this bench sends the *same* frozen prompt. Ollama
caches the KV prefix, so from the second identical request onward the prefill is not
recomputed. The API still reports the full `prompt_eval_count` (537 tokens) with a near-zero
`prompt_eval_duration`, and the arithmetic that falls out is physically impossible:

| quantity | warm (cached prefix) | cold (nothing cached) |
|---|---|---|
| reported `prompt_eval_count` | 537 | 537 |
| `prompt_eval_duration` | 0.205 s | **15.97 s** |
| implied prefill rate | 2,613 tok/s | **33.6 tok/s** |
| time to first token | **211 ms** | **20,319 ms** |

2,613 tok/s of prefill on a 9B model is roughly 47 TFLOPS. A 12-core mini-PC processor does
not do 47 TFLOPS. **That impossible number is what gave the cache away** — the check that caught it
is "divide the claimed rate by what the silicon can physically do," and it is worth running on
any prefill figure anyone hands you, ours included.

So: the published prefill column is computed from the `cold_load` entries only. The
`derived.prefill_tok_s` field inside every warm-run JSON is a **cached-prefix value by
construction** and is not tabulated anywhere.

```
python3 -c "
import json,statistics
j=json.load(open('runs/cpu-mini-m1.json'))
print('cold ', statistics.median(c['prefill_tok_s'] for c in j['cold_load']))
print('WARM (do not publish)', statistics.median(r['prefill_tok_s'] for r in j['runs']))"
→ cold  33.5085      WARM (do not publish) 2613.037
```

## Rule 3 — Four latency numbers, never one "TTFT", and each carries its denominator

The bench reports first-token latency four ways. They differ by up to 96×, so the label
matters more than the number:

1. **Cached-prefix first token** — the warm-run `ttft_ms`. The honest *"I re-sent an identical
   prompt"* number, and the only one that looks fast. **Do not quote this as TTFT.**
2. **Fresh prompt, model already loaded** — cold TTFT minus `load_duration`. The honest
   *"I asked a new question"* number.
3. **Fully cold** — load plus prefill: the whole `cold_load[].ttft_ms`.
4. **Decode throughput** (rule 1) — unaffected by the cache and fully valid throughout.

**The multiplier depends on which pair you divide, and the page states its pair.** On the cell
that gave the trap away — the 9B model on the mini — the two live readings are:

```
76×  = fresh prompt, MODEL ALREADY LOADED  16,031.1 ms  ÷  cached 210.8 ms  ← the page's rule
96×  = cold TTFT INCLUDING the model load  20,375.3 ms  ÷  cached 210.8 ms  ← the ledger's rule
310× = the worst cell (24B dense, same machine), model-loaded rule: 97,673.1 ÷ 314.9

(Each of those is the median of that arm's cold runs and the median of its scored warm runs —
 rule 1's statistic applied to latency. Taking a single cold rep instead moves 76× to 75.8×
 and 96× to 96.4×; the finding does not move, but say which you took.)
```

Both are real; they differ only in denominator. The page uses the model-loaded rule because it
is the like-for-like comparison — same model resident, same machine, only the prompt changes.
The durable bench record carries the 96× figure as its section header. **If you find one
number in one place and the other in another, this is why**; neither is a correction of the
other.

```
python3 -c "
import json,statistics as st
j=json.load(open('runs/cpu-mini-m1.json'))
warm  = st.median(r['ttft_ms'] for r in j['runs'])
cold  = st.median(c['ttft_ms'] for c in j['cold_load'])
fresh = st.median(c['ttft_ms']-c['load_duration_ns']/1e6 for c in j['cold_load'])
print(f'cached {warm:.1f} ms | fresh-loaded {fresh:.0f} ms | cold {cold:.0f} ms')
print(f'model-loaded rule {fresh/warm:.1f}x | cold-start rule {cold/warm:.1f}x')"
→ cached 210.8 ms | fresh-loaded 16031 ms | cold 20375 ms
→ model-loaded rule 76.1x | cold-start rule 96.7x
```

## Rule 4 — Thread count: the default is not the machine's best, and the gap is 78%

**The second trap.** Ollama sets no thread count; llama.cpp's own
`common_cpu_get_num_math()` chooses one, and on a hybrid CPU it applies an *unconditional*
"skip the hyperthread sibling" step. On a hybrid part with no SMT, that step steps over a real
performance core. Both Intel boxes here were handed **4 threads** — on the laptop that is
**half** its performance cores, before its efficiency cores are even considered.

Measured cost of the default, dense models, laptop: **1.75–1.91×** left on the table. An
earlier best-of-2 sweep first flagged the same laptop at **1.78×**; this kit's median-of-5
sweep reads a hair below it, and both figures are in the record (see `prereg.md`, DS3).

**The consequence for anyone comparing machines:** every cross-machine ratio in this kit is
**configuration-inclusive, not silicon-pure.** Two boxes running at different effective thread
counts are not a clean hardware comparison, and this kit says so rather than normalising it
away. Where a tuned column exists it is published beside the default one, both labelled.

**And the lever reverses on mixtures of experts.** The same `num_thread` that makes a dense
model 1.75–1.91× faster makes an MoE **collapse**: forced to the dense optimum, the two MoE
models run at **0.50–0.64×** their default speed, and a dedicated sweep of the 30B MoE on the
mini falls from 14.4 tok/s at t=4 to **1.27 tok/s at t=16** — an 11× fall. A single "tuned"
number for a machine does not exist; it is per-model.

```
python3 -c "
import json,statistics
m=lambda p: statistics.median(r['decode_tok_s'] for r in json.load(open(p))['runs'])
print('laptop 9B  default', m('runs/cpu-laptop-m1.json'), '-> tuned t=20', m('supplement-runs/tuned-cpu-laptop-m1.json'))
print('laptop 30B default', m('runs/cpu-laptop-m6.json'), '-> forced t=20', m('supplement-runs/tuned-cpu-laptop-m6.json'))"
→ laptop 9B  default 7.45 -> tuned t=20 13.052        (1.75x)
→ laptop 30B default 15.35 -> forced t=20 8.529       (0.56x)
```

## Rule 5 — Two spread columns, two different denominators, both named where they stand

This kit publishes two "spread" figures and **they are not the same statistic**. Neither is
wrong; quoting one under the other's name is.

| table | rule | worked example |
|---|---|---|
| `analysis/dispersion.md` — per-cell decode spread | **(max − min) ÷ MEDIAN** | widest cell: 5.99%. Dividing by the minimum instead gives 6.25%. |
| `analysis/bandwidth.md` — triad best-vs-worst spread | **(best − worst) ÷ WORST** | laptop: (59.31 − 49.21) ÷ 49.21 = 20.5%. Over the best instead: 17.0%. |

## Rule 6 — Comparisons this data does not support, stated as such

- **Model quality is not ranked here.** The field exam saturated — all six models scored 57 to
  59 out of 60 and every one passed every bar. That is a **floor check**, not a ranker, and a
  2-point gap on a saturated 60-item instrument is not a quality ordering.
- **The moderation set is not a ranking either.** The widest agreement gap across five models
  is 5 items out of 374, every disagreement falls in the `clean` set, and no model missed a
  single one of the 20 offensive or 25 hard-negative items. We decline to rank on it.
- **Cross-machine ratios are configuration-inclusive** (rule 4).
- **Load times are floors, not true cold reads.** The page cache could not be dropped (no root
  on two of the three boxes), so every "cold load" here is a warm-disk reload.
- **The mini's t=12 optimum is a tie broken on stability, not speed** — t=12 and t=16 are
  within 0.3% on pooled median; t=16 is bimodal (per-run spread to 11%), t=12 reproduces to
  0.14%. See `prereg.md`, DS5.

## Rule 7 — The GPU receipt, and exactly which arms carry it

Three receipts per arm: `size_vram: 0` from Ollama's `/api/ps`; `cpu_only_receipt: true`; and
the `nvidia-smi` device snapshot recorded before and after the arm, compared byte for byte.

**Twenty of the 32 files in `runs/` carry all three. Nineteen of those twenty are the scored
decode-and-prefill arms** — six models × three machines, plus the 70B — **and all nineteen
have byte-identical before/after snapshots.**

**The twentieth is the arm the bench threw away.** `cpu-server-m1.CONTAMINATED.json` is the
only file in the pack whose snapshot is *not* identical: `0, 38764 MiB` before and
`0, 46719 MiB` after, because a second job started on that machine's card partway through. Its
own `size_vram` stayed `0`, so the arm never used the card itself — the snapshot check simply
**fired**. It is kept in the pack, named, because a check that has never fired is a check
nobody has tested.

The remaining twelve files — the concurrency pair, the three embedder arms, the memory-fit
arm, the two thread sweeps and the four cold-path probes — carry the frozen prompt's hash and
their timings but **not** the GPU snapshot. They ran through the same CPU-only servers, between
arms that carry the full receipt on either side, but their own records cannot prove it and the
fields cannot be re-derived after the fact. Where the page publishes a finding from one of
those twelve, that is what it rests on.

```
python3 -c "
import json,glob
r=[(p, {'size_vram','nvidia_before','nvidia_after'} <= set(json.load(open(p))))
   for p in sorted(glob.glob('runs/*.json'))]
print('all three:', sum(1 for _,v in r if v), 'of', len(r))
print('without  :', [p.split('/')[-1] for p,v in r if not v])"
```

## Rule 8 — Everything is UTC, and one field needed converting to keep it that way

Every timestamp in this kit is Z-stamped UTC. **One field was not, in the source records:**
Ollama's `/api/ps` returns the model's `expires_at` in the serving machine's own local clock
with a numeric offset, and two of the three machines' arms carried it that way — sitting
beside `started_utc` and `finished_utc`, which were already UTC. Subtract one from the other
and the machines' timezone falls out to the second.

Those 42 fields, across 22 files, are **converted to the same instant, Z-stamped**, under the
named transform `local-offset-to-utc` counted per file in `provenance.json`. Nothing else in
those files moved. If you are ever reading this harness's raw output somewhere other than this
kit: **check `expires_at` before you trust the clock on it.**