# Pre-registration, deviations and honest limits

The bench registered its comparison rules before it drew any conclusion from them, and this
file is that registration together with every place the run departed from it. It is an extract
of the bench's own working ledgers, redacted for publication under the rules named in
`README.md` — the measurements, rules and deviations are reproduced verbatim; machine
identities are given as classes.

**Nothing in this file was added after the fact to explain a result.** Where a deviation made
a published figure worse for us, the worse figure is the published one; where two records
disagree, both are printed and neither is reconciled away.

---

## The three machines, by class

| published name | filename slug | class |
|---|---|---|
| a server with 128G of ECC RAM and a server-class processor | `cpu-server` | 64 logical CPUs (32 physical), not a hybrid part; a GPU is present in the box and is **parked** — never used, receipted per arm |
| test Rig B — the laptop | `cpu-laptop` | 24 logical CPUs: 8 performance cores with no hyperthreading, 16 efficiency cores; hybrid Intel part; a 24G-VRAM-class GPU is present and parked |
| a mini PC with 32G RAM | `cpu-mini` | 16 logical CPUs: 4 performance cores with hyperthreading (8 performance logical CPUs), 8 efficiency cores; hybrid Intel part; **no discrete GPU at all** |

The server's core, cache and clock detail beyond the above stays unpublished. Its bandwidth
figure is a plain GB/s number with no hardware attached, which is what makes it publishable.

## Method, registered before the run

- **Prompt**: one frozen prompt, `frozen-p512.txt` (its text is **not** in this kit — held
  pending a licensing call; see README), sha256
  `90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c`, 2,101 bytes / 396 words.
  Identical file on all three boxes, sha-verified per box. It **tokenises to 537 tokens, not
  512** — the filename is historical; the token count is the one to quote, and it differs per
  model (see `NOTES.md`).
- **Generation**: `num_predict` 256, `temperature` 0, `seed` 0, streaming. `think:false` where
  the model accepts it, probed per model and recorded in `think_mode` / `think_note`. All six
  models accepted it on all three boxes and every scored run returned `thinking_chars = 0`, so
  no arm here carries reasoning tokens and `eval_count` is 256 answer tokens throughout.
- **TTFT counts the first streamed chunk carrying *either* `response` *or* `thinking`.**
  Counting only `response` would report a fictional first-token time for any model that
  streams a reasoning channel first — a trap a sibling bench hit.
- **Per arm**: 1 discarded warmup, then 5 scored runs (3 for the 70B). Recorded per run: TTFT,
  decode tok/s, total wall seconds, `eval_count`, and the raw `prompt_eval_*` fields.
- **Cold load**: after the scored runs the model is evicted (`keep_alive: 0`), `/api/ps` is
  polled until empty, then a single request is timed. **This is a page-cache-warm reload, not a
  true cold read** — dropping the page cache needs root and this bench had no sudo on two of
  the three boxes. Reported as a floor everywhere. 2 reps per model, 1 for the 70B.
- **Context window**: `OLLAMA_CONTEXT_LENGTH` unset, so every model loads at Ollama's default
  **4,096-token context**, confirmed per arm by `/api/ps`. This is not the model's maximum (the
  9B advertises 262,144) and it is not what the production seats use. **Anyone reproducing
  these numbers must match the 4,096 allocation** — a larger context changes both the KV-cache
  RAM cost and the load time.
- **Parallel slots**: one loaded model, one parallel slot, for every scored arm. The single
  exception is the concurrency row, which necessarily runs a two-slot server and therefore
  re-measures its own concurrency-1 baseline on that same server.
- **Engine**: Ollama **0.32.15 on all three boxes**; version parity was enforced.
- **Quantisation**: every model is the public `Q4_K_M` tag, digest-checked for parity across
  all three machines before benching.

## Deviations and honest limits

- **DS1 — the mini has no C compiler and no passwordless sudo.** Rather than install a
  toolchain on that box, both measurement binaries were **cross-built on the laptop and copied
  over**: `coretype` statically (pure CPUID/affinity code, ISA-independent) and `triad`
  statically to the mini's own ISA target. `-march=native` on the mini resolves to that same
  target, so a native build would have compiled the identical instruction set. On the other two
  boxes `-march=native` was used directly. **Nothing was installed on the mini.**
- **DS2 — the laptop's triad has a wide spread** (59.31 best / 49.21 worst; the other two boxes
  repeat within 0.5%). OpenMP spreads 24 threads across 8 performance and 16 efficiency cores
  on a static schedule, so a rep's time depends on how the halves interleave. The best-of figure
  is reported per STREAM convention, but **the honest read is "roughly 50–59 GB/s"**, not a
  tight number. It is published un-smoothed.
- **DS3 — this kit's thread sweep is scored differently from the parent bench's, and reads
  slightly lower.** The parent used `num_predict` 128 and **best-of-2**; this one uses
  `num_predict` 256 and **median-of-5**, i.e. the bench's own scoring rule for everything else.
  Best-of-2 takes the optimistic tail, so the parent's sweep sits ~3% above these. Both are
  correct measurements of different statistics; **the conservative one is the published one**,
  and the discrepancy is stated rather than reconciled. (This is why an earlier record flags the
  laptop at 1.78× and this kit at 1.75×.)
- **DS4 — the two sweeps disagree on where the laptop's plateau peaks** (16 vs 20 threads).
  With the wider sweep (4/8/12/16/20/24) and median-of-5, 20 wins and 24 regresses. Both sweeps
  agree on the shape — big gains to 16, flat after — they disagree only on where the flat top
  peaks. Published as "the plateau is 16–20 cores."
- **DS5 — the mini's optimum is a tie broken on stability, not on speed.** t=12 and t=16 are
  within 0.3% on pooled median; **t=16 is bimodal** across repeats (medians 10.074 then 10.239,
  per-run spread up to 11%) while t=12 reproduces to 0.14% (10.086 then 10.100). t=12 was taken
  as the optimum. Both settings' full numbers are in `supplement-runs/`; nothing was dropped.
- **DS6 — one model failed its first pull** on the laptop with a `503` from the registry.
  Retried once, succeeded, digest `08ae7ec1744b` — the same digest the main bench used on all
  three machines. All six tags were digest-checked before benching; all six match.
- **DS7 — the "default" column comes from the main bench, not from a fresh default run**,
  except where a same-session `t=4` control is shown. The main-bench arms *are* the
  default-threads measurement, taken with identical rules on the same digests. **A same-session
  control was also run** and is in this kit (`supplement-runs/default4-cpu-mini-*`): it puts the
  mini's defaults within **5.5%** of the main bench on the worst cell — 8.63 vs 9.11 tok/s on
  the 9B, which is 5.5% of the smaller figure and 5.2% of the larger; **the denominator is the
  smaller one**, and the other three cells agree to 0.5%, 1.7% and 3.3%. So the published gains
  of 1.11–1.22× read as **1.17–1.22×** if you compute them entirely within one session. Both
  columns are in the pack.
- **DS8 — the page cache still cannot be dropped, and the CPU governors are unequal across the
  three boxes** (root-gated on two of them). Neither affects decode-at-fixed-threads
  comparisons *within* a box, which is what the tuned-vs-default table is. It does mean every
  **cross-machine** ratio here is configuration-inclusive rather than silicon-pure — see
  `counting-rules.md`, rule 4.

## The incident: one arm was discarded, and it is in this kit

The server was double-booked mid-bench: a second job started loading a model onto that
machine's parked GPU while an arm was running. The arm was **discarded and kept**, named
`cpu-server-m1.CONTAMINATED.json`, and it is the one file in the pack whose GPU snapshot is
not byte-identical before and after (`0, 38764 MiB` → `0, 46719 MiB`). Its own `size_vram`
stayed `0`.

Per-arm loadavg contention receipts were added mid-bench in response, and every arm from that
point carries one.

## One reading that would not reproduce, and was thrown out

A direct probe of the 26B MoE returned **1,809 tok/s** of prefill. That figure was judged
physically implausible on the same impossible-FLOPs check that caught the prefix cache
(`counting-rules.md`, rule 2), did not reproduce on re-run (44.26 and ~47 tok/s), and was
discarded as a cache hit wearing a fresh-prompt costume. Both the outlier and its re-runs are
in `runs/` — `firstpass-gemma4-26b.json`, `fresh-gemma4-26b.json`,
`fresh-gemma4-26b-rerun.json`.