# PREREG-QUANT · one model at five quantization levels, each whole on one card

*What this file is, for a reader who scrolled straight here.* The pre-registration of the quantization-ladder
bench: one model, Mistral NeMo 12B Instruct (2407), from one publisher (the Ollama library), at five
quantization levels, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K and Q8_0, each loaded whole on benchbox's GeForce RTX 3090
24 GB, read through three house instruments that already hold rows: the speed ladder (the language bank of
/six-cards-the-same-questions/ and /a-laptop-asked-the-desktops-questions/), the 108-question six-way set and
the doorman's 48 (the cost-of-intelligence instrument). It fixes, before any scored call, what is measured,
what each figure is compared with (by file), what is predicted, which gates void a reading and what the page
can and cannot say. **INTERNAL, not public-bound**: it names a box and paths. Written by lane `quant-ladder`
(Opus 5.5) from 2026-10-14T20:38:52Z (UTC, `date -u`); the close stamp is the FROZEN line at the end. Every
time here is UTC.

**Authority, verbatim** (estate ledger D-20261014-047, estate `origin/master` `b6ccc2fd`). an operator 2026-10-14
~20:4xZ: "all 3 are a go, and let's release one-per-day, starting on Friday (withheld-2026-10-16)" and "and roll with all your recs on all of it", answering the lead's ask "go quant ladder": "A
quantization ladder bench on benchbox. It's a natural follow-up to the GGUF article: one model at five
quantization levels, from about 3 bits per weight up to 8, each loaded whole on the card. Each level gets the
doorman's 48 questions, the 108-question cost-of-intelligence set and the speed ladder. That's about 3-4 hours
on the card." The standing law (CLAUDE.md, "Benches run NOW, in the daytime"): a first measured arm within
about an hour of the go; this file and the kit harden beside the running instrument.

## 1 · The question

What does each step down in bits cost in answers, and what does it buy in speed and memory, for one model on
one card? Five levels, one card, one runtime, one store, the instruments unchanged. Only the weights'
quantization moves: the five tags share the template, licence and params blobs byte for byte (registry
manifests read 2026-10-14T20:47Z: template `438402dd…`, licence `43070e2d…`, params `ed11eda7…` on all five).

## 2 · Recon (read 2026-10-14 20:38-20:51Z; estate files from `origin/master` `b6ccc2fd` by `git show`)

| what | finding | source |
|---|---|---|
| benchbox's card | a GeForce RTX 3090 24 GB at 300 W with persistence on: the EVGA RTX 3090 XC3 Ultra measured 2026-10-01 (D-20261001-032, -041), "card 1" under a lock (D-20261001-060); a 2026-10-09 recon says "benchbox's two 3090s" with 30 GiB of system memory. The roster row (`estate-roster.yml`, benchbox) still lists two 3080 Tis from 2026-09-25 and is stale. **The stage reads the card; nothing here assumes it.** | ledger rows named; `plans-2026-10-09/qwen-image-turbo/RECON.md` l.248 |
| power-on | powered on demand, by an operator at the box (no remote wake is recorded); a dark box's UPS reads nothing (NUT runs on benchbox) | `estate-roster.yml` benchbox row; memory `overnight-orders-2026-10-01.md` l.169 |
| 3090 posture | 300 W + persistence Enabled "from now on" (R-12, an operator 2026-09-23); the instrument refuses any other reading | memory `cost-of-intelligence-train.md`; `coi3090.py` `EXPECT_CAP_W`, `EXPECT_PERSISTENCE` |
| bench conditions | open frame, three USB fans, turned up by hand for every bench | memory `benchbox-slots-and-clock-2026-09-22.md` |
| kernel rule | before trusting a boot: `uname -r` against `/lib/modules/*/kernel/nvidia-*`, `apt-mark showhold` lists the kernel metas (held since 2026-09-24) | memory `unattended-upgrade-kernel-without-nvidia-module.md` |
| Vulkan trap | a one-card instance on a two-board box sets `CUDA_VISIBLE_DEVICES=<uuid>`, `OLLAMA_VULKAN=0`, `GGML_VK_VISIBLE_DEVICES=-1`, blank HIP/ROCR; residency is proven by growth on the named board | memory `ollama-enumerates-the-second-board-over-vulkan.md` |
| card lock | `~/CARD-LOCK` on benchbox, first word = the holder; taken only on an empty card at its posture | estate `bench/ollama-vs-vllm-2026-09-28/dtt/tools/ovv_dtt_lock_take.sh` |
| UPS | NUT `upsc pr1500@localhost`, the 80 % stop | roster benchbox row; tier harness `stopguard.py` |
| disk | benchbox's root read 92 % (50 GiB free) on 10-07 and 94 % on 10-08 | D-20261007-045, D-20261008-027 |
| Ollama | benchbox's binary of record is 0.32.13, sha256 `39c8dea520be…` (A-09); the 2026-10-01 receipt also names a 0.34.4 copy | `cost-of-intelligence-2026-09-23` A-09; D-20261001-041 |

### 2.1 The instruments, by path at estate `origin/master`

1. **The speed ladder** (the language bank). `bench/gpu-3080ti-benchbox-2026-09-18/bench_gpu.py` (sha256
   `c70b5fb55b824c88…`) and its `benchlib.py` (`56c96d7fb8adc1b2…`), BYTE-IDENTICAL, the leg that already ran
   on benchbox and reads its card from the environment. Mode `--ladder`, arm `control-3090` (its arm for one
   GeForce RTX 3090 24 GB on this box, scored under the strict `whole` fit rule). The frozen prompt
   `frozen-p512.txt` (2,101 bytes, sha256 `90eedd0c53f9554ae3837674504fcb7090013d9a432a653352183c0f25a7ce5c`),
   `num_predict` 256, temperature 0, seed 0, streamed, a one-token load call then three scored runs per
   window, decode tok/s from Ollama's counters, TTFT to the first chunk, board power at 2 Hz, the contention
   gate (5 %), the spread gate (15 %), the fit rule (`size_vram == size` plus the runner-RSS allowance), and a
   ladder that stops at the first window that does not fit whole. The arm's label string in the file is the
   harness's historical prose ("comes OUT of this x16 seat … before the swap"); this file, not that string,
   says what the arm is.
2. **The 108-question six-way and the doorman's 48** (the cost-of-intelligence instrument).
   `bench/cost-of-intelligence-2026-09-23/rtx-3090-reading-0/instrument/coi3090.py` (repo sha256
   `d16088cdc4acfbe8…`, the bytes that ran every benchbox arm from 17:46Z on 2026-09-23), with ONE documented
   delta, `tasks/INSTRUMENT-DELTA.md`: the constants block (host, card, card name, box class, store, disk
   floor) reads the kit's params instead of literals; no measuring line moves. It drives, unchanged:
   `bench/jev-2026-09-21/run.py` (`withheld-2026-10-16`), `bench/jev-2026-09-21/benchbox/bench-jev-2026-09-21/run_addenda.py`
   (`a34aca61…`, task d, DOORMAN_SYSTEM v1.2), `kit/task_c.json` (`61a0b1d6…`, 108 items),
   `kit/doorman_planted.json` (`4daf059c…`, 48 items, **PRIVATE**: counts and scores only, never an item,
   question or answer), the tier harness at estate `51be5b25` (`benchlib.py` `28adb0c5…`, `stopguard.py`,
   `stopreceipt.py`). Think off (`think: false`, `num_predict` 8, run.py's body byte for byte), `num_ctx`
   4,096, the strict parser `v2-benchbox-1` (`analyse.py`), the probes (decode 3 × 256, prefill ≥ 12 s, TTFT on
   12 prompts).

### 2.2 The rows that anchor the ladder (benchbox's 3090, 300 W, Ollama 0.32.13, 2026-09-23)

`mistral-nemo:12b-instruct-2407-q4_K_M`, digest `daf67374171218…` (the same manifest the Q4_K_M rung pulls),
arm `b-mistral-nemo-12b` of `bench/cost-of-intelligence-2026-09-23/rtx-3090-reading-0/rows/`: six-way strict
**47/108** (43.5 %, Wilson 34.5-52.9 %), doorman **33/48** (in contract 21/36), decode probe **87.7 tok/s**
at 264.4 W board, prefill probe 3,253 tok/s, median 0.599 s per decision, growth 7,827 MiB (`FIGURES.md`
l.23, l.76, l.316, l.371). That arm ran `coi3090.py` `3c0db236…`, the predecessor of `d16088cd…` (the
`x_digest` field added at 17:4xZ).

## 3 · The model, and why

**`mistral-nemo:12b-instruct-2407`, the Ollama library's five builds** (manifest sha256 prefixes, pinned;
store bytes from the manifests):

| level | tag | manifest sha256 (prefix, pinned) | store bytes | GiB |
|---|---|---|---:|---:|
| Q3_K_M | `mistral-nemo:12b-instruct-2407-q3_K_M` | `0549a17f596f` | 6,083,102,669 | 5.67 |
| Q4_K_M | `mistral-nemo:12b-instruct-2407-q4_K_M` | `daf673741712` | 7,477,217,229 | 6.96 |
| Q5_K_M | `mistral-nemo:12b-instruct-2407-q5_K_M` | `cf4d8cc50281` | 8,727,644,109 | 8.13 |
| Q6_K | `mistral-nemo:12b-instruct-2407-q6_K` | `b6435edbd791` | 10,056,222,667 | 9.37 |
| Q8_0 | `mistral-nemo:12b-instruct-2407-q8_0` | `e9b8ba00d2a1` | 13,022,382,027 | 12.13 |

Why this one: (a) Q8_0 is 12.13 GiB, whole on a 24 GB card with room for the window ladder, and large enough
that the top levels press on memory at long windows, so the ladder can show what bits buy in window; (b) the
Q4_K_M build is the exact manifest the house already read on this box, card class and instrument (2.2), so
the Q4 rung is a replication check before any comparison; (c) its six-way score sits mid-range (43.5 %), with
room to move both ways; (d) one publisher ships all five levels in identical packaging (section 1). Rejected:
the gemma4 and qwen3.x lines (the library ships q4_K_M and q8_0 only), phi4 (no q3, q5 or q6),
`mistral:7b-v0.3` (18.5 %, near the floor), gemma2 27B and qwen2.5 32B (Q8_0 does not fit 24 GB).
`llama3.1:8b-instruct` (all five levels, house row 43/108) is the registered fallback if a NeMo level cannot be
pulled.

## 4 · Arms, order, posture

- **Arms = the five levels.** Order: Q4_K_M (the anchor) → Q8_0 → Q6_K → Q5_K_M → Q3_K_M. The anchor runs first
  so the instrument is checked before anything is compared; then from the top down, so the level most at risk
  of spilling (and the biggest pull against the disk) runs while an operator is around.
- **Per level, in order:** pull (digest checked against section 3 before and after) → the speed ladder at
  4,096, 8,192, 16,384, 32,768, 49,152, 65,536, 98,304, 131,072 (it stops at the first window that does not fit
  whole; 131,072 is NeMo's advertised 128K window, so a level that fits there has "topped out", never "a
  ceiling") → the six-way and the doorman with the probes → delete the tag. One level on the card at a time;
  the model is unloaded between instruments and levels.
- **Card:** benchbox's GeForce RTX 3090 24 GB, chosen at stage by the rule in `bin/qlib.sh` `ql_pick_card`
  (name exactly `NVIDIA GeForce RTX 3090`, memory ≥ 24,000 MiB, the x16 seat when two qualify; anything else
  REFUSES). Its UUID lives only in the box-local state file, never in this folder.
- **Posture:** enforced limit 300.00 W and persistence Enabled, read at every stage (R-12); the USB fans up
  (an operator's standing bench condition, recorded as his receipt line).
- **Runtime:** Ollama 0.32.13 (binary sha256 prefix `39c8dea520be`), the bench's own transient user unit on
  loopback, its own store `~/bench-quant-ladder/store`, the Vulkan-hide set beside the CUDA pin. Two instance
  postures, each its instrument's own: SPEED = KV `q8_0`, flash attention on, parallel 1, one model loaded
  (the language bank's posture); TASKS = KV f16, flash attention on, keep-alive -1, context 131,072 (the
  cost-of-intelligence posture). The unit restarts between postures.

## 5 · Predictions (directions and bars, registered before any scored call)

- **P1 speed.** Decode tok/s at 4,096 rises at every step down from Q8_0 to Q4_K_M, each adjacent pair
  separated by more than the larger of the two rungs' run spreads. Bar: Q8_0 48-60, Q6_K 58-72, Q5_K_M 66-82,
  Q4_K_M 78-95 tok/s.
- **P2 the bytes law.** decode tok/s × store bytes is within ±15 % of its five-level median for Q4_K_M up to
  Q8_0; Q3_K_M falls BELOW that band (its gain over Q4_K_M is smaller than the 1.23× bytes ratio).
- **P3 window.** Q8_0's ladder stops below 131,072 (largest whole window 98,304); Q6_K, Q5_K_M, Q4_K_M and
  Q3_K_M top out at 131,072.
- **P4 the anchor.** Q4_K_M today makes the same choice as the 2026-09-23 row on ≥ 104 of 108 six-way items
  and ≥ 46 of 48 doorman items, strict score 47 ± 2.
- **P5 answers.** Six-way strict score: Q3_K_M is the lowest of the five, at least 3 items under Q8_0. No
  level from Q4_K_M to Q6_K differs from Q8_0 at p < 0.05 by an exact McNemar test on the paired items.
- **P6 the doorman.** Q3_K_M scores lowest; every level from Q4_K_M up is within ±3 of Q8_0.
- **P7 short calls.** Median seconds per six-way decision differ by under 20 % across the five levels (a
  decision is 8 tokens out; prefill and per-call overhead dominate), so fewer bits buy little time there.
- **P8 memory.** Board growth at the 4,096-token TASKS load is within 1.2 GiB above the store bytes at every
  level; the five growths order exactly as the store bytes.

## 6 · Gates (a gate that fires voids what it names; a level that spills is "not tested", never scored)

| gate | rule | what it voids |
|---|---|---|
| G-0 identity | tag manifest prefix = section 3 (before the pull and on `/api/tags` after); binary sha256 prefix `39c8dea520be`; frozen prompt `90eedd0c…`; every instrument file's sha256 = `SHA256SUMS` | refuses the step before its first call |
| G-1 card and posture | the card picked at stage is the card every stage reads (UUID, name, bus id); 300.00 W + Enabled | refuses the stage (rc 4) |
| G-2 the lock and foreign work | `~/CARD-LOCK` first word `quant-ladder` for every stage; no compute process on the card but the bench's runner; no other board grows ≥ 200 MiB; the contention gate (5 %) before every scored run | refuses before the load; mid-stage the watcher interrupts the stage and writes a stop |
| G-3 whole on the card | speed: the harness's `whole` rule per window (a window that spills ends that level's ladder: the largest whole window is the figure, the spilled one is "not tested"); tasks: growth ≥ 90 % of the store bytes on the named board, `/api/ps` corroborating | the window, or the level's tasks ("not tested") |
| G-4 UPS stop | 80 % load, read every 2 s by `stopguard.py` | stops the stage; the queue halts; no re-run without the lead |
| G-5 temperature | 83 °C core on any board (`stopguard.py`) | same as G-4 |
| G-6 throttle | a scored run with `hw_slowdown`, `hw_thermal_slowdown` or `sw_thermal_slowdown` active in its trace | that run is flagged; a rung whose median rests on a flagged run is reported "throttled", not as a speed (the 300 W `sw_power_cap` is the posture, not a gate) |
| G-7 spread | the harness's 15 % spread gate on three runs | the rung is reported with its spread and no headline |
| G-8 disk | free − manifest bytes ≥ 20 GiB before every pull; the instrument's floor 20 GiB before every arm | refuses the pull or the arm (rc 4) |
| G-9 replication | P4 fails (fewer than 104/108 same choices or 46/48) | every cross-level answer comparison is labelled "instrument drift" until the lead reads why; the speed rows stand |

A failed request is never a verdict: an HTTP failure writes FAILED.json with the server's sentence and the
level is re-queued by the lead, never relabelled.

## 7 · What the page can say, and what it cannot

**Can:** for this one model on this one card at 300 W, what each level scored on the two frozen sets (with
Wilson intervals and the paired agreement with Q8_0), its writing speed and first-token time at each window it
held whole, the largest window each level held whole, the board energy per answer and per 1,000 tokens, each
number read from its file with its gates; and that only the weights' quantization differed between levels.

**Cannot:** say a level is "as good as" another beyond the paired test it ran; carry the result to another
model, another publisher's quantization, another runtime or another card; call a topped-out level's 131,072 a
ceiling; read a WALL figure from a window under a minute; print any doorman item, question or answer; name the
box, a path, a port or a card UUID.

## 8 · The way back

Every stage stops on its own road: the model is unloaded, the instance unit stopped, the lock released, the
store's tags deleted, the bench folder left with its rows. Nothing outside `~/bench-quant-ladder/` and the
user units named `ql-*` is touched: no sudo, no seat unit, no system Ollama, no cap change by the kit (a cap
or persistence change is an operator's own paste).

## 9 · Amendments

Any change is an amendment appended below the FROZEN line, stamped by `date -u`, ending with its own line of
the same shape over every byte above it, and written before the first scored call it touches; one written
after that call voids the comparison or gate it touches. The stage's card read, the posture read and the pull
receipts are receipts, not amendments.

---

FROZEN 2026-10-14T20:52:44Z · sha256 of every byte above this line: withheld-2026-09-29