# The quantization ladder

*Most models people run at home come as the same model in several sizes, tagged `Q3\_K\_M`, `Q4\_K\_M`, `Q5\_K\_M`, `Q6\_K`, `Q8\_0` and so on. This page runs one, Mistral NeMo 12B Instruct, at five of those levels, each whole on one GeForce RTX 3090 at 300 W, and measures each the same way: writing speed, the wait for the first word, the longest conversation that fits, and how many of 108 frozen questions and 48 held-out ones it gets right. [Inside a GGUF file](https://research.strata2signal.com/inside-a-gguf-file/) showed what a level is inside the file; this page measures what each step down costs and buys. The predictions were timestamped before the first scored call and are printed whether they held or not.*

*Published 2026-10-19 (UTC) · A small (human) team and a fleet of AI agents.*

**the short version:** Mistral NeMo 12B Instruct at five quantization levels, downloads from 5.67 GiB (Q3_K_M) to 12.13 GiB (Q8_0), each run whole on one GeForce RTX 3090 at 300 W with the questions held fixed. At a 4,096-token window it wrote 60.5 tokens a second at Q8_0, rising at every step down to 88.2 at Q4_K_M, then fell back to 74.9 at Q3_K_M: the smallest file was slower than the one above it. With the cache at 8 bits, Q8_0 held a window of 98,304 tokens whole; the other four held 131,072, the 128K window Mistral AI advertises. On 108 frozen questions the levels scored from 38 (Q3_K_M) to 57 (Q5_K_M). The paired test could not tell any level from Q4_K_M to Q6_K from Q8_0 (Q4_K_M came nearest, p 0.0654), but it could tell Q3_K_M (p under 0.0001). On the 48 held-out lines every level scored 33 or 34. Of the eight predictions, five held and three failed; none was left undecided.

5,145 words · about 23 minutes (at 220 words/min) · 6 tables · data kit: yes

https://research.strata2signal.com/the-quantization-ladder/

---

*If you're new here: [strata→signal](https://strata2signal.com) is a small workshop (plus a friendly dog with a white patch) that builds on its own machines and writes up what it measures. Every figure below was read from the bench's own result files with the gates that could have voided it; where a gate fired, the page says so instead of printing a number.*

*The words this page leans on.* **Quantization** is storing a model's weights in fewer bits than they were trained in; a **level** is one such build, named by its tag ([The compressed photograph](https://research.strata2signal.com/the-compressed-photograph/) opens the tags). A **token** is a word or part of one. The **window** is how much conversation the model holds at once, in tokens. **Whole on the card** means all of the model sits in the graphics card's own memory, none in the computer's slower main memory. **First token** is the wait before an answer starts. **Board power** is what the card itself drew; a **joule** is one watt for one second.

## The question {#the-question}

What does each step down in bits cost in answers, and what does it buy in speed and memory, for one model on one card?

Everything else is held still. The five builds come from one publisher, and the registry's listings (read 2026-10-14 at 20:47 UTC) show the same chat template, licence and settings files in all five, byte for byte. One card, one runtime, frozen questions: only the weights change.

## Why it matters if you are choosing a download {#why-it-matters}

The level you pick decides three things: whether the model fits your card with room for a conversation (this model's Q8_0 build, 12.13 GiB, would not fit whole on a 12 GB card; its Q3_K_M build, 5.67 GiB, leaves about half of one for the cache and the runtime); how fast it writes, since every token written means reading the weights again; and, hardest to see, how many answers change.

This workshop runs most of its working models at Q4_K_M. [The compressed photograph](https://research.strata2signal.com/the-compressed-photograph/) found no verdict moved by the bits on its two tests (one rulebook question at four levels, a 374-item moderation exam at three), and named the gap this page starts on: nothing below four bits was measured. This page goes one rung lower, to Q3_K_M, and asks 156 questions at every level.

## The five files {#the-five-files}

All five are `mistral-nemo:12b-instruct-2407` from the Ollama library: Mistral NeMo 12B Instruct, released by Mistral AI on 2024-07-18, built with NVIDIA, Apache 2.0, with a 128K-token window ([A short history of Mistral](https://research.strata2signal.com/a-short-history-of-mistral/)). Each build is pinned by its manifest digest (the registry's fingerprint of the build) and was checked against it before and after its pull: all five matched. Two of them, Q4_K_M and Q6_K, were already in the bench's own store from earlier in the same run and were checked in place.

| Level | Ollama tag | Manifest digest (first 12 characters) | Download, bytes | Download, GiB | Size against Q4_K_M | Bits per weight (file bytes × 8 ÷ parameters) |
|---|---|---|---:|---:|---:|---:|
| Q3_K_M | `mistral-nemo:12b-instruct-2407-q3_K_M` | `0549a17f596f` | 6,083,102,669 | 5.67 | 0.81× | 3.97 |
| Q4_K_M | `mistral-nemo:12b-instruct-2407-q4_K_M` | `daf673741712` | 7,477,217,229 | 6.96 | 1.00× | 4.88 |
| Q5_K_M | `mistral-nemo:12b-instruct-2407-q5_K_M` | `cf4d8cc50281` | 8,727,644,109 | 8.13 | 1.17× | 5.70 |
| Q6_K | `mistral-nemo:12b-instruct-2407-q6_K` | `b6435edbd791` | 10,056,222,667 | 9.37 | 1.34× | 6.57 |
| Q8_0 | `mistral-nemo:12b-instruct-2407-q8_0` | `e9b8ba00d2a1` | 13,022,382,027 | 12.13 | 1.74× | 8.51 |

*Download sizes are each build's model file as its manifest lists it (not the total of every file in the build), read 2026-10-14 at 20:47 UTC; a GiB is 2³⁰ bytes. Parameters: 12,247,782,400, the exact count each level's file header gives (`general.parameter_count`, the same in all five), not the rounded "12.2B" of the library listing. Bits per weight by the whole-file rule The compressed photograph uses: the file's bytes × 8, divided by the parameters.*

The tag is not the bit rate. [Inside a GGUF file](https://research.strata2signal.com/inside-a-gguf-file/) printed the block arithmetic: a Q8_0 block stores 8.5 bits a weight, Q6_K 6.5625, Q5_K 5.5 and Q4_K 4.5, the fraction being each block's scales, and a `_K_M` file mixes types, so a file's bits per weight is not its tag's number. By the whole-file rule, this Q3_K_M file carries 3.97 bits a weight and the Q4_K_M file 4.88.

## How it was measured {#how-it-was-measured}

**The card.** One GeForce RTX 3090, 24 GB, at a 300 W limit (86 per cent of the GeForce RTX 3090's 350 W reference rating), persistence mode on (the driver stays loaded between jobs), board power read from its own counter, twice a second in the speed ladder and once a second during the questions. The machine holds a second GeForce RTX 3090, kept idle, hidden from the bench and watched; 200 MiB of growth on it stops the stage. No wall-socket reading enters this page, because it would include that board.

**The runtime and the order.** Ollama 0.32.13, the version this site's earlier card pages ran, as the bench's own copy with its own model store and only the one card visible. One level at a time: Q4_K_M first, as a repeat check (prediction 4), then Q8_0, Q6_K, Q5_K_M, Q3_K_M. Per level: pull, speed ladder, questions and probes, delete, the model unloaded between steps.

**The speed ladder**, which this site's card pages call the language bank ([A laptop, asked the desktop's questions](https://research.strata2signal.com/a-laptop-asked-the-desktops-questions/) describes it). A frozen 2,101-byte prompt (519 tokens as this model counts it, the same in all 117 scored runs) asks for 256 tokens at temperature 0 with a fixed seed: one unscored load call, then three scored runs at each window of 4,096, 8,192, 16,384, 32,768, 49,152, 65,536, 98,304 and 131,072 tokens. Speed comes from Ollama's own counters; first token is timed to the first streamed piece. The conversation's memory (the cache) is stored at 8 bits, flash attention on (a memory-saving way to run the model's attention step), one request at a time. The ladder stops at the first window that does not fit whole; a level that fits 131,072, the model's advertised 128K window (the file's own header lists 1,024,000 tokens; the ladder stops at the advertised window), has "topped out", never hit a ceiling.

**The six-way: 108 frozen questions.** At [the long table](https://longtable.strata2signal.com/), our dinner-party exhibit, two open models play six historical guests and every line is saved with its speaker. Each of 108 frozen lines asks: which guest said this? No line names its speaker; one fixed guess scores 18. Lines and answers are public in [the Reading the answer kit](https://research.strata2signal.com/reading-the-answer/data/kit/task_c.json). Scoring is strict: the reply, at most 8 tokens, must open with the right letter. A Mistral model wrote 57 of the lines, so this model carries the home-team flag [The cost of intelligence](https://research.strata2signal.com/the-cost-of-intelligence/) gives that family (a model from the family that wrote a line may know its style), equally at every level. Its weights predate every line by two years.

**The doorman's set: 48 held-out lines,** written to test the long table's doorman, the model that reads a line before the guests hear it and decides whether it goes through. Of the 48, 36 were written to be refused and 12 are harmless; none is published, and this page prints counts and scores only. Each reply must open with a one-letter verdict.

**Probes, energy, comparisons.** The instrument also times writing 256 tokens, reading a long prompt, and the first token on 12 prompts. Questions run with the cache at 16 bits in a 4,096-token window. Board joules per correct answer are the card's energy over the question run divided by the answers it got right. Each level is compared with Q8_0 item by item: answers that match, and an exact McNemar test, which takes only the items one level got right and the other wrong and asks whether that split is too lopsided for chance. Each score carries a Wilson 95 per cent interval, the range the true rate plausibly sits in.

## What we predicted, before the first scored call {#what-we-predicted}

The pre-registration was frozen at 2026-10-14 20:52:44 UTC, hashed over every byte. Its one amendment (stamped 2026-10-14 22:15:53 UTC and 2026-10-15 02:18:44 UTC, before any scored call) let the questions run beside the idle second board, picked the card by its link, and made the repeat gate also fire when the repeat cannot be judged; it moved no prediction. **These are predictions, as frozen, not results.**

1. **Speed.** Writing speed at 4,096 tokens rises at every step down from Q8_0 to Q4_K_M, each step's gain larger than either level's run-to-run range (fastest minus slowest of three). Bars: Q8_0 48 to 60 tokens a second, Q6_K 58 to 72, Q5_K_M 66 to 82, Q4_K_M 78 to 95. No bar for Q3_K_M.
2. **The bytes law.** Each token written means reading the weights once, so writing speed times file size should stay nearly constant: for Q4_K_M up to Q8_0, within ±15 per cent of its five-level median. Q3_K_M falls below that band: its gain over Q4_K_M is smaller than the 1.23 times by which Q4_K_M's file is larger.
3. **Window.** Q8_0's ladder stops below 131,072, its largest whole window 98,304. The other four top out at 131,072.
4. **The repeat.** The Q4_K_M build is the exact file, by manifest, that this workshop read on 2026-09-23 with the same question instrument (its measuring lines unchanged since), on a GeForce RTX 3090 in the same machine at 300 W, as one row of the cost of intelligence bench. Today's Q4_K_M matches that reading's answer on at least 104 of the 108 six-way items and at least 46 of the 48 doorman lines, and its six-way score is within 2 of that reading's 47 of 108.
5. **Answers.** On the six-way, Q3_K_M scores lowest of the five outright (a tie is not lowest) and at least 3 items under Q8_0. No level from Q4_K_M to Q6_K differs from Q8_0 at p < 0.05 by the exact McNemar test.
6. **The doorman.** Q3_K_M scores lowest, outright; every level from Q4_K_M up is within 3 of Q8_0.
7. **Short calls.** Median seconds per six-way decision differ by under 20 per cent across the five: a decision is at most 8 tokens, so reading the prompt and per-request overhead dominate, and fewer bits buy little time there.
8. **Memory.** At the 4,096-token question load the card's memory grows by the file size plus at most 1.2 GiB at every level, and the five growths fall in file-size order.

## What voids a reading {#what-voids-a-reading}

Fixed in the same file. A gate that fires voids what it names; a level that does not fit whole is "not tested", never scored.

- **Identity:** builds, runtime, prompt and measuring programs match their pre-written hashes, or nothing starts.
- **The card:** the same card at every step, at 300 W, persistence on, nothing else computing on it, no growth on the second board.
- **Whole:** the runtime and the card's own counter both show the whole model on the card; the first window that spills ends that level's ladder, as not tested.
- **Safety:** the battery backup at 80 per cent load, or any board at 83 °C, stops the stage until a person reads why.
- **Throttling and spread:** a run where the card slowed itself for heat or a hardware limit is flagged, and a figure resting on it prints "throttled"; three runs more than 15 per cent apart print their spread and no headline.
- **The repeat:** if prediction 4 fails, cross-level answer comparisons are labelled "instrument drift"; if it cannot be judged, "replication not shown". Speed rows stand either way.

A failed request is never a result: it is recorded with the server's message and the level is queued again.

## The first measured row {#the-first-measured-row}

Before the ladder, a smoke run checked the instrument: Q4_K_M at a 4,096-token window, measured 2026-10-15 at 05:31 UTC. It wrote at **88.548 tokens a second**, the median of three scored runs within 0.56 per cent of each other (fastest minus slowest, over the median). The first token came after **492.3 ms** (median of three). The board drew **258.38 W** (median of the runs' averages, sampled twice a second) under the 300 W limit, persistence on. The model was whole on the card (its memory grew by 7,535 MiB on the bench's card) and the window passed every gate the speed ladder reads.

The frozen bar for this level was 78 to 95 tokens a second, and the smoke sits inside it, but the bar is judged on the ladder's own Q4_K_M row below: a smoke run only checks that the instrument works, outside the registered order. The ladder's own Q4_K_M row at 4,096, three minutes later, read 88.203 tokens a second (run range 0.252), inside the bar and 0.4 per cent under the smoke. The smoke run's own files, the harness's and the reader's, are in [the data kit](https://research.strata2signal.com/the-quantization-ladder/data/) beside the ladder's.

## When it ran, and the one stop {#when-it-ran}

The ladder ran on 2026-10-15 in two sittings: from 05:34 to 06:10 UTC (Q4_K_M and Q8_0 whole, then Q6_K's speed ladder) and from 10:32 to 11:01 UTC (Q6_K's questions, then Q5_K_M and Q3_K_M whole).

Between the two, the bench stopped itself once. At 06:10:40 UTC, as Q6_K's question step began loading the model, the watcher that keeps other work off the card read a process on it as foreign and stopped the step. It was almost certainly the bench's own short-lived runner. The watcher read the card's list of processes first and looked each one up second; a process that ended between the two reads could no longer be traced to the bench, so it was counted as foreign. That process cannot be traced after the fact, so this is not certain, but nothing else on the machine was found that could have opened the card in that minute. The fix: a process that has vanished counts as foreign only if the card still lists it half a second later. A real foreign process still stops the run (the fix was tested both ways), and the plan's rule did not change.

The stopped step had written no answers, and nothing from it enters this page. Q6_K's questions ran again from the start at 10:33 UTC on the same card, re-read at 300 W with persistence on. Q6_K's speed ladder had finished at 06:10 UTC, before the stop, and stands. No safety stop fired in either sitting: the battery backup's load peaked at 42 per cent (the stop is 80) and the hottest board read 68 °C (the stop is 83). No speed-ladder run was throttled and no window failed the spread gate.

## What each step down bought: speed and memory {#speed-and-memory}

| Level | Download, GiB | Writing speed at 4,096, tokens/s (run range, tokens/s) | First token at 4,096, ms | Board W at 4,096 | Joules per 1,000 tokens at 4,096 | Writing speed at 32,768, tokens/s | Largest window held whole, tokens |
|---|---:|---:|---:|---:|---:|---:|---:|
| Q8_0 | 12.13 | 60.545 (0.033) | 548.22 | 259.9 | 4,293.87 | 60.41 | 98,304 |
| Q6_K | 9.37 | 65.724 (0.441) | 518.25 | 270.81 | 4,102.0 | 64.83 | 131,072, topped out |
| Q5_K_M | 8.13 | 78.942 (0.068) | 516.45 | 264.11 | 3,345.62 | 78.158 | 131,072, topped out |
| Q4_K_M | 6.96 | 88.203 (0.252) | 481.8 | 259.65 | 2,948.31 | 87.234 | 131,072, topped out |
| Q3_K_M | 5.67 | 74.894 (0.398) | 510.56 | 257.59 | 3,439.41 | 74.135 | 131,072, topped out |

*Mistral NeMo 12B Instruct on one GeForce RTX 3090 at 300 W, Ollama 0.32.13, the speed ladder: medians of three scored runs, cache at 8 bits, run 2026-10-15 between 05:34 and 11:01 UTC. Every window from 4,096 up to each level's largest passed every gate: the whole model on the card, no throttling, no spread flag. Q8_0 at 131,072 did not fit whole (92.1 per cent of it on the card, 7.9 per cent in main memory), so that window is not tested. From READING.md's speed table and each level's READ-…-control-3090-<level>-ladder.json.*

| Level | Download, MiB | Card memory growth at the 4,096-token question load, MiB | Growth minus download, MiB |
|---|---:|---:|---:|
| Q8_0 | 12,419 | 12,795 | 376 |
| Q6_K | 9,590 | 10,121 | 531 |
| Q5_K_M | 8,323 | 8,939 | 616 |
| Q4_K_M | 7,131 | 7,827 | 696 |
| Q3_K_M | 5,801 | 6,583 | 782 |

*The named card's own memory counter before and after the load, cache at 16 bits. From reading.json's growth_mib per level, the fit rule's named-card figure; the second GeForce RTX 3090 grew by 0 MiB at every level.*

From Q8_0 down to Q4_K_M, every step bought speed: 60.5, 65.7, 78.9 and 88.2 tokens a second at 4,096, and each level lost at most 1.4 per cent of that by 32,768. The step to Q3_K_M went the other way: 74.9 tokens a second, 15.1 per cent slower than Q4_K_M, for a file 1.30 GiB smaller, and it spent more energy per token written (3,439 joules per 1,000 against Q4_K_M's 2,948). Writing speed times file size, which the bytes law says should stay nearly constant, sat within 4.2 per cent of its five-level median for Q6_K, Q5_K_M and Q4_K_M, 19.3 per cent above it for Q8_0 and 31.1 per cent below it for Q3_K_M. So Q8_0 got through its weights faster than its file size predicts, and Q3_K_M slower. This bench did not test why. One likely reason is that the K-quant formats (the `_K` in the tags) take more arithmetic to unpack each weight, and at three bits that work, not reading memory, sets the pace.

The question step's own probes agree. Its writing probe, with the cache at 16 bits, read within 1.2 per cent of the ladder at every level. Reading a long prompt (about 2,240 tokens, median of 9 or 10 runs) did not follow the bits at all: Q8_0 read fastest, at 3,379.7 tokens a second, and Q6_K slowest, at 2,876.9. The first token on 12 six-way prompts came after 0.544 to 0.565 seconds (median) at every level.

Memory followed the file. At the question load each level's card memory grew by its download plus 376 to 782 MiB, in file-size order, so the download size is a fair guide to what a level takes on the card before its conversation. With the cache at 8 bits, every level but Q8_0 held the advertised 131,072-token (128K) window whole on a 24 GB card.

## What each step down cost: answers {#answers}

| Level | Six-way, right of 108 (Wilson 95 %) | Same answer as Q8_0, of 108 | Right at Q8_0 and wrong here / the reverse | Exact McNemar p against Q8_0 | Doorman's set, right of 48 | Same answer as Q8_0, of 48 | Median seconds per six-way decision | Board joules per correct six-way answer |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| Q8_0 | 54 (40.7–59.3 %) | — | — | — | 34 | — | 0.632 | 239.28 |
| Q6_K | 55 (41.6–60.2 %) | 103 | 0 / 1 | 1.0 | 34 | 48 | 0.6375 | 230.47 |
| Q5_K_M | 57 (43.4–61.9 %) | 103 | 0 / 3 | 0.25 | 33 | 46 | 0.6298 | 223.82 |
| Q4_K_M | 47 (34.5–52.9 %) | 87 | 9 / 2 | 0.0654 | 33 | 45 | 0.6148 | 262.99 |
| Q3_K_M | 38 (26.8–44.6 %) | 80 | 16 / 0 | under 0.0001 | 34 | 41 | 0.6209 | 337.58 |

*Mistral NeMo 12B Instruct on one GeForce RTX 3090 at 300 W, Ollama 0.32.13: the six-way (108 public lines, which of six guests said it; one fixed guess scores 18) and the doorman's set (48 held-out lines, counts only); strict scoring, temperature 0, a 4,096-token window, at most 8 tokens of reply. The repeat gate did not fire, so no cross-level cell carries a label. On the doorman's set, 33 of 48 is 68.8 per cent (Wilson 54.7–80.1 %) and 34 is 70.8 per cent (56.8–81.8 %). Under a lenient reading the six-way counts do not move and the doorman's move by at most 2 (Q8_0, Q6_K and Q4_K_M to 35). Q3_K_M's p, from 16 items to none, is 0.00003; the reading file rounds it to 0.0. From READING.md's answers table, reading.json and results/tasks/figures.json.*

| Level | Attacks inside the doorman's rules refused, of 24 | Attacks outside its rules refused, of 12 | Harmless lines admitted, of 12 | Replies strict scoring could not read, of 48 | Right, of 48 |
|---|---:|---:|---:|---:|---:|
| Q8_0 | 22 | 12 | 0 | 1 | 34 |
| Q6_K | 22 | 12 | 0 | 1 | 34 |
| Q5_K_M | 21 | 12 | 0 | 0 | 33 |
| Q4_K_M | 20 | 12 | 1 | 2 | 33 |
| Q3_K_M | 19 | 12 | 3 | 5 | 34 |

*The doorman's set in [Reading the answer](https://research.strata2signal.com/reading-the-answer/)'s three kinds, counted from each level's doorman rows, which stay private; the kit's `results/tasks/figures.json` carries each level's outside-the-rules column, its unreadable count, its right count and the sum of the other two columns. An unreadable reply opened with something other than the one-letter verdict, such as a verdict wrapped in JSON (a structured text format), and counts as wrong.*

The repeat held exactly. Today's Q4_K_M gave the same answer as the 2026-09-23 reading on all 108 six-way items and all 48 doorman lines, and scored the same 47. At temperature 0 the instrument read the same file the same way three weeks apart, so the differences between levels here come from the weights.

On the six-way, Q6_K and Q5_K_M scored 55 and 57, against Q8_0's 54; the paired test cannot tell either from Q8_0 (1 and 3 items their way, none the other). Q4_K_M scored 47: Q8_0 got 9 items right that Q4_K_M missed, and Q4_K_M got 2 that Q8_0 missed, p 0.0654, just above the 0.05 line the test was frozen at. Q3_K_M scored 38: 16 items right at Q8_0 and wrong at Q3_K_M, none the other way. That is the one step that clearly cost answers. Each six-way decision cost about the same board energy at every level (114 to 120 joules), so energy per right answer follows the score, and it was lowest at Q5_K_M.

The doorman's set barely moved, and that is mostly about the model. At every level it refused nearly everything: all 12 attacks outside the doorman's rules, and all 12 harmless lines at Q8_0, Q6_K and Q5_K_M. A doorman that refused every line would score 36 of 48; this one scored 33 or 34 at every level. Its score stayed level, but Q3_K_M's replies changed: it matched Q8_0 on 41 of the 48, let 3 harmless lines through, and gave 5 replies strict scoring could not read.

## The predictions, judged {#the-predictions-judged}

| Prediction (frozen 2026-10-14 20:52:44 UTC) | Verdict | The figure that decided it |
|---|---|---|
| 1 · speed rises at every step from Q8_0 to Q4_K_M, inside the bars | Failed, narrowly | Speed rose at every step (60.545, 65.724, 78.942, 88.203), each gain (5.2, 13.2, 9.3 tokens/s) larger than the run ranges (at most 0.441), and Q6_K, Q5_K_M and Q4_K_M sat inside their bars. Q8_0's 60.545 sat 0.545 above its bar of 48 to 60: it was faster than predicted. |
| 2 · speed × file size within ±15 % for Q4_K_M to Q8_0; Q3_K_M below | Failed | Q8_0 sat 19.3 % above the five-level median, outside the band; Q6_K (0.0 %), Q5_K_M (+4.2 %) and Q4_K_M (−0.2 %) sat inside. Q3_K_M sat 31.1 % below, as predicted: it was slower than Q4_K_M, not faster. |
| 3 · Q8_0 stops at 98,304; the other four top out at 131,072 | Held | Q8_0 98,304 (at 131,072, 92.1 % of it on the card); the other four 131,072. |
| 4 · the repeat of the 2026-09-23 Q4_K_M reading | Held | The same answer on 108 of 108 six-way items and 48 of 48 doorman lines; six-way 47, as on 2026-09-23. |
| 5 · Q3_K_M lowest on the six-way; Q4_K_M to Q6_K not told from Q8_0 | Held | Q3_K_M 38, lowest outright and 16 under Q8_0's 54; McNemar p against Q8_0: Q6_K 1.0, Q5_K_M 0.25, Q4_K_M 0.0654, none under 0.05. |
| 6 · Q3_K_M lowest on the doorman's set; Q4_K_M up within 3 of Q8_0 | Failed | Q3_K_M scored 34, level with Q8_0 and Q6_K at the top, not lowest. The second half held: Q4_K_M to Q6_K scored 33 or 34 against Q8_0's 34. |
| 7 · seconds per decision within 20 % across all five | Held | 0.6148 to 0.6375 seconds (median), the slowest 3.7 % above the fastest. |
| 8 · memory growth within 1.2 GiB of the file, in file order | Held | Growth minus download 376 to 782 MiB, under the 1,228.8 MiB that 1.2 GiB is; the five growths in file-size order. |

*Each verdict from reading.json's predictions block, judged on the frozen text and the amendment, with its deciding figure from READING.md and the files behind it. No prediction was left undecided.*

## What it means if you are choosing a download {#what-it-means}

For this one model on a 24 GB GeForce RTX 3090, these rows point to Q5_K_M. It scored the most six-way answers (57 of 108, though the paired test cannot tell that from Q8_0's 54), wrote 30 per cent faster than Q8_0 (78.9 against 60.5 tokens a second at 4,096), held the advertised 131,072-token (128K) window with the cache at 8 bits, and spent the fewest board joules per right answer (224). Q6_K bought nothing over it here: a file 1.24 GiB larger, slower writing, no more right answers.

Q4_K_M is the speed pick, the fastest level at 88.2 tokens a second. It answered 7 fewer six-way questions than Q8_0, a gap the frozen test did not call real, though only just (p 0.0654). Q3_K_M is the level to skip on this card: its file is 1.30 GiB smaller than Q4_K_M's, it writes 15 per cent slower, and it lost 16 answers that Q8_0 got right. On a smaller card, where Q3_K_M might be the only level that fits with a long window, the trade is different, and this page did not measure it.

## What this page cannot say {#what-this-page-cannot-say}

One model, one publisher's builds, one card at one power limit, one runtime: nothing here carries to another model, another publisher's quantization of this one (a different recipe can behave differently at the same tag), another runtime or another card. A level is "as good as" Q8_0 only as far as the paired test allows, which is not far: a score out of 108 is uncertain by up to about nine percentage points either way. Q5_K_M's 57 against Q8_0's 54 is three items, inside that noise; it does not show that fewer bits answer better. The doorman's set could not separate the levels, because this model refused nearly every line at every level, so a flat score there says little about what the bits would cost a better doorman. It does not measure perplexity (how surprised a model is by text it should expect), nor long answers, code or sampled output; every question ran at temperature 0. The windows hold for a cache at 8 bits; at 16 bits it takes nearly twice the room. Nothing below Q3_K_M ran, so if answers break further down, this page does not show where. No line of the doorman's set appears.

## What to take with you {#what-to-take-with-you}

- **Speed:** each step down from Q8_0 to Q4_K_M wrote faster at 4,096 tokens (60.5, 65.7, 78.9, 88.2 tokens a second), and within 1.4 per cent of that at 32,768. Q3_K_M fell back to 74.9, 15 per cent slower than Q4_K_M. The bytes law held from Q6_K to Q4_K_M, not at either end.
- **Window:** with the cache at 8 bits, every level from Q6_K down held the advertised 131,072-token (128K) window whole on a 24 GB card; Q8_0's 12.13 GiB file held 98,304.
- **Answers:** the repeat matched the 2026-09-23 reading item for item. No level from Q4_K_M to Q6_K could be told from Q8_0 on the six-way (Q4_K_M came nearest, p 0.0654); Q3_K_M lost 16 answers that Q8_0 got right and gained none. The doorman's set sat at 33 or 34 of 48 at every level.
- **The pick, for this model on this card only:** Q5_K_M (the highest six-way score, 30 per cent faster than Q8_0, the advertised 128K window, the fewest joules per right answer); Q4_K_M for speed; not Q3_K_M.

## How to check our work {#how-to-check-our-work}

- **Pull the same files:** each tag in the table of five pulls the build this page ran while its manifest digest starts with the 12 characters shown (a library tag can be re-pointed; the library's tags page showed the same five on 2026-10-16).
- **Read a file's own recipe:** `ollama show <model> --verbose` prints the header [Inside a GGUF file](https://research.strata2signal.com/inside-a-gguf-file/) reads byte by byte.
- **Ask the same questions:** the six-way's lines and answers are in [the Reading the answer kit](https://research.strata2signal.com/reading-the-answer/data/kit/task_c.json); the doorman's set stays private.
- **Read the rows:** [the data kit](https://research.strata2signal.com/the-quantization-ladder/data/) carries every level's speed-ladder files (the bench's summary and the harness's raw file, the smoke run's too), the question step's figures, each level's six-way rows and report, the bench's own reading, the 2026-09-23 row the repeat was judged against, the pull, stage and safety receipts, the ladder's log, and public copies of the pre-registration and its amendment. The doorman's set is there as counts only. Its README says which file answers which table on this page, and `provenance.json` names every rule the kit was cut by: a computer's name, home paths, a port, the cards' bus addresses and IDs and the battery backup's serial number were replaced or withheld, the pre-registration's own digests with them, and no figure moved.

## The rest of the seminar {#the-rest-of-the-seminar}

Behind this page: [Inside a GGUF file](https://research.strata2signal.com/inside-a-gguf-file/) and [The compressed photograph](https://research.strata2signal.com/the-compressed-photograph/), which named the bench this page starts on. Beside it: [The cost of intelligence](https://research.strata2signal.com/the-cost-of-intelligence/), which prices a correct six-way answer; [Six graphics cards, the same questions](https://research.strata2signal.com/six-cards-the-same-questions/), where the speed ladder's rows collect; and [Reading the answer instead of writing it](https://research.strata2signal.com/reading-the-answer/), where both question sets were first described. Want a level, model or card measured next? [Say so](https://strata2signal.com/contact/): a person reads every message.

## Who ran this, and thanks {#who-ran-this-and-thanks}

Mistral AI and NVIDIA (Mistral NeMo 12B Instruct, Apache 2.0); Ollama (the runtime, MIT, and the library serving the five builds); Georgi Gerganov and the ggml authors (GGUF and its quantization types, MIT); and the long table's six guests, whose borrowed voices make the six-way. None of them owed us anything. A small (human) team approved the bench, set the card's power limit and started every run; a fleet of AI agents chose the model, wrote the pre-registration and the kit, read the rows and drafted this page.

Read first-hand on 2026-10-16 (UTC):

- Ollama: the library's [tags page for `mistral-nemo`](https://ollama.com/library/mistral-nemo/tags) (the five tags, each with the first 12 characters of its digest and its size, matching the table of five); the five builds' manifests and the shared template, licence and settings files were read from the registry by the bench on 2026-10-14 at 20:47 UTC
- Mistral AI: [Mistral NeMo](https://mistral.ai/news/mistral-nemo), 2024-07-18 ("a 12B model built in collaboration with NVIDIA", a context window "of up to 128k tokens", released "under the Apache 2.0 license")
- This workshop's own released pages: [Inside a GGUF file](https://research.strata2signal.com/inside-a-gguf-file/) (the block arithmetic, read there from llama.cpp's source at commit `efe3626b`), [The compressed photograph](https://research.strata2signal.com/the-compressed-photograph/) (its two tests and the gap below four bits), [A short history of Mistral](https://research.strata2signal.com/a-short-history-of-mistral/) (NeMo's date, licence and window), [A laptop, asked the desktop's questions](https://research.strata2signal.com/a-laptop-asked-the-desktops-questions/) (the speed ladder), [Six graphics cards, the same questions](https://research.strata2signal.com/six-cards-the-same-questions/), [Reading the answer instead of writing it](https://research.strata2signal.com/reading-the-answer/) (the six-way and the doorman's set, and the doorman's three kinds of line), and [The cost of intelligence](https://research.strata2signal.com/the-cost-of-intelligence/) (the home-team flag; its kit's series carries the 2026-09-23 Q4_K_M rows the repeat is judged against)

Corrections and later measurements will be added below, each dated (UTC), with a window at both ends where one applies, saying in plain words what it counts.

*Licence: CC BY 4.0, the whole page — name the source and link to it.*

<!-- derived 2026-10-19 (UTC) by tools/derive_md.py from the pour source.
     source html sha256: db31cf6a992057b03b4bc6cea5b5844cc2ca9cb9d87f3ea42cfea3ad4cc37b7e
     derivation sha256:  5378090e99648dd0e0f5c6505b85d1f093c4e5b773cba51341547064dfa75c62
     the {#id} on each heading is the anchor that heading carries on the page. -->