The compressed photograph

Five words this page leans on. Quantization is storing a model's weights in fewer bits than they were trained in — the same numbers, written smaller. A weight's precision is how many bits it gets. A block is a small group of neighboring weights that get compressed together, sharing a scale — a separate number, kept at higher precision than the weights it stretches, that says how to blow the small stored values back up; this page calls it the block's ruler. And the tag (Q4_K_M, Q8_0, F16) is the label a model file wears to summarize all of the above — a summary, this page will show, in the same way "a3b" was a summary. One note on the road ahead: the page runs in bench-first order on purpose — first the measurement part two promised, then how the trick actually works, then the receipts.

The same brain at two compressions

Two files on our shelf hold the same model — 11.9 billion weights, whatever the "12b" in its name rounds to. One stores its weights in about four bits apiece (gemma4:12b, tagged Q4_K_M, 7.56 GB on disk — decimal GB, as the runtime reports it; 7.04 GiB); the other in about eight (gemma4:12b-it-q8_0, tagged Q8_0, 12.84 GB — 70% more file). Same training, same 667 tensors: the runtime's own inventory — ollama's, in our case — returns identical tensor-name sets and an identical 338-tensor full-precision set for both files, so the only difference it can see is storage width, and the kit prints both inventories side by side. We'll write q4 and q8 for short from here.

The previous page ended by promising we had asked them both the question this seminar keeps as a guest star — can the robber go back to the desert?, Catan's favorite argument, which our rules product settles live as ruling 4615 with the actual rulebook page attached. Regular readers have met it twice; here it is at two compressions of one brain. Paying up, under the pre-registered rule (in the kit: two question phrasings — the simple ask is the bare question, the harder ask buries the same question in a multi-part game scenario — five scored calls per build per phrasing after a warm-up, temperature 0, run serially beside live traffic; the rows are an illustration under that rule, never a ranking):

Measured 2026-08-19, 22:19–22:23 UTC · ollama 0.32.13 · temperature 0, seed 0, num_ctx 32768 · one unscored warm-up, then five scored calls per build per phrasing, serial, beside live traffic · each figure is the median of its five (simple-ask ranges 126.75–129.24 and 91.88–92.44 tok/s) · rates from the runtime's own nanosecond counters · full rows in the kit.

Same opening sentence, word for word, at both compressions — and at temperature zero each build repeats its reply byte-identically, five runs out of five. The two replies are not identical texts: the q4 answer runs half again as long (98 tokens to 64) and adds a friendly aside the q8 skips; where they overlap, they differ only in phrasing.

Now the part the table's last column politely defers. The referee — ruling 4615, the cited answer linked above — says yes: the rulebook's line is that the robber moves to "any other terrain hex or on the desert." Both compressions said no, and both backed it with a requirement the cited page doesn't state (that the robber needs a hex with a building on it). So compression preserved this model's confident wrong answer down to the opening sentence — which is a cleaner finding than agreement would have been: quantization damage and being wrong are different failures, and only one of them showed up here. (It is also, once again, the whole reason the product answers from a cited page instead of from a model's memory.)

And the harder phrasing delivered the strangest receipt of the bench: asked the multi-part version, both builds flipped — the q4 build: "Yes, you can move the robber onto a desert hex during your turn if you choose to do so." — agreeing with each other again, disagreeing with their own answer to the simpler phrasing, and, on the harder phrasing, agreeing with ruling 4615, which the simpler phrasing had them contradicting. Same brain, same day, same temperature; the wording of the question decided which side of the referee they landed on. The q8 build also volunteered two extra confident clauses of its own along the way — a discard rule nobody asked about, and a parenthetical requiring a 7 or a Knight card. On this bench, the phrasing of the question moved the answer; the compression never did. All four replies, verbatim, are in the kit.

The speed columns are the other half of the story. The 8-bit copy is 28% slower in tokens per second — which is the same thing as 1.39× the time per token, the form the arithmetic below wants — and the reason is the whole of part one's physics in one line: every written token pays a trip through the weights — one trip per token here, no guesser on either build — and the q8 copy has 70% more bytes to haul. (These are short-prompt runs beside live traffic; they read faster than our published steady-state numbers for this model, the two are not comparable, and the rule travels in the kit.)

More bytes should have cost more than they did. 1.7× the bytes, but only 1.39× the time — and the gap is honest: some per-token work doesn't grow with the file at all. The notes in the margin (the running record a model keeps of the conversation so far — part one anatomizes it; the KV cache, if you've seen the term), the sampling, the launch overhead — none of it gets heavier when the weights do, so the bigger file's penalty is diluted. More faithful storage is more luggage; the luggage is most, not all, of the trip.

The same question, down the whole ladder

The morning after the pair above ran, we extended it to every precision the vendor ships for this model in a format our runtime loads — four builds of the same 11.9 B brain, from a trained-4-bit build up to the uncompressed 24 GB original — and asked the same two robber questions under the same registered rule (both registrations in the kit, n=5 per build per phrasing, temperature 0, one build loaded at a time beside the live set):

Measured 2026-08-20, 13:27–13:31 UTC · ollama 0.32.13 · same decoding as the table above · medians of five, byte-identical replies 5/5 in every cell · simple-ask ranges 134.06–135.00, 126.10–129.43, 92.11–92.24, and 48.49–58.12 tok/s (that last low first call is the load-warm artefact the registered warm-up absorbs everywhere else) · bits/weight = file bytes × 8 ÷ parameter count, the whole-blob rule defined below — the blob carries a vision projector the parameter count omits, which is why even bf16 reads above 16 · the q4 rung re-ran a point faster than the 2026-08-19 pair's 128.4 in its own fresh window — same bytes out, different clock · full rows, verbatim replies, load receipts, and all three pre-registrations in the kit.

One inventory note before the receipts, because this page opens files rather than trusting tags: we ran the closing section's command on all four builds (the kit holds all four inventories). The trained build's recipe is the tell — 328 of its tensors sit at plain, uniform Q4_0 with only the token embeddings bumped, none of the 45-tensor six-bit favoritism the compressed build needs. A model trained at four bits doesn't need protecting from them — which is also exactly why its file is the smallest on the ladder at 4.80 bits per weight.

Three receipts worth the whole morning. First: the referee's question is settled at full precision. The uncompressed original — every bit the vendor trained — answers "No, you cannot move the robber back to the desert" and backs it with the same settlement-rule requirement the rulebook doesn't state. The wrongness is the model's, not the compression's; ruling 4615's cited page corrects all four builds equally. Second: on the simple ask, eight bits bought a byte-perfect copy of sixteen. The Q8_0 build's answer to the simple ask is byte-identical to the uncompressed original's — same sha, every run — and the two 4-bit builds each keep the verdict while re-wording it their own way. Compression down this whole ladder never once moved what the model believes; it only, at the 4-bit end, reworded how it says it. Third: the speed column is part one's physics, four times over. Decode orders exactly by file size, top to bottom — the trained-4-bit build, carrying the least luggage at 4.80 bits per weight, is the fastest rung on this ladder at 134.4 tok/s, and the uncompressed original hauls 3.4× its bytes at 0.43× its speed.

The replies themselves — the simple ask, down the ladder (paragraph breaks preserved; the kit holds the exact bytes and their shas):

qat (trained 4-bit, 111 tokens): The short answer is: No, you cannot.

In the official rules of Catan, the robber must be moved to a hex that contains at least one settlement or city. Since the Desert hex is empty (it has no buildings on it), it is an illegal move.

The goal of moving the robber is to block someone's production or steal a resource, so the rules require you to place it on a hex where it can actually affect another player's progress. Just pick any other hex that has a building on it!

Q4_K_M (the seat build, 98 tokens): No, you cannot move the robber back to the desert.

In Catan, the robber must be moved to a hex that contains at least one settlement or city. Since the desert is empty, it doesn't count as a valid spot.

Think of the desert as "off-limits" for the robber—it's a place where no one lives, so there is no one there to be blocked! You must pick a hex with a building on it.

Q8_0 and BF16 (byte-identical to each other, 64 tokens): No, you cannot move the robber back to the desert.

In Catan, the robber must be moved to a hex that contains at least one settlement or city. Since the desert is empty, it is not a valid destination. You must choose a hex that has a building on it to place the robber.

And on the harder phrasing, all four builds flip to yes on the move-back question — agreeing with ruling 4615, disagreeing with their own simpler answer — while each decorates the flip differently: the 8-bit adds its 7-or-Knight-card parenthetical, the uncompressed original volunteers that the robber may move "to any adjacent hex" (a restriction found nowhere in the rulebook — full precision inventing rules of its own), and the trained-4-bit answers cleanest of the four. The phrasing moved the verdict at every precision; the compression never did, anywhere on the ladder. All eight replies are in the kit, verbatim.

Two honesty notes from the same morning, both with receipts in the kit. The bench's first attempt is banked beside the second: midway through it, the image pipeline serving the live game re-took a large share of the card, the builds fell to partial residency, and the run collapsed mid-flight — the resident 4-bit build's own cells fall from 125.24 to the ~25 tok/s we have measured before as this card's half-offload signature, and the heaviest rung to 2.67 — so attempt one's speed rows are refused. It left two genuinely interesting marks: under partial residency the 8-bit build's harder-phrasing reply changed bytes relative to the day before, and the uncompressed build's changed by exactly one word — a discard clause present offloaded, absent seated. Under full residency both came back byte-stable — same weights, same seed, different arithmetic path, different words. And during that first contended attempt, the operator asked the live rules product a question from a tablet: it answered in full — 10.8 seconds end to end, about 5.3 of them queued behind the contended card, then generation at 35.6 tokens a second (its own debug ledger, row 4727, 13:14 UTC) — slowed, never dark. Benches here run beside the products, and both directions of that bargain get measured.

So the two-sentence version of this whole page: quantization trades bytes for speed and memory, and on everything we have measured — down a whole ladder of it — it traded away astonishingly little else. The rest is how the trick works, what the tag actually says, and the receipts — including the one where we paid three times the bits and got back nothing at all.

What the tag actually says

Read Q4_K_M as three claims:

• Q4 — the headline precision class: roughly four bits per weight for the bulk of the model. Roughly, because the honest number is a division this page's reader can do: file bytes × 8 ÷ parameter count. Against the whole blob as the runtime reports it, our "4-bit" file works out to 5.08 bits per weight; strip the ~191 MB vision projector that the parameter count never included and the text weights alone run 4.95. The "8-bit" file works out to 8.63 on the same whole-blob rule. Either way, the tag is not the number — the overhead is the rulers and the tensors the recipe treats better than the headline.
• K — the compression format: the "K-quants," which pack weights into 256-weight super-blocks whose per-block rulers are themselves stored compactly. This family replaced the older plain formats (the _0 and _1 types — one ruler per small block, no nesting) and is what almost every modern local model ships as.
• M — the mix: Small/Medium/Large, the dial for how many tensors get bumped above the headline bits. Medium is why 45 tensors in our "4-bit" file secretly sit at six bits — the reveal two sections down.

Our own shelf carries the zoo: Q4_K_M on most of the working models, Q8_0 on a couple, F16 on the embedder (no quantization to speak of). Every tag names a different bargain between faithfulness and luggage.

Fewer bits is not "rounding off"

Here is the picture most people carry: take each weight, chop its decimals until it fits in four bits, accept the blur. That picture is wrong in a way that matters.

Four bits can only count to sixteen. If every weight had to be one of sixteen fixed values, the model would be lobotomized. The actual trick is the block and scale: weights are grouped into small neighborhoods, and each neighborhood carries its own ruler — a higher-precision scale (16-bit in these formats) that its four-bit values are read against. A neighborhood of tiny weights gets a finely graded ruler; a neighborhood with one huge weight gets a coarse one.

The K-quants nest this one level deeper: a 256-weight super-block carries one 16-bit master ruler, and its eight 32-weight sub-blocks carry small cheap rulers of their own, read against the master. Hence this page's title: every patch of the picture gets the same number of tones — but each patch picks those tones from its own range, so the sky's forty nearly identical blues are told apart as finely as the face's shadows. (The rulers are also part of why "4-bit" files run well above their headline four bits in practice — the rulers ride inside the tensor bytes.)

The reveal: a "4-bit model" is not 4-bit

Now the part we went to our own files to check, the way the previous page checked "a3b." Ask ollama for the full tensor inventory of our 11.9 B at Q4_K_M — 667 tensors — and read what precision each one actually got:

Read from the runtime's /api/show tensor list, ollama 0.32.13; the identical command is printed in the closing section.

The middle row is the plot. The recipe singles out the model's dictionary (the embeddings) and then reaches into two specific working roles — the value projections and the down projections, two of the handful of matrices every layer pushes its numbers through, and you do not need to know what they do to see the point — but takes only half of each: 20 of the 40 value projections and 24 of the 48 down projections, layer by layer (the kit names all 45). Those get six bits, quietly. That is what the tag's "M" bought, and the half-and-half pattern is the tell: the quantizer's shipped heuristic is choosing layers, not just roles — its authors bump the places where their accuracy measurements reacted worst to compression. We can read what the recipe did to our file; the why is their measurement, not ours. (The F32 row, by contrast, is a format convention, not a judgment call: small structural tensors are stored at full width in every GGUF we have read here — our Q8_0 build carries the identical 338, and the kit prints both name lists. And no, the rulers are not in that row — they live packed inside the quantized tensors' own bytes.)

The mixtures from the previous page push the mixing further still, each in its own way: in our 25.8 B mixture's Q4_K_M file, the routed experts' gate and up projections sit at Q4_K while every expert's down projection sits at Q8_0; in the 32.9 B whose tag reads a3b — part two's subject, meaning only about 3 billion of its weights get read for any one token — the ups sit at Q5_0 and the downs at Q8_0 (part two's kit holds both accountings). Part of that is the same importance heuristic — and part is plumbing: tensors whose shapes don't divide the K-format's 256-weight blocks fall back to the older types. Recipe by importance, seasoned by geometry.

Either way the lesson stands: the tag is a recipe's center of gravity, not its contents — exactly like the "3b" in a3b was a bookkeeping convention, not a measurement. The file itself is the only honest inventory, one command reads it (closing section), and the kit carries our counts, names included.

The receipt: we bought three times the bits and got nothing

Theory says the recipe protects what matters. Our best receipt says it louder than we expected, and it was published before this page existed.

When we were choosing which model would hold our moderation classifier's seat — this workshop's word for a job, so the job can outlive whichever model is doing it — its five-gate exam, 374 items and three passes, was re-run at three precisions: the production q4_K_M (the original published bench, in that page's own casing), then q8_0 and bf16 against the same prompt and fixture shas in a maintenance window — 1,122 scored calls per precision, temperature 0 throughout. The result: every verdict identical. All 1,122, across all three. Not similar — identical, including the two clean-corpus false positives the exam already knew about, held unanimously on every arm. The extra bits did not fix the errors either. And the bill for the extra faithfulness was real: about three times the bits per weight by this page's own division — that seat's q4 runs 5.20 bits, bf16 runs 16; the tags say four times, but the tags always say four — three times the resident memory (17.5 → 29.2 → 53.1 GB), 77% more median latency (406 → 560 → 719 ms). The exhibit's own words: "On this instrument, at these sizes, extra precision bought no accuracy at all." The seat keeps running q4 — the way we'd put it: precision we cannot measure is precision we do not buy. (One transparency note carried forward from that page: the classifier's raw fixtures stay closed — they hold real slur specimens — so those rows are cited by pre-registration sha and run id rather than being openly downloadable; the page explains the closure.)

The fence, stated as plainly as the receipt: that is one task — a binary gate, the narrowest judgment a model makes. Longer-form work is where quantization damage would hide if it hides anywhere, and there our shelf holds a subtler receipt: the voice trials scored this same 11.9 B at both compressions on narration — blind panels of LLM judges, ten-point scale — and the q8 build took one round 9.0 to 8.0; the next round, a re-bake on a repaired prompt that the page marks as a different ruler rather than a rerun, flipped the order, q4 ahead 7.6 to 7.0. Two rounds, two rulers, opposite signs — only the gap inside each round is readable, and it changes sign between them: nothing a quantization effect could show through. The kits' counting rule stays the last word on reading too much into it: "Comparing across quantizations is a confound we name, not one we equalize." Finding where quantization damage does become measurable is a bench this workshop now owes — registered as owed work, not promised for a date.

What the freed memory buys

The point of spending fewer bytes per weight is what you do with the bytes you kept. The menu, priced from our own published residency rows (the chair trials' and the August arrivals'):

• A bigger brain in the same card. A dense 28.9 B at Q8_0 holds 33.30 GiB resident; a 32.9 B mixture at Q4_K_M holds 23.64 GiB — both at 32k context, and yes, that comparison crosses model, family, and bits at once (a confound we name, not one we equalize) — but as a price list it is exact: the q8 midsize costs more card than the q4 large. Same-model arithmetic from our own pair: at q4's 5.08 bits per weight against q8's 8.63, about 1.7× the parameters fit in the same weight budget — less once the notes in the margin take their share, since they don't shrink when the weights do.
• Roommates. Our whole standing set — a 25.8 B mixture, a dense 27.3 B, a dense 11.9 B, and a 137 M embedder, four models warm at once — fits in one card because the working models ride q4. We know what happens past the edge because we did it by accident once: loading a fourth model of the 27 B class at deployment context evicted a live seat, and a reader-facing product briefly answered at ~25 tokens a second against its usual 200-plus. The q4 recipe is the reason that edge is as far away as it is.
• A chair that was decided this way. Our narration seat went to the q4 build of this very 11.9 B over its q8 sibling — not on voice (the q8 had won that round's blind panel) but on residency: at ~13.5 GB steady-state the q8 left the image pipeline so little headroom that the painter paid ~54-second model reloads after every idle gap — and the q8 wasn't even faster. The bytes this page is about are the bytes that picked that chair.

What this page does not say

It does not say q4 is free everywhere — one measured moderation gate, one rulebook question measured down a four-build ladder, one noisy narration pair, honest fences above. It does not say more bits ever measurably improved an answer — and on the harder phrasing the uncompressed build invented a restriction the 4-bit ones didn't, an observation we print, not a score; on our rows, the extra bits' only measured costs were speed and memory. It does not rank the four builds into a winner — they are one model written four ways, and the kit that carries the rows forbids reading them as a scoreboard. It never attributes a quality or speed difference to quantization across two different models — where a cross-model number appears above, it is priced as a menu with its confound named. And it does not measure perplexity — the number most quantization comparisons you have read online are built on. Perplexity scores how surprised a model is by text it should have expected; it is a fine instrument, sensitive to damage this page's methods would miss entirely, and we do not have it. Ours measure verdicts, replies, and clocks, and this page stays inside what they can see.

What to take with you

Five things, each with its number from this page:

• The tag is a recipe's center, not its contents. Our "4-bit" file really runs 5.08 bits per weight against the blob (4.95 on text weights alone), keeps 338 structural tensors unquantized, and secretly stores its embeddings and half its value and down projections at six bits.
• Blocks and rulers, not rounding. Four-bit values are read against higher-precision per-neighborhood scales, nested two deep in the K-quants — every patch of the photograph gets its own palette.
• We bought three times the bits and got nothing. 1,122 verdicts at q4, q8, and bf16: identical, every one — same two known false positives included — for triple the resident memory and 77% more latency. One task, honestly fenced; the source exhibit prints the rows.
• More bits is more luggage, all the way up. The same 11.9 B answered the same question 28% slower at q8 than q4 in the 08-19 pair, and the uncompressed original slower still — 134.4 → 129.4 → 92.2 → 58.1 tok/s down the four-build ladder, medians of five under a registered rule — with the same confident wrong answer at every rung, byte-identical between 8-bit and 16, that ruling 4615's cited page corrects.
• Freed bytes become brain or roommates. About 1.7× the parameters fit at our q4's real bit-rate — and our four-model standing set only co-exists on one card because the working models ride q4.

How to check our work — and see it live

Your own model's real recipe, one command, if you run ollama (recent enough that /api/show returns the tensor list — ours is 0.32.13):

curl -s http://localhost:11434/api/show -d '{"model": "<your-model>"}' \
  | python3 -c "import json,sys,collections; \
t=json.load(sys.stdin)['tensors']; \
print(collections.Counter(x['type'] for x in t))"

On our 11.9 B it prints Counter({'F32': 338, 'Q4_K': 284, 'Q6_K': 45}) — and one reading note before you try yours: those are tensor counts, not bytes. F32 leads the tally while being almost none of the file, because those are one-dimensional norms, a few thousand numbers each; a single Q4_K weight matrix holds millions. Expect a mix, not one number — and now you know why.

The kit ships the whole morning: all four builds' tensor inventories, the pair bench and every ladder rung's rows with verbatim replies, the refused first attempt banked beside the second, the availability record that closed registration #1 unrunnable, and all three pre-registrations with their pinned shas — this page's kit; the precision addendum's rows are printed on The new kid, exhibit fourteen.

And the living receipt is the same as ever: the q4-quantized models this page just anatomized are serving RuleSage and narrating RealKeep right now. The compression this page explained is the reason they all fit.

The rest of the seminar

This is the machines' journey, part three. Everyone on the payroll, three at the table is part two — who shows up for each token — and it ended on the thread this page opened with. Reading is fast, writing is slow is part one and the physics under both: the trip through the weights that sparsity shrinks and quantization lightens. Three librarians and a careful reader is where the robber's question actually gets answered with a cited page. Still ahead on this road, tracked in a register we keep of owed work: what a token costs and why — the bill that reading, writing, sparsity, and compression all feed into — plus the controlled dense-versus-sparse stopwatch part two promised, and the true bottom of the quantization ladder — this page ran every rung the vendor ships for its model in our runtime's format, and nothing lower than four bits exists there, so where compression finally breaks stays honestly unmeasured. The whole shelf holds every bench behind every claim, failures included. If one of those is the piece you want next, say so — the suggestion box is read.

Licence: CC BY 4.0, the whole page — name the source and link to it.