The free speed wasn't free

Five terms used on this page: a seat is a named job on our box that a model is measured into and can be swapped out of; decode rate is how fast a model produces tokens once its answer has started, as opposed to how long it takes to start; the toll is the per-token cost of the hidden work this page traces; a sampler is one stage in the pipeline that turns a model's scores into the one token you see; temperature 0 means always take the top-scoring token, so a run repeats itself exactly.

The day the box got faster

On 2026-08-16, one of us looked at the numbers from the restart window and typed this to the assistant working alongside them:

one more thing: after we updated ollama last night, i started see over 200tok/sec for gemma4:26b from [the-gpu-box], :D is that just from ollama upgrade? how is that possible?

(Our box's internal name is replaced with its public alias; the message is otherwise as typed, typos and all.)

Some context for that disbelief. gemma4:26b holds our busiest seat: it answers rules questions in RuleSage and narrates walking tours in Amble, on a single 96 GB-class workstation card. For weeks, every well-controlled median of its decode rate had landed between 138 and 146 tokens per second: a production median of 138.6 tok/s over 1,982 live rulings on 2026-08-15 (the strongest single pre-upgrade receipt we own) and a sealed bench run on 2026-08-14 at 139.55 (n=6, range 137.75–150.49). Live serving-journal lines from 08-13 sit in the same neighborhood but aren't cleanly comparable (variable prompts, some generations only a few tokens long, and the source record stamps its own attribution UNCONFIRMED), and one bench reading sits lower — 124.975 — because it was taken at a much deeper context tier. The kit that ships with this page carries every dated reading with its posture and comparability flags.

Then the runtime — ollama, the server program that loads the model and turns your question into its answer — was upgraded from version 0.32.9 to 0.32.13. The service restarted at 2026-08-16T01:36:27Z, and the same model on the same card started printing 202–214 tok/s.

A model doesn't get half again as fast because you restarted it. Something real changed, and the honest first answer that day was: we don't know what. We filed three candidate causes before anyone celebrated. A genuine runtime improvement. Contention on the old readings — an early note blamed a large co-resident model for holding the card back, a premise we later checked against the residency record, found nothing to support, and formally retracted (the retraction is in the kit). And a silently changed context setting — closed by receipt, since every controlled reading on both sides pinned the same context length. The forensic that follows settled the contention question the other way entirely: the fast box was the busier one.

The hunt: change one thing

The way to separate candidates like these is boring and decisive: hold everything constant and toggle one thing. Later that day (2026-08-16T21:31Z), a forensic pass replayed a byte-identical sealed request — same model, same runtime binary, same card, twelve seconds apart — with exactly one request option changed. We'll call this experiment the toggle:

One option, repeat_penalty: 1.1, took back the entire speed jump, and then some. 209.37 ÷ 136.13 = 1.538×. Worth saying plainly, because the same upgrade moved two things: it also carried a 27-commit bump of the llama.cpp engine underneath. The toggle bounds whatever that bump contributed to at most a couple of percent of the jump — the one option alone over-explains it. (And llama.cpp had dropped this same 1.1 default back in 2024; the value we were still paying was one their side had already retired.)

And for once we didn't have to infer the mechanism, because the server prints its own sampling pipeline into its log. Before the upgrade, and in the slow arm above, the chain reads logits -> penalties -> ...; after the upgrade, logits -> ?penalties -> .... The ? marks a stage the daemon skipped; every other disabled stage on the same line carries it too. The stage that vanished is called penalties, and it had been running on every token our unpinned models generated.

The two-line commit

Why did it vanish? Because of ollama commit 6a261db7d, "api: stop applying repeat_penalty 1.1 to models that don't set one", authored by Jesse Gross on 2026-08-11 and shipped in v0.32.10 the next day. (Full sha, raw metadata, and the complete message are in the kit.) The change is two lines: a default value in one source file goes from 1.1 to 1.0, and a documentation row is updated to match — that row is worth noting, because it had read (Default: 1.1) all along. The value was published; what was invisible was its effect. It went in without a PR thread — no pull-request number in the subject, no comments on the commit page — which means there is no discussion to read alongside it. The commit message is the record. It happens to be an unusually complete one. The release note gives it one bullet:

"Models that don't set a repeat_penalty now default to 1.0 (off) instead of 1.1, matching other engines and speeding up speculative decoding; set a per-model parameter if an older model repeats itself."

The commit message is worth reading in full, because the author explains the reasoning and names our model family while doing it. (We fetched it raw from the source and ship it verbatim in the kit.) Two lines of diff; behind them, an audit of the parameter defaults of every model in the library, and two benchmarked models to justify the flip. His claims, in brief:

• Request options are layered over server defaults, so the old 1.1 "reaches every model whose parameters leave it unset."
• The default matches no one: "No maker of the library's current models recommends 1.1: their generation configs either omit the penalty, meaning 1.0, or pin 1.05." llama.cpp dropped it in 2024; vLLM, SGLang, and transformers apply no penalty at all.
• An always-on penalty "distorts output that legitimately repeats tokens, such as code, JSON, and long reasoning traces."
• The unpinned models now matching their makers' no-penalty recommendation include, verbatim, "gemma2 through gemma4." That's us. (qwen3, qwen3.6, and qwen3-coder pin their own values and are unchanged; qwen3.5 had been paying the default too, stacked on top of its own presence_penalty of 1.5, and moved. Cloud models never receive these defaults — the tax was never universal.)

Notice what kind of argument this is: a correctness argument. The default was wrong on the merits — it changed what models say, in ways their makers never asked for — and the speed was a side effect of fixing it. His figure came scoped, and he says so: "13-16% of end-to-end throughput", on a different model, on Apple hardware, through speculative decoding (cheap tokens drafted ahead of time for the model to confirm in one pass), a mechanism our box doesn't even use — our server log confirms no draft model loaded. Inverted, a 13–16% cost is roughly a +15–19% gain when removed; that arithmetic is ours, not his. Our route to the same fix simply wasn't in his scope, and — on the narrower measure of decode rate alone, which flatters us — it turned out to pay this box considerably more.

What the tax actually was

Here is the mechanism in plain English; the source behind every claim in it is in the kit.

Every time a model generates one token, it first produces a raw score for every entry in its vocabulary — for our model, 262,144 of them, read first-party off the server's own load log. A pipeline of small sampler stages then turns those scores into the one token you see. The penalties stage is first in that pipeline, before anything narrows the field — so when it runs, it runs over the whole vocabulary. The engine that executes GGUF models like ours under ollama is llama.cpp, at the build this runtime ships (that lineage is receipted in the kit), and it does this per token: walk the entire candidate array, do a hash-table lookup per entry asking "is this token among the last 64 we've seen?" — the runtime prints that window as repeat_last_n = 64 — and adjust the few that are, then hand the array on flagged unsorted. That flag costs nothing here — penalties runs first, so nothing had sorted the array yet — but it is the tell that this stage was never meant to sit in front of a quarter-million candidates: everything downstream of top-k works on 64 of them.

A quarter-million lookups to adjust a handful of numbers, on the CPU, between GPU passes, every single token. Divide the toll we measured by the vocabulary and it comes to about 9.5 nanoseconds per candidate — an ordinary rate for a hash lookup and a strided read, which is the sanity check that says this loop is big enough to explain the bill. What we measured is what the stage costs in place; whether some part of it comes from a second effect — upstream documents that a sampler chain beginning with penalties forfeits GPU-side sampling for the whole chain, and we did not confirm whether this runtime enables that path — is an open question in the kit, not a claim here. When the penalty is 1.0 the work does not happen — either the stage never joins the chain, which is what the ? in the log suggests, or it joins and returns on its first line, effectively if disabled, return. We could not trace which, and for the bill it makes no difference.

That pass has a price. We'll call it the toll: the difference in per-token time between penalty-on and penalty-off on the same binary. We measured it in four cells across two days — three of them a dedicated run at temperature 0 we'll call the toll probes, the fourth the toggle, at the model's own default temperature:

(Medians of n=3–4 per arm; every raw sample is in the kit's TOLL-PROBE file.) Call it ~2.5 milliseconds per generated token, across cells that varied prompt length 83-fold and ran on both a loaded card and a warm-resident one.

Why our gain was so much bigger than the author's

That flat toll is the key to the whole story. It doesn't scale with how big the model is, or with how long the prompt was — it's the same ~2.5 ms whether the model's own work costs 4 ms or 13 ms per token. So the faster your model, the larger the share of its remaining budget the tax was eating — on our seat, more than half of what the model's own arithmetic cost:

Two tolls within 7% of each other in absolute milliseconds; two shares almost 3× apart — 54% of one model's own work against 19% of the other's. (What sets the toll's length is the vocabulary, and we hold a first-party vocabulary receipt only for gemma4:26b — 262,144. So read those two tolls as two readings that landed close, not as proof that the toll is one constant across models. The flatness we did establish is within one model, across an 83-fold prompt-length spread.) Our headline seat is a sparse mixture-of-experts, and it activates only about a seventh of its weights per token, which is what makes it fast — the very efficiency that let a fixed CPU-side pass eat that share. We found no mixture-of-experts-specific work anywhere in the release window (a read of the commit titles across the runtime's 16 commits and its engine's 27: a title sweep, not a proof, and the kit says so). It's arithmetic: a flat toll takes the biggest bite from the smallest budget. Same-family corroboration, directional only: our smaller gemma4:12b — a slower, non-sparse model with a bigger per-token budget — gained about +20% across the same window in shipped data, the smaller share a flat toll takes from a bigger budget.

One honesty note on that table: the qwen row's toll came through a different knob — that model ships its own presence_penalty 1.5 in its ollama parameters (first-party, kit: perf-probe-qwen-penalty — the same knob the commit flags on qwen3.5), which keeps the same penalties stage alive — and its card doesn't state an active-parameter count, so we call it "the other 27B-class seat" rather than claiming it as a true non-sparse control. Directionally it lands exactly where a flat toll says it should.

The version pair, and a twist

Everything above varies one option on one binary. The claim "the upgrade did it" needs one more experiment: both runtime versions, side by side, same hour, same card, same prompt, same settings. We'll call it the version pair. During a maintenance window on 2026-08-21 we ran the old 0.32.9 release next to the live 0.32.13 and asked both the same fixed question ten times each at temperature 0. One deviation from the pre-registration, disclosed here as in the kit: the plan specified a container; the box has no container runtime, so the old release ran as the official binary in an isolated second instance — its own port, its own byte-for-byte copy of the model store, its served version read back as 0.32.9 at staging. It also ran the model as a five-minute visitor rather than the pinned resident our other readings use, which will matter in a moment.

+31.7% from the runtime alone. (n=10 per arm, one fixed prompt, temperature 0, seed 0 — a design that suppresses variance by construction, which is why we print the ranges rather than lean on them.) But that's 1.32×, not the 1.54× the toggle measures — and the toll probe run on this same prompt, the last row of the toll table above, explains the gap with a genuinely surprising answer. On the same prompt:

• 0.32.9 running the penalty path (its default): 154.09 tok/s
• 0.32.13 forced onto the penalty path: 139.76 tok/s — about 9% slower at this work than the old runtime was. Two things about that number: the arms sat on different load postures, so treat its size as approximate — though the correction runs the wrong way for the old runtime, since 139.76 came from the faster warm-resident posture and 154.09 from the slower fresh-load one. And we never ran the cell that would place the regression precisely: 0.32.9 with repeat_penalty: 1.0 sent explicitly. Without it, "the newer runtime is slower at the penalty work" is the most economical reading of the gap, not an isolated measurement of the stage. That cell is one short run, and it is owed.
• 0.32.13 skipping the penalty path (its default): 203.00–213.66 tok/s — 203.00 at the version pair's fresh-load posture, 213.66 with the model warm-resident; same binary, same prompt, same day

The upgrade didn't make the penalty path cheaper — in every arm we measured, it was slower at it. It won by not doing the work at all. And the gap between the two designs decomposes, to the thousandth of a millisecond, into three terms this section has already met: 1.564 ms/token implied by the version pair, plus 0.665 for the newer runtime's slower penalty path, plus 0.246 for fresh-load versus warm-resident. That is 2.475 ms/token — the within-binary toll on this exact prompt. The within-binary toll is the clean one. Depending on which experiment you weight more heavily, the dividend on this seat is somewhere between 1.32× and 1.54×; three comparisons, three denominators — the toggle isolates the sampler stage, the version pair isolates the release, and the production-median jump (~1.46×, 138.6 → 203.00) mixes both with real traffic.

And one comparison across our own kits deserves printing: a 2026-08-19 cell of this exact prompt, taken beside live traffic with the model pinned resident, came in at 213.07 — about 5% above the version pair's fresh-load arm on byte-identical work (4.7% measured as per-token time). That sits right beside the 213.66 resident reading above — two resident readings agreeing, with the fresh-load arm below both. The posture gap is the 0.246 ms term in the decomposition, but we have not measured it as its own controlled experiment, and until we do, its cause stands open.

The part that isn't about speed

At temperature 0 with a fixed seed, the version pair's two arms each produced perfectly stable output — ten byte-identical replies — but they did not produce the same text as each other. 128 tokens on the old runtime, 130 on the new, different content hashes. Two caveats before that becomes a claim: these arms differ by a whole release, not only by the penalty default, and two builds of an inference engine can diverge at temperature 0 for reasons that have nothing to do with sampling. The clean version of this test — same binary, penalty on versus off, temperature 0, replies hashed — is one short run, and we have not done it. What is not in doubt is the direction: a repetition penalty exists to change which token wins, at the steps where a recently-used token was in front, and turning it off changed the scores every unpinned model on this box picks from.

So the same two-line commit that bought us a third more speed also changed, in some measurable way, what those models say. The commit author says the old default was distorting output — code, JSON, long reasoning — and every other major engine agrees with the new behavior. We believe the new default is the correct one. But "correct" was decided upstream — documented in the API reference, announced plainly in a release note, and invisible in every request we send, which is not the same as hidden. It was never concealed; it was just never felt, until it stopped. And no side-by-side scoring of penalty-on versus penalty-off answers is on record for any of our product seats. Until one runs, the honest status of the quality half of this change is: unmeasured. (The author names the trade plainly too: some smaller, older models may repeat themselves more without the penalty masking it — "the remedy is a per-model parameter, not a penalty applied to every model.") Which is an uncomfortable place to end a section: we are printing the bill before we have paid it. We would rather publish an unmeasured cost we can name than a clean story that quietly omits one.

We got a faster box and a quiet change to every answer it gives, in the same restart. Only one of those showed up on a dashboard.

The same conveyor delivers regressions

If this reads like an advertisement for blindly upgrading, here is the rest of our ledger from the same few weeks, same vendor, same conveyor:

• The same release family that gave us the dividend also shipped a win we couldn't have. v0.32.10's notes accurately scope a prefill improvement to a model format that runs through an Apple-only engine path this box doesn't have — and when we later tried to pull one of those tags ourselves (2026-08-17), the registry answered 412: this model requires macOS. One family of releases, one gift, one closed door.
• Request flags can silently interact. On some models and releases, setting think: false makes the runtime stop enforcing the structured-output format you asked for — no error, just unformatted text. We reproduced it on nemotron3:33b (schema held 2/2 with think: true, 0/2 with think: false) and saw it consistent with one other 27B-class model; we have not reduced it to a filable upstream repro, which is on us. That trap once made two models look untestable in a seat trial — we recorded them as unmeasured rather than publishing a wrong number, which is the rule, but it was still a wrong exclusion — and it has to be re-probed per model, per release, because it appears and disappears across versions. (The same model the author benchmarked his change on, as it happens, failed a tryout on our box the same month — on day-one packaging, a chat template leaking control tokens, not on ability. We have not re-screened it since, and that's our gap, not the model's verdict.)
• And the dividend is not durable. On 2026-08-17, one day after the free speed arrived, the same seat fell to 24.6 tok/s — an 8× loss against the ~203 it had been running — because a neighboring process reclaimed video memory and half the model spilled to the CPU. Free speed is a property of a runtime; keeping it is a property of how you run the box.

That's the honest shape of owning your own inference: the conveyor belt hands you gifts and regressions with the same motion. The discipline that turns it net-positive is the same either way — pin your versions, measure across every upgrade, and re-probe the traps you know about, because "it held last release" is not a receipt.

What this means if you run your own models

The practical check, because this may be running on your box right now: if you run ollama below 0.32.10, every model that doesn't pin its own repeat_penalty is paying the tax. If you can't move versions today, sending repeat_penalty: 1.0 in your request options turns the stage off from the client side — request options are layered over the server defaults, which is the same layering that let the old default reach these models in the first place. On 0.32.10 or later, look at the sampler-chain line in your server log: ?penalties means the stage is off; penalties without the ? means something — your client's defaults, your request, or the model's own vendor parameters — is turning it back on.

A two-line default change by Jesse Gross — argued on correctness, not on speed — riding an open-source release train from ollama and the llama.cpp engine underneath it, made our rules app and our walking-tour guide roughly a third faster at generating their answers while we slept. He was fixing what models say; the speed came along for free, to a box he has never heard of. That is also, incidentally, the claim we've made before — that owning the machine means the whole industry's engineering keeps upgrading it for you — caught on camera, with a name attached. We ran no fine-tune, bought no hardware, changed no config. The noticing was human; the hunt — the commit archaeology, the timeline reconstruction, the probes — was the work of a small human team and a fleet of AI agents, all in the same day. Our part was to notice it, measure it properly, and say thank you by name.

And the conveyor hasn't stopped: v0.32.15, two releases past the 0.32.13 this box runs as of 2026-08-21, claims model-metadata caching that roughly halves time-to-first-token — the delay before the first word appears, which our interactive apps feel more than they feel decode rate. That upgrade will get what this article's lesson demands: a pre-registered leg, measured across the jump, with the quality gates watching for whatever else moves. When it runs, its numbers will land in this exhibit's kit either way.

Every number in this piece carries its receipt in the kit, the evidence bundle published alongside this page: the sealed-request toggle; the version pair's raw rows with reply hashes; the toll-probe cells with every sample; the commit message fetched raw from the source; the timeline of every dated reading we hold, with each one's posture and comparability flags; and the full recon reports this article was distilled from, including the claims we could not verify, labeled as such.

The rest of the seminar

This is the machines' journey, part four. Reading is fast, writing is slow is part one and the physics under everything here: the trip through the weights that decides what a token costs. Everyone on the payroll, three at the table is part two — who shows up for each token, which is why our sparse seat had the smallest budget for the tax to eat. The compressed photograph is part three, what the quantization tags do to the weights themselves. This page is what happened when the building around all of that moved on its own. Still ahead on this road, tracked in the kit as owed cells rather than promises: the one run that would place the newer runtime's penalty-path regression precisely; the same-binary penalty-on-versus-off comparison that would settle what the quality half actually changed — first hashed, then judged; the fresh-load-versus-resident gap measured in isolation, which would close this page's one standing anomaly; and the v0.32.15 first-token dividend, pre-registered before it runs. The whole shelf holds every bench behind every claim, failures included. If one of those owed cells is the piece you want next, say so — the suggestion box is read.

Licence: CC BY 4.0, the whole page — name the source and link to it.