# the-gpu-box's +50%: what actually happened > SANITIZED AT PUBLICATION (2026-08-21): estate-internal box names, home > paths, and private overlay addresses in this working document are replaced > with public aliases ([workspace], the-gpu-box, the-dev-laptop, the-vps, > [private-peer], [agent-memory]); two receipt filenames beginning with a box > name are renamed to speed-forensic-2026-08-16.md / speed-probe-2026-08-16.json. > Quoted clock strings are re-expressed in UTC (same instants) and no longer > byte-match their sources; every load-bearing timestamp is stated in UTC. > Measurements, counts, shas, and quoted third-party text are untouched. *Forensic, 2026-08-16. Question from the operator: `gemma4:26b` on the-gpu-box jumped from ~145 tok/s to ~205 tok/s around the ollama `0.32.9 → 0.32.13` upgrade. Is it the upgrade, and what exactly caused it?* **Answer: yes, it is the upgrade — and it is one line of it.** Ollama **v0.32.10** stopped applying a default `repeat_penalty` of 1.1 to models that don't set one. `gemma4:26b` sets none. With the penalty off, ollama drops the `penalties` stage out of the sampler chain entirely, and this model — a 128-expert MoE that only activates 8 experts per token — was spending **2.57 ms of every 4.8 ms token** inside that stage. Contention, CPU offload, `num_ctx` drift and keep-alive effects are all **refuted by receipts below**, not merely argued away. The whole thing was reproduced on the-gpu-box in **12 seconds of GPU time**, by replaying the sealed 08-14 request byte-for-byte and toggling the one option back. --- ## 1. The decisive experiment — same bytes, one option The 2026-08-14 open-call round froze its request bodies. `S1-S1-ask-A`'s body is on disk with its hash, and the sample record that produced the historical 140 tok/s rows carries **the same hash**: ``` sealed/S1-S1-ask-A.bundle.json wire_sha256 = 66e1a1989bd745fc34242e62d948ecb32f7fe5d5a2c0e4a08db5a05425021dbe samples/S1-the-gpu-box/local-gemma4-26b.json records[0].wire_sha256 = 66e1a1989bd745fc34242e62d948ecb32f7fe5d5a2c0e4a08db5a05425021dbe ``` That exact body was replayed against the-gpu-box today through the same house-law builder (`ollama_houselaw.build_chat_body`, `num_ctx` 32768, `think:false` explicit, `format:"json"`, `stream:false`, no `keep_alive`), twice: once as-sent, and once with `options.repeat_penalty = 1.1` — the value ollama ≤ 0.32.9 supplied on its own. | arm | runtime | `repeat_penalty` on the wire | tok/s per call | **median** | ms/token | | --- | --- | --- | --- | --- | --- | | **08-14 sealed leg** (historical) | 0.32.9 | none sent → daemon applied **1.100** | 140.14 · 140.10 · 137.79 · 137.75 · 139.00 · 150.49 | **139.55** | 7.166 | | **today, DEFAULT** | 0.32.13 | none sent → daemon applied **1.000** | 209.79 · 208.31 · 209.37 | **209.37** | 4.776 | | **today, RP-1.1** | 0.32.13 | **1.1 sent explicitly** | 136.13 · 137.25 · 136.06 | **136.13** | 7.346 | - DEFAULT ÷ RP-1.1 = **1.538×** - DEFAULT ÷ the 08-14 sealed median = **1.500×** - RP-1.1 recovers **97.5%** of the pre-upgrade rate, on today's runtime, on today's box. - The sampler tax is **2.570 ms per generated token** — 35% of the old per-token budget. Both of today's arms ran within **12 seconds of each other** (probe window `2026-08-16T21:31:56Z → 21:32:08Z`), against the same daemon, same residents, same prompt-cache state. Whatever the box was doing, it was doing it to both arms. Per-call spread is tight in both arms (DEFAULT CV ≈ 0.4%, RP-1.1 CV ≈ 0.5%), so neither number is a queueing artefact. --- ## 2. Per-arm, per-era medians (every local arm with counters on disk) `tok_s_eval` = `eval_count ÷ eval_duration`, the daemon's own counters, decode only — model load and prompt prefill excluded. | era | when (UTC) | ollama | box | arm | n | **median tok/s** | min–max | | --- | --- | --- | --- | --- | --- | --- | --- | | open-call sealed the-gpu-box leg | 2026-08-14T13:43Z | 0.32.9 † | the-gpu-box | `local-gemma4-26b` | 6 | **139.55** | 137.75–150.49 | | open-call sealed the-gpu-box leg | 2026-08-14T13:43Z | 0.32.9 † | the-gpu-box | `local-qwen3.6-27b` | 6 | **68.19** | 64.05–68.22 | | open-call sealed leg | 2026-08-14T12:01Z | 0.32.9 † | the-dev-laptop | `local-gemma4-12b` | 6 | **68.19** | 67.7–69.0 | | qwen3.8 addendum | 2026-08-16T01:50Z | **0.32.13** (pinned) | the-gpu-box | `local-qwen3.8-27b` | 6 | **130.53** | 84.42–152.97 | | thirteen bench | 2026-08-16T14:50–15:13Z | 0.32.13 | the-gpu-box | all four local arms | 336 | **— (no rate)** | — | | **forensic replay** | 2026-08-16T21:31Z | 0.32.13 | the-gpu-box | `gemma4:26b` DEFAULT | 3 | **209.37** | 208.31–209.79 | | **forensic replay** | 2026-08-16T21:32Z | 0.32.13 | the-gpu-box | `gemma4:26b` @ rp 1.1 | 3 | **136.13** | 136.06–137.25 | † The sealed round's runtime was never written to a file. `RUNTIME-PINS.md` labels it *inferred*. This forensic **upgrades that inference to a receipt**, by process identity: the-gpu-box's unit log shows **pid 189678** serving the sealed leg's calls at `2026-08-14T13:43:20Z`; that same pid was still serving at `2026-08-16T01:00:01Z`; its `/api/version` was read as `0.32.9` at `2026-08-16T01:27Z`; and `systemctl` puts its replacement (pid 231004) at `2026-08-16T01:36:27Z`. One continuous process, version read from it directly — the sealed round ran on 0.32.9. **Two instrument gaps, disclosed rather than papered over:** 1. **The thirteen bench cannot answer this question.** `harness/thirteen_run.py:343` records only `prompt_eval_count`, `eval_count`, `done_reason`, `total_duration`, `load_duration` for local arms — `eval_duration` is dropped, so no decode rate can be computed. (The cloud arms keep it; the locals don't.) Worse, its median local `eval_count` is **6–7 tokens** ("the site doesn't say"), so even a wall-clock proxy would be pure overhead. Nothing in `results/journal.jsonl` bears on the jump. **Owed fix:** add `eval_duration` / `prompt_eval_duration` to the local counter dict — the open-call harness already does this and the two harnesses should not disagree. 2. **`local-gemma4-12b` ran on the-dev-laptop, not the-gpu-box** (`hardware_class: "a 24 GB-class portable card"`). It is listed for completeness and is **not** comparable to the the-gpu-box rows. **Corroboration that ~140 was the settled pre-upgrade rate, not a contended outlier** — three independent measurements, three weeks, different prompts and paths, all on 0.32.9: | date | measurement | tok/s | | --- | --- | --- | | 2026-07-24 | amble guide bake-off, prod payload on the-gpu-box, warm | **146** | | 2026-07-24 | amble, full the-vps → tunnel → the-gpu-box path | **142** | | 2026-08-14 | open-call sealed the-gpu-box leg (this table) | **139.55** | | 2026-08-15 06:00Z | session note, "gemma4:26b runs around 143 tok/s on the-gpu-box" | **~143** | --- ## 3. The mechanism, printed by the daemon itself the-gpu-box's `ollama` unit logs its sampler chain per request. A `?` prefix means the stage is **skipped**. Across the upgrade boundary, on the same box: **Before — pid 189678, ollama 0.32.9, `2026-08-16T01:00:01Z`:** ``` sampler chain: logits -> penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ... repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 1.500 ``` **The sealed leg itself — pid 189678, `2026-08-14T13:43:20Z`, client [private-peer] (the-dev-laptop):** ``` sampler chain: logits -> penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> ?temp-ext -> dist repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000 ``` **After — pid 231004, ollama 0.32.13, `2026-08-16T01:42:22Z`:** ``` sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> ... repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000 ``` **Today's two probe arms, twelve seconds apart:** ``` 15:31:57 ?penalties … repeat_penalty = 1.000 init sampler, took 0.21 ms → 209 tok/s 15:32:01 penalties … repeat_penalty = 1.100 init sampler, took 8.88 ms → 136 tok/s ``` `penalties` runs **before** `top-k` in the chain, i.e. over the full logit vector, once per generated token. `gemma4:26b` is `expert_count = 128`, `expert_used_count = 8`, `block_count = 30`, `embedding_length = 2816` (`/api/show`) — an MoE whose per-token matmul work is tiny, so a fixed full-vocab sampler pass is a *large* share of its token budget. That is why this model gained ~50% where a dense model of the same footprint would gain far less. **The architecture reading is inference; the 2.57 ms/token is measured.** And the reason the default reached this model at all — `/api/show gemma4:26b`: ``` PARAMETER temperature 1 PARAMETER top_k 64 PARAMETER top_p 0.95 ``` No `repeat_penalty`. The v0.32.10 change targets exactly "models that don't set one". --- ## 4. Release notes — verbatim **v0.32.10** — published 2026-08-12T22:36:49Z — > "Models that don't set a `repeat_penalty` now default to 1.0 (off) instead of 1.1, > matching other engines and speeding up speculative decoding; set a per-model parameter > if an older model repeats itself." > "Faster prefill on NVFP4 MLX models with a global scale, about 7–8% on Qwen3.6 and Muse Glimmer." > "Fixed blob verification being skipped when an OCI manifest's config and layer share a digest." **v0.32.11** — — no performance items: `ollama launch dsh` (DeepSeek Harness), `ollama launch muse` (Muse Code), web search in the OpenAI-compatible Responses API, Muse Glimmer template updates. **v0.32.12** — > "This release adds the support of Qwen 3.8 27B. Qwen3.8 delivers substantial gains across > coding, professional work, research, and long-horizon agentic tasks." > "For Apple Silicon devices, Ollama has in particular optimized for maximum performance and > output quality suitable for repeated tasks and coding agents." **v0.32.13** — published 2026-08-14T19:16:07Z — > "## What's Changed > * qwen3.8: support developer instructions" **The full window** — — 16 commits. Relevant ones: `api: stop applying repeat_penalty 1.1 to models that don't set one`; `llama.cpp bump (#17702)` — **`LLAMA_CPP_VERSION` b10353 → b10380**; `nn: speed up prefill on double-scale nvfp4 models`; `mlx: enable CUDA backend in CUDA builds (#17688)`; `MLX update (#17704)`. **Reading of the notes.** The only headline performance line that touches a Q4_K_M GGUF on CUDA is the `repeat_penalty` default. The NVFP4 prefill win is MLX-path and is a *prefill* win; our delta is entirely in `eval_duration` (prefill was 69–76 ms in **both** of today's arms). The vendor's own framing — *"speeding up speculative decoding"* — undersells it and does not describe our case: the-gpu-box's log prints `draft: 0.000 MiB`, so no draft model is in play. What we measured is the plain per-token sampler cost the same change removes. The `llama.cpp` b10353 → b10380 bump is the one candidate this forensic **cannot fully exclude**, but it is bounded: the RP-1.1 arm on the *new* llama.cpp reproduces 97.5% of the old rate, so ≤ 2.5% of the jump can live there. --- ## 5. Apportionment across the candidates | # | candidate | verdict | confidence | receipt | | --- | --- | --- | --- | --- | | **A** | genuine runtime gain in 0.32.10–13 | **CONFIRMED — the whole jump.** Specifically v0.32.10's `repeat_penalty` default, worth **1.538×** on this model. Any residual llama.cpp b10353→b10380 contribution is **≤ 2.5%**. | **high** | §1 replay A/B; §3 sampler chain across the boundary; §4 release note | | **B** | contention (llama3.3:70b at 48 GB + litellm) | **REFUTED as the cause of the baseline.** The 08-14 leg that produced 139.55 recorded `resident models ['gemma4:26b']` — gemma was the *only* resident. Today's 209.37 was measured with **three** residents (`gemma4:26b` 16.27 GiB + `gemma4:12b` 7.55 GiB + `nomic-embed-text` 0.30 GiB, all 100% GPU) and **live traffic in flight** — `/api/embeddings` and `/api/chat` from [private-peer] seconds either side of the probe. Today is the *busier* box and it is the *faster* one. | **high** | sample-record `contention` field; `/api/ps` before+after in the probe JSON; the-gpu-box unit log | | **B′** | *the premise itself* | **The 70B/litellm state was real but was never a gemma measurement.** the-gpu-box was multi-tenant with `llama3.3:70b` resident at 2026-08-16T01:27–01:33Z — minutes *before* the upgrade — but no gemma tok/s was taken in that window. The ~145 lineage is 07-24 (146, 142), 08-14 (139.55), 08-15 (~143), none of them under the 70B. | high | scribe transcript 01:27–01:33Z; the four-row corroboration table in §2 | | **C** | partial CPU offload then, 100% GPU now | **REFUTED.** Both of today's arms ran on the same 100%-GPU residency (`size_vram == size`), twelve seconds apart, and differ by 1.538×. Offload cannot differ between two calls that share a daemon and a minute. The 08-14 rows also show GPU-class prefill (1617 tokens in 69–226 ms). | high | probe `ps_before`/`ps_after`; 08-14 counters | | **D** | `num_ctx` drift | **REFUTED.** 32768 in both eras: the 08-14 record's `deltas.law.num_ctx = 32768`; today's `/api/ps` reports `context_length: 32768`; the probe body is built by the same house-law module that raises on any other value. | high | `deltas.law`; `/api/ps`; `ollama_houselaw.NUM_CTX` | | **E** | keep_alive / reload effects | **REFUTED.** `tok_s_eval` excludes load by construction, and `load_duration` was 0.394–0.407 s in **both** arms today — resident-model bookkeeping, not a reload. No call in either era sent `keep_alive` (law 3). `gemma4:26b`'s pin is intact (`expires_at` 2318). | high | probe rows; `deltas.law.keep_alive = null` | | **F** | *(added)* measurement mismatch — different prompt/output shape | **EXCLUDED by design.** The replay sent the sealed 08-14 body byte-for-byte (`wire_sha256 66e1a198…`), `prompt_eval_count = 1617` in both eras, output lengths 130–169 tokens against the historical 135–188. | high | §1 | | **G** | *(added)* different weights | **EXCLUDED.** Resident digest today `5571076f3d70050487b26b341705799e0ab29b808164f90d20d4cf84f699d251` — identical to `RUNTIME-PINS.md`'s `gemma4:26b` row, whose mtime (2026-07-24) predates the seal. | high | `/api/ps`; `RUNTIME-PINS.md` | ### The verdict **The jump is real, it is the upgrade, and it is one commit.** Ollama v0.32.10 stopped supplying a default `repeat_penalty` of 1.1 to models whose Modelfile sets none; `gemma4:26b` is such a model, and with the penalty at 1.0 the daemon drops the `penalties` stage out of the sampler chain altogether. That stage was costing **2.570 ms of every 4.776 ms token** — because `gemma4:26b` is a 128-expert MoE using 8 experts per token, its actual matmul work is small enough that a once-per-token full-vocab sampler pass dominated the budget. Replaying the sealed 2026-08-14 request byte-for-byte on today's runtime gives **209.37 tok/s**; adding back the single option `repeat_penalty: 1.1` gives **136.13 tok/s**, or 97.5% of the historical 139.55 — same daemon, same weights, same context, twelve seconds apart. The contention story, which was the most plausible sceptical alternative, runs the wrong way: the slow historical number was measured with gemma as the *only* resident on the-gpu-box, while today's fast number was measured with three residents and live embedding/chat traffic in flight. Context, offload, keep-alive, weights and prompt shape are each pinned identical across the two eras by their own receipts. The `llama.cpp` b10353 → b10380 bump in the same window is the one thing this cannot fully separate, and it is bounded at **≤ 2.5%** by the RP-1.1 arm reproducing the old rate on the new binary. Confidence: **high**, and it is the rare case where the mechanism is not inferred but *printed by the daemon* — `?penalties` versus `penalties`, on either side of a service restart at `2026-08-16T01:36:27Z`. *(Timing correction for the record: the cutover was **2026-08-16T01:36:27Z** (`2026-08-16T01:36:27Z`), per `systemctl show ollama -p ActiveEnterTimestamp`, not ~21:45.)* --- ## 6. What this changes operationally - **The gain is free and it is already everywhere** that talks to the-gpu-box without penalty options: today's live callers log `repeat_penalty = 1.000` and `?penalties`. rulesage-live and amble sit downstream of this daemon, which is why the operator saw it on both paths. - **It is not everywhere.** 92 requests today still ran the `penalties` stage — 4 of them this probe's RP-1.1 arm, the other 88 because a caller sends **`presence_penalty = 1.500`**, and *any* non-zero penalty keeps the stage in the chain regardless of the new `repeat_penalty` default. Those calls still pay the ~35% tax. Today's instances came from `[private-peer]` (the-dev-laptop) at 14:50Z; the same `presence_penalty = 1.500` signature appears in pre-upgrade traffic on 08-15. **Worth a follow-up:** find that caller and decide whether 1.5 is load-bearing or inherited. - **Quality is a real question, not a formality.** The vendor's own note hedges — *"set a per-model parameter if an older model repeats itself."* Nothing here measured output quality; a 50% speed win that reintroduces looping is not a win. The estate's judged benches are the instrument for that, and no bench has yet compared 1.0 vs 1.1 output. - **Comparability caveat for the open-call round:** every sealed local arm ran with the penalties stage on; anything re-measured after 2026-08-16T01:36Z did not. The qwen3.8 addendum (130.53 tok/s) is a **post-change** number sitting in a **pre-change** table. Its *latency* rows are not comparable to the sealed arms'; its scored content is unaffected. - **The the-gpu-box page's speed line needs a number and a date.** Suggested: "≈209 tok/s (`gemma4:26b`, ollama 0.32.13, `num_ctx` 32768, 1617-token prompt, measured 2026-08-16); ≈140 tok/s on 0.32.9 before ollama stopped defaulting `repeat_penalty` to 1.1." --- ## 7. What we did NOT do, and why **A clean downgrade A/B** — pin the-gpu-box back to 0.32.9, replay, re-upgrade — would add the last 2.5%: it would separate the `llama.cpp` b10353 → b10380 bump from the sampler change with no inference at all. **We are not doing it**, for three reasons: 1. **the-gpu-box is a prod pin.** It serves rulesage-live and amble; a downgrade means a service restart that evicts every resident and drops in-flight generation. The standing rule is never to restart the daemon during an ingest saga, and the box was serving throughout this forensic (581 + 1578 requests today from two other clients). 2. **0.32.9 → 0.32.13 was a one-way gate.** `qwen3.8:27b` cannot be pulled below 0.32.12 and is now a bench arm; downgrading would strand it. 3. **The counterfactual is already measured.** `repeat_penalty: 1.1` on 0.32.13 recovers 97.5% of the 0.32.9 rate. Any llama.cpp contribution is inside that 2.5% band — smaller than the historical run's own spread (137.75–150.49). A downgrade would resolve a quantity we have already bounded below the noise floor. The honest residual: **≤2.5% of the jump is unattributed** between the llama.cpp bump and ordinary run-to-run variance. The other ~97.5% has a name and a commit. --- ## 8. Receipts index | what | where | | --- | --- | | probe output (both arms, `ps` before/after, version, bundle hash) | `[workspace]/rs-thirteen-2026-08/recon/speed-probe-2026-08-16.json` | | frozen 08-14 request body | `[workspace]/rs-open-call-2026-08-14/results/sealed/S1-S1-ask-A.bundle.json` | | historical gemma rows w/ `tok_s_eval` | `[workspace]/rs-open-call-2026-08-14/results/samples/S{1,2,3}-the-gpu-box/local-gemma4-26b.json` | | contention observed at 08-14 run time | same files, `contention` field | | 0.32.13 pin + weight digests | `[workspace]/rs-open-call-2026-08-14/results/addendum-qwen38/RUNTIME-PINS.md` | | thirteen local counters (the gap) | `[workspace]/rs-thirteen-2026-08/harness/thirteen_run.py:343` | | upgrade timestamp | the-gpu-box: `systemctl show ollama -p ActiveEnterTimestamp` → `2026-08-16T01:36:27Z` | | sampler chain across the boundary | the-gpu-box: `journalctl -u ollama --since "2026-08-15 19:00" --until "2026-08-15 20:10"` | | model parameters (no `repeat_penalty`) | `curl -s http://[gpu-box-daemon]/api/show -d '{"model":"gemma4:26b"}'` | **Probe cost and posture.** 8 generations total (2 discarded warmups + 6 scored), ~1.3 s each, **12 s of GPU wall in one window**. Read-only everywhere else: `/api/version`, `/api/ps`, `/api/show`, and `journalctl`/`systemctl show` over ssh. No model was loaded or evicted, no pin was rewritten (no `keep_alive` on any call), `:9809` was never touched.