{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "the-free-speed-wasnt-free",
 "title": "The free speed wasn’t free — the runtime dividend, traced to one commit",
 "dek": "One day our busiest model started producing tokens roughly a third faster than it ever had, and nobody had touched it. The cause was a two-line upstream commit ending a ~2.5 ms-per-token tax nobody configured — and, in the same stroke, changing what every unpinned model says. The receipts, the mechanism, and the honest unmeasured half.",
 "published": "2026-08-22",
 "series": [
  "bench",
  "notes"
 ],
 "licence": "CC BY 4.0 for the kit's records; the upstream commit message and the raw API response it was read from are the upstream project's own, quoted verbatim as research receipts",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/the-free-speed-wasnt-free/",
  "md_url": "https://research.strata2signal.com/the-free-speed-wasnt-free/index.md",
  "md_sha": "ae6d39d58b327b66f6c59ac6991c46e132d84891bbcab46343008825b09a8b59",
  "html_sha": "658c5400250c2d145e0600a02ad25c3f2d45e64ce62dba459a0780fa21ce9a62",
  "anchors_sha": "063a1604a59cf6cd52988458ddc1c4ae83f7feb39b391351d9fec74640429205"
 },
 "short": {
  "paragraph": "One day our busiest model started producing tokens roughly a third faster than it ever had, and nobody had touched it. The cause was a two-line upstream commit ending a ~2.5 ms-per-token tax nobody configured - and, in the same stroke, changing what every unpinned model says. The receipts, the mechanism, and the honest unmeasured half.",
  "from_dek": true,
  "counts": {
   "words": 4363,
   "minutes": 20,
   "tables": 4,
   "kit": true
  },
  "bullets": [
   {
    "text": "the production median for gemma4:26b on 2026-08-15 was 138.6 tok/s",
    "figure": "138.6 tok/s",
    "cite": "the-day-the-box-got-faster",
    "quote": "For weeks, every well-controlled median of its decode rate had landed between 138 and 146 tokens per second: a production median of 138.6 tok/s over 1,982 live rulings on 2026-08-15 (the strongest single pre-upgrade receipt we own) and a sealed bench run on 2026-08-14 at 139.55 (n=6, range 137.75-150.49).",
    "span": [
     2765,
     2779
    ]
   },
   {
    "text": "the toggle experiment showed that re-adding the penalty option resulted in a rate of 136.13 tok/s",
    "figure": "136.13 tok/s",
    "cite": "the-hunt-change-one-thing",
    "quote": "We'll call this experiment the toggle: arm options sent decode rate (median of 3) as-is (new defaults) {\"numctx\": 32768} (context window, held constant) 209.37 tok/s one option re-added {\"repeatpenalty\": 1.1, \"numctx\": 32768} 136.13 tok/s One option, repeatpenalty: 1.1, took back the entire speed jump, and then some.",
    "span": [
     5055,
     5071
    ]
   }
  ]
 },
 "sections": [
  {
   "id": "the-day-the-box-got-faster",
   "heading": "The day the box got faster",
   "level": 2,
   "span": [
    1962,
    4391
   ],
   "chunks": [
    [
     1962,
     4391
    ]
   ],
   "chars": 2429,
   "digest": "on 2026-08-16, an upgrade to ollama from 0.32.9 to 0.32.13 caused gemma4:26b to jump from a production median of 138.6 tok/s to between 202–214 tok/s. the authors investigated whether this was due to a runtime improvement, contention, or a changed context setting.",
   "digest_skipped": null
  },
  {
   "id": "the-hunt-change-one-thing",
   "heading": "The hunt: change one thing",
   "level": 2,
   "span": [
    4391,
    6076
   ],
   "chunks": [
    [
     4391,
     6076
    ]
   ],
   "chars": 1685,
   "digest": "the authors performed a toggle experiment by replaying a byte-identical sealed request with one option changed. re-adding repeat_penalty: 1.1 dropped the decode rate from 209.37 tok/s to 136.13 tok/s, a 1.538× difference, revealing that the penalty stage had been skipped in the new version.",
   "digest_skipped": null
  },
  {
   "id": "the-two-line-commit",
   "heading": "The two-line commit",
   "level": 2,
   "span": [
    6076,
    9229
   ],
   "chunks": [
    [
     6076,
     9229
    ]
   ],
   "chars": 3153,
   "digest": "ollama commit 6a261db7d changed the default repeat_penalty from 1.1 to 1.0. the author, Jesse Gross, argued this change improves correctness for models like gemma2 through gemma4 by matching makers' recommendations and avoiding distortion of output like code or JSON.",
   "digest_skipped": null
  },
  {
   "id": "what-the-tax-actually-was",
   "heading": "What the tax actually was",
   "level": 2,
   "span": [
    9229,
    15104
   ],
   "chunks": [
    [
     9229,
     15104
    ]
   ],
   "chars": 5875,
   "digest": "the penalty stage requires a CPU-side hash-table lookup across the entire vocabulary of 262,144 entries for every token. this creates a toll of ~2.5 ms/token. because gemma4:26b is a sparse mixture-of-experts, this fixed toll consumed 54% of its per-token work, resulting in a 1.538× speedup.",
   "digest_skipped": null
  },
  {
   "id": "the-version-pair-and-a-twist",
   "heading": "The version pair, and a twist",
   "level": 2,
   "span": [
    15104,
    19092
   ],
   "chunks": [
    [
     15104,
     19092
    ]
   ],
   "chars": 3988,
   "digest": "a version pair experiment showed ollama 0.32.13 provided a +31.7% speedup over 0.32.9. however, 0.32.13 was actually slower at the penalty work itself. the total gain came from skipping the penalty path entirely, which the new runtime does by default.",
   "digest_skipped": null
  },
  {
   "id": "the-part-that-isnt-about-speed",
   "heading": "The part that isn't about speed",
   "level": 2,
   "span": [
    19092,
    21343
   ],
   "chunks": [
    [
     19092,
     21343
    ]
   ],
   "chars": 2251,
   "digest": "the change in penalty defaults also altered model output, as the two runtime versions produced different content hashes at temperature 0. while the authors believe the new default is correct, the impact on output quality remains unmeasured.",
   "digest_skipped": null
  },
  {
   "id": "the-same-conveyor-delivers-regressions",
   "heading": "The same conveyor delivers regressions",
   "level": 2,
   "span": [
    21343,
    23673
   ],
   "chunks": [
    [
     21343,
     23673
    ]
   ],
   "chars": 2330,
   "digest": "upgrades can bring both gifts and regressions, such as Apple-only engine paths or silent interactions where setting think: false stops structured-output enforcement. additionally, speed is not durable if a neighboring process reclaims video memory, causing a drop to 24.6 tok/s.",
   "digest_skipped": null
  },
  {
   "id": "what-this-means-if-you-run-your-own-models",
   "heading": "What this means if you run your own models",
   "level": 2,
   "span": [
    23673,
    26379
   ],
   "chunks": [
    [
     23673,
     26379
    ]
   ],
   "chars": 2706,
   "digest": "users on ollama below 0.32.10 can turn off the penalty stage by sending repeat_penalty: 1.0 in request options. the authors note that the speedup was a side effect of a correctness fix that changed how models generate answers.",
   "digest_skipped": null
  },
  {
   "id": "the-rest-of-the-seminar",
   "heading": "The rest of the seminar",
   "level": 2,
   "span": [
    26379,
    27932
   ],
   "chunks": [
    [
     26379,
     27932
    ]
   ],
   "chars": 1553,
   "digest": null,
   "digest_skipped": "credits"
  }
 ],
 "chips": [
  {
   "id": "c-b37bd390",
   "q": "how much did the decode rate increase after the ollama upgrade?",
   "a": "The decode rate for the gemma4:26b model increased from a production median of 138.6 tok/s to between 202–214 tok/s following the upgrade from version 0.32.9 to 0.32.13.",
   "cites": [
    "the-day-the-box-got-faster"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-381a22c7",
   "q": "what specific parameter change caused the speed jump?",
   "a": "The speed increase was traced to the removal of a default repeat_penalty of 1.1. Re-adding the repeat_penalty: 1.1 option reduced the decode rate from 209.37 tok/s to 136.13 tok/s.",
   "cites": [
    "the-hunt-change-one-thing"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-759e7313",
   "q": "what was the reasoning behind the two-line commit?",
   "a": "The commit changed the default repeat_penalty from 1.1 to 1.0 to match other engines and avoid distorting output for models like gemma2 through gemma4 that do not recommend a 1.1 penalty.",
   "cites": [
    "the-two-line-commit"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-5496da6b",
   "q": "how much time does the penalties stage cost per token?",
   "a": "The toll for the penalties stage was measured at approximately 2.5 milliseconds per generated token, a cost that remains relatively flat regardless of prompt length or model size.",
   "cites": [
    "what-the-tax-actually-was"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-9f2c29c3",
   "q": "how did the newer runtime compare to the old version in a side-by-side test?",
   "a": "In a version pair test, the 0.32.13 release achieved a median of 203.00 tok/s compared to 154.09 tok/s for version 0.32.9, representing a 31.7% increase from the runtime alone.",
   "cites": [
    "the-version-pair-and-a-twist"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-d74c5b98",
   "q": "how did the change affect the actual text generated by models?",
   "a": "The change in the repeat_penalty default altered the scores used for token selection, resulting in different content hashes and different text lengths between the old and new runtimes.",
   "cites": [
    "the-part-that-isnt-about-speed"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-4e650f57",
   "q": "what risks are associated with upgrading inference runtimes?",
   "a": "Upgrades can introduce regressions, such as silent interactions where request flags like think: false cause the runtime to stop enforcing structured output, or performance drops if video memory is reclaimed.",
   "cites": [
    "the-same-conveyor-delivers-regressions"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-6de8bf33",
   "q": "how can users manually disable the penalty stage in older versions?",
   "a": "Users running ollama below 0.32.10 can turn the stage off by sending repeat_penalty: 1.0 in their request options, as request options are layered over server defaults.",
   "cites": [
    "what-this-means-if-you-run-your-own-models"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "four-homes-for-one-reranker",
   "why": "shares ground with § The hunt: change one thing · § A question on game night"
  },
  {
   "slug": "a-laptop-asked-the-desktops-questions",
   "why": "shares ground with § The day the box got faster · § How it was measured"
  },
  {
   "slug": "two-hours-on-battery",
   "why": "shares ground with § What the tax actually was · § The wait: prefill is the real CPU tax"
  }
 ],
 "thanks": "",
 "kit": {
  "url": "https://research.strata2signal.com/the-free-speed-wasnt-free/data/",
  "licence": "CC BY 4.0",
  "files": [
   "AB-LEG-PREREG.md",
   "PREREG-INDEX.txt",
   "ab-runs.json",
   "AB-RUN-2026-08-21.md",
   "TOLL-PROBE-2026-08-21.md",
   "RECON-A-upstream.md",
   "RECON-B-timeline.md",
   "RECON-C-legend.md",
   "COMMIT-6a261db7-MESSAGE-VERBATIM.txt",
   "COMMIT-6a261db7-RAW-API.json",
   "VOCAB-RECEIPT-2026-08-21.md",
   "speed-forensic-2026-08-16.md",
   "speed-probe-2026-08-16.json",
   "perf-probe-qwen-penalty.json",
   "leg2b_ab.py"
  ]
 },
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 2,
  "digests_written": 8,
  "digests_kept": 8,
  "chips_written": 8,
  "chips_kept": 8,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 0,
   "figure": 1,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0
  },
  "new_noun_tokens_checked": [
   "0",
   "0.32.10",
   "0.32.13",
   "0.32.9",
   "1.0",
   "1.1",
   "1.538×",
   "136.13",
   "138.6",
   "144",
   "154.09",
   "2.5",
   "202-214",
   "2026-08-15",
   "2026-08-16",
   "203.00",
   "209.37",
   "24.6",
   "262",
   "26b",
   "31.7%",
   "54%",
   "6a261db7d",
   "Apple-only",
   "CPU-side",
   "Gross",
   "JSON",
   "Jesse",
   "gemma2",
   "gemma4"
  ],
  "new_noun_tokens_withheld": 0,
  "figure_definition": "v3",
  "drops": [
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the measured toll for the penalty stage was approximately ~2.5 ms/token",
    "detail": "terminal token not verbatim in the cited span"
   }
  ]
 },
 "generated_utc": "2026-09-09T01:27:46Z"
}