{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "temperature-zero-same-answer-twice",
 "title": "Does temperature zero give the same answer twice?",
 "dek": "We sent the same ten prompts again at temperature zero, under rules frozen first: at home a new llama.cpp server repeated itself on 10 of 10 until a prompt cache was reused or other requests arrived; hosted, none of 100 read prompts over 11 models came back the same three times.",
 "published": "2026-11-15",
 "series": [
  "bench"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/temperature-zero-same-answer-twice/",
  "md_url": "https://research.strata2signal.com/temperature-zero-same-answer-twice/index.md",
  "md_sha": "05f3c80660e7ef85da283fca5310302cf5a0f1808ad0929d06bf8ade4aba86e9",
  "html_sha": "5dc404ce8e0ce90186c642749110ce475a8073a424034f9cf672c473348ad573",
  "anchors_sha": "87dabf43ff2fa051177955dc4a98d66a69d5ffa3783d4557d1c0fe132cdd5f2a"
 },
 "short": {
  "paragraph": "At temperature zero a sampler takes the highest-scoring token every time, so the same scores give the same text. Whether the scores come out the same is another matter: on the one build and model we read at home, it turned on the server's state, not on the setting. On 2026-11-13 (UTC) we sent ten prompts of our own, at temperature 0 with seed 0, two ways. At home (llama.cpp v0.9.0, Ministral 3 8B Instruct 2512 at Q4KM, one NVIDIA GeForce RTX 3090 (24 GB) capped at 250 W, answers capped at 512 tokens), a new server's first answer came back token for token on 10 prompts of 10 from another new server, from a warm one with the prompt cache off, and from a four-slot one with nothing else running. With the cache on it came back on 3 of 10 after a different prompt and on 0 of 10 after the same prompt; with other requests arriving, on 0 of 10. Short answers were mostly spared: at 16 tokens, 9 or 10 of 10 still matched in every condition. Hosted (21 models on Ollama's cloud, thinking on, three passes), 11 models could be read, and none of their 100 read prompts gave the same answer three times; 73 of the 100 had parted within 50 characters. The two are not one experiment: home compares the tokens of one model, hosted compares characters written after thinking, across eleven. Each wrote its bets down first and each lost some. Neither says why an answer moved, which answer is better, or how often any of this happens: ten prompts give k of 10, never a rate.",
  "from_dek": false,
  "counts": {
   "words": 3479,
   "minutes": 16,
   "tables": 7,
   "kit": false
  },
  "bullets": []
 },
 "sections": [
  {
   "id": "the-question",
   "heading": "The question",
   "level": 2,
   "span": [
    2821,
    3881
   ],
   "chunks": [
    [
     2821,
     3881
    ]
   ],
   "chars": 1060,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "two-benches",
   "heading": "Two benches, and how they differ",
   "level": 2,
   "span": [
    3881,
    5945
   ],
   "chunks": [
    [
     3881,
     5945
    ]
   ],
   "chars": 2064,
   "digest": "two benches were pre-registered to test whether scores come out the same. the home bench used llama.cpp v0.9.0 on an NVIDIA GeForce RTX 3090, while the hosted bench used Ollama's cloud. the home bench compared token ids and bytes, while the hosted bench compared characters of the answer.",
   "digest_skipped": null
  },
  {
   "id": "at-home",
   "heading": "At home: the same answer when the prompt is read fresh and alone",
   "level": 2,
   "span": [
    5945,
    9982
   ],
   "chunks": [
    [
     5945,
     9982
    ]
   ],
   "chars": 4037,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "hosted",
   "heading": "Hosted: no prompt came back the same three times",
   "level": 2,
   "span": [
    9982,
    15241
   ],
   "chunks": [
    [
     9982,
     15241
    ]
   ],
   "chars": 5259,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "how-soon",
   "heading": "How soon the answers part",
   "level": 2,
   "span": [
    15241,
    17894
   ],
   "chunks": [
    [
     15241,
     17894
    ]
   ],
   "chars": 2653,
   "digest": "at home, short answers were mostly safe, with all ten prompts matching through 128 tokens when neighbours were on the server. hosted answers parted quickly, with 73 of 100 parting within 50 characters, and 21 of them at the first character.",
   "digest_skipped": null
  },
  {
   "id": "the-bets",
   "heading": "The bets, and the ones that lost",
   "level": 2,
   "span": [
    17894,
    19871
   ],
   "chunks": [
    [
     17894,
     19871
    ]
   ],
   "chars": 1977,
   "digest": "the home bench results showed that a reused opening can change long answers, causing a failed bet. the hosted bench showed that short answers are not spared, as no model managed to stay the same at 50 characters on 8 or more prompts.",
   "digest_skipped": null
  },
  {
   "id": "what-neither-can-say",
   "heading": "What neither bench can say",
   "level": 2,
   "span": [
    19871,
    21007
   ],
   "chunks": [
    [
     19871,
     21007
    ]
   ],
   "chars": 1136,
   "digest": "the benches cannot explain why scores change, nor can they say anything about weights. they cannot predict another day, route, or build, nor can they determine which answer is better or provide a rate.",
   "digest_skipped": null
  },
  {
   "id": "what-to-do-with-this",
   "heading": "What to do with this",
   "level": 2,
   "span": [
    21007,
    21871
   ],
   "chunks": [
    [
     21007,
     21871
    ]
   ],
   "chars": 864,
   "digest": "if a test needs the same answer twice, it must own the server's state. at home, this means the prompt cache is off and nothing else is in flight. for hosted services, one should treat one answer as one draw.",
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work",
   "heading": "How to check our work",
   "level": 2,
   "span": [
    21871,
    23162
   ],
   "chunks": [
    [
     21871,
     23162
    ]
   ],
   "chars": 1291,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    23162,
    23845
   ],
   "chunks": [
    [
     23162,
     23845
    ]
   ],
   "chars": 683,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "sources",
   "heading": "Sources",
   "level": 2,
   "span": [
    23845,
    25118
   ],
   "chunks": [
    [
     23845,
     25118
    ]
   ],
   "chars": 1273,
   "digest": null,
   "digest_skipped": null
  }
 ],
 "chips": [
  {
   "id": "c-4f1acb57",
   "q": "how do language models select specific tokens?",
   "a": "A language model produces scores for every known token, and an external sampler converts these scores into a choice. At temperature zero, the sampler simply selects the token with the highest score.",
   "cites": [
    "the-question"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-31084aa3",
   "q": "how do the two experimental benches differ?",
   "a": "The home bench uses a local llama.cpp build with a single model and compares token ids and bytes against a reference. The hosted bench uses Ollama's cloud with various models, comparing character agreement among three passes without a reference.",
   "cites": [
    "two-benches"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-cc963219",
   "q": "how often do hosted models produce identical answers?",
   "a": "None of the 100 prompts across the read models produced three identical answers. Agreement was limited to 27 prompts at 50 characters, 10 at 200 characters, and 1 at 800 characters.",
   "cites": [
    "hosted"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-c35b8e26",
   "q": "how quickly do answers diverge in different settings?",
   "a": "At home, answers remain stable through 128 tokens when neighbors are present, but diverge much faster when the cache is reused. In the hosted bench, 73 of 100 prompts parted within 50 characters.",
   "cites": [
    "how-soon"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-36fc79e8",
   "q": "what limitations exist regarding the causes of divergence?",
   "a": "The article cannot determine why scores change, nor can it provide information about specific model weights, different builds, or different days. It also cannot determine which of the differing answers is better.",
   "cites": [
    "what-neither-can-say"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-f8da2c34",
   "q": "how should one handle regression testing for models?",
   "a": "Tests requiring identical answers must own the server's state, including turning the prompt cache off and ensuring no other requests are in flight. For hosted services, users should treat each answer as a single draw and measure differences against the noise floor.",
   "cites": [
    "what-to-do-with-this"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "flash-next-why-the-first-word-comes-sooner",
   "why": "shares ground with § Two benches, and how they differ · § What else the bench read"
  },
  {
   "slug": "flash-next-on-three-cards",
   "why": "shares ground with § Two benches, and how they differ · § The three cards, and how they were run"
  },
  {
   "slug": "reading-the-answer",
   "why": "shares ground with § Two benches, and how they differ · § What we asked it"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nThanks to the [llama.cpp](https://github.com/ggml-org/llama.cpp) project (MIT), whose release, manual and source the home bench stood on; to Mistral AI for Ministral 3 8B Instruct 2512 and its own GGUF file (Apache 2.0); to NVIDIA for the CUDA runtime that binary uses (proprietary); to Ollama for its client (MIT) and its cloud; and to the makers of the 21 models it served, named above by tag. Both harnesses are plain Python on its standard library. A small human team asked for these benches and signed the numbers; AI agents wrote the registrations, built and ran the harnesses, recounted the rows and drafted this page.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 0,
  "digests_written": 9,
  "digests_kept": 5,
  "chips_written": 8,
  "chips_kept": 6,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 0,
   "figure": 0,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0,
   "identifier": 1,
   "bare_figure": 1,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 7
  },
  "new_noun_tokens_checked": [
   "0",
   "1",
   "10",
   "100",
   "128",
   "200",
   "21",
   "250",
   "27",
   "3090",
   "4",
   "50",
   "73",
   "8",
   "800",
   "AI",
   "GeForce",
   "Mistral",
   "NVIDIA",
   "Ollama",
   "Ollama's",
   "RTX",
   "W",
   "v0.9.0"
  ],
  "new_noun_tokens_withheld": 0,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "bullet",
    "reason": "identifier",
    "item": "the home bench used an NVIDIA GeForce RTX 3090 capped at 250 W",
    "detail": "nvidia is in the cited section only as code or in a table cell"
   },
   {
    "kind": "bullet",
    "reason": "bare_figure",
    "item": "the hosted bench sent each prompt three times to each of 21",
    "detail": "the cited section writes this figure as '21 models' and the bullet prints '21' alone"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 42,
    "text_sha": "0207aacd5447",
    "figure": "10",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "0 of 10 equalled the reference and the three runs were one text on 2 of 10",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: 0 of 10 equalled the reference and the three runs were one text on 2 of 10. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 287,
    "text_sha": "f6e2b746dbcd",
    "section": "the-question",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "the engine we read does not promise the same digits from batches of different sizes.",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: the engine we read does not promise the same digits from batches of different sizes. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 238,
    "text_sha": "660c6ea69a42",
    "section": "at-home",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "It equalled the reference on 3 prompts of 10",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: It equalled the reference on 3 prompts of 10. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 254,
    "text_sha": "85fadbd9a5eb",
    "section": "hosted",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "No prompt came back the same three times: 0 of 100.",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: No prompt came back the same three times: 0 of 100. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 138,
    "text_sha": "238e6ef63a04",
    "section": "sources",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "to NVIDIA for the CUDA runtime that binary uses (proprietary)",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: to NVIDIA for the CUDA runtime that binary uses (proprietary). Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 251,
    "text_sha": "917552413dac",
    "chip_id": "c-93c0900e",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "Ten prompts make this a sketch, not a guarantee.",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: Ten prompts make this a sketch, not a guarantee. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 313,
    "text_sha": "b1c1aad81c8e",
    "chip_id": "c-ee304f92",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-11-14",
    "why": "and its three runs, each after a different prompt, were one text on 4",
    "detail": "removed from this pack by hand on 2026-11-14 (UTC), after the writer ran and after the gates had counted: and its three runs, each after a different prompt, were one text on 4. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-11-14T06:06:12Z"
}
