# Why we write the bet down first *Before one of our benches runs, we write down what we expect, what counts as a pass and what we will print if it fails, then seal the file with a fingerprint and a time. This guide says why, where the habit comes from in medicine and physics, the night it caught a fault that no other check in the run pointed to, and a five-line version you can use on your own machine.* *Published 2026-11-11 (UTC) · A small (human) team and a fleet of AI agents.* **the short version:** A benchmark on your own hardware is cheap to repeat, and that is the problem: a second run, a different flag, a warmer cache or a kinder subset is one more try until the number looks right. Pre-registration is the habit of writing the question, the predictions, the pass marks and the failure wording down before the data exist, so the data can only confirm or refute them. Medicine adopted it for clinical trials, particle physics adopted a stricter cousin called blind analysis, and the same discipline fits a one-person bench. We use four habits: a plan sealed with a sha256 fingerprint and a UTC time; controls, runs whose answer we already know; amendments, dated and written before the run they touch, kept outside the sealed text (our recent plans keep them in a file beside it); and losing bets printed as losses, with no table of what the loser would have needed. On 2026-10-29 one of those controls, the first step of a scheduled overnight run, read "0.0 % on card": the sandbox could not see the GPU, the model ran on the CPU at about a tenth of its usual speed, and no other check in the run pointed to the card. The control was registered to report, not to stop the night, so the night still measured nothing; its lesson became a gate for the next bench, written and stamped before that bench's first scored run, and the gate read the card on its first night. The lost night's re-run was registered by a signed amendment on 2026-11-10 and had not run on 2026-11-10 (UTC), when this page was written. 4,038 words · about 18 minutes (at 220 words/min) · 2 tables · data kit: no https://research.strata2signal.com/write-the-bet-down-first/ --- *If you're new here: [strata→signal](https://strata2signal.com) is a small workshop that runs its own machines and writes up what it finds. Most pages on this shelf are benches. This one is a guide to the method behind them, for a reader who has met our labels (FROZEN, VOID, NOT TESTED, "registered", "refuted") and wants to know what they commit us to.* ## The night the known answer came back wrong {#the-night} On 2026-10-29 a scheduled window on our inference server, whose larger card is an NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB), took the live services down and started a row of model steps inside a sandbox. The first step was an anchor: gemma4:26b on Ollama 0.32.13, a model we have measured many times, run first so the night's numbers could be checked against a figure we already trust. The night's plan, frozen on 2026-10-26 at 18:21:27Z, said what the anchor was for. The anchor finished with exit code 0 in 167.7 seconds, and our harness printed its placement label: "SPILL 0.0 % on card". None of the model was on the card. It had written at 19.5 tokens a second, where the same step on the same card and build wrote 201.82 on 2026-10-27: about a tenth. Inside its sandbox the model could not see the GPU and had loaded into system memory. The main model never answered: its server exited before its first reply. From the scheduler's side the night was green, "EARLY-OK", the services restored at 07:39:14Z. Here is the honest part. The anchor was registered as a control that reports, not one that stops. The plan's words, where A-1 is the anchor: "A NOT RUN or VOID A-1 is printed and the row goes on." It printed the fault in the first three minutes, and the row went on to the main model, which failed too; that it failed for the same cause is our inference, since its server's log was never read. Without the known answer we would have had a clean exit code and a speed nobody was awake to read; with it we had a diagnosis to start from and a rule to change. The diagnosis, wrong turn included, belongs with that bench's results, and there are none yet. This page is about the rule. ## What pre-registration is {#what-it-is} Pre-registration is writing the question, the predictions, the pass marks and the analysis down before the data exist, in a form you cannot quietly change afterwards. Nosek and colleagues put the point in a pair of words, prediction against postdiction: "In prediction, data are used to confront the possibility that the prediction is wrong." A postdiction is made after the data are in, to explain them, and nothing in the data can confront it. "Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions." The habit has three lineages. Clinical trials: the committee of medical journal editors set out its registration policy in 2004, and today it recommends that journals require registration in a public registry "at or before the time of first patient consent for enrollment as a condition of consideration for publication". Registered reports, a format introduced at the journal *Cortex* in 2013: the plan is peer-reviewed before the outcomes are observed, and one that passes gets "an in-principle acceptance that will not be revoked based on the outcomes". And blind analysis in particle physics, which Klein and Roodman describe as preventing experimenters "from knowing the answer until the analysis is complete". They quote Dunnington, measuring a constant of nature in 1932: "It is easier than is generally realized to unconsciously work toward a certain value." Machine learning tried the format in 2020. A NeurIPS 2020 workshop took 73 pre-registered proposals and accepted 38; of the 25 that came back with results, 23 were accepted. Its organisers' reason: "since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted". ## Why a home bench needs it {#why-a-home-bench} The danger on your own desk is rarely fraud; it is quieter. Gelman and Loken called it the garden of forking paths: "researchers can perform a reasonable analysis given their assumptions and their data, but had the data turned out differently, they could have done other analyses that were just as reasonable in those circumstances." On a desk the forks are the engine build, the flags, the context length, warm or cold, which three of five runs count, and which card. Each choice is defensible; the trouble is choosing after the readings are in, with the number you hoped for in view. That reading of their argument is ours; they wrote about research papers, not benchmarks. A bench is also cheap to re-run, which a trial is not, and the temptation needs no bad intent. Feynman put it in one sentence: "The first principle is that you must not fool yourself—and you are the easiest person to fool." There is a measurement of what writing the bet down does to results. Kaplan and Irvin read 55 large trials funded by the US National Heart, Lung, and Blood Institute: 17 of 30 published before 2000 reported a significant benefit on their primary outcome, against 2 of 25 published after, and "pre-registration in clinical trials.gov was strongly associated with the trend toward null findings". They are careful, and so are we: "the design of the study does not allow causal inferences". It is an association, and the reading that registration took away the room to pick a winning outcome afterwards is theirs and ours, not a proof. ## The FROZEN line {#the-frozen-line} The night's plan, like our recent ones, is one text file. It carries the question; the arms (each engine, model file and setting that runs); the gates (pass-or-fail tests); the bands (a numeric range predicted for each reading before it exists); what a failure prints; and what the results can and cannot say. Its last line seals it. Its shape, with the digest and the plan's file name left out on purpose: ``` FROZEN · sha256 <64 hex characters> over every byte above this line (check: `sed '$d' | sha256sum`) · 2026-10-26T18:21:27Z · mode 0444 ``` sha256 is a hash function from the NIST Secure Hash Standard: it turns any file into a 64-character fingerprint, and a change of one byte anywhere gives a different one. The standard's own words: "The digests are used to detect whether messages have been changed since the digests were generated." The check in the line is the whole procedure: drop the last line, hash what is left, compare. We re-ran it on the night's plan on 2026-11-10 (UTC), fifteen days after the freeze, and it matched. The mode, 0444, makes the file read-only, which stops an accidental edit and nothing more. Our older plans sealed differently: the ceiling page's prints its seal near its top and carries its two dated amendments appended to its tail. A hash proves a change, not a date. The time on the line is a claim, and it binds only when the fingerprint also lives somewhere you cannot quietly edit: a pushed commit, a message to a friend, a ledger with its own history. Ours go into our repository's history and our decision ledger. We have not used an outside registry; one would add a dated copy held by someone else, which we could not change and a stranger could check without trusting our records. One more thing to know when you open our kits. Some public FROZEN lines read `withheld-`, and the public copy of a sealed file is re-cut for privacy before it ships, so its digest does not reproduce from the file you can download. A fingerprint of a file holding guessable private details, an operator's name or a home directory path, lets a stranger put the guess back, hash and confirm it; in October 2026 we withheld two such seals for that reason. The fingerprint is a promise we make to ourselves and to anyone we hand the sealed file to, not yet one a stranger can check from the kit alone. One of our own pages said otherwise for seven weeks: [The ceiling is not the corpus](https://research.strata2signal.com/the-ceiling-is-not-the-corpus/) told readers they could re-derive its plan's seal from its data kit, while the kit's README said the published copy "does not hash to it". Drafting this guide found the mismatch; the page has carried a dated correction since 2026-10-30. ## Controls: runs whose answer you already know {#controls} A positive control must come back positive, or the instrument is broken; a negative control must come back null, or the instrument is finding things that are not there. Lipsitch and colleagues describe negative controls as a routine precaution of the biology laboratory, carried into epidemiology. [Why the first word comes sooner](https://research.strata2signal.com/flash-next-why-the-first-word-comes-sooner/) uses "control" more narrowly, for an arm with no band and no gate run to answer one question about another arm; here a control is a run whose answer is known before it starts. The grandest positive control we know of is LIGO's blind injection. The gravitational-wave collaborations "train and test their search procedures with many simulated signals that are injected into the detectors, or directly into the data streams", and in 2010 a few members secretly injected zero, one or more signals and sealed the answer in an envelope. A strong signal arrived on 16 September 2010 and was worked up for nearly six months as a candidate detection; the envelope was opened on 14 March 2011, and it was an injection. The procedure had passed its control. Ours are smaller. The anchor above is one. The same night's plan registered three faults its render check (the test that our server turns each request into the same bytes and token ids as the model maker's own encoder does) had to catch before it could pass: a request in thinking mode, the closing think marker swapped for the opening one, and a second beginning-of-sequence token; "PASS needs … all three controls caught". On [Two new frontier models at the rules desk](https://research.strata2signal.com/two-new-frontier-models-at-the-rules-desk/) a known fact was planted at both ends of one prompt of about 30,000 tokens by our own estimate and read before any scored call; the local model, the one that answers our users, missed the one at the front, which told us it had not read the whole prompt and kept twelve cells out of the scored table. We have also written a control that could not fail. A receipt counted a service's activity from the system journal, and its "positive control" turned out to be the journal tool's own "no entries" line, printed when a query matches nothing: the count had read nothing, and the control passed anyway. The fix made the control a line the service itself prints every thirty seconds, and a control reading zero now stops the step. A check you have never seen fail is not yet a check, which is why our test suites plant faults in their own tools and count the catches: 89 of 89 planted mutants killed by one kit's suite in October 2026. ## Amendments: change the plan before the run, never after {#amendments} Plans meet reality and change; the question is when. In medicine the failure is outcome switching: a trial registers one primary outcome, measures several, and reports the one that moved. The COMPare project read 67 trials in five journals as they were published; 58 had discrepancies that needed a correction letter, with a mean of 5.4 undeclared added outcomes per trial. Nosek's paper is clear that changing a plan is not the sin: "Deviations do not necessarily rule out testing predictions effectively", if they are documented, and before the outcomes are seen is the easy case. Our rule, from the night's plan: "After the freeze, any change goes into \[a dated amendments file\] beside this file, stamped in UTC and written before the first scored run it affects. This file is never edited." The plan of [the quantization ladder](https://research.strata2signal.com/the-quantization-ladder/), in that page's data kit, states the other half: an amendment written after a scored call "voids the comparison or gate it touches". Three examples: - **The anchor's lesson, made a gate.** The control reported and the night went on. The same afternoon, at 13:28:27Z on 2026-10-29, an amendment to the next bench's plan added a probe that runs inside the sandbox before any model loads, asks the GPU driver to initialise and count devices, and on a failure marks every step NOT RUN with a line that opens "CUDA sees no device inside the fence". The plan it amends stayed untouched and read-only; the kit was re-sealed under the amendment at 14:37:34Z, before its first scored run. On that bench's first night, 2026-10-30, the probe counted one device and the anchor read whole on the card. (A different check then failed closed and left that night's main steps unscored.) - **A lost night, run again.** The re-run of the 2026-10-29 night is itself an amendment, signed 2026-11-10 at 06:12:32Z. One item is headed "Why a re-run is not a data peek": no reading of the main model exists to steer the choice, and "Every bar, item, nonce, pin, cap, bound and stop is unchanged." If the model then fails its start-up check for the engine's own reasons, "there is no third attempt under this registration". On 2026-11-10 (UTC), when this page was written, the re-run had not run. - **An amendment that needed its own correction.** [Speeds from RAM alone](https://research.strata2signal.com/speeds-from-ram-alone/) ran under a plan frozen 2026-11-01 at 15:09:22Z and "amended twice in writing on 2026-11-02 and 2026-11-03, before the small desktop ran": that desktop had been recorded as another machine. The first amendment fixed its memory's size and wrote down a stick count nobody had read, two of 16 GB; the slots, read on 2026-11-08, hold four of 8 GB. The amendment took a dated correction in the same file, and the page one on 2026-11-09. No figure moved: the plan measured the memory's speed and never worked it out from the sticks. ## When the bet loses {#when-the-bet-loses} A registration is worth something only if it can lose, and ours do. On [Small open deciders, measured](https://research.strata2signal.com/small-open-deciders-measured/#our-fine-tune-that-voided-itself) we registered a fine-tune, T1, with three negative controls: models trained the same way on shuffled labels, with almost nothing true to learn, each of which had to read 25 of 108 or less. They read 18, 29 and 24, so T1 was VOID by its own rule. Its three final models, one trained, one cut at step 127 and one never trained, will not be read, because "reading one would invite a story told after the fact". [Our page on a ZOTAC GeForce RTX 5080 Solid CORE OC (16 GB) against an NVIDIA GeForce RTX 3080 Ti Founders Edition (12 GB)](https://research.strata2signal.com/rtx-5080-vs-rtx-3080-ti/) registered fourteen predictions before its first reading, prints the four that missed or were refuted in their own table, and says of the ten that held that they "were identity checks, one-sided floors … or bands about a fifth or more wide": the easy ones. A predicted memory clock of 15,001 MHz read 14,801 on every run, and is printed as refuted. Two pages of November 2026 show a bet read exactly as written. [Reciprocal rank fusion](https://research.strata2signal.com/reciprocal-rank-fusion/) registered five comparisons before its scored run, "each with the words it would print"; a win needed 10 more questions and a p at or below 0.05 after adjusting for five tests. The first gained 16, 416 against 400 of 470, at an unadjusted p of 0.0139; adjusted it was 0.0693, and "the rule written first reads that as no material difference". All five read so. [Speeds from RAM alone](https://research.strata2signal.com/speeds-from-ram-alone/) registered a rule to hold between 0.20 and 0.28; a laptop read 0.2445 and a desktop 0.6366, "above", "a verdict this page prints rather than explains". The registration can bind the prose too. [The ceiling is not the corpus](https://research.strata2signal.com/the-ceiling-is-not-the-corpus/) wrote its "forbidden sentences" before the numbers existed, "and they bind now that the numbers are unflattering"; on the same page a check whose pass mark was never filed in time is reported as description, because "an unfiled criterion becomes descriptive rather than silently becoming a pass". Our own slips get the same treatment: corrected on the record, never edited away. On 2026-10-29 the cause we first recorded as found for the anchor's reading was wrong, and stood on the record for about twenty minutes before the probe that corrected it. The row that was wrong stands, and the row that corrects it says so; the stick count and the ceiling page's sentence, above, were handled the same way. ## Why a failed rung gets no threshold table {#no-threshold-table} A candidate that fails its gate gets its verdict in the registered words (FAIL, VOID or NOT TESTED) and nothing built to rescue it. The night's plan says it for a model that fails its start-up check: it "reads NOT TESTED, with one line, no table, and no 'what it would have needed'". No lowered bar, no friendlier subset, no table of the margins by which it would have passed. That table is tempting, and it is a postdiction. Kerr's name for a hypothesis written after the result is known is HARKing, and a bar fitted to a reading already seen is one, on our reading of his term; Simmons, Nelson and Simonsohn showed how far undisclosed flexibility of that kind raises the rate of false findings. A losing rung that deserves another try gets its own registration, written fresh: the ruling that closed T1 says the next attempt "is not a patch to T1". The four habits, in one table: | Habit | The failure it prevents | Where it comes from | Our example on this shelf | | --- | --- | --- | --- | | Seal the plan (a FROZEN line) | Moving the pass mark after seeing the data | Trial registration; registered reports | The quantization ladder's plan, frozen 2026-10-14 at 20:52:44 UTC | | Run a known answer (controls) | An instrument that is quietly broken | Laboratory controls; LIGO's blind injections | The anchor that read 0.0 % on card; the fact planted at both ends of a prompt | | Amend before, never after | Outcome switching | COMPare's 58 of 67 | A re-run registered before it runs; an amendment that took its own dated correction | | Print the loss, no rescue table | HARKing; forking paths | Kerr; Gelman and Loken | T1 VOID with its finals unread; the refuted clock on the two-card page; a gain of 16 read as no material difference | ## A pre-registration for your next bench, in five lines {#five-lines} You do not need our tooling. The examples are ours, from plans on this shelf. | Line | Write down | An example from our own plans | | --- | --- | --- | | 1 · Question | One sentence | What does each step down in bits cost in answers, and what does it buy in speed and memory, for one model on one card? | | 2 · Prediction | A number and a band, for each arm | Q8\_0 48 to 60 tokens a second; Q4\_K\_M 78 to 95 | | 3 · Gate | Pass, fail, and what you print on a fail | Pass if each step down the ladder gains more than either level's run-to-run range and every level lands inside its band; on a miss, print "Failed" and the figure that decided it | | 4 · Control | A run whose answer you know, and the reading that means "instrument broken" | `gemma4:26b` on this card wrote 201.82 tokens a second on 2026-10-27; our control only reported a reading far below that; the next plan stops the night if the sandbox cannot see the card | | 5 · Freeze | `sha256sum plan.md` and `date -u`, kept somewhere you cannot quietly edit; changes go in `amendments.md`, dated, before the run they touch | A pushed commit, or a message to a friend | ## What this cannot say {#what-this-cannot-say} - **That registration makes a measurement right.** The control caught the fault and the night still measured nothing; its re-run is registered and, on 2026-11-10 (UTC), had not run. A plan says what the numbers may mean; it cannot make them. - **That our seals are checkable by strangers.** Our registrations live in our own files and history, and we have not used an outside registry; the public copies of sealed files are re-cut for privacy. A reader can redo our arithmetic from the numbers we print; the fingerprints, and the times beside them, rest on our own records. - **That predictions which hold prove much.** Ten of fourteen held on the two-card page, and the page itself says those ten were identity checks, one-sided floors or wide bands. Two of our card benches also registered a self-refutation, a sentence, written in advance, that would have become the headline had a named reading come in; neither fired, which proves the headline no further than the predictions beside it do. - **That exploratory figures are banned.** A reading with no band and no gate is printed with the label "exploratory" and confirms nothing. Labelled, not hidden. ## Who ran this, and thanks {#who-ran-this-and-thanks} Thanks to the authors whose work this guide leans on, named below, and to the Center for Open Science, the LIGO Scientific Collaboration and the Caltech archives for keeping their pages open. Software this page's own examples stood on: Ollama 0.32.13, Python 3, GNU coreutils (`sha256sum`, `sed`, `date`), git and bubblewrap. A small human team set the rules this page describes and ruled on the calls put to it, the re-run among them; a fleet of AI agents drafted the plans, applied the rules case by case, ran the checks, kept the records and drafted this page under that team's rulings. ## Sources {#sources} Outside sources read 2026-10-29 (UTC) unless dated otherwise, every link re-checked 2026-11-10 (UTC); our own pages and records read again on 2026-11-10 (UTC). Quotations are from the open copies linked; two paywalled papers are cited and not quoted. - **Our records:** the pre-registration and amendment files of the benches named, the decision ledger rows that record each day in order, and the pages linked above. The FROZEN line's check was re-run on 2026-11-10. The 2026-10-29 night and its registered re-run have no page of their own yet; what this page says of them is from those files and rows. - **Speeds from RAM alone** (2026-11-07, with dated corrections of 2026-11-09 and 2026-11-10): https://research.strata2signal.com/speeds-from-ram-alone/ - **Reciprocal rank fusion** (2026-11-09), its section on LongMemEval-S and its data kit: https://research.strata2signal.com/reciprocal-rank-fusion/ - Nosek, Ebersole, DeHaven and Mellor, "The preregistration revolution", *PNAS* 115(11), 2018, doi 10.1073/pnas.1708274114 (open copy): https://pmc.ncbi.nlm.nih.gov/articles/PMC5856500/ - Gelman and Loken, "The garden of forking paths", 2013: https://sites.stat.columbia.edu/gelman/research/unpublished/p\_hacking.pdf - Klein and Roodman, "Blind Analysis in Nuclear and Particle Physics", *Annual Review of Nuclear and Particle Science* 55, 2005 (Dunnington's 1932 sentence is quoted as they quote it): https://inspirehep.net/files/123bf8d8ec8f80e60be89076c65dbdbd - LIGO Scientific Collaboration, "'Blind injection' stress-tests LIGO and Virgo's search for gravitational waves": https://www.ligo.org/blind-injection/ - Feynman, "Cargo Cult Science", Caltech commencement address, 1974: https://calteches.library.caltech.edu/51/2/CargoCult.htm - Goldacre and colleagues, "COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time", *Trials* 20:118, 2019, doi 10.1186/s13063-019-3173-2 (CC BY 4.0): https://researchonline.lshtm.ac.uk/id/eprint/4653668/1/COMPare.pdf - Kaplan and Irvin, "Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time", *PLOS ONE*, 2015, doi 10.1371/journal.pone.0132382: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382 - Chambers, "Registered Reports: A new publishing initiative at Cortex", *Cortex* 49(3), 2013, doi 10.1016/j.cortex.2012.12.016 (record): https://pubmed.ncbi.nlm.nih.gov/23347556/ · the format today, Center for Open Science: https://www.cos.io/initiatives/registered-reports - ICMJE, "Clinical Trial Registration": https://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html · the 2004 statement, De Angelis and colleagues, *NEJM* 351:1250-1251, doi 10.1056/NEJMe048225 (cited by record) - Bertinetto, Henriques, Albanie, Paganini and Varol, preface to the *NeurIPS 2020 Preregistration Workshop*, PMLR 148, 2021: http://proceedings.mlr.press/v148/bertinetto21a/bertinetto21a.pdf - NIST, FIPS 180-4, *Secure Hash Standard*, 2015: https://csrc.nist.gov/pubs/fips/180-4/upd1/final - Lipsitch, Tchetgen Tchetgen and Cohen, "Negative controls: a tool for detecting confounding and bias in observational studies", *Epidemiology* 21(3), 2010, doi 10.1097/EDE.0b013e3181d61eeb (abstract read; cited, not quoted) - Simmons, Nelson and Simonsohn, "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant", *Psychological Science* 22(11), 2011, doi 10.1177/0956797611417632 (cited, not quoted) - Kerr, "HARKing: Hypothesizing After the Results are Known", *Personality and Social Psychology Review* 2(3), 1998, doi 10.1207/s15327957pspr0203\_4 (cited, not quoted) *Four habits, one night that needed them, and five lines to take home. Every outside quotation re-read at its source on 2026-10-29 (UTC); every quotation from our own pages and records on 2026-11-10 (UTC).* Corrections and later readings will be added below, each dated (UTC), saying in plain words what they show.