# The Jev bench — three gaps, closed on stored rows and one CPU model *Working file for the 2026-09-22 gap lane. **No GPU, no seat, no network call to any box in the estate.** Everything below is arithmetic over row files that were already written, plus one 22-million-parameter cross-encoder that runs on this laptop's CPU. `inferencebox`, `the public host` and `benchbox` are not touched.* **The law this file is written under.** Each gap is **pre-registered in full before its first number is computed** — what is computed, from which rows, and what would count as a finding — and the pre-registration is committed before the computation runs. Results are appended *under* the pre-registration they belong to, and any place where the result and the pre-registration disagree is named in its own line rather than quietly reconciled. Final tables go to `TABLES-GAPS.md`, drawn by `tables_gaps.py`, which reads only row files and invents nothing. The three gaps are the three the draft article names as open (`~/estate/docs/article/reading-the-answer-v2.md`, "What this page does not know" and "A model of our own"): 1. **The worked example** — what one decision actually looks like, end to end. 2. **The temperature refit** — "What a temperature refit would do … was not done." 3. **The reranker baseline** — "a cross-encoder reranker, which this workshop already runs as a seat, is a model that scores an option without writing a word", never measured against the decision model on the same items. --- ## GAP 1 — THE WORKED EXAMPLE (pre-registration, written 2026-09-22) ### What is computed Two dinner-table decisions, shown whole: the exact bytes that were sent, the six raw letter logprobs that came back, the arithmetic that turned them into probabilities, the letter that won, the true speaker, and what the base chat model did with the identical prompt. ### From which rows * **The decision arm:** `rows-arm3/openjev-fp8-largecard.c.jsonl` — arm 3, `openjev/openjev-FP8` whole on one card, 108 rows, 0 floored, accuracy 93.5 %. * **The control arm:** `rows-addenda/gemma4-fp8-readout.c.jsonl` — arm 10, the estate's own chat checkpoint read the same way on the same runtime, 108 rows, 103 of them with at least one floored letter. * **The prompt:** rebuilt from `kit/task_c.json` + `kit/manifest.json` through `run.py`'s own `readout_prompt()`, `INSTR_C` and `task_c()` — imported, not re-typed — and the rebuilt text's sha256 is **asserted equal** to the row's stored `prompt_sha256`. A mismatch aborts the gap. * **The join:** `kit/task_c.json`'s `id` field is **not unique** (108 lines under 50 ids — README R6). Every join here is **by position** in the file, with `prompt_sha256` asserted equal across the two arms at that position, exactly as `tables_addenda.py` does it. An id is never used as a key. ### The selection rule, fixed now so the example cannot be cherry-picked * **The "right" item:** the **first row by file position** (0-indexed) in `openjev-fp8-largecard.c.jsonl` where `choice == label` and the winning probability `max(probs.values())` lies in **[0.60, 0.90]**. * **The "wrong" item:** the **first row by file position** where `choice != label`. * If either criterion matches no row, that is reported as found and no substitute criterion is invented. ### The calibration, reproduced rather than re-derived `run.py::decide`, the `readout` branch, lines 416–430, verbatim: ```python raw, detail = letter_scores(resp, len(options)) # the six letters at position 0 z = [(FLOOR if v is None else v) / READOUT_T for v in raw] # FLOOR = -30.0, READOUT_T = 0.85 p = softmax(z) ``` with `READOUT_T = 0.85` (`run.py:72`) and `FLOOR = -30.0` (`run.py:78`), the model card's published constants, unchanged. `noul` is **not** applied on task (c) — it is a six-way choice, and `task_c()` passes no `noul=True`. The reproduced probabilities are **asserted equal** to the row's stored `probs` to within 1e-12; a mismatch aborts the gap. ### What would count as a finding * The reproduction is exact (the stored `probs` fall out of the stored `letter_logprobs` and the two published constants) — which is what makes every probability on this page auditable by a reader with a calculator. * The two arms' behaviour on the **same bytes** is visible in one table: whether the base model's letters are present at all, and what a floored letter does to the probability it reports. ### The public-words rule for this section No box name, no card name, no visitor identifier, no path outside the bench kit. The dinner lines are on the long table's **public** wall and are quotable; the kit records the wall's address as `wall_url` in `kit/manifest.json` — `https://longtable.strata2signal.com/api/wall` — and `kit/build_kit.py` composes each line's address out of `(guest, edition, seat, course, line_index)`, which is the citation given for every quoted line. --- ### GAP 1 — RESULT (computed 2026-09-22, after the pre-registration above was committed) **The joins and the assertions, all four of which had to hold before a number was printed.** | assertion | what it checks | outcome | |---|---|---| | row counts | `kit/task_c.json`, `rows-arm3/openjev-fp8-largecard.c.jsonl` and `rows-addenda/gemma4-fp8-readout.c.jsonl` are all 108 | PASS | | prompt rebuild | the prompt rebuilt from the kit through `run.py::readout_prompt` + `INSTR_C` hashes to the row's own `prompt_sha256` — **at all 108 positions, in both arms** | PASS, 0 mismatches | | cross-arm join | the two arms carry the **same** `prompt_sha256` at the same position, so the pair is the same bytes | PASS at all 108 | | calibration | the stored `probs` fall out of the stored `letter_logprobs` through `T = 0.85`, `FLOOR = −30` | PASS — max absolute error over all 216 rows is **0.0** (bit-exact) | **The two items the pre-registered rule selected.** No judgement was applied; these are the first rows by file position that match. | | the one it got right | the one it got wrong | |---|---|---| | row (position, 0-indexed) | **14** of 108 | **7** of 108 | | true speaker | Charles Darwin | Charles Darwin | | decision model chose | Charles Darwin, p = **0.7240** | Ibn Sina, p = **0.8963** | | chat model, same bytes, chose | Carl Sagan, p = **0.8288** (wrong) | Ibn Sina, p = **0.9123** (wrong) | | letters the chat model had at position 0 | 5 of 6 | 5 of 6 | | the chat model's first token | `[` | `[` | | seconds (decision / chat) | 0.085 / 0.109 | 0.086 / 0.116 | **The findings this section was written to expose, and both are visible in the tables.** 1. **The probabilities are reproducible from the rows.** Bit-exact, over 216 rows, from two published constants. A reader does not have to believe the bench; they can recompute it. 2. **"Cannot hallucinate" is narrower than it sounds, and row 7 is the picture of it.** The wrong answer was on the list, well-formed, and given **0.8963**. It is a line about a man too preoccupied with his next meal to contemplate, spoken to Socrates — and both models read it as Ibn Sina. A schema removed the malformed answer, not the mistaken one. 3. **A floored letter is an absent opinion, not a low one.** In both items the chat model never put `F` (Socrates) on the table at all; in the item the decision model got right, the chat model's whole six-way distribution is a choice among the five letters it happened to surface, and it chose wrong with 0.83 of its mass. **Where the result and the pre-registration disagree — one place, named.** The pre-registration says each line is addressed by `(guest, edition, seat, course, line_index)`, quoting `kit/build_kit.py`. The **frozen kit that was actually run** predates that fix: its `source` object carries `edition`, `course` and `line_index` only — `seat` was added to `build_kit.py` as part of the R6 repair, *after* this kit was built. The tables therefore print the three-field tuple the kit really holds and quote the row's **position** beside it as the address that is unique. Nothing numeric moved; the citation format did. --- ## GAP 2 — THE TEMPERATURE REFIT (pre-registration, written 2026-09-22, before any fit) The draft article lists this under what the page does not know: *"What a temperature refit would do. The calibration setting was fitted for OpenJev; every row keeps its raw letter scores, so a refit for any other model costs no card time and was not done."* This gap does it. ### What is computed For each readout arm, on each labelled task, a **single scalar temperature** `T` fitted by minimising multi-class negative log-likelihood on one half of the items, and the calibration measured on the **other** half, before and after. ### From which rows — the six arms and the two tasks, named | arm | what it is | task (a) rows | task (c) rows | |---|---|---|---| | `jev-readout` | arm 1, the community adapter, everyday server | `rows/jev-readout.a.jsonl` | `rows/jev-readout.c.jsonl` | | `base-readout` | arm 1, the chat model, everyday server | `rows/base-readout.a.jsonl` | `rows/base-readout.c.jsonl` | | `openjev-fp8-readout` | arm 2, OpenJev-FP8 across two cards | `rows/openjev-fp8-readout.a.jsonl` | `rows/openjev-fp8-readout.c.jsonl` | | `openjev-bf16-largecard` | arm 3, OpenJev 16-bit, one card | `rows-arm3/openjev-bf16-largecard.a.jsonl` | `rows-arm3/openjev-bf16-largecard.c.jsonl` | | `openjev-fp8-largecard` | arm 3, OpenJev 8-bit, one card | `rows-arm3/openjev-fp8-largecard.a.jsonl` | `rows-arm3/openjev-fp8-largecard.c.jsonl` | | `gemma4-fp8-readout` | arm 10, the chat model on OpenJev's own server | `rows-addenda/gemma4-fp8-readout.a.jsonl` | `rows-addenda/gemma4-fp8-readout.c.jsonl` | Task (b) is excluded: 1 of 42 rows carries a label (§3), and a fit needs labels. Generating arms are excluded: a written letter has no letter scores to refit. Rows marked `refused` are excluded from every count, as everywhere else in this bench; the 8-bit and 16-bit arms and arm 10 each refused the same 10 overlong pages on (a), so their (a) denominator is **53**, not 63, and the tables print it. ### The split `random.Random(20260921)`, **stratified by label**: within each label class, in sorted class order, the class's row positions are shuffled once and the first `floor(n/2)` go to the **fit** half and the rest to the **held** half. The split is a function of `(task, label counts)` only, so it is **the same items for every arm** and the arms are comparable row for row. The exact n per half is printed. ### The fit The objective is the mean multi-class NLL of the true letter over the fit half: ``` z(T) = [(FLOOR if v is None else v) / T for v in raw] # FLOOR = -30.0 p(T) = softmax(z(T)) NLL = mean over fit rows of -log p(T)[label] ``` minimised over `T` by golden-section search on `log T` in **[0.05, 20.0]**, confirmed against a 400-point log-grid over the same interval. **A fit that lands within 1 % of either bound is declared DEGENERATE and reported as "no temperature identified", with the bound named** — it is not dressed up as a fitted value. **This is pre-registered as the expected outcome on task (a), and it is a property of the data, not a failure of the method.** Every arm reads task (a) at **100 % accuracy**. On a set the model never gets wrong, NLL falls monotonically as `T → 0`: the likeliest calibration is infinite confidence. So (a) is expected to degenerate for every arm, and that is itself the finding — *a saturated task cannot calibrate anything*. ### What it is compared against * **On task (c)**, the published `READOUT_T = 0.85` (`run.py:72`), unchanged. * **On task (a)**, NOT 0.85. Task (a) is a `noul` decision: `run.py::decide` applies `softmax(raw / 0.85)` and *then* pushes `logit(p_yes)` through `NOUL_T = 1.829074` with `NOUL_BIAS = 0`. For a two-option softmax those compose exactly — `logit(p_yes) = (z₀ − z₁)/T`, so dividing by `NOUL_T` afterwards is the same as having used `T × NOUL_T` in the first place. **The published effective temperature on task (a) is therefore 0.85 × 1.829074 = 1.554713**, and that is the number the refit is placed beside. This equivalence is **asserted numerically** against the stored `probs` (max error < 1e-9) before it is quoted; the assertion accounts for the helper's `1e-4` clip on `p_yes`. ### The floor, treated explicitly — two variants, and a third column so they can be read together * **Variant A — the floor as it ran.** A floored letter enters the fit at −30, exactly as the protocol produced it. Every labelled row is used. * **Variant B — floored letters excluded.** A floored letter is dropped from the option set and the softmax renormalised over the letters that actually surfaced. A row whose **true letter is the floored one** has no likelihood to contribute and is dropped from that variant's fit *and* its evaluation; the dropped count is printed. A row where every letter floored is likewise dropped. * **Variant A|B — variant A, restricted to variant B's evaluation subset**, so the two are read on one denominator instead of two. Brier under variant B is a sum over a *reduced* option set and is therefore not the same quantity as the full-support Brier; the tables say so in place rather than in a footnote. **Which is honest — the criterion, fixed now.** The two variants answer two different questions and the report will say which is which: **A** is what a refit would actually buy an arm *as it ran*, because on the untargeted path the floor is part of the instrument and the caller cannot know in advance which letters will surface; **B** is the model's opinion among the letters it happened to put on the table, which is a diagnostic and not something that can be deployed. The report will name A as the deployable number and B as the diagnostic, and will say plainly if the two point in different directions. ### What is reported, per arm × task × variant Held half only: **n**, accuracy before and after (**asserted equal** — a positive scalar temperature cannot move an argmax; the assertion is a real check, not a comment, and the argmax stability of variant B is asserted too), **mean NLL**, **multi-class Brier** (`run.py::brier`'s form), and **ECE** — ten equal-width bins on the winning probability, `ECE = Σ (n_b/N)·|mean p_b − accuracy_b|`, the same binning `run.py::reliability` draws — each before and after, plus the fitted `T`. ### What would count as a finding The question is the article's: **does a refit close the calibration gap between the chat model and OpenJev, or is the chat model's problem the missing letters (README R4) that no temperature can fix?** The rule is fixed here: * **CLOSED** if, on task (c) variant A, the refit brings `gemma4-fp8-readout`'s held-half **ECE within 0.02** and **Brier within 0.02** of `openjev-fp8-largecard`'s own refitted values. * **NOT CLOSED** otherwise — and in that case the report states how much of the residual is accuracy (13 points, which no temperature touches) and how much is the floor, by reading variant A against variant A|B and B. A secondary finding, reported either way: **how far the fitted `T` sits from the published 0.85**, per arm, and whether the two OpenJev checkpoints (8-bit and 16-bit) agree on it — two runs of the same weights at different precisions disagreeing about their own best temperature would be worth knowing. ### GAP 2 — RESULT (computed 2026-09-22, after the pre-registration above was committed) Full tables in `TABLES-GAPS.md`. The headline, on task (c), variant A (the floor as it ran), held half of 54 items, published `READOUT_T = 0.85`: | arm | floored rows | fitted T | held accuracy | ECE before → after | Brier before → after | |---|---|---|---|---|---| | `jev-readout` | 0 of 108 | **0.711** | 88.9 % | 0.1459 → **0.1035** | 0.2458 → 0.2316 | | `base-readout` | 101 of 108 | **5.301** | 81.5 % | 0.1769 → **0.1016** | 0.3457 → 0.2484 | | `openjev-fp8-readout` | 0 of 108 | **1.152** | 94.4 % | 0.0451 → **0.0316** | 0.1144 → 0.1166 | | `openjev-bf16-largecard` | 0 of 108 | **1.168** | 94.4 % | 0.0466 → **0.0393** | 0.1132 → 0.1152 | | `openjev-fp8-largecard` | 0 of 108 | **1.145** | 94.4 % | 0.0622 → **0.0459** | 0.1211 → 0.1228 | | `gemma4-fp8-readout` | 103 of 108 | **5.433** | 85.2 % | 0.1408 → **0.0851** | 0.2754 → 0.2375 | Accuracy was **asserted** unchanged by the refit in all 24 fits (12 arm×task under variant A, 12 under variant B); the assertion is a real check in `tables_gaps.py`, and it passed in every one. **The pre-registered question, answered by the pre-registered rule.** ECE distance after refit 0.0392, Brier distance 0.1147, both outside 0.02 → **NOT CLOSED**. **What the numbers say beyond the verdict.** 1. **The refit is worth a lot and is not the answer.** It removes 40 % of the chat model's calibration error on task (c) for no card time at all, and drops its mean NLL from 1.2989 to 0.5097 — that collapse is the floor being undone. 2. **The residual is accuracy, not calibration.** After the refit the chat model's Brier is 0.2375 against 0.1228. The two arms are 9.3 points apart on the held half (51 right against 46 of 54), and that accuracy difference alone is worth 0.1626 on the Brier scale — *larger than the entire 0.1147 residual*. The refitted chat model is already beating the rank-blind anchor for its own accuracy. **So: a temperature fixes the chat model's probabilities and cannot fix its answers, and R4's missing letters are on the answers' side of that line.** 3. **The chat model needs a temperature near 5.4 and the decision model needs 1.15.** That ratio is the floor, quantified: four of six letters at −30 make a softmax that claims near-certainty, and only a large flattening temperature spreads it back out. 4. **The two OpenJev checkpoints agree about their own temperature** — 1.168 at 16-bit and 1.145 at 8-bit, 2.0 % apart on the same card and the same items. The quantisation does not move the calibration. 5. **The published 0.85 is not the best temperature for any arm measured here.** Five of six want to be flatter; the one exception is `jev-readout` at 0.711, which wants to be sharper. 6. **Task (a) is degenerate for every arm and that is the finding.** All six arms read it at 100 %, so the likelihood wants infinite confidence and all twelve (a) fits run out of mantissa before they run out of interval. **A saturated task cannot calibrate anything.** **Which floor treatment is honest — the ruling, against the criterion fixed above.** **Variant A** is the number to publish: on the untargeted path the floor *is* the instrument, a caller cannot know in advance which letters will surface, and A is what a refit would actually buy an arm as it ran. **Variant B** is a diagnostic and is reported as one — it says the chat model's opinion *among the letters it did surface* is better calibrated than variant A makes it look (ECE 0.1077 → 0.0803 at T = 3.011 on the 52 held rows B can keep), which is a fact about the model and not a number anybody can deploy. The two do **not** point in different directions on the verdict: under B the chat model's ECE is still far from the decision model's, and its accuracy is untouched. **Where the result and the pre-registration disagree — three places, named.** 1. **The degeneracy test was wrong as written and is corrected in the code.** The pre-registration declares a fit degenerate if it lands within 1 % of a bound. That test does not fire here: on a task an arm never gets wrong the NLL falls monotonically toward 0 and reaches *floating-point* zero well inside the interval, so the search halts at an interior point that looks like an optimum and is only where the mantissa ran out — the first run of this gap duly printed six confident-looking task-(a) temperatures between 0.065 and 0.271. The test now used is **whether either bound is as good as the claimed optimum** (NLL within 1e-9); it fires on all twelve (a) fits and on none of the twelve (c) fits. The *prediction* the pre-registration made — that (a) would degenerate for every arm — is confirmed; the *detector* it specified would have missed it. 2. **The split is not "the same items for every arm" on task (a).** It is on task (c), where all six arms scored all 108. On (a) the four arms served on a 16,384-token window refused the same ten overlong rows (six distinct pages), so they score 53 where arm 1's two arms score 63, and a stratified half of 53 is not a subset of a stratified half of 63. The tables print both denominators and say that a cross-group comparison on (a) is two sets, not one; the code asserts that arms sharing a label sequence share a split. 3. **One column was added that the pre-registration did not name:** the *rank-blind Brier at this accuracy*, an anchor that puts an accuracy difference on the Brier scale. It was added because the pre-registration promised to state "how much of the residual is accuracy" and offered no instrument for doing it. It is named an anchor and not a floor, because the refitted chat model beats its own. --- ## GAP 3 — THE RERANKER BASELINE (pre-registration, written 2026-09-22, before the model was loaded) The draft article says, of the small encoder trained in ten seconds: *"There is an older name for this. A classifier is a decision model, and a cross-encoder reranker, which this workshop already runs as a seat, is a model that scores an option without writing a word."* It never puts one on the same items. This gap does. ### What is computed The seat's own reranker — **`cross-encoder/ms-marco-MiniLM-L6-v2`**, 22 M parameters, Apache 2.0, six transformer layers, trained in 2020 to score a (query, passage) pair for passage ranking — asked the bench's task (a) by the only means it has: score the question against the page, and threshold the score. **It runs on this laptop's CPU, from the copy already in `~/.cache/huggingface`.** No GPU, no seat, no box. The wall seconds and the thread count are reported. ### From which rows, and the casting * **The items:** `kit/task_a.json` — 63 questions, 42 `on-page` (label yes) and 21 `off-page` (label no). The same frozen file every arm was asked from. * **The page:** `kit/articles.json`, field `state` — byte for byte the span the deployed docent seat reads, which is what every arm was given. * **The casting.** A cross-encoder takes a short query and a short passage; a page here runs to 82,500 tokens and the model's window is 512. So the page is **split into the sections the kit already names**: the `state` is cut on level-2 ATX headings (`## …`), each section keeping its heading line, with the text before the first heading kept as its own chunk. Any section longer than **180 words** is further cut into 180-word windows with **40 words of overlap**, so no text is silently truncated by the tokenizer. The item's score is the **maximum** over the page's chunks — the standard retrieval reading of "does this page contain the answer", and the one that gives the reranker its best case. * The chunk count per page and in total is reported. Scoring is deterministic (evaluation mode, no sampling); no seed is needed and none is claimed. ### The split and the threshold The **same split rule as gap 2** — `random.Random(20260921)`, stratified 50/50 by label, fit half and held half — so the reranker is thresholded on exactly the items the temperature refit was fitted on. The threshold is chosen on the **fit** half as the value maximising accuracy there: candidate thresholds are the midpoints between adjacent sorted fit-half scores, and where several tie for best accuracy the **median** of the tied candidates is taken. That threshold, and only that threshold, is applied to the **held** half. ### What is reported * **AUC** — the area under the ROC curve, by the Mann-Whitney rank formula with midranks for ties — over all 63 items, and over the held half alone. * **Accuracy at the fitted threshold, on the held half**, beside the fit half's own. * The same two, restricted to the **53 items the 16,384-token arms reached**, so a reader can compare against OpenJev on the denominator OpenJev actually scored. * The **always-yes floor of 66.7 %**, which is the only floor this task has. * The comparators, quoted from `TABLES.md` and `TABLES-ADDENDA.md`, never re-run: `jev-readout` and `base-readout` at **100.0 %** on 63; `openjev-fp8-readout`, `openjev-bf16-largecard`, `openjev-fp8-largecard` and `gemma4-fp8-readout` at **100.0 %** on 53. ### Task (c) — pre-registered as SKIPPED, with the reason A six-way "which guest said this?" would have to be cast as the line scored against six **voice cards**. **The frozen kit has no such text.** `kit/manifest.json`'s `c_roster` carries a guest as an `id` and a display name and nothing else, and the six options as every arm actually saw them were literally `[A] darwin: Charles Darwin`. Scoring a line against the string "Charles Darwin" with a passage-ranking cross-encoder is not a casting of the task; it is a measurement of nothing. Writing voice cards for the six guests would be inventing an instrument mid-bench and handing the baseline text no arm ever saw. **So task (c) is skipped and this paragraph is the report.** ### What would count as a finding, and what the comparison licenses * **MATCHES** if the reranker's held-half accuracy is **100.0 %**, the comparators' own figure; **CLOSE** if ≥ 95 %; **DOES NOT MATCH** otherwise. If it does not match, the AUC says which half failed: a high AUC with a poor accuracy is a *threshold* problem, a low AUC is a *ranking* problem, and the two have different consequences for anyone thinking of putting a reranker on this gate. * **What this licenses, stated before the number exists.** The reranker is an older kind of decision model and the comparison is deliberately unfair in both directions. Unfair to it: it was trained for passage ranking, not for a plain-language yes/no question; it has no notion of a question being *unanswerable*, which is exactly what the 21 `off-page` items are; and it needs a threshold **fitted on labels**, where every model arm needed none. Unfair to the model arms: the reranker is 22 M parameters on a CPU against 26–27 B on a card. So a reranker win would say **this gate is a retrieval-similarity task in disguise** and should be served by a 22 M-parameter model; a reranker loss says only that this particular old instrument does not fit this particular question — not that small models cannot decide. * **The denominator is small and the report says so in place**: 63 items, a held half of 32, where **one flip is 3.1 points**. ### GAP 3 — RESULT (computed 2026-09-22, after the pre-registration above was committed) `cross-encoder/ms-marco-MiniLM-L6-v2`, on this laptop's CPU, from the copy already in the local model cache. 42 pages cut into **2,363 chunks** (8 smallest page, 37 median, 290 largest); 63 items scored in **42.2 s**, median 0.41 s per item, 24 CPU threads. Rows in `rows-gaps/rerank.a.jsonl`, written by `rerank_baseline.py`. | | the reranker | the model arms | |---|---|---| | ROC AUC, all 63 | **0.982** | — (a readout right every time has no ROC) | | ROC AUC, fit half (31) | 0.952 | — | | ROC AUC, held half (32) | **1.000** | — | | accuracy, fit half at its own best threshold | 93.5 % | — | | **accuracy, held half, threshold carried over** | **90.6 %** | **100.0 %**, all six arms | | always-yes floor | 66.7 % | 66.7 % | | held half at its OWN best threshold (an upper bound, not a result) | 100.0 % | — | Fitted threshold **−5.565**, chosen on the fit half and the only number carried across. On the 53 items the short-window arms reached: AUC 0.979, accuracy 92.5 % at the same threshold. **Verdict against the pre-registered rule: DOES NOT MATCH** (90.6 % < 95 %). **The finding, and it is sharper than the verdict.** With an AUC of 0.982 overall and **1.000 on the held half**, the reranker's problem is not ranking — on the held half it puts *every* answerable question above *every* unanswerable one — it is **thresholding**. It still gets 3 of 32 wrong, because the cut came from the other half and did not transfer. **A model that ranks perfectly and decides at 90.6 % is the difference between a score and a decision**, and that difference is precisely what a decision model is sold as removing: it is asked the question and answers it, with no threshold to carry and none to fit. The reranker also cannot be asked at all without 31 labelled rows to cut with; every model arm needed none. **What the comparison licenses, as pre-registered.** The only strong reading available was a reranker *win* — which would have said this gate is a retrieval-similarity task in disguise, servable by 22 M parameters on a CPU for nothing. That reading is not available. The loss says only that this old instrument does not fit this question, not that small models cannot decide — and the bench's own small-encoder arm says the same thing from the other side: the ceiling is made of labels. **Task (c): SKIPPED, as pre-registered, and the reason is unchanged.** The frozen kit carries a guest as an `id` and a display name; there is no voice-card text to score a line against, and writing some would be inventing an instrument mid-bench. **Where the result and the pre-registration disagree — nothing material, one addition.** The pre-registration did not name the *held half at its own best threshold* row. It was added because the held-half AUC came back at 1.000 and the ranking/thresholding split the pre-registration promised to diagnose could not be stated without it. It is labelled in place as an upper bound and never as a result. --- ## The three gaps, in one table | gap | the question the article left open | the answer | where it came from | |---|---|---|---| | 1 | what does one decision actually look like? | two rows shown whole; the published probabilities reproduce **bit-exactly** from the stored letter scores | `rows-arm3/openjev-fp8-largecard.c.jsonl`, `rows-addenda/gemma4-fp8-readout.c.jsonl`, `kit/task_c.json` | | 2 | what would a temperature refit do? | it removes **40 %** of the chat model's calibration error on task (c) for no card time, and the gap stays open: the residual is accuracy, not calibration — **NOT CLOSED** | the `letter_logprobs` field of 6 readout arms × 2 tasks | | 3 | how does the workshop's own cross-encoder do on the same items? | ranks the held half **perfectly** (AUC 1.000) and decides at **90.6 %** against 100 % — **DOES NOT MATCH**, and it is a threshold problem | `kit/task_a.json` + `kit/articles.json` → `rows-gaps/rerank.a.jsonl` | **One sentence that ties them.** The bench's three open questions all resolve the same way: the readout's *probabilities* are cheap to fix — reproducible from stored rows, refittable for free, and worth 40 % of the chat model's calibration error — and the readout's *answers* are not, which is why the thing being sold is a model trained to put the decision at the first position rather than a temperature, a threshold, or a prompt. --- ## Correction, 2026-09-22 — GAP 1 said "the same server" and that was wrong The first draw of `TABLES-GAPS.md` introduced GAP 1's two decisions as *"put to two models on the same server"* and headed the chat-model column *"(same server)"*. **They did not run on the same server.** Read from the rows themselves: | | `openjev-fp8-largecard` (the decision model) | `gemma4-fp8-readout` (the chat model) | |---|---|---| | `card_uuid` | `GPU-25bc3288…` | `GPU-46890836…` | | `power_cap_w` | 420 | 250 | | posture | one large card, whole | two consumer cards, tensor-parallel | What the two rows **do** share is the thing the section needs: the same prompt bytes, asserted by sha256 at every one of the 108 positions. So the section now says so, names the two postures, marks the seconds row as not comparable and does not compare it, and quotes the **like-for-like control** under each table — the same decision model run on the chat model's own server and cards (`rows/openjev-fp8-readout.c.jsonl`, arm 2), which chose the same guest on both example rows and scores **93.5 %** across all 108, identical to the one-card arm. `tables_gaps.py` now **asserts** that parity, so if a future re-run breaks it the control sentence fails loudly instead of quietly becoming false. This correction is recorded rather than edited away because the claim was in a committed file for the length of one commit, and because the mistake is the exact one this bench keeps catching: a comparison that changes two things at once.