# Decision models vs the generating seat — the Jev bench, benchbox, 2026-09-21 *The pre-registration below was written and committed BEFORE any row was measured (`git log` is the receipt). Results are appended under it as they land; nothing above the results line was edited after the first call.* **Pre-registration stamped 2026-09-21T16:51:34Z** · box **benchbox** · card 0 `GPU-46890836-d8f7-e868-1977-c1ce28db0d7a` (PCI 04:00.0, x16, cap 250 W) · card 1 `GPU-7aa0be10-974f-6430-52fe-09018a0e2e08` stays dark and is checked for it. --- ## 1. The question Does a **decision model** — Jev-style: typed answers read off the scores of option letters at the first output position, no generation — match the **generating seat** on decisions the estate already makes, at a fraction of the latency and energy? Arm 1, today: the community adapter `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` against **its own base**, `gemma4:26b` — the same 26B-A4B mixture, the same quantisation class, the same card, one at a time, residency proved each time. ## 2. The four arms | arm | model | how it is asked | what it isolates | |---|---|---|---| | `jev-readout` | jevify-gemma4-26b-a4b Q4_K_M | letters read at position 0 | the thing under test | | `base-readout` | gemma4:26b Q4_K_M | the same readout | **the readout control** — how much is the adapter and how much is the protocol | | `jev-generate` | jevify-gemma4-26b-a4b Q4_K_M | the same prompt, written out, temperature 0 | the adapter without the readout | | `base-generate` | gemma4:26b Q4_K_M | the same prompt, written out, temperature 0 | **the seat's normal way** — today's baseline | The four share one prompt builder, one option order, one seed (20260921) and one kit. An arm differs from its neighbour in the model string or in the readout, never in both at once. ## 3. The three task sets All three are frozen in `kit/` before the first call, with a sha256 per state, and every row carries the sha of the prompt it was asked. **(a) The docent bank as a typed decision — 63 items, 42 yes / 21 no.** Each of the docent's 63 bank questions, recast as one `noul` (yes/no) decision: *does the article answer this question?* The label is the bank's own kind — `on-page` → yes, `off-page` → no — so the set is the docent's abstain contract with the prose removed. The **state is what the docent gives its seat**: `prompt.fenced_article(article.prompt_body)`, the fenced twin body with the synthetic short-version section at its foot, byte for byte, loaded through the docent's own `corpus.load`. **(b) The docent's cite decision — 42 items, choice over that page's own section ids.** *Which section contains the answer?* over the page's cite enum (its `anchor_ids`, 2–25 options). **This task is UNLABELLED and is declared so before it runs**: the bank carries an expected section for exactly **1** of its 63 questions. It is measured for cost, for stability, and for agreement between arms — never for accuracy, and it is **not under the gate**. A scored (b) needs 42 hand labels from the content owner; that is named as the gap, not filled by a model's own answer (un-judged model text cannot grade a model — the docent bench's own ruling). **(c) Six voices at the long table — 108 items, 6-way choice, balanced 18 per guest.** Built from the long table's public wall JSON (`https://longtable.strata2signal.com/api/wall`, sha in `kit/manifest.json`): *which of the six guests said this line?* Chance is 16.7 %. Held out of every prompt: **no speaker label ever enters a state**, the line stands alone with the six names as options, and any line whose own text names its own speaker is **dropped before sampling** (`NAME_TOKENS` in `kit/build_kit.py`) so the label cannot leak. Lines under 80 characters and withheld lines are dropped. The sample is `random.Random(20260921)`, 18 per guest (the floor guest's count), option order fixed alphabetically for every item and every arm. ## 4. The readout, exactly OpenJev's, from its model card and `helper/shim.py`: assign a letter per option, ask the question, read the scores of **exactly those letters** at the first output position, softmax. The prompt is the helper's text-lane layout verbatim: ``` State: Question: Options: [A] : ... Answer with the letter of the best option only. ``` Calibration is the card's published constants, **unchanged**: `READOUT_T = 0.85`, and for yes/no `READOUT_NOUL_T = 1.829074`, `READOUT_NOUL_BIAS = 0` applied to the logit of p(yes) after the softmax, which is the order the helper applies them in. **Three things this bench states rather than hides.** 1. **It is on the OLD (untargeted) path.** `READOUT_TARGETED=1` needs vLLM's `allowed_token_ids` / `logprob_token_ids`; ollama has neither, so scores come from the raw top-20 and a letter outside it takes the helper's −30 floor. Floored rows are counted per arm. 2. **The calibration was fitted for a different model on the other path.** Brier and the reliability table are therefore *measurements of these constants on this model*, not a claim that the model is well calibrated. **Every row keeps its raw per-letter logprobs**, so any temperature can be refitted afterwards without spending a GPU second. 3. **The first output position is not the first token of the reply.** This model ships the Gemma 4 canonical template, whose assistant turn opens on a thinking channel: untouched, the position-0 token is `<|channel>` and every option letter sits ~13–18 nats below it (measured 16:45Z, receipts in `receipts/`). The template's own `enable_thinking=false` writes `<|channel>thought\n` into the generation prompt and lands position 0 on the answer — this is ollama's `think: false` and it is what the OpenJev helper sends vLLM as `chat_template_kwargs={"enable_thinking": false}`. **Every arm sets it**, the generating controls included, so readout and generation differ in the readout and in nothing else. ## 5. The gate, pre-registered > The adapter must reach **≥ the base readout's accuracy − 2 points** on **(a) AND (c)**. > If it does not, the rung is **REJECTED** and gets **no threshold table**. > Latency and energy are reported either way. Task (b) is not under the gate (§3). ## 6. What is measured, per arm × task Accuracy against labels · multi-class Brier · a 10-bin reliability table (mean confidence vs accuracy, with the count in each bin) · median and p95 seconds per decision · prompt and eval tokens · **energy as the board's own watts × seconds**, integrated by trapezoid from a 1 Hz `nvidia-smi` sampler pinned to the card's UUID, reported both raw and net of the idle draw read for ten seconds immediately before the arm. ## 7. The standing laws this run is built on * **D-20260921-001 — one model whole on one card; residency proved by memory growth on the NAMED card, never by `/api/ps`.** `verify_pin` refuses in both directions: a named card that did not grow, and any other card that did. Measured before the first call: card 0 1 MiB → 18,020 MiB while card 1 held at 1 MiB. * **D-20260920-133 — ollama enumerates the other board over Vulkan.** The bench unit sets `CUDA_VISIBLE_DEVICES=`, `OLLAMA_VULKAN=0`, `GGML_VK_VISIBLE_DEVICES=-1`; its discovery log names exactly one device, `filter_id=GPU-46890836…`, `library=CUDA`, and no Vulkan device. * Seeds fixed (20260921), concurrency 1, every row a receipt: model tag, quant, card UUID, 250 W cap, context, timestamp, prompt sha256. ## 8. One deviation from the brief, named The bench unit was started with `OLLAMA_CONTEXT_LENGTH=8192`. **The docent's own bank does not fit in it**: the states run from 6.4 KB to 241 KB, median ≈ 7.9 k tokens, with two pages at ≈ 62 k and ≈ 67 k. Truncating would change the task and could flip an on-page label to a genuinely unanswerable one. So every call sets `options.num_ctx = 81920` on `/api/chat` (which is also why this bench uses `/api/chat` and not `/v1/chat/completions` — the OpenAI endpoint takes no `num_ctx`; both return identical logprobs). **Measured, not assumed**: at num_ctx 81920 the model sits at 19,306 MiB on card 0, whole, with card 1 at 1 MiB — the law holds. ## 9. Disk, and what arm 2 needs benchbox is at 97 % (≈ 10 GB free after the adapter). The two models are ≈ 17.6 GB each, so **they cannot both be resident on disk**; arm 1 runs the adapter's four task-arms first, then the adapter is removed and the base pulled. The re-pull is ≈ 16 GB at the measured ≈ 115 MB/s (≈ 2.5 min), which is cheaper than asking for space. **Arm 2 — `openjev/openjev-FP8` on both cards under vLLM tensor-parallel — is NOT started.** What it needs, measured on the box today: **disk ≈ 29 GB for the FP8 checkpoint** (benchbox has ≈ 10 GB — short by ≈ 19 GB before any HF cache); **vLLM is absent** (`command -v vllm` finds nothing; `~/ComfyUI/venv` and `~/lab/venv` are the only venvs and neither serves); and **the two 3090s have no NVLink**, so tensor-parallel would run over PCIe with one card on **x4** (2B:00.0) — an all-reduce on the slow link, which must be measured and not assumed. --- ## RESULTS *(appended as arms land; nothing above this line is edited)* **What was measured, and when.** Four arms — the community adapter `jevify-gemma4-26b-a4b` Q4_K_M and its own base `gemma4:26b` Q4_K_M, each asked twice: once by reading option-letter scores at the first output position (the OpenJev readout) and once by letting it write the letter — over three task sets: 63 typed yes/no decisions on the docent's own frozen bank, 42 cite decisions on the same pages, and 108 balanced six-way voice decisions from the long table's public wall. **852 decisions in all, between 2026-09-21T17:00:54Z and 2026-09-21T17:58:17Z**, sequentially on one RTX 3090 capped at 250 W on benchbox. There is no prior reading to compare against: this is the first run of this instrument. Total energy drawn by that card over the 57 minutes: **618.5 kJ, 0.172 kWh**. ## The verdict, in one paragraph **The gate passes and the promise does not.** The adapter matched or beat its own base on both gated tasks — a tie at 100 % on the docent bank and **+2.8 points** on the long table — so by the rule written before the first call the rung is **ACCEPTED**. But the reason to want a decision model was *"a fraction of the latency and energy"*, and on this substrate that is **not true**: with the cache-warmed rows removed, reading the answer instead of writing it is worth between **0.1 % and 2.8 %** of the time per decision. The cost of a decision here is prefill plus about 870 ms of server overhead; the tokens the seat does or does not write are noise beside both. What the adapter *does* buy is real and is not speed — see §R4. ## The gate | gated task | base readout | adapter readout | delta (points) | floor (base − 2) | verdict | |---|---|---|---|---|---| | (a) does the article answer this question? | 100.0 % | 100.0 % | +0.0 | 98.0 % | **PASS** | | (c) which of the six guests said this line? | 80.6 % | 83.3 % | +2.8 | 78.6 % | **PASS** | Verdict: **ACCEPTED** ## Accuracy | task | arm | accuracy | correct / labelled | Brier | unparsed | floored rows | |---|---|---|---|---|---|---| | (a) does the article answer this question? | `jev-readout` | 100.0 % | 63 / 63 | 0.0038 | 0 | 5 | | | `base-readout` | 100.0 % | 63 / 63 | 0.0001 | 0 | 19 | | | `jev-generate` | 100.0 % | 63 / 63 | 0.0001 | 0 | 0 | | | `base-generate` | 100.0 % | 63 / 63 | 0.0001 | 0 | 0 | | (b) which section is the answer in? | `jev-readout` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 0.9461 | 0 | 7 | | | `base-readout` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 39 | | | `jev-generate` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 0 | | | `base-generate` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 0 | | (c) which of the six guests said this line? | `jev-readout` | 83.3 % | 90 / 108 | 0.2875 | 0 | 0 | | | `base-readout` | 80.6 % | 87 / 108 | 0.3635 | 0 | 101 | | | `jev-generate` | 83.3 % | 90 / 108 | 0.3333 | 0 | 0 | | | `base-generate` | 82.4 % | 89 / 108 | 0.3596 | 1 | 0 | _Brier here is the MULTI-CLASS form — the squared error summed over every option — so on a two-option task it is twice the familiar binary Brier and must not be read against a published binary number. Lower is better; 0 is perfect. A generating arm has no probabilities, so its Brier is just its error rate doubled and its reliability table is one bin at 1.0: that is a property of writing a letter instead of scoring one, and it is the point._ ## Task (a), split — the abstain half is the half that matters | arm | on-page answered yes (of 42) | off-page answered no (of 21) | accuracy | always-yes would score | |---|---|---|---|---| | `jev-readout` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % | | `base-readout` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % | | `jev-generate` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % | | `base-generate` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % | ## Latency, tokens and energy | task | arm | median s | p95 s | prompt tokens | written tokens | mean W | J / decision | J / decision net of idle | idle W | |---|---|---|---|---|---|---|---|---|---| | (a) does the article answer this question? | `jev-readout` | 3.527 | 12.182 | 825,919 | 63 | 214.4 | 1181.3 | 464.8 | 130.1 | | | `base-readout` | 3.746 | 12.943 | 825,919 | 63 | 211.1 | 1240.1 | 490.0 | 127.7 | | | `jev-generate` | 3.621 | 28.089 | 825,919 | 232 | 218.4 | 1362.7 | 657.4 | 113.0 | | | `base-generate` | 3.841 | 29.896 | 825,919 | 128 | 215.4 | 1429.8 | 646.6 | 118.0 | | (b) which section is the answer in? | `jev-readout` | 3.766 | 12.210 | 538,426 | 42 | 217.1 | 1298.0 | 584.4 | 119.4 | | | `base-readout` | 4.010 | 12.969 | 538,426 | 42 | 214.4 | 1363.5 | 606.6 | 119.0 | | | `jev-generate` | 3.848 | 12.214 | 538,426 | 84 | 217.5 | 1305.0 | 567.4 | 122.9 | | | `base-generate` | 4.024 | 13.052 | 538,426 | 90 | 214.4 | 1368.6 | 601.6 | 120.2 | | (c) which of the six guests said this line? | `jev-readout` | 1.017 | 1.093 | 30,905 | 108 | 148.0 | 150.7 | 25.8 | 122.6 | | | `base-readout` | 1.025 | 1.116 | 30,905 | 108 | 146.9 | 151.0 | 28.2 | 119.4 | | | `jev-generate` | 1.031 | 1.090 | 30,905 | 216 | 146.8 | 150.9 | 27.1 | 120.4 | | | `base-generate` | 1.060 | 1.116 | 30,905 | 398 | 149.9 | 158.3 | 30.6 | 120.9 | ## Agreement between arms | task | pair | agree | of | rate | |---|---|---|---|---| | (a) does the article answer this question? | `jev-readout` vs `base-readout` | 63 | 63 | 100.0 % | | | `jev-readout` vs `jev-generate` | 63 | 63 | 100.0 % | | | `jev-readout` vs `base-generate` | 63 | 63 | 100.0 % | | | `base-readout` vs `jev-generate` | 63 | 63 | 100.0 % | | | `base-readout` vs `base-generate` | 63 | 63 | 100.0 % | | | `jev-generate` vs `base-generate` | 63 | 63 | 100.0 % | | (b) which section is the answer in? | `jev-readout` vs `base-readout` | 31 | 42 | 73.8 % | | | `jev-readout` vs `jev-generate` | 42 | 42 | 100.0 % | | | `jev-readout` vs `base-generate` | 31 | 42 | 73.8 % | | | `base-readout` vs `jev-generate` | 31 | 42 | 73.8 % | | | `base-readout` vs `base-generate` | 41 | 42 | 97.6 % | | | `jev-generate` vs `base-generate` | 31 | 42 | 73.8 % | | (c) which of the six guests said this line? | `jev-readout` vs `base-readout` | 89 | 108 | 82.4 % | | | `jev-readout` vs `jev-generate` | 108 | 108 | 100.0 % | | | `jev-readout` vs `base-generate` | 95 | 108 | 88.0 % | | | `base-readout` vs `jev-generate` | 89 | 108 | 82.4 % | | | `base-readout` vs `base-generate` | 102 | 108 | 94.4 % | | | `jev-generate` vs `base-generate` | 95 | 108 | 88.0 % | ## Task (c): the floor an arm has to beat 103 of 108 lines name at least one other guest; crossing those off and guessing among the rest is worth **24.0 %**, against a six-way chance of 16.7 % ### Reliability — `jev-readout`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.8–0.9 | 4 | 0.851 | 100.0 % | -0.149 | | 0.9–1.0 | 59 | 0.985 | 100.0 % | -0.015 | ### Reliability — `jev-readout`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.2–0.3 | 3 | 0.289 | 33.3 % | -0.044 | | 0.3–0.4 | 11 | 0.357 | 45.5 % | -0.097 | | 0.4–0.5 | 11 | 0.458 | 81.8 % | -0.360 | | 0.5–0.6 | 12 | 0.541 | 75.0 % | -0.209 | | 0.6–0.7 | 5 | 0.650 | 80.0 % | -0.150 | | 0.7–0.8 | 10 | 0.763 | 90.0 % | -0.137 | | 0.8–0.9 | 15 | 0.853 | 100.0 % | -0.147 | | 0.9–1.0 | 41 | 0.971 | 92.7 % | +0.044 | ### Reliability — `base-readout`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 | ### Reliability — `base-readout`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.4–0.5 | 2 | 0.432 | 50.0 % | -0.068 | | 0.7–0.8 | 3 | 0.770 | 33.3 % | +0.437 | | 0.8–0.9 | 6 | 0.863 | 50.0 % | +0.363 | | 0.9–1.0 | 97 | 0.999 | 84.5 % | +0.153 | ### Reliability — `jev-generate`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 | ### Reliability — `jev-generate`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 108 | 1.000 | 83.3 % | +0.167 | ### Reliability — `base-generate`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 | ### Reliability — `base-generate`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.1–0.2 | 1 | 0.167 | 100.0 % | -0.833 | | 0.9–1.0 | 107 | 1.000 | 82.2 % | +0.178 | ### Task (c) confusion — `jev-readout` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **17** | 0 | 0 | 1 | 0 | 0 | | `einstein` | 1 | **13** | 1 | 0 | 3 | 0 | | `hypatia` | 1 | 0 | **12** | 3 | 2 | 0 | | `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 | | `sagan` | 1 | 1 | 0 | 0 | **16** | 0 | | `socrates` | 0 | 0 | 0 | 2 | 2 | **14** | ### Task (c) confusion — `base-readout` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **13** | 0 | 1 | 2 | 2 | 0 | | `einstein` | 0 | **14** | 0 | 0 | 3 | 1 | | `hypatia` | 0 | 3 | **12** | 0 | 2 | 1 | | `ibn_sina` | 0 | 0 | 2 | **14** | 1 | 1 | | `sagan` | 1 | 0 | 0 | 0 | **17** | 0 | | `socrates` | 0 | 0 | 0 | 0 | 1 | **17** | ## Residency, per arm | arm | task | card 0 MiB after | named grew | others quiet | ok | |---|---|---|---|---|---| | `base-generate` | a | 21472.0 | False | True | True | | `base-generate` | b | 21472.0 | False | True | True | | `base-generate` | c | 21472.0 | False | True | True | | `base-readout` | a | 21472.0 | False | True | True | | `base-readout` | b | 21472.0 | False | True | True | | `base-readout` | c | 21472.0 | False | True | True | | `jev-generate` | a | 20336.0 | False | True | True | | `jev-generate` | b | 20336.0 | False | True | True | | `jev-generate` | c | 20336.0 | False | True | True | | `jev-readout` | a | 20336.0 | False | True | True | | `jev-readout` | b | 20336.0 | False | True | True | | `jev-readout` | c | 20336.0 | False | True | True | --- ## R1 — The gate, read as written Pre-registered: *the adapter must reach ≥ the base readout's accuracy − 2 points on (a) AND (c).* * **(a) 100.0 % vs 100.0 %, delta +0.0 → PASS.** * **(c) 83.3 % (90/108) vs 80.6 % (87/108), delta +2.8 → PASS.** * **Verdict: ACCEPTED.** A threshold table is earned, and §R5 carries it. **And task (a) has to be said honestly: it is saturated and it discriminates nothing.** All four arms answered 63 of 63 — every on-page question yes, every off-page question no, with no arm making a single mistake. A gate that both sides clear perfectly is a gate that told us nothing about either side; what it *did* tell us is that the docent's abstain contract, recast as one typed decision over the page the seat already reads, is **easy** — easy enough that a 26B-A4B at Q4 does not miss, and easy enough that the docent's own prose-and-five-gates machinery is doing something harder than this bench asked of it. The whole of the gate's discriminating power was in (c), and that is where it should be read. ## R2 — The promise that did not survive contact The question this bench was built on was latency and energy, and the pre-registered comparison is arm-level. Arm-level, the readout looks 5–8 % cheaper: | arm | wall, all three tasks | energy, all three tasks | J / decision (852 total) | |---|---|---|---| | `jev-readout` | 11.8 min | 145.2 kJ | 681.8 | | `jev-generate` | 12.6 min | 157.0 kJ | 736.9 | | `base-readout` | 12.5 min | 151.7 kJ | 712.2 | | `base-generate` | 13.4 min | 164.7 kJ | 773.0 | **Most of that gap is an artefact of this harness and is named rather than banked.** The census — a full prefill of all 42 pages, run once per model — sits immediately before that model's *first* arm and leaves ollama's single KV slot warm. Exactly two rows in each readout arm came back prefix-cached, and they are the only two rows in the whole run where a readout beat its generating twin by more than five seconds: | item | prompt tokens | readout | its prefill | generate | its prefill | |---|---|---|---|---|---| | `on-01` | 82,531 | **3.0 s** | 694 ms | 39.2 s | 38,130 ms | | `on-02` | 26,735 | **3.5 s** | 546 ms | 11.6 s | 8,999 ms | Drop those two and pair every remaining item with itself across the two arms — same model, same prompt, same card, one differing only in whether the letter was read or written: | task | pair | clean pairs | median delta | share | |---|---|---|---|---| | (a) | `jev-readout` vs `jev-generate` | 61 of 63 | +0.026 s | **+0.7 %** | | (a) | `base-readout` vs `base-generate` | 61 of 63 | +0.017 s | **+0.4 %** | | (b) | `jev-readout` vs `jev-generate` | 42 of 42 | +0.004 s | **+0.1 %** | | (b) | `base-readout` vs `base-generate` | 42 of 42 | +0.031 s | **+0.8 %** | | (c) | `jev-readout` vs `jev-generate` | 108 of 108 | +0.013 s | **+1.3 %** | | (c) | `base-readout` vs `base-generate` | 108 of 108 | +0.029 s | **+2.8 %** | **Reading the answer instead of writing it is worth about one to three percent here, not a fraction.** The reason is visible in the shortest task, where prefill cannot hide it: on task (c)'s 286-token prompts every arm sits at about 1.02–1.06 s per decision, of which the server's own `prompt_eval` is **128–152 ms**. The other ~870 ms is ollama's per-request overhead, and it is the same whether the model emits one token or four. OpenJev's published figure — about 80 ms for a short text decision on one H100 under vLLM with prefix caching — **is not reachable through this server at all**, and the gap is the server's, not the card's. Where the readout *does* pay is the long tail, and only through a warm cache: a 1-token decode on an 82 k-token KV is fast, a 4-token decode on the same KV is not — 38 s of that row's 39 s was prefill, so the saving lives in the prefill being reused, which is a caching property and not a readout property. **A decision model on a hot page is a real saving; a decision model on a cold page is the same prefill anybody pays.** ## R3 — Task (b), as declared: measured, not scored The pre-registration said (b) would not be scored, and it is not. 42 decisions over each page's own cite enum (2–25 options, median 12). The bank carries an expected section for **1** of them and all four arms missed that one, which is one row and is not a rate. What (b) *can* say: * The two readout arms **disagree with each other on 11 of 42** pages (73.8 % agreement) — the largest disagreement anywhere in this run, and the place a hand-labelled set would buy the most. * The adapter's readout and the adapter's generation agree **42 of 42**. Within a model, reading the letter and writing it are the same decision (see §R4). * **The gap this bench cannot close: 42 hand labels from the content owner.** They would turn the estate's own cite decision into a scored task, and this run says that is the single highest-value hour anyone could spend on this instrument. ## R4 — What the adapter actually buys, and it is not speed The finding that survives every control is this one: **the base model does not put the option letters where a readout can see them, and the adapter does.** | arm | task | rows with ≥1 floored letter | median floored | |---|---|---|---| | `base-readout` | (a), 2 options | 19 of 63 | 0 of 2 | | `base-readout` | (b), median 12 options | 39 of 42 | 6 of 12 | | `base-readout` | (c), 6 options | **101 of 108** | **4 of 6** | | `jev-readout` | (a) | 5 of 63 | 0 | | `jev-readout` | (b) | 7 of 42 | 0 | | `jev-readout` | (c) | **0 of 108** | 0 | A "floored" letter is one that never appeared in the server's raw top-20 at position 0 and took the protocol's −30 floor. On six-way decisions the base needs that floor on four of its six options in 94 % of rows; the adapter needs it **never**. That is precisely what OpenJev says tuning is for — *"trained so that this single position carries the decision"* — and it is the one thing in this run that separates the adapter from its base by a wide margin. **Two consequences, and the second is a trap.** 1. The base's readout probabilities are **not usable**. Its task-(a) Brier of 0.0001 looks like superb calibration and is nothing of the kind: with one of two letters floored at −30, the softmax returns ≈1.0 by construction. **A low Brier produced by the floor is an artefact of the floor, not a property of the model**, and any table that prints the two side by side without this sentence is misleading. The adapter's 0.0038 on the same task is the honest number of the two. 2. The base's *accuracy* survives anyway (80.6 % on (c) with four letters floored), because the argmax among the letters that did surface was usually right. Accuracy tolerates the floor; probability does not. **The readout and the written answer are the same decision.** Within the adapter they agree on every single row of every task — 63/63, 42/42, 108/108. Within the base they agree 63/63, 41/42, 102/108. So the readout is not a different or worse answer; it is the same answer with a probability attached, at the same cost. ## R5 — The threshold table the gate earned Read against the floor a reader without ears already clears — 24.0 %, not the six-way 16.7 % (see the task-(c) floor above) — and stated only for the tasks the gate covered: | decision | arm | accuracy | over the elimination floor | calibration usable? | |---|---|---|---|---| | six-way voice, 108 balanced items | `jev-readout` | **83.3 %** | +59.3 points | **yes** (0 floored rows) | | six-way voice | `base-readout` | 80.6 % | +56.6 | no (101 of 108 floored) | | six-way voice | `jev-generate` | 83.3 % | +59.3 | none — a written letter has no probability | | six-way voice | `base-generate` | 82.4 % | +58.4 | none; **1 row refused to answer in the form asked** | | article answers this question?, 63 items | all four | **100.0 %** | — | saturated; see §R1 | The adapter's reliability on (c) is under-confident across the board — every bin from 0.2 to 0.9 is more accurate than it claims — and slightly over-confident at the top (0.971 claimed against 92.7 % at 41 rows). The published `READOUT_T = 0.85` was fitted for OpenJev on the targeted extraction path; **every row in this run keeps its raw per-letter logprobs, so a temperature refit for this model costs no GPU time at all** — `rows/*.jsonl`, field `letter_logprobs`. ## R6 — What refused, and what was fixed * **The bench unit's 8,192 context refused the docent's own corpus.** The longest page is **82,500 tokens** as the docent fences it. `num_ctx` moved to 131,072, measured to keep the model whole on card 0 at 20,336 MiB (adapter) / 21,472 MiB (base), card 1 at 1 MiB. Under §8 this was the declared deviation and it is the reason the run uses `/api/chat` rather than `/v1/chat/completions`. * **The first output token is `<|channel>`, not a letter.** Untouched, every option letter sits 13–18 nats below the template's thinking-channel marker. `think: false` is the fix and every arm sets it. Full receipt in `receipts/2026-09-21-substrate.md`. * **`chat_template_kwargs` is ignored by ollama** (the channel token came back anyway); `reasoning_effort: "none"` on `/v1` is equivalent to `think: false`. * **The frozen kit's task-(c) ids are not unique — 108 distinct lines under 50 distinct ids.** Every wall entry is edition 1, so `(guest, edition, course, line_index)` collides across dinners. Blast radius, checked field by field: **the agreement table only**, which first printed rates over 50 of 108 rows. Accuracy, Brier, reliability, latency, tokens and energy all read the row lists and never the id, so none of them moved. The kit was **not** rebuilt — that would have broken the instrument this run was measured against — so `tables.py` now pairs arms by position, asserting the label sequences match rather than assuming it, and `kit/build_kit.py` puts the entry in the id and refuses to emit a kit with duplicates. * **One generated answer refused the form.** `base-generate` on `c-darwin-…-dessert-03` wrote *"The speaker addresses Hypatia in the second"* instead of a letter. It is counted as unparsed, not silently guessed. **A readout cannot fail this way** — it scores letters whether or not the model would have written one — and that is a robustness argument for the readout that has nothing to do with speed. * **Nothing else refused.** 852 decisions, **0 errors**, 0 timeouts, no card but card 0 ever above 1 MiB. ## R7 — Arm 2, and one number that moved *What this is, for a reader who scrolled here: arm 2 is `openjev/openjev-FP8` served under vLLM across both of benchbox's 3090s, tensor-parallel. It was **not started**, as briefed.* Re-read on benchbox at 18:04Z, and **one precondition has changed since §9 was written at 16:56Z**: the disk is no longer the blocker. | precondition | at 16:56Z (§9) | at 18:04Z | verdict | |---|---|---|---| | free disk on `/` | ≈ 10 GB | **54 GB** | **clears** — the FP8 checkpoint is ≈ 29 GB | | `vllm` on the box | absent | **still absent** | blocks — `command -v vllm` empty, no venv bin, `import vllm` → `ModuleNotFoundError` | | NVLink between the two 3090s | assumed absent | **measured absent** — `nvidia-smi nvlink -s`: *all links are inActive*; `topo -m`: **PHB** | blocks a cheap tensor-parallel | | the second card's link | x4 (2B:00.0) | unchanged | an all-reduce would cross PHB with one card on x4 | So arm 2 needs: **a vLLM install (nothing else)**, and its tensor-parallel all-reduce must be *measured* over PHB with a card on x4 rather than assumed — on a two-3090 box with no NVLink that is the number that decides whether arm 2 is worth its seat, and this run cannot guess it. Note also what §R2 implies for arm 2: OpenJev's own latency figures are vLLM figures with prefix caching, and the ~870 ms of per-request overhead this bench measured is **ollama's**. An arm 2 under vLLM is therefore not just a different model — it is the only configuration in which the latency half of the original question can actually be tested. ## R8 — The kit, and how to re-run it ``` bench/jev-2026-09-21/ README.md this file: pre-registration, then results run.py the harness; `ARMS` is where arm 2 goes tables.py reads the reports, calls nothing kit/build_kit.py freezes the instrument from its sources kit/articles.json 42 states, verbatim, sha256 each kit/task_a.json 63 items kit/task_b.json 42 kit/task_c.json 108 kit/manifest.json bank sha, wall sha, seed, counts, roster rows/..jsonl every decision, with its raw letter logprobs rows/..report.json summary, energy, idle, residency, census rows/..watts.csv the 1 Hz board reading the joules came from rows/census..json every page's real prompt length receipts/2026-09-21-substrate.md the pin, the context ladder, the channel trap receipts/run.log the driver's own output, all four arms ``` The instrument that ran was the committed one: `run.py` on benchbox hashed `26023ce42de02214…`. That content hash is the durable pointer and it is what a re-runner should check; the commit this file first named, `b0c46605`, was orphaned by a later rebase and will not survive a `gc`. **The same bytes are on the branch at `c2982673`** — verified, not assumed: `git show c2982673:bench/jev-2026-09-21/run.py | sha256sum` prints `26023ce4…`. (`run.py` has since grown arm 2's runtime and is no longer those bytes; arm 2 names its own instrument hash in §R2.8.) ```bash # on benchbox, with the bench unit up and pinned to card 0 python3 run.py --unload-between \ --arm jev-readout --arm jev-generate --arm base-readout --arm base-generate \ --task a --task b --task c --out rows python3 tables.py rows/ > TABLES.md ``` --- # §Arm 2 — the pre-registration (`openjev/openjev-FP8` under vLLM, both cards) *This section was written and committed **before any arm-2 row was measured** — `git log` is the receipt, and nothing in it was edited afterwards. It sits below arm 1's results because it was written after them; it sits above arm 2's results because it was written before those. **What this is, for a reader who scrolled straight here:** arm 2 puts the real decision model — `openjev/openjev-FP8`, the FP8 file of OpenJev — on benchbox's **two** RTX 3090s under vLLM, tensor-parallel, and asks it the same three task sets arm 1 asked a community adapter and its base on one card under ollama.* **Pre-registration stamped 2026-09-21T18:15:14Z** · box **benchbox** · card 0 `GPU-46890836-d8f7-e868-1977-c1ce28db0d7a` (PCI 04:00.0, **x16**, cap 250 W) · card 1 `GPU-7aa0be10-974f-6430-52fe-09018a0e2e08` (PCI 2B:00.0, **x4**, cap 250 W) · **PHB topology, no NVLink** (`nvidia-smi nvlink -s`: all links inActive; `topo -m`: PHB). Both cards are read, by UUID, in every reading this arm takes. ## A2.1 — The question arm 2 asks Arm 1 measured a *community adapter* of a generating seat, on one card, through ollama, and found (§R2) that the latency half of the original question **could not be tested there**: about 870 ms of every decision was the server's own per-request overhead, the same whether the model wrote one token or four. Arm 2 changes both halves of that substrate at once and says so: a model **built** as a decision model rather than adapted into one, and the runtime its published numbers were measured on. It therefore answers one question arm 1 could not: > **Does a purpose-built decision model, served the way its own card serves it, hold its > accuracy on the estate's own decisions — and what does a decision actually cost on two > consumer cards with no NVLink and one of them on a four-lane link?** ## A2.2 — The model, and the licence that bounds it | | | |---|---| | repository | `openjev/openjev-FP8` (the FP8 file of `openjev/openjev`) | | architecture | `Qwen3_5ForConditionalGeneration`, `model_type: qwen3_5` — a hybrid linear/full-attention VL model, 64 layers, full attention every 4th | | quantisation | FP8 e4m3, **128×128 weight blocks, dynamic activations** (`quantization_config.quant_method: fp8`) | | size on disk | **30.41 GB** over 12 safetensors shards (the card says "about 29 GB") | | revision | recorded from the download, not from this page | | **licence** | **CC BY-NC 4.0 — research and non-commercial use only.** | **The licence is a fence, not a footnote. This arm is BENCH-ONLY. These weights never become a seat**: no estate service, no rulesage, no docent, no beat lab, no cove, nothing a visitor or a customer can reach, now or later. If arm 2 passes its gate, what it licenses is a *finding about decision models*, never this checkpoint in production. Any future production use needs either a commercial arrangement with the authors or a differently licensed model, and that is a decision for the operator, not for a bench. ## A2.3 — What arm 2 keeps identical to arm 1, and what it cannot **Identical** — the frozen kit and nothing rebuilt: `kit/task_a.json` (63), `task_b.json` (42), `task_c.json` (108), the same `state_sha256` per page, the same option order, the same seed (20260921), the same prompt builder (`run.py::readout_prompt`, the OpenJev helper's text-lane layout), the same calibration constants applied unchanged (`READOUT_T = 0.85`, `READOUT_NOUL_T = 1.829074`, `READOUT_NOUL_BIAS = 0`), the same `FLOOR = -30`, the same `top_logprobs = 20` read at the **first output position**, concurrency 1, one request at a time. *Why 20 and not the 64 this server can serve:* arm 1's headline finding (§R4) is a **floor** count, and a floor counted against a wider top-K is not the same measurement. 20 keeps the protocol byte-identical to arm 1. Every row still records `letters_in_topk`, so if OpenJev ever needs the floor at all, the number is in the file rather than in an argument. **Cannot be identical, and each is named rather than absorbed:** 1. **Different runtime.** vLLM, not ollama. Arm 1's ~870 ms per-request overhead is ollama's; arm 2's is vLLM's. Any latency row of arm 2 read against an arm-1 row is a **cross-runtime** comparison and every table that does it says so in its own header. 2. **Different tokenizer and different model family.** Prompt-token counts are not comparable across arms; the *decisions* are. 3. **Arm 1's adapter is NOT re-run here.** The GGUF is not vLLM's format, and converting it would make a third thing that is neither arm. Arm 1's rows stand as arm 1 measured them. 4. **The apples-to-apples base is not served.** The latency claim would be cleanest with arm 1's base (`gemma4:26b`) under vLLM too — but vLLM needs HF weights and the box has only the GGUF. A >20 GB download for one table is out of scope for this arm, **so that comparison is NOT MADE**, and it is named here rather than quietly substituted. OpenJev's decision latency is reported as **its own row**, against arm 1's ollama rows, with the runtime difference stated in the table header. 5. **Two cards, not one.** Energy is the sum over **both** boards by UUID; idle is read on both for ten seconds immediately before each task. 6. **Python 3.12, not the box's 3.14.** benchbox's system interpreter is 3.14.4 and vLLM 0.29.0's own dependency set is not built for it; the venv is built on the uv-managed 3.12.14 already on the box. Declared here as a deviation from "Python 3 there". ## A2.4 — The serving posture, exactly The model card's own recipe, bound to loopback and to two cards: ```bash vllm serve --served-model-name openjev-fp8 \ --host 127.0.0.1 --port 8000 \ --tensor-parallel-size 2 \ --max-model-len 16384 --gpu-memory-utilization 0.90 \ --enable-prefix-caching --max-num-seqs 1 --max-logprobs 64 \ --limit-mm-per-prompt '{"image":0}' --trust-remote-code \ --gdn-prefill-backend triton --quantization fp8 ``` * **127.0.0.1 only.** The port is never bound to any other address. * **`--max-num-seqs 1`**: one request at a time, as arm 1 ran. * **FP8 on Ampere is weight-only.** sm86 has no native FP8 arithmetic, so vLLM will pick a weight-only kernel (Marlin or a block-scaled triton path). **Which one it picks is recorded verbatim from the startup log** rather than assumed, and if it refuses the checkpoint outright that is the arm's answer. * **Residency is proved on BOTH cards**, by memory growth read from `nvidia-smi` by UUID before and after load. A tensor-parallel load that grew only one board is a failed pin and the arm stops. * **No home-brew fallbacks.** If TP=2 refuses on PHB/x4 or OOMs, the only alternatives this arm may try are vLLM's own (a narrower `--gpu-memory-utilization`, or a 4-bit path vLLM itself offers for this checkpoint). CPU offload, layer-splitting by hand, or any hand-rolled sharding is **out of bounds** — the arm reports a refusal instead. ## A2.5 — The context ceiling, and the refusal it forces **`--max-model-len 16384`** — the card's own figure and the model's stated request limit ("Prompts up to 16,384 tokens"). The docent's own corpus does not fit in it: arm 1 measured pages from 1,772 to **82,500** tokens (`rows/census.gemma4-26b.json`). **Pre-registered policy, and it is a refusal rather than a repair:** an item whose prompt exceeds the served context is recorded as **`refused-overlong`** with its measured token count, and is **excluded from every rate**. It is **never truncated** — truncating a docent page can turn a genuinely on-page question into an unanswerable one, which is arm 1 §8's ruling and it stands. Every table prints the denominator it actually scored. Expected coverage, by arm 1's *gemma* token census and therefore an estimate, not a claim — the Qwen tokenizer's real counts are measured by a census before the first row: | task | items expected to fit 16,384 | of | |---|---|---| | (a) | ≈ 53 (35 on-page, 18 off-page) | 63 | | (b) | ≈ 35 | 42 | | (c) | **108 — all of them** (286-token prompts) | 108 | **The gated task is untouched by this ceiling**, which is why the gate is where it is. ## A2.6 — THE GATE, pre-registered > **OpenJev's readout must reach ≥ 81.3 % on task (c)** — arm 1's adapter readout scored > **83.3 %** (90 of 108) and the margin is two points. > If it does not, the rung is **REJECTED** and gets **no threshold table**. > **Latency, energy and the PHB number are reported either way.** Two tasks are deliberately outside the gate, each for a reason arm 1 established: * **(a) is reported, not gated.** All four arm-1 arms scored 63 of 63; a gate both sides clear perfectly discriminates nothing (§R1). It is reported for coverage and for the abstain split, and a *fall* on it would be a finding worth stating — but it cannot earn a pass. * **(b) is not scored at all**, as in arm 1: the bank carries an expected section for 1 of 63 questions. Cost, stability and cross-arm agreement only. ## A2.7 — What is measured, per task Exactly arm 1's list, with both boards where arm 1 had one: accuracy against labels on the scored denominator · multi-class Brier · a 10-bin reliability table · median and p95 seconds per decision · prompt and written tokens · **energy as each board's own watts × seconds, integrated by trapezoid from a 1 Hz `nvidia-smi` sampler pinned to that board's UUID, summed across the two cards**, reported raw and net of the idle draw read on both boards for ten seconds immediately before the task · residency on both boards · `refused-overlong` counts · `letters_in_topk` and floored letters per row · the raw per-letter logprobs, so any temperature can be refitted later without spending a GPU second. ## A2.8 — The PHB number, pre-registered as a required receipt Two 3090s, no NVLink, one on x4: a tensor-parallel all-reduce crosses the host bridge on every layer, and **on this box that price must be a measured number or arm 2's latency means nothing.** Before or after the task arms, this arm runs **one all-reduce microbench** — vLLM's own startup collective check if it emits a figure, otherwise a bounded `torch.distributed` `all_reduce` sweep over the tensor sizes a 5,120-hidden model actually reduces — and reports **achieved bus bandwidth (GB/s) and per-call latency (µs) by message size**, with the measurement command in the receipts. A bench that reports a decision latency over PHB without the PHB number is reporting a number it cannot explain. ## A2.9 — Announcements, and the box's other tenant * **Every task arm announces `box · card(s) · arm · task · start stamp` at the moment it starts**, timestamped, in the run log and in the report. * **Arm 1's ollama unit (`ollama-bench-c0`, pinned to card 0, store `~/bench-store/ollama`) holds card 0.** It is **stopped** (`systemctl --user stop ollama-bench-c0`) before arm 2 asks for card 0, the stop is stamped, and the unit is **left as arm 2 found it** when the arm ends. The vLLM server is left **stopped**. * No sudo is used anywhere in this arm. ## A2.10 — What counts as a refusal, decided now rather than later So that nothing is chosen after seeing a number: the arm **STOPS and reports** — it does not improvise — if any of these happen. Each is reported with the exact output that produced it. 1. **The install cannot complete**: no wheel for this interpreter or this CUDA, the disk fills, or the install exceeds **45 minutes**. 2. **The driver is too old for the wheel's CUDA runtime** (benchbox reads **595.84**; checked against the wheel's own requirement before the install is trusted). 3. **vLLM refuses the checkpoint** — the architecture, the block-scaled FP8, or the GDN prefill backend. 4. **TP=2 refuses or OOMs**, after the two in-bounds alternatives of §A2.4. 5. **The pin fails**: a board that should have grown did not, or a board that should have been quiet was not. 6. **The download does not complete** or its revision cannot be recorded. **Disk, stated before it is spent:** benchbox has **54 GB** free. The checkpoint is 30.41 GB and the venv is expected near 10–15 GB, which leaves single-digit GB of headroom. If the install and the download together cannot both fit, that is refusal (1) and the arm says so rather than deleting anything it did not create. --- ## ARM 2 — RESULTS *(appended as arm 2 landed; nothing above this line, including the arm-2 pre-registration, was edited after the first arm-2 call.)* **What was measured, and when.** Two arms — `openjev/openjev-FP8` asked to **read** the option letters at the first output position, and the same model on the same server asked to **write** the letter — over the same three frozen task sets arm 1 used: 63 typed yes/no decisions on the docent's own bank, 42 cite decisions on the same pages, and the same 108 balanced six-way voice decisions from the long table's public wall. **426 decisions attempted and 392 reached, between 2026-09-21T18:36:45Z and 2026-09-21T19:01:27Z**, one request at a time, tensor-parallel across **both** of benchbox's RTX 3090s at 250 W caps. The 34 that were not reached were **refused as overlong** and never truncated (§A2.5). Energy drawn by the two boards together over the 23.6 minutes the arms held them: **690.0 kJ, 0.192 kWh**. Exit code 0, zero errors, zero timeouts. ## The verdict, in one paragraph **The gate passes by ten points, and this time the latency promise is half-kept — by the model, not by the box.** OpenJev-FP8 read **93.5 %** on the long table's six voices against a pre-registered floor of 81.3 %, which is **+10.2 points over arm 1's community adapter** and +12.9 over arm 1's base, with **not one floored letter** anywhere in (a) or (c). It is also **2.9× faster per decision** than arm 1's adapter on that task — 0.350 s against 1.017 s — so the ~870 ms of per-request overhead arm 1 measured was indeed ollama's, and vLLM does not charge it. But OpenJev's own published figure for a short text decision is **about 80 ms**, and this box cannot get near it: **the host bridge between two GeForce cards with no NVLink and no peer-to-peer is the cost of everything here**, and §R2.6 puts a number on it. What the readout buys over writing the letter is now visible where arm 1 could not see it: **10.5 % of the time and 11.6 % of the energy** on short prompts, and essentially nothing on long ones — because a long prompt is prefill, and prefill on this box is wire. ## The gate — arm 2 | gated task | arm 1's adapter readout (ollama, one card) | the floor (that − 2) | OpenJev-FP8 readout (vLLM, two cards) | delta (points) | verdict | |---|---|---|---|---|---| | (c) which of the six guests said this line? | 83.3 % (90 / 108) | **81.3 %** | 93.5 % (101 / 108) | +10.2 | **PASS** | Verdict: **ACCEPTED** ## Coverage — what the ceiling refused, before anything was scored | task | items in the kit | reached | refused as overlong | scored against a label | |---|---|---|---|---| | (a) does the article answer this question? | 63 | 53 | 10 | 53 | | (b) which section is the answer in? | 42 | 35 | 7 | 0 | | (c) which of the six guests said this line? | 108 | 108 | 0 | 108 | _A refused item is in the row file with its measured token count and out of every rate. It was never truncated: a truncated docent page can turn a genuinely on-page question into an unanswerable one, which is arm 1 §8's ruling and it stands._ ## Accuracy | task | arm | accuracy | correct / scored | Brier | unparsed | rows with a floored letter | letters floored | |---|---|---|---|---|---|---|---| | (a) does the article answer this question? | `openjev-fp8-readout` | 100.0 % | 53 / 53 | 0.0059 | 0 | 0 | 0 | | | `openjev-fp8-generate` | 100.0 % | 53 / 53 | 0.0001 | 0 | 0 | 0 | | (b) which section is the answer in? | `openjev-fp8-readout` | **not a rate** — 1 of 35 reached rows carry a label | 0 / 1 | 0.5159 | 0 | 5 | 20 | | | `openjev-fp8-generate` | **not a rate** — 1 of 35 reached rows carry a label | 0 / 1 | 1.0000 | 0 | 0 | 0 | | (c) which of the six guests said this line? | `openjev-fp8-readout` | 93.5 % | 101 / 108 | 0.1219 | 0 | 0 | 0 | | | `openjev-fp8-generate` | 93.5 % | 101 / 108 | 0.1296 | 0 | 0 | 0 | ## Latency and energy — BOTH boards, summed | task | arm | median s | p95 s | prompt tokens | written tokens | card 0 mean W | card 1 mean W | both, J / decision | net of idle, J / decision | idle W (both) | |---|---|---|---|---|---|---|---|---|---|---| | (a) does the article answer this question? | `openjev-fp8-readout` | 6.458 | 14.200 | 373,760 | 53 | 241.9 | 245.4 | 3521.7 | 1836.1 | 233.2 | | | `openjev-fp8-generate` | 6.501 | 14.325 | 373,760 | 106 | 242.7 | 245.1 | 3553.5 | 1508.6 | 280.7 | | (b) which section is the answer in? | `openjev-fp8-readout` | 7.643 | 14.553 | 261,452 | 35 | 241.8 | 244.9 | 3936.6 | 1526.5 | 298.0 | | | `openjev-fp8-generate` | 7.658 | 14.559 | 261,452 | 70 | 242.4 | 245.5 | 3959.6 | 1707.9 | 277.4 | | (c) which of the six guests said this line? | `openjev-fp8-readout` | 0.350 | 0.417 | 31,303 | 108 | 244.2 | 245.5 | 167.9 | 72.6 | 277.9 | | | `openjev-fp8-generate` | 0.392 | 0.459 | 31,303 | 216 | 243.6 | 244.6 | 189.9 | 82.5 | 276.1 | ## Reading the letter against writing it, on the same model | task | paired items | median readout s | median generate s | median delta | share | same choice | |---|---|---|---|---|---|---| | (a) does the article answer this question? | 53 | 6.458 | 6.501 | +0.031 s | 0.5 % | 53 / 53 | | (b) which section is the answer in? | 35 | 7.643 | 7.658 | +0.021 s | 0.3 % | 35 / 35 | | (c) which of the six guests said this line? | 108 | 0.350 | 0.392 | +0.041 s | 10.5 % | 108 / 108 | ## Latency ACROSS RUNTIMES — not a like-for-like comparison _Arm 1 ran on **ollama, one card, a Q4\_K\_M GGUF of a 26B-A4B**; arm 2 runs on **vLLM 0.29.0, two cards tensor-parallel, an FP8 checkpoint of a different model family**. Arm 1's base was NOT served under vLLM (README §A2.3.4), so **the apples-to-apples latency comparison is not made**. These rows sit in one table because a reader needs the scale; they do not sit in one table because they are the same measurement._ | task | arm | server · cards | model | median s | p95 s | decisions it is a median of | |---|---|---|---|---|---|---| | (a) does the article answer this question? | `jev-readout` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 3.527 | 12.182 | 63 | | | `base-readout` | ollama · 1 card | `gemma4:26b` | 3.746 | 12.943 | 63 | | | `jev-generate` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 3.621 | 28.089 | 63 | | | `base-generate` | ollama · 1 card | `gemma4:26b` | 3.841 | 29.896 | 63 | | | `openjev-fp8-readout` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 6.458 | 14.200 | 53 | | | `openjev-fp8-generate` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 6.501 | 14.325 | 53 | | (b) which section is the answer in? | `jev-readout` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 3.766 | 12.210 | 42 | | | `base-readout` | ollama · 1 card | `gemma4:26b` | 4.010 | 12.969 | 42 | | | `jev-generate` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 3.848 | 12.214 | 42 | | | `base-generate` | ollama · 1 card | `gemma4:26b` | 4.024 | 13.052 | 42 | | | `openjev-fp8-readout` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 7.643 | 14.553 | 35 | | | `openjev-fp8-generate` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 7.658 | 14.559 | 35 | | (c) which of the six guests said this line? | `jev-readout` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 1.017 | 1.093 | 108 | | | `base-readout` | ollama · 1 card | `gemma4:26b` | 1.025 | 1.116 | 108 | | | `jev-generate` | ollama · 1 card | `hf.co/mradermacher/jevify-gemma4-26b-a4b-GGUF:Q4_K_M` | 1.031 | 1.090 | 108 | | | `base-generate` | ollama · 1 card | `gemma4:26b` | 1.060 | 1.116 | 108 | | | `openjev-fp8-readout` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 0.350 | 0.417 | 108 | | | `openjev-fp8-generate` | vLLM 0.29.0 · 2 cards | `openjev-fp8` | 0.392 | 0.459 | 108 | ## Agreement with arm 1's arms | task | pair | agree | of comparable | rate | |---|---|---|---|---| | (a) does the article answer this question? | `jev-readout` vs `openjev-fp8-readout` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `jev-readout` vs `openjev-fp8-generate` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `base-readout` vs `openjev-fp8-readout` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `base-readout` vs `openjev-fp8-generate` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `jev-generate` vs `openjev-fp8-readout` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `jev-generate` vs `openjev-fp8-generate` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `base-generate` vs `openjev-fp8-readout` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `base-generate` vs `openjev-fp8-generate` | 53 | 53 | 100.0 % | | (a) does the article answer this question? | `openjev-fp8-readout` vs `openjev-fp8-generate` | 53 | 53 | 100.0 % | | (b) which section is the answer in? | `jev-readout` vs `openjev-fp8-readout` | 20 | 35 | 57.1 % | | (b) which section is the answer in? | `jev-readout` vs `openjev-fp8-generate` | 20 | 35 | 57.1 % | | (b) which section is the answer in? | `base-readout` vs `openjev-fp8-readout` | 23 | 35 | 65.7 % | | (b) which section is the answer in? | `base-readout` vs `openjev-fp8-generate` | 23 | 35 | 65.7 % | | (b) which section is the answer in? | `jev-generate` vs `openjev-fp8-readout` | 20 | 35 | 57.1 % | | (b) which section is the answer in? | `jev-generate` vs `openjev-fp8-generate` | 20 | 35 | 57.1 % | | (b) which section is the answer in? | `base-generate` vs `openjev-fp8-readout` | 22 | 35 | 62.9 % | | (b) which section is the answer in? | `base-generate` vs `openjev-fp8-generate` | 22 | 35 | 62.9 % | | (b) which section is the answer in? | `openjev-fp8-readout` vs `openjev-fp8-generate` | 35 | 35 | 100.0 % | | (c) which of the six guests said this line? | `jev-readout` vs `openjev-fp8-readout` | 91 | 108 | 84.3 % | | (c) which of the six guests said this line? | `jev-readout` vs `openjev-fp8-generate` | 91 | 108 | 84.3 % | | (c) which of the six guests said this line? | `base-readout` vs `openjev-fp8-readout` | 89 | 108 | 82.4 % | | (c) which of the six guests said this line? | `base-readout` vs `openjev-fp8-generate` | 89 | 108 | 82.4 % | | (c) which of the six guests said this line? | `jev-generate` vs `openjev-fp8-readout` | 91 | 108 | 84.3 % | | (c) which of the six guests said this line? | `jev-generate` vs `openjev-fp8-generate` | 91 | 108 | 84.3 % | | (c) which of the six guests said this line? | `base-generate` vs `openjev-fp8-readout` | 89 | 108 | 82.4 % | | (c) which of the six guests said this line? | `base-generate` vs `openjev-fp8-generate` | 89 | 108 | 82.4 % | | (c) which of the six guests said this line? | `openjev-fp8-readout` vs `openjev-fp8-generate` | 108 | 108 | 100.0 % | ## The PHB receipt — what the host bridge costs (README §A2.8) | what one all-reduce carries | bytes on the wire | median | p95 | bus bandwidth | |---|---|---|---|---| | one decode step (1 token × 5,120 × bf16) | 10.0 KiB | 50.2 µs | 55.3 µs | 0.20 GB/s | | a 4-token prefill chunk (4 tokens × 5,120 × bf16) | 40.0 KiB | 82.9 µs | 87.0 µs | 0.49 GB/s | | a 16-token prefill chunk (16 tokens × 5,120 × bf16) | 160.0 KiB | 155.6 µs | 160.8 µs | 1.05 GB/s | | a 64-token prefill chunk (64 tokens × 5,120 × bf16) | 640.0 KiB | 423.9 µs | 435.2 µs | 1.55 GB/s | | a 256-token prefill chunk (256 tokens × 5,120 × bf16) | 2.5 MiB | 1.5 ms | 1.5 ms | 1.75 GB/s | | a 1024-token prefill chunk (1024 tokens × 5,120 × bf16) | 10.0 MiB | 5.8 ms | 5.9 ms | 1.80 GB/s | | a 4096-token prefill chunk (4096 tokens × 5,120 × bf16) | 40.0 MiB | 23.1 ms | 23.2 ms | 1.82 GB/s | | a 8192-token prefill chunk (8192 tokens × 5,120 × bf16) | 80.0 MiB | 46.1 ms | 46.4 ms | 1.82 GB/s | | a 16384-token prefill chunk (16384 tokens × 5,120 × bf16) | 160.0 MiB | 92.0 ms | 92.4 ms | 1.82 GB/s | **The floor this link puts under one written token: 6.4 ms** — 128 all-reduces (2 per layer × 64 layers) at 50.2 µs each, before the model computes anything. torch 2.13.0+cu130, NCCL 2.29.7, measured 2026-09-21T18:22:15Z. ### Reliability — `openjev-fp8-readout`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.8–0.9 | 4 | 0.884 | 100.0 % | -0.116 | | 0.9–1.0 | 49 | 0.959 | 100.0 % | -0.041 | ### Reliability — `openjev-fp8-readout`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.4–0.5 | 1 | 0.465 | 0.0 % | +0.465 | | 0.5–0.6 | 1 | 0.501 | 0.0 % | +0.501 | | 0.6–0.7 | 4 | 0.659 | 100.0 % | -0.341 | | 0.7–0.8 | 5 | 0.741 | 100.0 % | -0.259 | | 0.8–0.9 | 9 | 0.853 | 77.8 % | +0.075 | | 0.9–1.0 | 88 | 0.987 | 96.6 % | +0.021 | ### Reliability — `openjev-fp8-generate`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 53 | 0.994 | 100.0 % | -0.006 | ### Reliability — `openjev-fp8-generate`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 108 | 1.000 | 93.5 % | +0.065 | ### Task (c) confusion — `openjev-fp8-readout` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **16** | 0 | 0 | 1 | 1 | 0 | | `einstein` | 0 | **16** | 2 | 0 | 0 | 0 | | `hypatia` | 0 | 0 | **17** | 0 | 0 | 1 | | `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 | | `sagan` | 1 | 0 | 0 | 0 | **17** | 0 | | `socrates` | 0 | 0 | 1 | 0 | 0 | **17** | ### Task (c) confusion — `openjev-fp8-generate` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **16** | 0 | 0 | 1 | 1 | 0 | | `einstein` | 0 | **16** | 2 | 0 | 0 | 0 | | `hypatia` | 0 | 0 | **17** | 0 | 0 | 1 | | `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 | | `sagan` | 1 | 0 | 0 | 0 | **17** | 0 | | `socrates` | 0 | 0 | 1 | 0 | 0 | **17** | ### Reliability — `openjev-fp8-readout`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.8–0.9 | 4 | 0.884 | 100.0 % | -0.116 | | 0.9–1.0 | 49 | 0.959 | 100.0 % | -0.041 | ### Reliability — `openjev-fp8-readout`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.4–0.5 | 1 | 0.465 | 0.0 % | +0.465 | | 0.5–0.6 | 1 | 0.501 | 0.0 % | +0.501 | | 0.6–0.7 | 4 | 0.659 | 100.0 % | -0.341 | | 0.7–0.8 | 5 | 0.741 | 100.0 % | -0.259 | | 0.8–0.9 | 9 | 0.853 | 77.8 % | +0.075 | | 0.9–1.0 | 88 | 0.987 | 96.6 % | +0.021 | ### Reliability — `openjev-fp8-generate`, task a | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 53 | 0.994 | 100.0 % | -0.006 | ### Reliability — `openjev-fp8-generate`, task c | bin | n | mean confidence | accuracy | gap | |---|---|---|---|---| | 0.9–1.0 | 108 | 1.000 | 93.5 % | +0.065 | ### Task (c) confusion — `openjev-fp8-readout` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **16** | 0 | 0 | 1 | 1 | 0 | | `einstein` | 0 | **16** | 2 | 0 | 0 | 0 | | `hypatia` | 0 | 0 | **17** | 0 | 0 | 1 | | `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 | | `sagan` | 1 | 0 | 0 | 0 | **17** | 0 | | `socrates` | 0 | 0 | 1 | 0 | 0 | **17** | ### Task (c) confusion — `openjev-fp8-generate` | said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` | |---|---|---|---|---|---|---| | `darwin` | **16** | 0 | 0 | 1 | 1 | 0 | | `einstein` | 0 | **16** | 2 | 0 | 0 | 0 | | `hypatia` | 0 | 0 | **17** | 0 | 0 | 1 | | `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 | | `sagan` | 1 | 0 | 0 | 0 | **17** | 0 | | `socrates` | 0 | 0 | 1 | 0 | 0 | **17** | ## Residency — both boards, per arm | arm | task | card 0 MiB after | card 1 MiB after | both grew | no other board moved | ok | |---|---|---|---|---|---|---| | `openjev-fp8-readout` | a | 21246.0 | 21246.0 | True | True | True | | `openjev-fp8-readout` | b | 21246.0 | 21246.0 | False | True | True | | `openjev-fp8-readout` | c | 21246.0 | 21246.0 | False | True | True | | `openjev-fp8-generate` | a | 21246.0 | 21246.0 | False | True | True | | `openjev-fp8-generate` | b | 21246.0 | 21246.0 | False | True | True | | `openjev-fp8-generate` | c | 21246.0 | 21246.0 | False | True | True | --- ## R2.1 — The gate, read as written Pre-registered (§A2.6): *OpenJev's readout must reach ≥ 81.3 % on task (c)* — arm 1's adapter scored 83.3 % and the margin is two points. The floor was **derived from `rows/jev-readout.c.report.json` by `tables_arm2.py`, not typed into it**. * **(c) 93.5 % (101 / 108) against a floor of 81.3 % → PASS by 12.2 points.** * Against the arms it is compared to: **+10.2** over arm 1's adapter readout (83.3 %), **+12.9** over arm 1's base readout (80.6 %), and **+69.5 points over the elimination floor** a reader without ears already clears (24.0 %, arm 1's task-(c) floor). * **Verdict: ACCEPTED.** **And task (a) is saturated again, exactly as arm 1 warned.** All six arms across both runs answered every reached item correctly — 53 of 53 here, 63 of 63 there. A gate both sides clear perfectly discriminates nothing, which is why (a) was reported and not gated (§A2.6), and the whole of the gate's power was in (c) where it belongs. ## R2.2 — What a purpose-built decision model does that an adapter does not Arm 1's sharpest finding (§R4) was a **floor count**: its base model did not put the option letters where a readout could see them, needing the −30 floor on four of six letters in 94 % of six-way rows, while the community adapter needed it never. Arm 2 puts a third model on the same axis, on the same protocol, at the same `top_logprobs = 20`: | arm | (a), 2 options | (b), median 12 options | (c), 6 options | |---|---|---|---| | `base-readout` (arm 1) | 19 of 63 rows floored | 39 of 42 | **101 of 108** | | `jev-readout` (arm 1's adapter) | 5 of 63 | 7 of 42 | **0 of 108** | | **`openjev-fp8-readout` (arm 2)** | **0 of 53** | 5 of 35 | **0 of 108** | OpenJev needs the floor on **neither gated task, not once** — cleaner than the adapter on (a) (0 rows against 5) and equal on (c). The 5 floored rows on (b) are the long-tail case the protocol was always going to have: pages with up to 25 section options, where twenty slots cannot hold them all. **This is the one number that says "built as a decision model" rather than "told to behave like one"**, and it is why its probabilities are usable where the arm-1 base's were an artefact of the floor. Its calibration is honest too, and better than anything in arm 1: on (c) the readout's top bin holds 88 of 108 rows at 0.987 claimed confidence against 96.6 % actual — a gap of **+0.021**, where arm 1's adapter was +0.044 and arm 1's base +0.153. The two rows it got wrong while claiming under 0.6 are the only badly-calibrated cells on the page, and there are two of them. ## R2.3 — The readout's saving is real, and only visible where prefill is not Arm 1 could not test this: its ~870 ms of ollama per-request overhead swamped the difference, and reading instead of writing was worth 0.1–2.8 % (§R2). On vLLM, the same pairing — same model, same prompt, same cards, differing **only** in whether the letter was read or written — gives: | task | mean prompt | paired items | readout | generate | the readout's saving | |---|---|---|---|---|---| | (a) | 7,052 tokens | 53 | 6.458 s | 6.501 s | 0.5 % | | (b) | 7,470 tokens | 35 | 7.643 s | 7.658 s | 0.3 % | | **(c)** | **290 tokens** | **108** | **0.350 s** | **0.392 s** | **10.5 %** | and in energy on (c), **167.9 J against 189.9 J per decision — 11.6 % less**. **The rule this run establishes is a ratio, not a percentage.** Reading the answer saves exactly what the extra decode tokens would have cost and nothing else; whether that is noticeable depends entirely on how much prefill it is set against. On a 290-token decision it is a tenth of the bill. On a 7,000-token page it is a rounding error. **A decision model earns its keep on short states asked often — which is precisely the shape of the estate's routing, triage and judging work, and precisely not the shape of the docent's 7,000-token pages.** ## R2.4 — The readout and the written letter are the same decision, again Within arm 2 the two arms agree on **every single row of every task**: 53/53, 35/35, 108/108, and their (c) confusion matrices are identical cell for cell. Arm 1 found the same thing within its adapter. Two models, two runtimes, two quantisations, 852 + 426 decisions: **a readout is not a cheaper, worse answer; it is the same answer with a probability attached.** The robustness argument from arm 1 §R6 repeats in the negative, too — arm 1's `base-generate` refused the form once, writing prose instead of a letter. **Arm 2's generating arm never did: 0 unparsed in 196 written decisions.** A better model writes the letter you asked for; a readout cannot fail that way at all. The first is a property you re-earn with every model swap; the second is structural. ## R2.5 — Where arm 2 and arm 1 disagree, and what that is worth | task | arm 1's adapter vs arm 2 | what it means | |---|---|---| | (a) | **53 / 53, 100 %** | a saturated task two good models cannot disagree on | | (b) | **20 / 35, 57.1 %** | the largest disagreement in either run | | (c) | **91 / 108, 84.3 %** | and arm 2 is right 10.2 points more often | **Task (b) is the finding the estate should act on.** Two capable models, given the same page and the same list of that page's own section ids, pick a different section on **43 % of pages** — and *nobody can say which is right*, because the bank carries an expected section for **1 of 63** questions. Arm 1 called 42 hand labels "the single highest-value hour anyone could spend on this instrument"; arm 2 raises the price of not spending it. The estate's own cite decision is the one place where its two best available answers disagree almost half the time and the bench has nothing to grade them with. ## R2.6 — The host bridge is the cost of everything on this box The pre-registration required the PHB price as a number (§A2.8) rather than an adjective, and the number turns out to explain nearly every latency row above. **What NCCL found, in its own words:** `P2P is disabled between connected GPUs 0 and 1`. Two GeForce cards do not do peer-to-peer over PCIe, so every byte of every all-reduce is staged **card → host RAM → card**. Measured ceiling: **1.82 GB/s**, flat from 4,096 tokens upward. Measured floor: **50.2 µs** for a one-token all-reduce, of which the model runs **128 per output token** (2 per layer × 64 layers) — **6.4 ms of wire under every written token, before the model computes anything.** **An estimate, built from two measured numbers and labelled as one.** Multiply each task's mean prompt by 5,120 × bf16, look the resulting message size up in the microbench, multiply by 128: | task | mean prompt | 128 all-reduces cost about | median decision | share that is wire | |---|---|---|---|---| | (a) | 7,052 tokens | ~5.08 s | 6.458 s | **~79 %** | | (b) | 7,470 tokens | ~5.38 s | 7.643 s | **~70 %** | | (c) | 290 tokens | ~0.22 s | 0.350 s | **~62 %** | This is an estimate and not a direct measurement — chunked prefill overlaps some communication with compute, so the true share is somewhat lower. **The server's own numbers bound it from the other side and agree:** vLLM logged a peak prefill throughput of **1,689.7 tokens/s** (median 752.4), and the link alone would cap a 7,000-token prefill at about **1,390 tokens/s** — the observed peak sits just above that, which is what "mostly wire, with some overlap" looks like. **What this licenses, and what it does not.** * It explains the gap to OpenJev's published figures. The card says ~80 ms for a short text decision and 227 ms for a 1.1k-token page, **on one H100** where there is no all-reduce at all. Arm 2 measured 350 ms and 6.5 s. **That gap is the interconnect, not the model and not the quantisation** — the model's accuracy travelled intact. * It does **not** say tensor-parallel was the wrong call: 30.41 GB of weights do not fit one 24 GB board, so TP=2 was not a choice between fast and slow, it was the only configuration in which this checkpoint runs on this box at all. * It **does** say what a third arm would have to change. Not the model, not the runtime, not the quantisation — **the link**. One card with enough memory, or two cards with NVLink, moves every latency row on this page and none of the accuracy ones. ## R2.7 — Energy, and a comparison that does not flatter arm 2 Arm 2 spends **more** energy per decision than arm 1 on every task, and the reason is two boards rather than one: | task | arm 1 adapter readout, J / decision | arm 2 readout, J / decision | net of idle | |---|---|---|---| | (a) | 1,181.3 | **3,521.7** | 464.8 → 1,836.1 | | (c) | 150.7 | **167.9** | 25.8 → 72.6 | On short decisions the two are close — arm 2 is 11 % more energy for a 2.9× faster answer, which is a trade most seats would take. **On long-prompt decisions arm 2 costs about three times the energy**, because two cards sit at ~245 W each for 6.5 seconds while most of that time is a PCIe transfer. This is §R2.6 wearing different units: on this box, a long prompt under tensor-parallel is an expensive way to move data, not an expensive way to think. **Read the idle column before reading the net column.** Both boards idle at 233–298 W with the model resident under vLLM, against 113–130 W for arm 1's single card under ollama; the net-of-idle figures subtract a large and rather variable number, and the raw joules are the sturdier of the two. ## R2.8 — What refused, what was missing, and what is owed * **benchbox has no C compiler.** Not `cc`, `gcc`, `clang`, or anything under `/usr/lib/gcc` — and a lane has no sudo. vLLM's engine died on `Failed to find C compiler`, because triton JITs a C launcher stub per kernel. Closed inside the venv with `ziglang` 0.16.0 plus a shim translating two gcc-isms zig does not take (`-fno-sanitize=all`, and `-l:libcuda.so.1` resolved to an absolute path). Proved on a real triton kernel before the engine was restarted. * **benchbox has no CUDA toolkit either**, and flashinfer's sampler is a JIT CUDA kernel: `Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist`. Closed with vLLM's own switch, `VLLM_USE_FLASHINFER_SAMPLER=0`. **It cannot touch a number here**: every arm samples greedily at temperature 0 and the readout reads raw logprobs, so the sampler only decides which token comes back as text. * **Ten of 63 task-(a) items and 7 of 42 task-(b) items were refused as overlong** against the 16,384-token ceiling, on the server's own `/tokenize` count, before a card spent a second on them. Seven of the docent's 42 pages do not fit; the longest is **80,515 tokens** as OpenJev's tokenizer sees it. Nothing was truncated, and every table prints the denominator it scored. * **The topology in arm 1's §R7 and in this arm's brief is backwards, and the correction is measured.** Sampling `pcie.link.width.current` *under load* (idle readings downtrain and tell you nothing): **card 0 trains to gen 3 × 4 and card 1 to gen 4 × 16.** The four-lane card is card 0 — the one arm 1 ran its entire bench on. `lspci -vv`'s `LnkCap` would settle it in a line and is root-only here. This changes nothing about arm 1's numbers (a single-card arm makes no collective traffic) and everything about how they are explained. * **The ollama unit could not be put back, and this lane did not fabricate one.** `ollama-bench-c0` was **active** when arm 2 found it and was stopped at 18:21:00Z so arm 2 could have card 0, exactly as briefed. At 19:02:01Z, `systemctl --user start` answered **`Unit ollama-bench-c0.service not found`**: the unit file is gone from `~/.config/systemd/user` and from `list-unit-files`. This lane only ever ran `stop`, so the file went between those two stamps by something else. Its 57 GB model store at `~/bench-store/ollama` is intact. **Writing a replacement unit from memory would be worse than leaving it down** — one wrong environment line re-opens D-20260920-133's Vulkan trap, the exact failure arm 1's pin exists to catch. **Owed to the operator: re-create or restore `ollama-bench-c0.service`.** The vLLM server is stopped and both boards read 1 MiB. * **Nothing else refused.** 426 decisions, RC=0, 0 errors, 0 timeouts, 0 unparsed, and no board but the two named ones ever moved. ## R2.9 — The instrument, and how to re-run it The harness that ran hashed **`c9e0b33e967b70a5617708c61e8d873f7b574df87b23ae272df4042df70ead04`** on benchbox, byte-identical to `run.py` in this tree at the commit that adds this section. The frozen kit was verified identical on both boxes by sha256 before the first call. Arm 1's twelve summaries re-score byte-for-byte under the extended `summarise`, checked after every change to it. ```bash # on benchbox, both cards free, the ollama unit stopped # with CC=~/bench-arm2/shim/cc and VLLM_USE_FLASHINFER_SAMPLER=0 in the environment ~/bench-vllm-venv/bin/vllm serve $SNAP --served-model-name openjev-fp8 \ --host 127.0.0.1 --port 8000 --tensor-parallel-size 2 \ --max-model-len 16384 --gpu-memory-utilization 0.90 --enable-prefix-caching \ --max-num-seqs 1 --max-logprobs 64 --limit-mm-per-prompt '{"image":0}' \ --trust-remote-code --gdn-prefill-backend triton --quantization fp8 python3 run.py --arm openjev-fp8-readout --arm openjev-fp8-generate \ --task a --task b --task c --out rows python3 tables_arm2.py rows/ > TABLES-ARM2.md # the PHB receipt, with nothing else on either board ~/bench-vllm-venv/bin/torchrun --nproc_per_node 2 phb_allreduce.py \ --out receipts/phb-allreduce.json ``` Arm 2 adds to the tree: `tables_arm2.py`, `phb_allreduce.py`, `rows/openjev-fp8-*.{jsonl,report.json,watts.csv,idle.csv}`, `rows/census.openjev-fp8.json`, `TABLES-ARM2.md`, `receipts/2026-09-21-arm2-substrate.md`, `receipts/phb-allreduce.json`, `receipts/phb-linkwidth.csv`, `receipts/run-arm2.log`, and the three `receipts/vllm-serve*.log` files — including the two failed starts, which are kept because what a box does not have is a result. **The licence fence, restated where a reader finishes rather than only where one starts:** `openjev/openjev-FP8` is **CC BY-NC 4.0**. Arm 2 is a bench. **These weights never become a seat** — not rulesage, not the docent, not the beat lab, not the cove. What arm 2 licenses is a finding about decision models, and the finding is that the good ones are very good and this box's interconnect is what costs. --- # §Arm 3 — the pre-registration (`openjev/openjev` at BF16 and `openjev/openjev-FP8` natively, one card on inferencebox's largecard) *This section was written and committed **before any arm-3 row was measured** — `git log` is the receipt, and nothing in it is edited afterwards. It sits below arm 2's results because it was written after them; it sits above arm 3's results because it was written before those.* ***What this is, for a reader who scrolled straight here.*** *Arm 1 put a community adapter of a generating seat on one RTX 3090 under ollama. Arm 2 put the real decision model, `openjev/openjev-FP8`, on **two** 3090s under vLLM — tensor-parallel, because 30 GB of weights do not fit a 24 GB board — and found (§R2.6) that **62–79 % of every decision was the host bridge between the two cards**, not the model. Arm 3 takes the link away. It puts the same model on **one** card big enough to hold it whole — inferencebox's RTX PRO 6000 Blackwell, 97,887 MiB, the board this estate calls **largecard** — twice: once at the published **BF16** weights (54.73 GB) and once at the **FP8** file (30.41 GB), **natively**, because sm_120 has FP8 arithmetic that sm86 does not. It runs inside a **45-minute planned maintenance window** on a live box, and that window is part of the pre-registration.* **Pre-registration stamped 2026-09-21T19:26:50Z** · box **inferencebox** · card `GPU-25bc3288-319a-790c-a14e-82acdb8d15b8` (PCI 01:00.0, **sm_120**, 97,887 MiB, cap **420 W**) · **one card, `--tensor-parallel-size 1`, no all-reduce, no link.** inferencebox's other board — the RTX 3090 `the 3090 board`, `GPU-fde82ed0-be70-eb1f-2d4d-5538c5b82166` (PCI C1:00.0) — is **not named by either arm and is not stopped**: it carries the mistral seat on :11435, pinned resident, and it stays up through the whole window. ## A3.1 — The two questions arm 3 asks, and they are a pair > **(1) FP16 against FP8, on the same card, same protocol, same items.** Does the precision > the model was published at answer differently from the FP8 file that fits in a third less > memory — in accuracy, in calibration, in latency, in joules? > > **(2) What did arm 2's link cost?** Arm 2 measured **0.350 s** per six-way decision across > two 3090s over a host bridge with no peer-to-peer. The same model, the same 108 items, on > **one** card with no collective at all, is the other half of that subtraction. Arm 2 could > only *estimate* the link's share (§R2.6, ~62 % on task (c), built from a microbench and > labelled an estimate). **Arm 3's FP8 number turns that estimate into a measured difference** > — one that still mixes two variables (a different, much faster card *and* no link), which is > said here rather than discovered later. **What arm 3 cannot answer, named now:** it does **not** isolate the link by itself, because the card changes too. A clean link-only reading would need the same 3090 pair with NVLink, or one 3090 holding the whole model — neither exists in this estate. So arm 3 reports its own number and the difference from arm 2, and calls the difference *card-and-link*, never *link*. ## A3.2 — The two models, and the licence that bounds both | | BF16 arm | FP8 arm | |---|---|---| | repository | `openjev/openjev` | `openjev/openjev-FP8` | | **revision** | **`5ec9e5fd2f80a6fff386779b1e5ac7e389971889`** | **`4ec320f267401e67c9be04d5df1be4d2b6b64f10`** | | size on the hub | **54.73 GB**, 12 safetensors shards, 34 files | **30.41 GB**, 12 shards, 22 files | | precision | bfloat16 as published, **no `--quantization`** | FP8 e4m3, 128×128 weight blocks, dynamic activations | | architecture | `Qwen3_5ForConditionalGeneration`, `model_type: qwen3_5`, 64 layers, hidden 5,120 | the same model, quantised | | tokenizer | `tokenizer.json`, 19,989,325 bytes — **byte-identical across the two repos**, so the two arms tokenize every item the same way | same | | **licence** | **CC BY-NC 4.0** | **CC BY-NC 4.0** | | served-model-name | `openjev-bf16-largecard` | `openjev-fp8-largecard` | The FP8 revision is **the same one arm 2 measured** (`4ec320f2…`), so the FP8 arm is the same bytes on a different card and not a different checkpoint. **The licence is a fence, not a footnote, and it binds arm 3 exactly as it bound arm 2. BOTH repositories are CC BY-NC 4.0 — research and non-commercial use only. THIS ARM IS BENCH-ONLY. These weights never become a seat**: not the docent, not rulesage, not amble, not the long table, not the beat lab, not the cove, nothing a visitor or a customer can reach, now or later. The window this arm runs in **stops** seats; it never feeds one. If arm 3 passes its gate, what that licenses is *a finding about decision models* — never this checkpoint in production. ## A3.3 — What arm 3 keeps identical to arm 2, and what it cannot **Identical**, and nothing rebuilt: the frozen kit (`kit/task_a.json` 63, `task_b.json` 42, `task_c.json` 108, the same `state_sha256` per page, the same option order, the same seed 20260921), the same prompt builder (`run.py::readout_prompt`), the same calibration constants unchanged (`READOUT_T = 0.85`, `READOUT_NOUL_T = 1.829074`, `READOUT_NOUL_BIAS = 0`), the same `FLOOR = -30`, the same `top_logprobs = 20` read at the **first output position**, the same `chat_template_kwargs={"enable_thinking": false}`, `--max-model-len 16384`, `--max-num-seqs 1`, `--enable-prefix-caching`, concurrency 1, **vLLM 0.29.0 on Python 3.12** — the same pin, so the two boxes' runtimes match rather than merely rhyme. **Cannot be identical, and each is named rather than absorbed:** 1. **A different card, and a much faster one.** RTX PRO 6000 Blackwell (sm_120) against two RTX 3090s (sm86). Every latency and energy row of arm 3 read against arm 2 is a **cross-card** comparison as well as a cross-topology one, and every table that does it says so in its own header. 2. **One board, not two.** Energy is **largecard's own watts × seconds by UUID**, integrated by trapezoid from a 1 Hz sampler, raw and net of the idle draw read for ten seconds immediately before each task. Arm 2's figures are the **sum of two boards** and must never be compared to arm 3's without that sentence. 3. **A different power cap.** largecard caps at **420 W**; benchbox's boards were capped at 250 W by the operator's own paste. The cap is read from the board and recorded per row — it was a literal `250` inside the harness until this arm, and that literal is now an arm field (`Arm.power_cap_w`), because a number that is right on one box is a lie on the next. 4. **The idle reading is not an empty board.** The reranker seat (`rerank-seat-7997`, a MiniLM cross-encoder, **954 MiB** measured) **stays resident on largecard through the window** by the operator's ruling — rulesage degrades gracefully without it and it is small. So every idle reading, and therefore every net-of-idle joule, is taken **with that seat resident**. The raw joules are the sturdier of the two figures, as arm 2 also found (§R2.7). 5. **Neither of benchbox's two workarounds is needed, and their absence is a receipt.** inferencebox has **gcc 15.2.0** and **nvcc 13.1** (`/usr/local/cuda/bin/nvcc`), so triton's C launcher stub and flashinfer's JIT both have what they wanted and benchbox's `ziglang` shim is not used. **`VLLM_USE_FLASHINFER_SAMPLER=0` is set anyway, deliberately**, for two reasons stated before the run: it keeps the sampling path byte-identical to arm 2, and a cold flashinfer JIT inside a 45-minute window is minutes this arm does not have. It cannot touch a number — every arm samples greedily at temperature 0 and the readout reads raw logprobs, so the sampler only decides which token comes back as *text*. 6. **Arm 1's adapter and its base are not re-run, and arm 2's generating control is not repeated.** Arm 3 runs **readouts only**, two of them. Arm 2 settled the readout-versus- generation question on this model twice over (`53/53`, `35/35`, `108/108` identical choices, §R2.4); spending the window re-proving it would cost the gated task its margin. That is a deliberate narrowing and it is why §A3.6's gate reads on the readout alone. ## A3.4 — The serving posture, exactly One card, by UUID, bound to loopback. `$SNAP` is the snapshot directory under `~/hf-cache/hub/models--openjev--openjev{,-FP8}/snapshots/`. ```bash # common to both arms export HF_HOME=$HOME/hf-cache CUDA_HOME=/usr/local/cuda export PATH=/usr/local/cuda/bin:$PATH export CUDA_DEVICE_ORDER=PCI_BUS_ID export CUDA_VISIBLE_DEVICES=GPU-25bc3288-319a-790c-a14e-82acdb8d15b8 export VLLM_USE_FLASHINFER_SAMPLER=0 # THE BF16 ARM — no --quantization at all ~/bench-vllm-venv/bin/vllm serve $SNAP_BF16 --served-model-name openjev-bf16-largecard \ --host 127.0.0.1 --port 8000 --tensor-parallel-size 1 --dtype bfloat16 \ --max-model-len 16384 --gpu-memory-utilization 0.90 --enable-prefix-caching \ --max-num-seqs 1 --max-logprobs 64 --limit-mm-per-prompt '{"image":0}' \ --trust-remote-code --gdn-prefill-backend triton # THE FP8 ARM — the same everything, the checkpoint's own precision ~/bench-vllm-venv/bin/vllm serve $SNAP_FP8 --served-model-name openjev-fp8-largecard \ --host 127.0.0.1 --port 8000 --tensor-parallel-size 1 --quantization fp8 \ --max-model-len 16384 --gpu-memory-utilization 0.90 --enable-prefix-caching \ --max-num-seqs 1 --max-logprobs 64 --limit-mm-per-prompt '{"image":0}' \ --trust-remote-code --gdn-prefill-backend triton ``` * **127.0.0.1 only.** Neither server is ever bound to the a private network address the seats use. * **`--gpu-memory-utilization 0.90`** = 88,098 MiB of largecard's 97,887. vLLM V1 refuses to start when `total × util` exceeds what is FREE, so this is checked against the pre-flight reading (§A3.8) and not against the card's total. * **THE KERNEL LINE IS A REQUIRED RECEIPT.** Arm 2 recorded `Selected MarlinFP8ScaledMMLinearKernel for Fp8LinearMethod` — **weight-only**, because sm86 has no FP8 arithmetic, so arm 2 measured FP8's *memory* and never FP8's *math*. On sm_120 the server should select a **native block-scaled FP8 kernel**. Whichever line it prints is copied **verbatim** from the startup log into `receipts/`, alongside the attention backend and the GDN prefill/decode lines. **A Marlin line here is a finding, not a detail** — it would mean arm 3's pair differs in memory and not in arithmetic, and every sentence about FP8 on this card would have to say so. * **Residency is proved on largecard by memory GROWTH read by UUID**, before and after, by `verify_pin`, which refuses in both directions. `the 3090 board` is expected to sit unchanged at its mistral residency: it is not named, and `verify_pin` reads growth, so a board that stays resident is quiet by construction. **A board that GROWS mid-arm means that seat loaded a second model** — reported as a pin note in the report, never absorbed. * **No home-brew fallbacks.** If a start refuses or OOMs, the only alternatives this arm may try are vLLM's own (a narrower `--gpu-memory-utilization`). Hand-rolled offload or sharding is out of bounds; the arm reports a refusal instead. ## A3.5 — The context ceiling and the coverage it fixes **`--max-model-len 16384`**, arm 2's figure and the model card's own stated request limit, kept so the coverage table is comparable row for row. The refusal policy is arm 2's §A2.5 unchanged: an item whose prompt exceeds the served context is recorded as **`refused-overlong`** with its measured token count from the server's own `/tokenize`, excluded from every rate, and **never truncated**. Because the tokenizer is byte-identical to arm 2's, the expected coverage is arm 2's measured coverage and not an estimate — a census runs before the first row and any departure from this table is itself a finding: | task | reached, expected | refused as overlong | scored against a label | |---|---|---|---| | (a) does the article answer this question? | 53 of 63 | 10 | 53 | | (b) which section is the answer in? | 35 of 42 | 7 | 0 — unlabelled, as in arms 1 and 2 | | **(c) which of the six guests said this line?** | **108 of 108** | **0** | **108** | **The gated task is untouched by the ceiling**, which is why the gate is where it is. ## A3.6 — THE GATE, pre-registered > **Each of the two arms — `openjev-bf16-largecard` AND `openjev-fp8-largecard` — must reach > ≥ 91.5 % on task (c)**: arm 2's readout scored **93.5 %** (101 of 108) on the same 108 items > and the margin is **two points**. On 108 items that floor is **≥ 99 correct**. > **An arm that does not reach it is REJECTED and gets no threshold table.** The two arms are > gated **independently**: one may pass while the other fails, and that result — if it happens — > is the most interesting thing this arm could produce. > **Latency, energy, the kernel line and the coverage table are reported either way.** > > The floor is **derived from `rows/openjev-fp8-readout.c.report.json`** by the scoring script > at read time, never typed into it — the house law: derive from state, never type the number. Two tasks are deliberately outside the gate, each for a reason arms 1 and 2 established: * **(a) is reported, not gated.** Every arm of both previous runs answered every reached item correctly — 63/63 and 53/53. A gate both sides clear perfectly discriminates nothing (§R1, §R2.1). It is reported for coverage and for the abstain split, and a **fall** on it would be a finding worth stating — but it cannot earn a pass. * **(b) is not scored at all.** The bank carries an expected section for 1 of 63 questions. Cost, stability and cross-arm agreement only. **Arm 2 raised the price of the 42 missing hand labels** (§R2.5: two capable models disagree on 43 % of pages and nothing can grade them); arm 3 adds two more opinions to that disagreement and still cannot settle it. ## A3.7 — What is measured, per arm × task Arm 2's list with one board where arm 2 had two: accuracy on the scored denominator · multi-class Brier · a 10-bin reliability table · median and p95 seconds per decision · prompt and written tokens · **energy as largecard's own watts × seconds by UUID, integrated by trapezoid from a 1 Hz `nvidia-smi` sampler**, raw and net of the idle draw read for ten seconds immediately before each task (with the reranker seat resident, §A3.3.4) · residency by growth · `refused-overlong` counts · `letters_in_topk` and floored letters per row · the raw per-letter logprobs, so any temperature can be refitted later without spending a GPU second. **Task order is `c`, then `a`, then `b` — and the order is part of the pre-registration.** The gated task runs FIRST, so a window that overruns loses the ungated tail rather than the verdict. `a` before `b` preserves arm 2's adjacency, so the two share the same prefix-cache warmth they shared there. **(b) is declared droppable**: if the clock says it will not finish inside the window, it is **skipped and declared skipped**, never hurried. ## A3.8 — THE WINDOW, and it is part of the instrument **2026-09-22 04:00Z → 04:45Z** (the same forty-five minutes on the bench's own local clock), ruled by the operator. This arm runs on a **live 24/7 box**, so the window is pre-registered exactly as the protocol is. **What stops, and what it carries** (the unit list is read from the box in `RUNBOOK-ARM3-OUTAGE.md` §2, with the VRAM each holds): | unit | board | what it carries | in the window | |---|---|---|---| | `docent-vllm.service` | largecard | the research hub's assistant seat on :8001 (gemma-class FP8, 131,072 window) **and all four of the long table's gemma roles** — chairs, doorman, checks, MC | **STOPPED** | | `ollama.service` (its default port) | largecard | `gemma4:26b` — rulesage's answerer and amble's guide — plus `nomic-embed-text` | **STOPPED** | | `inferencebox-warm-seats.timer` | — | re-pins the residents every 5 min and **`Wants=ollama.service`**, so a tick inside the window would restart the seat under the bench | **STOPPED FIRST** | | `rerank-seat-7997.service` (user) | largecard | the rulesage reranker, **954 MiB** | **STAYS UP** — operator's ruling; rulesage degrades gracefully without it and the FP16 arm still fits beside it | | `ollama-11435.service` (:11435) | the 3090 board | `mistral-small3.2:24b` + `nomic-embed-text` — rulesage's moderation classifier, the beat lab's genie-mod, the long table's MISTRAL_CHAIRS | **STAYS UP** | | `estate-inference-fence.service` | — | the nft fence keeping the inference ports private-network-only | **STAYS UP** — it touches no board | **Every static page stays up throughout.** What refuses for ~40 minutes is the gemma-class answering: the hub's assistant, rulesage's answerer, amble's guide, and the long table's gemma chairs. **The quartermaster's door has no failover member and fails CLOSED by construction — its refusal rows ARE the window's record**, and it gets no flag. scribe's and gloss's timers retry on their own. **The free-VRAM pre-flight, and the bench does not start without it.** Read by UUID immediately after the stops: ``` nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv ``` must show **exactly one compute app on largecard** — the reranker's `/workshop/rerank-env/bin/python` at ~954 MiB — **and no vLLM and no llama-server**, with `memory.used` on `GPU-25bc3288…` at **≤ 2,000 MiB**. That leaves **≥ 95,800 MiB** free against the BF16 arm's need of 54.73 GB of weights plus its KV pool. **If any other process holds largecard, the bench does not start and the window is spent putting the seats back** — that is a refusal, not a problem to work around. ## A3.9 — What counts as a refusal, decided now rather than later So that nothing is chosen after seeing a number, the arm **STOPS and reports** — it does not improvise — on any of these, each reported with the exact output that produced it: 1. **The pre-flight fails**: any process but the reranker holds largecard after the stops. 2. **A server does not answer `/v1/models` inside its slot** (§ the runbook's clock): the BF16 arm by **04:14Z**, the FP8 arm by **04:32Z**. 3. **vLLM refuses a checkpoint** — the architecture, the precision, or the GDN prefill backend. 4. **An OOM**, after the one in-bounds alternative of §A3.4 (a narrower `--gpu-memory-utilization`). 5. **The pin fails**: largecard did not grow, or another board did. 6. **The clock reaches 04:35Z.** Whatever has been measured is what arm 3 has. **THE SEATS COME BACK AT 04:45Z WHATEVER HAPPENED** — that is the window's only hard law, and no measurement outranks it. 7. **A seat does not come back.** The arm's report is incomplete until every stopped unit is active and its health receipt is read (`RUNBOOK-ARM3-OUTAGE.md` §6), and a seat that will not return is escalated to the operator with its own journal tail rather than retried blind. **Disk, stated before it is spent:** inferencebox had **917 GB free on /** before this arm's preparation. The two checkpoints are 54.73 + 30.41 = **85.14 GB**, the venv is **7.6 GB**, and nothing is deleted that this arm did not create. --- ## Addenda arms 4–11 (2026-09-21, D-20260921-215) *Written and committed BEFORE the first addenda row was measured; `git log` is the receipt. Arms 4–9 were commissioned with the ruling; arms 10 and 11 were added by the orchestrator at 2026-09-21T20:38Z, before any addenda call, and are pre-registered here on the same terms. Nothing in this section was edited after its arm's first call.* **The server.** Arms 4, 5, 6, 7 run on arm 2's server, unchanged and restarted for the addenda at **2026-09-21T20:35:49Z**, ready **20:38:52Z**: `serve.sh` verbatim — vLLM 0.29.0, `openjev/openjev-FP8` rev `4ec320f2`, TP=2 across both benchbox 3090s at 250 W, `--max-model-len 16384`, `--gpu-memory-utilization 0.90`, `--max-num-seqs 1`, `--max-logprobs 64`, `VLLM_USE_FLASHINFER_SAMPLER=0`. Arms 8, 9 and 10 restart it with exactly one flag changed each, named in their sections. Arm 11 trains on the cards with no server at all. **What every addenda arm keeps from arm 2**, so its rows are comparable row for row: the prompt builder (`readout_prompt`), the letter protocol, `READOUT_T = 0.85`, `FLOOR = -30`, `top_logprobs = 20`, the untargeted (old) path, `chat_template_kwargs={"enable_thinking": false}`, `temperature 0`, seed 20260921, concurrency 1 (except arm 8, whose whole subject is concurrency), and the 1 Hz `nvidia-smi` sampler on BOTH boards by UUID with a 10-second idle read before each arm. Every row keeps: item id, prompt sha256, the per-letter raw logprobs, the floored letters, the chosen option, the label, seconds, prompt and eval tokens. **The standing refusal rules, restated and binding on all eight arms.** (1) **No rate for an unlabelled set** — an arm whose items carry no label reports agreement or answers, never accuracy. (2) A refusal — OOM, overlong, malformed, a server that will not start — is **written as a row with its reason and its measured number**, is excluded from every rate, and is never truncated, retried differently, or worked around by a repair invented mid-run. (3) **No home-brew fallbacks**: an arm that cannot run as written reports a refusal and stops. (4) **A dead arm gets no numbers** — not a threshold table, not an estimate, not a figure carried over from a neighbouring arm. **The runner.** A new `run_addenda.py` beside `run.py`, which it IMPORTS rather than forks: the arms, `decide`, `readout_prompt`, `Boards`, `verify_pin`, `summarise`, `refusal` and `run_arm` are arm 2's own code, and the addenda add only new task generators registered into `run.py`'s `TASKS`. `run.py` is not edited, so arms 1–3's record stays exactly as it was measured. Arm 8's concurrency driver (`run_arm8.py`) and arm 11's training driver (`run_arm11.py`) are separate because neither fits `run_arm`'s one-at-a-time shape. Tables go to `TABLES-ADDENDA.md` via a new `tables_addenda.py` (documented there); `tables_arm2.py` is not edited. Rows to `rows-addenda/`, receipts to `receipts/`. --- ### Arm 4 — the doorman's planted set **What is measured.** Whether OpenJev-FP8, read as a two-letter decision, makes the long table's *admission* decision as well as the gemma doorman that actually makes it. **The set.** `~/estate/bench/doorman-planted-2026-09-13/` — 36 planted hostile visitor lines + 12 controls = 48 items, with the planted/control class and the recorded verdict of the gemma doorman (DOORMAN_SYSTEM v1.2, guided JSON) frozen in `out/verdicts.jsonl`. The exact counts are re-verified from the file before the arm runs and the read count is reported; a count that is not 36/12 is reported as found rather than assumed. **The prompt.** The DOORMAN_SYSTEM v1.2 text, quoted verbatim from the long table's own `doorman.py` (its file and version string recorded in the report), is the `instructions`; the visitor line is the `state`. The readout letters are **[A] admit / [B] refuse**, in that fixed order for every item. **The labels — there are two, and they are not the same thing.** (1) The **planted label**: planted → refuse, control → admit. This is the set's designed ground truth and accuracy against it is a rate. (2) **Agreement with gemma's recorded verdict** — this is an agreement, NOT accuracy, and is reported under that word; gemma's verdict is another model's judgment. **Reported.** Refusals caught of 36; controls admitted of 12; the minimal pairs; accuracy and Brier against the planted label; agreement with gemma; latency; watts; J per decision — and **gemma's own numbers from `out/summary.json` quoted beside OpenJev's**. **Gate.** None — descriptive. **Refusal conditions:** a verdict file whose label field is absent or not binary (report the shape, run nothing); a DOORMAN_SYSTEM that cannot be found or whose version is not v1.2 (report which version was found, and quote it, rather than substituting a paraphrase); any item whose prompt exceeds 16,384 tokens (`refused-overlong`). --- ### Arm 10 — the base model on the SAME runtime **What is measured.** OpenJev's 93.5 % on task (c) has never been read against its base on the same runtime: arm 1's base was ollama-only (§A2.3.4), so every OpenJev-vs-base comparison so far confounds the model with the server. Arm 10 removes the server from the comparison. **The question: is the gap the model, or the runtime?** **The model.** The estate's assistant seat's own checkpoint, `gemma-4-26B-A4B-it-FP8-dynamic`, copied **read-only** from inferencebox (the only inferencebox touch in the whole addenda, and it is a read) to `benchbox:~/models/`. Served with arm 2's flags, TP=2, 16,384 window, `--served-model-name gemma4-fp8-vllm`; the exact command and the startup line naming the quantisation kernel are recorded verbatim in the report. **THE DISK CONSTRAINT, AND THE ORDER IT FORCES — recorded before the arm, not after.** Measured on benchbox at 20:39Z: **16 GB free**, the OpenJev FP8 cache **29 GB**, the gemma4 checkpoint **~30 GB**. *They cannot both be resident on disk* — the same constraint §9 records for arm 1's two models. Arm 10 therefore runs **after every OpenJev arm (4, 7, 5, 6, 8, 9)**, not immediately after arm 4 as commissioned, and the OpenJev cache is evicted to make room for it. The eviction is recoverable (the checkpoint re-downloads from HF) and is recorded with its stamp. **If the copy from inferencebox cannot complete before inferencebox's 04:00Z outage window, or the free space after eviction is still short of the checkpoint, arm 10 is a REFUSAL with the measured byte counts — not a substituted smaller model and not a shortened set.** **The tasks.** (c), (a) and (b), readout + generate — the same kit, letters, instructions and calibration as arm 2, so the rows table beside arm 2's. Reported per task: accuracy (where labelled), Brier, median/p95 latency, both cards' mean W, J per decision. Task (b) stays **unlabelled and ungated** (§3) for this arm exactly as for every other. **The thinking channel.** Gemma 4's template opens on a thinking channel (§4.3). Arm 10 sends the same `chat_template_kwargs={"enable_thinking": false}` every other vLLM arm sends; the report records the position-0 token actually observed, so a template that ignored the flag is visible rather than inferred. This is the vLLM twin of arm 1's ollama `think: false`. **Gate.** None — descriptive. **Refusal conditions:** the copy does not fit or does not finish; vLLM refuses the checkpoint or TP=2 OOMs (the §A2.4 in-bounds alternatives only, then refusal); a chat template that will not close its thinking channel (reported with the observed position-0 token, and the arm reports what it measured rather than silently comparing a channel marker against a letter). --- ### Arm 7 — task (c), name-blind **What is measured.** How much of arm 2's **93.5 %** on the six-voice task is the model recognising a *voice*, and how much is it reading an *address* — the lines in which one guest names another ("To Hypatia, I offer…"). **The set.** task (c)'s 108 items, unchanged in every respect but one: every one of the six guests' names and their obvious surface forms is removed **from the LINE text only** and replaced with the single word `someone`. The forms are `build_kit.py`'s own `NAME_TOKENS` (darwin/charles, hypatia, ibn sina/avicenna, sagan/carl, socrates/socratic, einstein/albert) plus the full roster names, matched case-insensitively on word boundaries. **The six options are NOT scrubbed** — they stay the six names, in the same fixed alphabetical order. The state line is unchanged. The number of lines whose text actually changed is reported, and the scrubbed kit is written out with a sha256 so the arm is re-runnable byte for byte. **Comparator and floor.** Arm 2's 93.5 % on the same 108 items, and the elimination floor. Because the options remain six and the scrub removes information rather than options, **the floor stays chance at 16.7 %** — arm 2's floor, unchanged, and it is stated rather than recomputed downward. **Reported.** Readout + generate accuracy, Brier, the delta from 93.5, latency, watts, and the accuracy split between the lines that changed and the lines that did not — the second being the arm's real payload: an unchanged line is its own internal control. **Gate.** None — descriptive. **Refusal conditions:** none expected (the set is arm 2's and fits); a scrub that changes zero lines is reported as such and the arm is declared uninformative rather than run to produce a duplicate of arm 2. --- ### Arm 5 — the field exam's stay-grounded task **What is measured.** Whether the readout can make the field exam's *stay-grounded* call. **The set.** `~/estate/bench/field-exam/`, `items/t*.json` with `task == "stay-grounded"` (20 items), against their own `expected` field, read under the definition of a correct answer in that bench's `PREREG.md` (quoted verbatim in the report). **THE PRE-REGISTERED FORK, decided now rather than after seeing a number.** `expected` is read FIRST, before any call: * **If it is a decision** — answerable from the excerpt as yes/no, or as abstain-vs-answer, or any small closed set — it is cast as a letter readout over exactly that closed set and scored against `expected` as a rate. * **If it is free text and not a decision**, the arm **runs nothing**, writes the reason and the observed shape of `expected`, and reports no number. A free-text expectation graded by a model is un-judged model text grading model text, which this bench has already ruled out (§3(b)). **grounded-qa, on the same fork.** Its 20 items are cast as "does the excerpt contain the answer?" **only if** `expected` yields a clean binary label. If it does not, the set is skipped with the reason stated and no rate. **Gate.** None — descriptive. **Refusal conditions:** the fork above; a count that is not 20 (reported as found); an excerpt over the 16,384 window (`refused-overlong`). --- ### Arm 6 — the judge-seat set **What is measured.** Whether the readout makes the judge seat's call. **The set.** `~/estate/bench/judge-seat-2026-09-09/out/judgeset-v2.json` (119 items), read through that bench's own `harness/` to learn what the judge decides and over what closed set. **THE PRE-REGISTERED THREE-WAY FORK, decided now rather than after seeing a number.** * **If items carry a human or otherwise fixed ground-truth label** and the judgment is a closed multiple-choice decision → cast it and score it as a rate. * **If the only available "label" is another model's judgment** (e.g. the `judge3-J-mistral-*` files) → report **AGREEMENT ONLY**, in those words, and state plainly in the table and in the prose that **this is not accuracy**. No Brier against it, no gate, no rate. * **If neither** — no closed set, or no label of any kind → **skip, with the reason**, and report no number. **Gate.** None — descriptive. **Refusal conditions:** the fork above; items over the window (`refused-overlong`). --- ### Arm 8 — concurrency **What is measured.** What the readout costs per decision when the server is allowed to batch. **The server.** Restarted with **`--max-num-seqs 8`** and every other flag identical to arm 2's. The restart, its stamp and its startup line are receipts. **The run.** Task (c) readout — the same 108 items, the same prompts — at **1, 4 and 8 concurrent streams** (asyncio), each level a complete pass over all 108. **Reported per level.** Median and p95 seconds per decision, **decisions per second**, accuracy, both cards' mean W, and J per decision. **The finding condition, pre-registered.** Accuracy at every level **must equal arm 2's 93.5 %** — the items, prompts, temperature and seed are identical and batching must not move a greedy readout. **A change is a FINDING and is reported as one**, with the differing item ids named, not smoothed into an average or attributed to noise. **Gate.** None — descriptive. **Refusal conditions:** the server will not start at `--max-num-seqs 8` (recorded with its own error); an OOM at a concurrency level (that level is a refusal row with its measured number; the completed levels still report). --- ### Arm 9 — the long pages **What is measured.** The 17 items arm 2 could not reach — **10 of task (a) and 7 of task (b)**, refused as overlong against the 16,384 window (§R2.8) — at a window wide enough to hold them. **The server.** Restarted with **`--max-model-len 32768`**, every other flag identical. **Pre-registered fallback, and it is the only one:** if it refuses to start for KV-cache room at `--gpu-memory-utilization 0.90`, try **0.95 exactly once**; if that also refuses, the arm is a **REFUSAL** recorded with the server's own sentence and both attempts' numbers. No third value, no narrower window, no smaller set. **The run.** ONLY those 17 items, readout + generate, identified by the ids in arm 2's own refusal rows so the set is arm 2's and not a fresh selection. **Reported.** Task (a): accuracy against its labels over the 10 items, stated with n = 10 beside it. Task (b): **the answers only — no rate**, because (b) is unlabelled (§3). Their measured prompt tokens, latency, watts, J per decision. An item that is *still* overlong at 32,768 is a `refused-overlong` row with its new measured count. **Gate.** None — descriptive. --- ### Arm 11 — training the estate's own decision model **What is measured.** Whether the estate's own labels, trained with the Apache-2.0 openjev framework on benchbox's own cards, beat a zero-shot frontier-ish decision model on the estate's own decisions. This is the arm that asks whether any of this needs a 26B model. **The framework.** `https://github.com/S1LV3RJ1NX/openjev` (Apache-2.0 — a *fine-tuning framework*, distinct from the CC BY-NC `openjev/openjev-FP8` **weights** the other arms bench, whose licence fence is unchanged: bench-only, never a seat). Cloned to `benchbox:~/bench-ownjev/` with its own uv venv on Python 3.12; **the commit sha is pinned in the report before training**. The **ModernBERT-base encoder** variant trains first (the cheap one); the Qwen3-1.7B + LoRA decoder variant runs **only** if the encoder trains cleanly and time remains. **The split, fixed before training.** Item-level, **seed 20260921**, **70/30 per task, stratified by label**. The held-out 30 % is written out with its ids before the first training step and **is never trained on** — the split file is a receipt and the training config is asserted against it. **The labels** (all from `~/estate/bench`): task (c) 108 six-way; task (a) 63 binary; the doorman planted set 48 binary (arm 4's); the field exam stay-grounded 20 (arm 5's, **only if arm 5 cast cleanly** — if arm 5 refused, this task is not in arm 11 either). **The metric.** Accuracy + multi-class Brier on the held-out 30 %, per task. **The three comparators, on the SAME held-out items** — filtered to the held-out ids, never re-run and never compared against a full-set figure: (1) **OpenJev-FP8 zero-shot**, from arm 2 and arm 4's own rows; (2) **the base gemma4**, from arm 10's rows; (3) the **elimination/majority floor** for that task. **Reported per task.** Held-out n, accuracy, Brier, training seconds, and the trained model's size on disk. **THE SMALL-n RULE, pre-registered.** **No number for a task whose held-out set is under 12 items** — the report says *"too small"* in the cell and no rate, no Brier, no comparator delta is computed for it. (At 70/30 this binds on the field exam's 20 items, whose held-out set is 6, before any of it is measured.) **Gate.** None — descriptive. **Refusal conditions:** the clone, the venv or the training run fails (recorded with its own error and no number); a comparator whose rows do not cover the held-out ids (that comparator cell is empty with the reason, never back-filled from the full set); inferencebox is not touched by any part of this arm. --- ### Addenda amendment 1 — arms 5 and 6, written before either arm's first call *Stamped 2026-09-21T20:47Z, after reading the two source benches and BEFORE a single call of arm 5 or arm 6. The forks above were pre-registered on the shape of a field named `expected`; this records what that field actually turned out to be in each bench, and which branch of each fork that selects. Arm 4 was already running (started 20:44:05Z); arms 5 and 6 had not been called.* **Arm 6 — fork branch 1 selects: a labelled rate.** `out/judgeset-v2.json` (sha256 `8e0a7c26f53afb5d91cc0626754bbcd310c270eaeb31a9967d02e673fe4c9a1f`, the same sha the judge run files record) carries `expected` on all 119 items, and it is **construction-derived ground truth, not a model's judgment**: `judgeset.py` assigns it per arm at build time (VERBATIM → `grounded`, PARAPHRASE → `grounded`, FIGURE-FLIP and NUMERAL-FABRICATION → `not_grounded`, OFF-PAGE → `off_page|not_grounded`, the `|` an alternation both of whose values score correct). So arm 6 **is scored as a rate**, over the judge's own closed three-value set, with the three recorded model judges (`J-mistral-v2`, `J-mistral-1s`, `J-qwen-1s`) carried alongside as **agreement, which is not accuracy** and is labelled so. Two caveats ride into the report rather than being smoothed away: 39 PARAPHRASE *claim strings* were written by a model (the labels were still assigned by construction), and the judge bench's own `setcheck.txt` flags `item_index 42` as the one label worth reading by hand. **Arm 5 — the fork's condition does not exist, and the amendment says so before it is run.** `expected` **is not a field on any `t1-*` or `t2-*` item**; only the 20 `t3-*` schema-extract items have one. Read literally, the pre-registered fork ("if it is free text and not a decision, run nothing") lands on *run nothing*, because there is no `expected` to be either. But the field exam's own `PREREG.md` §2 states the decision explicitly, and it is a decision: > **T1 and T2 are put to the model with the byte-identical prompt** and live in one > interleaved pool (`t1-*`, `t2-*` are ids for *us*, not a signal to the model). > The model is never told which kind it is looking at. Discriminating answerable > from unanswerable **is** the test. That makes the item's own `task` field a **balanced, construction-derived binary ground truth** — 20 `grounded-qa` (the excerpt answers the question) against 20 `stay-grounded` (it does not) — of exactly the same kind as arm 4's planted/control label, which is likewise derived from a structural field rather than stored under the name "label". **Amendment: arm 5 runs that 40-item binary**, cast as a two-letter readout, and it is named here before its first call rather than chosen after seeing a number. **What arm 5 therefore is NOT, stated so no reader mistakes it.** It is **not** the field exam's own T2 metric. That metric scores free text on three conditions at once — a decline regex, the absence of every `forbidden` token anywhere in the reply, and a ≤ 50-word limit — and a readout emits a letter, not a reply, so it cannot be scored that way and this arm does not claim to be. Arm 5 measures the **decision underneath** that metric: can the model tell an answerable excerpt from an unanswerable one. A comparison of this number against any T2 score in the field exam's own ledger would be comparing two different instruments, and the report says so in the table. **The set, and the letters.** All 40 T1+T2 items; the `excerpt` is the state, the `question` is the question, the letters are **[A] yes (the excerpt answers it) / [B] no (it does not)**, fixed for every item. The floor is the majority-class floor, **50.0 %**, because the set is balanced 20/20 by construction. Refusal conditions unchanged: an excerpt over the 16,384 window is `refused-overlong`; a count that is not 20/20 is reported as found. --- ### Addenda amendment 2 — arm 11's second reading, written before it was run *Stamped 2026-09-21T22:20Z. Arm 11's pre-registered run — the ModernBERT-base encoder trained from the raw backbone — had completed and is reported as measured. This amendment adds a SECOND reading and is written before that reading's first training step.* **What the first reading found.** The encoder trained cleanly on all three sets (rc=0, 9–14 seconds each) and landed **at or barely above the majority-class floor on every one**: longtable_voices 23.3 % against a 16.7 % floor, doorman_gate 73.3 % against a 73.3 % floor, fieldexam_grounded 50.0 % against a 50.0 % floor. That number is kept and reported. **Why a second reading is owed.** The framework's own README names a different command as **"the project's central claim"** — recipe **B**, which warm-starts from a published general checkpoint instead of the raw backbone, and reports *"same data, same time, +36 points"*. The first reading ran recipe **A**. Reporting only recipe A would be measuring a strawman: it would say "this framework cannot learn the estate's decisions" when what was actually measured is "a randomly-initialised scoring head cannot learn them from 24–66 examples", which is a claim about the sample size and about recipe A, not about the framework. **The second reading, exactly.** Identical in every respect to the first — same task dirs, same split file, same seed, same epochs (6), same batch size (8), same card, same leak check — with one flag added: **`--init-from`** pointing at the framework's published general **encoder** checkpoint, `s1lv3rj1nx/openjev-encoder-general`. (The README's recipe-B line names `openjev-general-lora`, which is the *decoder* adapter on a Qwen3-1.7B backbone; `train.py` refuses to init a ModernBERT run from it, and correctly so. The encoder's equivalent is the encoder checkpoint, and that substitution is recorded here rather than made silently.) **Both readings are reported side by side, and neither replaces the other.** Recipe A is what the estate gets from its own labels alone; recipe B is what it gets from its own labels plus someone else's 323,466-row pre-training run. They answer different questions and the table says which is which. **Refusal conditions:** the checkpoint does not download, or `train.py` refuses it on a backbone mismatch (recorded with its own sentence, and the arm keeps reading A only); the held-out floor rule and the leak check are unchanged and still binding.