# The Jev bench — gaps 4 and 5 (2026-09-22)

*What this is, for a reader who scrolled straight here: the Jev bench (`README.md`, benchbox, 2026-09-21) measured a decision model — `openjev/openjev-FP8`, CC BY-NC 4.0, bench-only — against the generating seat on this workshop's own decisions, and its draft article listed what it did not know. Two of those lines were claims on the model's own card. **Gap 4** shuffles the options; **gap 5** switches the model's eyes on. The pre-registration for both, written before any row here existed, is `GAPS-CARD.md`.*

*Drawn by `tables_gaps_card.py` from the row files in `rows-gaps-card`. It calls nothing and invents nothing; every cell is computed from a row that carries its own prompt sha256. Gaps 1–3 are a different lane's `TABLES-GAPS.md` and are not touched here.*

## Gap 4 — reorder consistency

The same **108** six-voice items from task (c), asked **six times**: the kit's own option order (`k = 0`, byte-identical to arm 2's prompts) plus five seeded shuffles, `random.Random(20260921 + k)`. **648 decisions per arm.** The question is whether the chosen **guest** moves — not whether the chosen letter does, which it must.

### `openjev-fp8-readout`

| figure | value | of | what it is |
|---|---|---|---|
| **items whose answer moved at all** | **13.9 %** | 15 / 108 items | the chosen guest is not the same under all six orders |
| **order-pairs that disagree** | **5.68 %** | 92 / 1620 pairs | the 15 pairs of the six orders, per item — **the figure comparable to the card's 2.3 %** |
| mean per-guest probability **range** | 0.0466 | 648 (item, guest) cells | max − min of that guest's probability over the six orders |
| mean per-guest probability **deviation** | 0.0135 | 648 (item, guest) cells | mean absolute deviation from that guest's own six-order mean |
| mean W, both cards | 494.1 | — | the arm's own 1 Hz sampler, summed |
| J / decision (net of idle) | 173.9 (163.1) | 648 decisions | idle read on both boards for ten seconds before the arm (30.8 W) |

**Accuracy, per presentation.** Same items, same labels, only the option order differs.

| order | accuracy | correct | Brier | median s | p95 s |
|---|---|---|---|---|---|
| **k = 0** — the kit's order (arm 2's) | 93.5 % | 101 / 108 | 0.1219 | 0.35 | 0.42 |
| k = 1 | 89.8 % | 97 / 108 | 0.1774 | 0.35 | 0.42 |
| k = 2 | 89.8 % | 97 / 108 | 0.1510 | 0.35 | 0.42 |
| k = 3 | 90.7 % | 98 / 108 | 0.1173 | 0.35 | 0.42 |
| k = 4 | 91.7 % | 99 / 108 | 0.1340 | 0.35 | 0.42 |
| k = 5 | 92.6 % | 100 / 108 | 0.1329 | 0.35 | 0.42 |

*Spread across the six orders: **89.8 % to 93.5 %**, a range of 3.7 points.*

**The k = 0 control, checked rather than asserted.** 108 of 108 prompts sha256-identical to arm 2's, and the same guest chosen on 108 of 108 — a difference there would be server nondeterminism, not the shuffle.

**Positional bias — which letter was chosen, over all 648 presentations.** A content-blind model would sit at 16.7 % in every column.

| letter | times chosen | share | times the CORRECT answer sat there | share |
|---|---|---|---|---|
| A | 78 | 12.0 % | 84 | 13.0 % |
| B | 114 | 17.6 % | 120 | 18.5 % |
| C | 123 | 19.0 % | 113 | 17.4 % |
| D | 102 | 15.7 % | 99 | 15.3 % |
| E | 114 | 17.6 % | 114 | 17.6 % |
| F | 117 | 18.1 % | 118 | 18.2 % |

*The right-hand pair is the schedule, not a result: it is where the correct answer happened to land once the seeds had shuffled. Read the left-hand share against it, not against 16.7 % alone.*


### Reliability — `openjev-fp8-readout`, all six orders

| confidence bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.4–0.5 | 11 | 45.3 % | 36.4 % | +0.089 |
| 0.5–0.6 | 12 | 54.7 % | 50.0 % | +0.047 |
| 0.6–0.7 | 18 | 65.5 % | 61.1 % | +0.044 |
| 0.7–0.8 | 23 | 75.3 % | 69.6 % | +0.057 |
| 0.8–0.9 | 48 | 85.8 % | 79.2 % | +0.066 |
| 0.9–1.0 | 536 | 98.7 % | 96.5 % | +0.023 |

### `openjev-fp8-generate`

| figure | value | of | what it is |
|---|---|---|---|
| **items whose answer moved at all** | **13.9 %** | 15 / 108 items | the chosen guest is not the same under all six orders |
| **order-pairs that disagree** | **5.68 %** | 92 / 1620 pairs | the 15 pairs of the six orders, per item — **the figure comparable to the card's 2.3 %** |
| mean per-guest probability **range** | 0.0478 | 648 (item, guest) cells | max − min of that guest's probability over the six orders |
| mean per-guest probability **deviation** | 0.0158 | 648 (item, guest) cells | mean absolute deviation from that guest's own six-order mean |
| mean W, both cards | 494.5 | — | the arm's own 1 Hz sampler, summed |
| J / decision (net of idle) | 195.4 (89.3) | 648 decisions | idle read on both boards for ten seconds before the arm (268.6 W) |

**Accuracy, per presentation.** Same items, same labels, only the option order differs.

| order | accuracy | correct | Brier | median s | p95 s |
|---|---|---|---|---|---|
| **k = 0** — the kit's order (arm 2's) | 93.5 % | 101 / 108 | 0.1296 | 0.39 | 0.47 |
| k = 1 | 89.8 % | 97 / 108 | 0.2037 | 0.40 | 0.47 |
| k = 2 | 89.8 % | 97 / 108 | 0.2037 | 0.40 | 0.47 |
| k = 3 | 90.7 % | 98 / 108 | 0.1852 | 0.40 | 0.47 |
| k = 4 | 91.7 % | 99 / 108 | 0.1667 | 0.40 | 0.47 |
| k = 5 | 92.6 % | 100 / 108 | 0.1481 | 0.40 | 0.47 |

*Spread across the six orders: **89.8 % to 93.5 %**, a range of 3.7 points.*

**The k = 0 control, checked rather than asserted.** 108 of 108 prompts sha256-identical to arm 2's, and the same guest chosen on 108 of 108 — a difference there would be server nondeterminism, not the shuffle.

**Positional bias — which letter was chosen, over all 648 presentations.** A content-blind model would sit at 16.7 % in every column.

| letter | times chosen | share | times the CORRECT answer sat there | share |
|---|---|---|---|---|
| A | 78 | 12.0 % | 84 | 13.0 % |
| B | 114 | 17.6 % | 120 | 18.5 % |
| C | 123 | 19.0 % | 113 | 17.4 % |
| D | 102 | 15.7 % | 99 | 15.3 % |
| E | 114 | 17.6 % | 114 | 17.6 % |
| F | 117 | 18.1 % | 118 | 18.2 % |

*The right-hand pair is the schedule, not a result: it is where the correct answer happened to land once the seeds had shuffled. Read the left-hand share against it, not against 16.7 % alone.*


### Reliability — `openjev-fp8-generate`, all six orders

| confidence bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 648 | 100.0 % | 91.4 % | +0.086 |

### The card's figure, beside this one — and they are not the same measurement

| reading | set | extraction | reorder rate |
|---|---|---|---|
| OpenJev's model card, tuned | the card's own | targeted (`allowed_token_ids`) | **2.3 %** |
| OpenJev's model card, untuned | the card's own | targeted | 18.5 % |
| **this bench**, `openjev-fp8-readout` | the long table, 108 items, 6 options | untargeted, `top_logprobs = 20` | **see the table above** |

*Quoted, not verified. A rate measured on a different set, with a different number of options, through a different extraction path is a different number that happens to share a name. What this bench can say is what this model did on this workshop's decision.*

**The base model's reorder rate is owed, and was not measured.** The control would be the same six orders on `gemma-4-26B-A4B-it-FP8-dynamic` — arm 10's checkpoint, which is the 27 GB this run deleted to make room for OpenJev (`GAPS-CARD.md` §0). benchbox cannot hold both. So there is no base column here, and the figure above is a rate for one model rather than a comparison between two.

---

## Gap 5 — a screenshot decision

36 screenshots at 1,280 × 800 of public pages this workshop publishes — **24 research-hub article pages** and **12 product front doors** — put to the same checkpoint arm 2 measured, served with **one flag changed**: `--limit-mm-per-prompt '{"image":1}'` instead of `'{"image":0}'`. The letters, the calibration, the temperature and the seed are arm 2's.

> **36 items is a small set and every table below prints its n.** A reading here says what this model did on these 36 pictures; it decides nothing about screenshot decisions in general.

| arm | decision | accuracy | correct | floor | Brier | median s | p95 s |
|---|---|---|---|---|---|---|---|
| `openjev-fp8-readout-mm` | (i) which of these six titles is this page? | **97.2 %** | 35 / 36 | 16.7 % (chance, 6 options) | 0.0356 | 1.81 | 1.88 |
| `openjev-fp8-generate-mm` | (i) which of these six titles is this page? | **97.2 %** | 35 / 36 | 16.7 % (chance, 6 options) | 0.0556 | 0.91 | 0.94 |
| `openjev-fp8-readout-mm` | (ii) is this a front door or an article? | **100.0 %** | 36 / 36 | 66.7 % (majority class, 24 of 36 are articles) | 0.0006 | 0.75 | 0.76 |
| `openjev-fp8-generate-mm` | (ii) is this a front door or an article? | **100.0 %** | 36 / 36 | 66.7 % (majority class, 24 of 36 are articles) | 0.0000 | 0.77 | 0.78 |

### What a picture costs

`prompt_tokens` is the server's own count for the call **with** the image. `text_tokens` is the same server's `/tokenize` asked the **text alone** under the same chat template, with no image attached — so the image column is a subtraction of two measured numbers, not an estimate.

| arm | decision | n | mean prompt tokens | mean text tokens | **mean image tokens** | mean W, both cards | J / decision (net) |
|---|---|---|---|---|---|---|---|
| `openjev-fp8-readout-mm` | (i) which of these six titles is this page? | 36 | 1206 | 204 | **1002** | 456.6 | 825.0 (763.3) |
| `openjev-fp8-generate-mm` | (i) which of these six titles is this page? | 36 | 1206 | 204 | **1002** | 453.6 | 403.4 (151.2) |
| `openjev-fp8-readout-mm` | (ii) is this a front door or an article? | 36 | 1102 | 100 | **1002** | 438.4 | 316.8 (117.8) |
| `openjev-fp8-generate-mm` | (ii) is this a front door or an article? | 36 | 1102 | 100 | **1002** | 446.8 | 335.4 (137.1) |

### Where the misses are, by kind of page

| arm | decision | articles (24) | front doors (12) |
|---|---|---|---|
| `openjev-fp8-readout-mm` | (i) which of these six titles is this page? | 100.0 % (24 / 24) | 91.7 % (11 / 12) |
| `openjev-fp8-generate-mm` | (i) which of these six titles is this page? | 100.0 % (24 / 24) | 91.7 % (11 / 12) |
| `openjev-fp8-readout-mm` | (ii) is this a front door or an article? | 100.0 % (24 / 24) | 100.0 % (12 / 12) |
| `openjev-fp8-generate-mm` | (ii) is this a front door or an article? | 100.0 % (24 / 24) | 100.0 % (12 / 12) |

### Every miss on (i) which of these six titles is this page? — `openjev-fp8-readout-mm`

| page | kind | the page's own title | chose | confidence |
|---|---|---|---|---|
| `design` | front_door | design — pick one at random | `memory` | 40.7 % |

### Every miss on (ii) is this a front door or an article? — `openjev-fp8-readout-mm`

*None. All 36 items correct.*


### Reliability — `openjev-fp8-readout-mm`, (i) which of these six titles is this page?

| confidence bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.5–0.6 | 1 | 50.6 % | 0.0 % | +0.506 |
| 0.8–0.9 | 1 | 87.3 % | 100.0 % | -0.127 |
| 0.9–1.0 | 34 | 99.8 % | 100.0 % | -0.002 |

### Reliability — `openjev-fp8-readout-mm`, (ii) is this a front door or an article?

| confidence bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 36 | 98.9 % | 100.0 % | -0.011 |

---

## Arm 12 — the Kev family, Apache-2.0

*What this arm asks, for a reader who scrolled straight here: every OpenJev number in this bench is fenced — **CC BY-NC 4.0, bench-only, those weights never become a seat**. Kev is the same category of model, a typed decision read off a pointer head with no generation, under **Apache-2.0** on an Apache-2.0 base. So the question is not which is better. It is whether a model this workshop **could** deploy lands within a few points of one it cannot.*

Kev-9B is ~19 GB in bf16 and fits **one 3090 whole**, so arm 12 has no tensor-parallel and no all-reduce — the host bridge arm 2 paid 6.4 ms a token for is simply absent. It is also a different server (transformers + FastAPI, not vLLM). **Every latency comparison below is cross-runtime and cross-topology**, and the decisions are what is comparable.

### task (c) — which of the six guests said this line?

| model | accuracy | correct | Brier | median s | p95 s | floor |
|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | **66.7 %** | 72 / 108 | 0.4803 | 0.11 | 0.14 | 16.7 % |
| **Kev-4B** — Apache-2.0, one 3090 | **55.6 %** | 60 / 108 | 0.6182 | 0.07 | 0.08 | 16.7 % |
| OpenJev-FP8 (arm 2, vLLM, TP=2) | 93.5 % | 101 / 108 | 0.1219 | 0.35 | 0.42 | 16.7 % |
| gemma 4 26B (arm 10, same runtime) | 80.6 % | 87 / 108 | 0.3381 | 0.12 | 0.15 | 16.7 % |

### task (a) — does the article answer this question?

| model | accuracy | correct | Brier | median s | p95 s | floor |
|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | **100.0 %** | 38 / 38 | 0.0102 | 1.59 | 2.50 | 66.7 % |
| **Kev-4B** — Apache-2.0, one 3090 | **100.0 %** | 38 / 38 | 0.0132 | 1.02 | 1.59 | 66.7 % |
| OpenJev-FP8 (arm 2, vLLM, TP=2) — **restricted to the items Kev reached** | 100.0 % | 38 / 38 | 0.0057 | 5.78 | 9.10 | 66.7 % |
| gemma 4 26B (arm 10, same runtime) — **restricted to the items Kev reached** | 100.0 % | 38 / 38 | 0.0002 | 2.01 | 3.21 | 66.7 % |

**What Kev could not reach, and why — 25 of 63 items.**

| reason | items | what it is |
|---|---|---|
| `overlong` | 10 | inherited from arm 2's own refusal rows, so the set started where arm 2's did (pre-registered) |
| `kev-422` | 15 | **the server refused the request**: `kev.serve` runs with `INFER_MAX_STATE = INFER_MAX_BRANCH = 8192` and `encode` rejects a question when `len(branch) > max_branch - len(state)`, so a page that fills the 8,192-token state window leaves a budget of zero and even a 56-token question is refused |

*The server's own sentence, carried on every such row: `{"detail":"branch too long: 56"}`*

**This is a real limit, not a harness problem, and it is the finding this set produced.** The estate's docent pages run from 1,772 to 82,500 tokens. A decision model that can hold 8,192 tokens of state cannot be asked *does this article answer this question?* about most of them without truncating the article — which this bench forbids (arm 1 §8), because truncating a page can turn a genuinely on-page question into an unanswerable one. Kev answered every page it could hold, and refused the rest in its own words rather than guessing at a fragment.

### the doorman's planted set (arm 4)

| model | accuracy | correct | Brier | median s | p95 s | floor |
|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | **56.2 %** | 27 / 48 | 0.5680 | 0.17 | 0.17 | 75.0 % |
| **Kev-4B** — Apache-2.0, one 3090 | **33.3 %** | 16 / 48 | 1.0120 | 0.10 | 0.11 | 75.0 % |
| OpenJev-FP8 (arm 4) | 93.8 % | 45 / 48 | 0.0911 | 0.58 | 0.60 | 75.0 % |
| gemma 4 26B (arm 10) | 72.9 % | 35 / 48 | 0.5185 | 0.21 | 0.21 | 75.0 % |

### the field exam, answerable or not (arm 5)

| model | accuracy | correct | Brier | median s | p95 s | floor |
|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | **100.0 %** | 40 / 40 | 0.0172 | 0.09 | 0.10 | 50.0 % |
| **Kev-4B** — Apache-2.0, one 3090 | **100.0 %** | 40 / 40 | 0.0081 | 0.07 | 0.07 | 50.0 % |
| OpenJev-FP8 (arm 5) | 100.0 % | 40 / 40 | 0.0057 | 0.28 | 0.30 | 50.0 % |
| gemma 4 26B (arm 10) | 100.0 % | 40 / 40 | 0.0001 | 0.10 | 0.11 | 50.0 % |

### the judge seat (arm 6)

| model | accuracy | correct | Brier | median s | p95 s | floor |
|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | **96.6 %** | 115 / 119 | 0.1566 | 0.10 | 0.56 | — |
| **Kev-4B** — Apache-2.0, one 3090 | **94.1 %** | 112 / 119 | 0.1942 | 0.08 | 0.30 | — |
| OpenJev-FP8 (arm 6) | 99.2 % | 118 / 119 | 0.1611 | 0.81 | 1.89 | — |

### Cost, and it is not a like-for-like latency

| model | set | n | median s | p95 s | mean W | idle W | J / decision (net) |
|---|---|---|---|---|---|---|---|
| **Kev-9B** — Apache-2.0, one 3090 | task (c) | 108 | 0.11 | 0.14 | 234.9 | 120.0 | 26.1 (12.8) |
| **Kev-9B** — Apache-2.0, one 3090 | task (a) | 38 | 1.59 | 2.50 | 239.8 | 115.3 | 441.8 (229.3) |
| **Kev-9B** — Apache-2.0, one 3090 | the doorman's planted set (arm 4) | 48 | 0.17 | 0.17 | 231.8 | 123.7 | 38.7 (18.0) |
| **Kev-9B** — Apache-2.0, one 3090 | the field exam, answerable or not (arm 5) | 40 | 0.09 | 0.10 | 198.0 | 122.2 | 14.9 (5.7) |
| **Kev-9B** — Apache-2.0, one 3090 | the judge seat (arm 6) | 119 | 0.10 | 0.56 | 220.7 | 126.1 | 50.1 (21.5) |
| **Kev-4B** — Apache-2.0, one 3090 | task (c) | 108 | 0.07 | 0.08 | 227.0 | 111.4 | 16.8 (8.6) |
| **Kev-4B** — Apache-2.0, one 3090 | task (a) | 38 | 1.02 | 1.59 | 243.2 | 114.7 | 256.0 (135.3) |
| **Kev-4B** — Apache-2.0, one 3090 | the doorman's planted set (arm 4) | 48 | 0.10 | 0.11 | 217.1 | 121.3 | 22.7 (10.0) |
| **Kev-4B** — Apache-2.0, one 3090 | the field exam, answerable or not (arm 5) | 40 | 0.07 | 0.07 | 166.0 | 122.5 | 8.3 (2.2) |
| **Kev-4B** — Apache-2.0, one 3090 | the judge seat (arm 6) | 119 | 0.08 | 0.30 | 236.8 | 125.4 | 27.9 (13.1) |

*Arm 12's watts are **one** board; arm 2's and arm 10's are the sum of two. A joules-per-decision figure here is not the same quantity as one in the OpenJev tables and must not be subtracted from it.*

### Kev's own published numbers, quoted

| reading | accuracy | Brier |
|---|---|---|
| Kev-9B, locked test (its own suite) | 0.852 | 0.237 |
| Kev-4B, locked test (its own suite) | 0.837 | 0.255 |
| hosted Jev, the same development items | 0.857 | 0.211 |

*Quoted, never verified — a different set, a different task mix and a different readout. Kev's published Brier is for the **raw logits**; the arms above run **as served**, with the checkpoint's own fitted temperature applied. The raw pass below is the only Brier in this file comparable with the column on the right.*

### Calibration — the same answers, a different Brier

| model | task (c) | accuracy | correct | Brier |
|---|---|---|---|---|
| Kev-9B | as served (the checkpoint's own T) | 66.7 % | 72 / 108 | 0.4803 |
| Kev-9B | `KEV_TEMPERATURE=1.0` (raw logits) | 66.7 % | 72 / 108 | 0.5395 |
| Kev-4B | as served (the checkpoint's own T) | 55.6 % | 60 / 108 | 0.6182 |
| Kev-4B | `KEV_TEMPERATURE=1.0` (raw logits) | 55.6 % | 60 / 108 | 0.7021 |

*Kev's temperature never changes an answer — the argmax is identical — so a pair of rows here that differ in accuracy would be a finding about the server, not about calibration.*

### Reorder consistency — Kev's own `/permute`, beside gap 4

| model | how | items whose answer moved | order-pairs that disagree | mean per-option spread |
|---|---|---|---|---|
| **Kev-9B** | its own `/permute`, `n_perm = 6`, seed 20260921 | **30.6 %** (33 / 108) | **15.43 %** (250 / 1620) | 0.1011 |
| **Kev-4B** | its own `/permute`, `n_perm = 6`, seed 20260921 | **30.6 %** (33 / 108) | **16.17 %** (262 / 1620) | 0.0875 |
| OpenJev-FP8 (gap 4) | gap 4's own six orders, seed 20260921 | **13.9 %** (15 / 108) | **5.68 %** (92 / 1620) | 0.0466 |
| OpenJev's model card, tuned / untuned | its own set, targeted extraction | — | 2.3 % / 18.5 % | — |
| Kev-4B's model card (Qwen3 generation) | its own suites | — | 6 % (its "option-order flip rate"; Jev 0 %) | — |

*The same statistic on different draws, and that is the whole caveat. Kev shuffles inside its own server with its own seeded `random.Random`; gap 4 shuffles with ours. The six orders are therefore not the same six — so this compares two rates measured the same way, not two runs of one schedule. The spread column is `max − min` of an option's probability across the orders, which is gap 4's per-guest range defined identically.*

---

## Residency — what switching the eyes on cost

Both boards, by UUID, read by each arm's own `verify_pin` before and after. Arm 2 served this checkpoint with `'{"image":0}'`; gap 5 served it with `'{"image":1}'` and nothing else changed.

| arm · task | posture | card 0 MiB | card 1 MiB | growth this task | other boards | resident |
|---|---|---|---|---|---|---|
| `openjev-fp8-readout` · c | arm 2 — `'{"image":0}'` | 21246 | 21246 | +0 / +0 | quiet | yes |
| `openjev-fp8-readout` · creorder | gap 4 — `'{"image":0}'` | 21056 | 21056 | +20 / +20 | quiet | yes |
| `openjev-fp8-readout-mm` · shot_title | **gap 5 — `'{"image":1}'`, first image** | 20910 | 20910 | +242 / +242 | quiet | yes |
| `openjev-fp8-readout-mm` · shot_kind | gap 5 — `'{"image":1}'`, second task | 20910 | 20910 | +0 / +0 | quiet | yes |

**The flag does not change the footprint; it changes how the budget is divided.** `--gpu-memory-utilization 0.90` is a ceiling, and vLLM fills it either way — so the steady-state figures above are within a few hundred MiB of each other. What the eyes actually cost is visible in the server's own startup lines, per card:

| posture | available KV cache | GPU KV cache size | max concurrency at 16,384 tokens |
|---|---|---|---|
| `'{"image":0}'` (arm 2's, and gap 4's) | 5.23 GiB | 132,285 tokens | 8.07× |
| `'{"image":1}'` (gap 5's) | **4.86 GiB** | **123,183 tokens** | **7.52×** |

*Switching the eyes on cost **0.37 GiB of KV cache per card** — 9,102 fewer tokens of context budget, about 6.9 % — plus the **+242 MiB per card** the vision tower allocated the first time an image arrived, which is the growth row above. TP=2 held on both boards in both postures.*