# Gaps 4 and 5 — the two things on the model card this bench never tested

*What this is, for a reader who scrolled straight here.* The Jev bench
(`README.md`, benchbox, 2026-09-21) measured a decision model — `openjev/openjev-FP8`
— against the generating seat on three of this workshop's own decisions. Its
draft article ends with a list called *"What this page does not know"*, and two
of those lines are claims **OpenJev's own model card makes** that the bench had
measured nothing about:

> *What the reorder-consistency claim looks like on our tasks. The option order
> never changed inside an arm.*
>
> *Anything about pictures or screenshots. Every decision here was text; Jev and
> OpenJev both claim web and screenshot decisions.*

This page closes those two. **Gap 4** shuffles the options. **Gap 5** switches the
model's eyes on. Both run on benchbox, on the same two RTX 3090s at 250 W caps,
against the same checkpoint at the same revision, with the same letters, the same
calibration constants and the same instrument every other arm used.

**Pre-registration stamped 2026-09-22T07:39:26Z** · box **benchbox** · card 0
`GPU-46890836-d8f7-e868-1977-c1ce28db0d7a` (PCI 04:00.0, cap 250 W) · card 1
`GPU-7aa0be10-974f-6430-52fe-09018a0e2e08` (PCI 2B:00.0, cap 250 W) ·
PHB topology, no NVLink. Written and committed **before any gap-4 or gap-5 row was
measured**; `git log` is the receipt, and nothing above the results line was edited
after the first call.

**LICENCE FENCE, restated rather than assumed from a neighbouring section.**
`openjev/openjev-FP8` is **CC BY-NC 4.0 — research and non-commercial use only**.
Gaps 4 and 5 are bench arms. These weights never become a seat: not the docent,
not rulesage, not amble, not the long table, not the beat lab, not the cove. A
finding here licenses a statement about decision models, never this checkpoint in
production.

---

## 0. The disk, before anything is measured

benchbox has one disk and the addenda spent it. Arm 10 evicted the OpenJev FP8
cache (≈ 29 GB) so a read-only copy of the estate's own gemma-4 seat checkpoint
(≈ 27 GB) could be measured on the same runtime — the only way both arms fit on
one box (README, arm 10, *"THE DISK CONSTRAINT"*). Arm 10 is measured and its rows
are committed, so:

1. **Arm 10's gemma copy at `~/models/gemma-4-26B-A4B-it-FP8-dynamic` is deleted**,
   on the operator's instruction. It has no further reader; every number it
   produced lives in `rows-addenda/` and in `TABLES-ADDENDA.md`.
2. **`openjev/openjev-FP8` is re-downloaded at revision
   `4ec320f267401e67c9be04d5df1be4d2b6b64f10`** — the revision every arm-2 row
   names — into `HF_HOME=/workshop/hf-cache`.
3. **The twelve shards are verified by sha256** against
   `bench-arm2/tree-fp8.json`, the Hugging Face file tree recorded *before arm 2's
   first call*. This is the only moment in the whole bench where a measured
   artefact left the box and came back, so it gets a computed receipt rather than
   a byte count (`verify_shards.py`; receipt in `receipts/`).

**Not touched, and named so that it is on the record**: the diffusion checkpoints
under `~/ComfyUI` (≈ 52 GB). They are not this bench's to spend.

**Refusal:** if the download does not complete, or any shard fails its sha256, or
the free space after the delete is still short of the checkpoint, **both gaps are
a refusal with the measured byte counts** — not a shortened set, not a different
revision, and not a checkpoint patched up by hand.

---

## Gap 4 — reorder consistency

### What the card claims, and why no arm here could see it

OpenJev's model card reports that a tuned model's chosen answer changes in
**2.3 %** of cases when the options are shuffled, against **18.5 %** untuned. Every
arm in this bench asked task (c) — *which of the six guests said this line?* — with
**one** option order. That is not an oversight in the arms; it is a property of the
frozen kit, and it is measured rather than assumed: `kit/task_c.json` holds 108
items and exactly **one distinct `options` tuple**, alphabetical by guest, the same
six names in the same six positions for every item and every arm. Under that kit no
arm could have seen a reorder effect at all.

### The set, and the six presentations

The same **108 items**, unchanged — same lines, same labels, same six guests. Each
is asked **six times**:

* **k = 0** — the kit's own order, untouched. These 108 prompts are **byte-identical
  to arm 2's**: verified before the run, **108 of 108 sha256 matches** against
  `rows/openjev-fp8-readout.c.jsonl`. This pass is the arm's own control, not a
  fresh measurement of a different prompt.
* **k = 1…5** — `random.Random(20260921 + k)`, walking the items in kit order, so
  the whole schedule is reproducible from two integers. The permutation is written
  into every row rather than recomputed from a seed by the reader.

**The shuffle moves the whole presentation, not the letter block alone.** `run.py`'s
`task_c` builds the state sentence ("A dinner table with six guests: …") from the
same option list the letters come from, so a kit that had drawn a different order
would have moved both. Moving only the letters would measure a prompt this bench
never asks; moving both is exactly the prompt arm 2 would have had under a
different draw. Stated here because it is the one design choice in gap 4 that a
reader could reasonably have made the other way.

### The arms

`openjev-fp8-readout` and `openjev-fp8-generate` — arm 2's own two arms, on **arm
2's server, verbatim** (`serve.sh`, `--limit-mm-per-prompt '{"image":0}'`). Gap 4
is a text arm and it runs under the text posture, so its k = 0 rows are comparable
to arm 2's in the serving flags as well as in the bytes.

**The base model's reorder figure is NOT measured, and it is owed.** The natural
control is the same six orders on `gemma-4-26B-A4B-it-FP8-dynamic` (arm 10's
checkpoint), which would say how much of any stability is OpenJev's tuning and how
much is the family. That checkpoint is the 27 GB this gap deletes to make room for
OpenJev, and benchbox cannot hold both (§0). So there is no base column in gap 4's
tables, **the absence is named here before the numbers exist**, and a bare OpenJev
reorder rate is reported as what it is: a rate for one model, not a comparison.

### What is measured, defined before it is computed

Every figure below is over the **chosen GUEST**, never the chosen letter, except
the one that is explicitly about letters.

| figure | definition |
|---|---|
| **items that moved** | of 108 items, the fraction whose chosen guest is not the same across all six orders |
| **order-pairs that disagree** | over the **15 unordered pairs** of the six orders, per item (1,620 pairs in total), the fraction where the two orders chose different guests — **this is the figure comparable to the card's 2.3 %** |
| **accuracy per shuffle** | accuracy against the label, separately for each of k = 0…5 (108 items each) |
| **mean per-guest probability shift** | per item and per guest, the probability that guest received under each of the six orders; reported as the mean **range** (max − min) and the mean **absolute deviation** from that guest's own six-order mean. Both, because a range is moved by one outlier and a deviation is not |
| **positional bias** | over all **648** presentations, how often the chosen **letter** was A, B, C, D, E, F — against the 16.7 % a content-blind model would show, and against the letter the correct answer actually sat on (which the schedule spreads 84–120 across the six letters, and exactly 18 each at k = 0) |

Also reported, as in every arm: Brier, median and p95 seconds, prompt and eval
tokens, both boards' mean watts and joules per decision, residency on both boards,
and the floored-letter counts.

**Gate.** None — descriptive. The card's 2.3 % was measured on the card's own set
with the card's own targeted extraction; this bench is on the **old (untargeted)
path** with `top_logprobs = 20`, so gap 4's number is *this model's reorder rate on
this workshop's decision*, reported beside the card's figure and never as a
verification of it.

**Refusal conditions.** A shuffle that is not a permutation of the item's own six
options (asserted before the call); an item whose prompt exceeds 16,384 tokens
(`refused-overlong`, excluded from every rate, never truncated); a server that
will not start as written — reported as a refusal, with no home-brew fallback.

---

## Gap 5 — a screenshot decision

### What the card claims, and what arm 2 switched off

Both Jev and OpenJev claim web and screenshot decisions. `openjev/openjev-FP8` is a
`Qwen3_5ForConditionalGeneration` — **a vision-language model**. Arm 2 served it
with `--limit-mm-per-prompt '{"image":0}'`: the eyes were switched off for every
arm in this bench, so the claim could not be tested at all. Gap 5 serves the same
checkpoint, the same revision, the same TP=2 across the same two cards, the same
16,384 window and the same letters — with **one flag changed**:

```
--limit-mm-per-prompt '{"image":1}'
```

**The VRAM that flag costs is a required receipt**, not an aside: admitting an
image makes vLLM allocate the vision encoder and a multimodal cache. The
difference is read from the two arms' own pin blocks — arm 2's
`memory_after_per_card_mib` against gap 5's — and reported whether it is large or
nil, along with the quantisation kernel vLLM names in its startup log and whether
TP=2 still holds.

### The set, built by construction because no labelled one exists

**36 screenshots at 1,280 × 800** of public pages this workshop publishes
(`build_shots.py`, kit frozen in `kit-gaps-card/`, manifest sha256
`bd01fa579f1f0a76…`):

* **24 research-hub article pages** — `research.strata2signal.com/<slug>/`, a
  seeded sample (`random.Random(20260921)`) of 24 from the 42 slugs in
  `kit/articles.json`, the same article set every other arm of this bench reads.
* **12 front doors** — the top page of twelve of the workshop's own products.

Every page returned HTTP 200 at capture. **Every URL is one a stranger can open**;
no private surface, no logged-in view, no internal host is photographed, and no
box name or private-network name appears in any prompt, option or row.

**The capture, exactly**: ungoogled-chromium 150.0.7871.46 headless,
`--force-device-scale-factor=1` (so the PNG is 1,280 × 800 device pixels, not a
retina multiple), `--hide-scrollbars`, `--virtual-time-budget=8000` so webfonts and
deferred paint land before the shutter, a throwaway user-data-dir so the operator's
own browser profile is never opened. The frame is the top of the page: what a
visitor sees before scrolling. Each page's measured pixel dimensions are asserted
against 1,280 × 800 at build time, and the **sha256 of every PNG is frozen in the
manifest and re-hashed at the moment of the call** — a file that changed between
the build and the run is a `refused-kit-moved` row carrying both hashes, never a
silently different measurement.

**The labels are what the pages ARE**, assigned by construction: `article` or
`front_door`. **The titles are the pages' own `<title>`**, read from the HTML at
capture time, entity-decoded and whitespace-collapsed — derived from the page, never
typed.

### The two decisions

**(i) Which of these six titles is this page?** — 36 items, six options, floor
**16.7 %**. The options are the page's own title and five distractors drawn by seed
from the other pages **of the same kind**. Same kind, and the reason is a leak:
every research article's `<title>` ends `· strata→signal research` and no front
door's does, so a cross-kind distractor could be discarded on its punctuation
without looking at the picture at all. The correct option's letter varies by seed.
The option shape is task (c)'s — `(key, description)` where the key is the page's
own short id and the description is its title; the key names the same page the
title does, so it can carry nothing about which option is right.

**(ii) Is this a front door or an article?** — 36 items, two options in a fixed
order for every item (`[A] front_door / [B] article`), majority-class floor
**66.7 %** (24 of 36 are articles). Not run through the yes/no calibration:
`READOUT_NOUL_T` is the card's constant for a `noul` yes/no decision, and this is a
two-way choice between two named kinds — arm 4's shape, which did not apply it
either.

**Said plainly before the numbers arrive**: (ii) is the easier of the two by
construction. Every article screenshot carries the research hub's masthead across
its top. (ii) is a legibility check — can this model see the picture at all — and
(i) is the decision that can actually be hard. At least one item in the set is
hard on purpose and by accident: one front door is a randomised splash that
renders no title text anywhere in the frame, so its answer has to come from the
picture rather than from reading.

### The readout, unchanged

Letters at the **first output position**, `temperature 0`, seed 20260921,
`top_logprobs = 20`, `FLOOR = -30`, `READOUT_T = 0.85`, the untargeted (old) path,
`chat_template_kwargs={"enable_thinking": false}`, concurrency 1 — arm 2's protocol
in every particular. The prompt is the OpenJev helper's text-lane layout with a
state sentence that is **identical for all 72 decisions** ("The screenshot attached
to this message."), so nothing in the text can separate one item from another; the
picture rides beside it on the same user turn.

Arms: **`openjev-fp8-readout-mm`** (the headline) and **`openjev-fp8-generate-mm`**
(the generating control on the same model, as every other arm carries one). They
are separate arm names from arm 2's because their rows were measured under a
different serving posture, and a row must never claim a posture it did not run
under.

### What is measured

Accuracy against the constructed labels, against the floors named above · Brier ·
a reliability table · median and p95 seconds · **prompt tokens including the image
tokens, and the image cost stated as a difference**: each row carries the server's
own `prompt_tokens` for the call *with* the picture, and a `text_tokens` count from
the same server's `/tokenize` asked the **text alone** under the same template, so
`image_tokens` is a subtraction of two measured numbers rather than an estimate ·
both boards' mean watts and joules per decision · residency on both boards · the
VRAM delta against arm 2's text-only posture.

**Gate.** None — descriptive; this is a capability reading, not a rung. What a
result licenses is bounded in advance: a set of 36 items decides nothing about
screenshot decisions in general, and the per-cell n is printed in every table for
that reason.

**Refusal conditions.** vLLM refuses `'{"image":1}'`, or the vision tower will not
load, or TP=2 OOMs with the encoder resident (the §A2.4 in-bounds alternatives only
— a narrower `--gpu-memory-utilization` — then refusal, and no hand-rolled
sharding); the server rejects the image part of the request (reported with the
server's own sentence); a prompt over the 16,384 window once image tokens are
counted (`refused-overlong`, excluded from every rate); a PNG whose sha256 at call
time differs from the frozen manifest (`refused-kit-moved`). **A dead arm gets no
numbers** — not a threshold table, not an estimate, not a figure carried over from
a neighbouring arm.

---

## The instrument

`run_gaps_card.py`, beside `run.py` and `run_addenda.py`, which it **imports rather
than forks**: `decide`, `readout_prompt`, `letter_scores`, the calibration, `Boards`,
`verify_pin`, `summarise`, `refusal` and `run_arm` are arm 2's own code, and this
module adds only new task generators registered into `run.py`'s `TASKS`. `run.py`
and `run_addenda.py` are not edited, so arms 1–11's record stays exactly as it was
measured.

The one hook gap 5 needs is deliberately the smallest one available. A picture has
to reach the user turn, and forking the request builder, the clock or the reply
normaliser to do it would make gap 5's latency a different measurement from every
other arm's. So exactly one function is patched — `run.post`, the JSON POST
underneath both endpoints — to add an `image_url` part to the user message when an
image is attached, **and only on `/v1/chat/completions`**. `/tokenize` therefore
still sees the text alone, which is what makes the image-token subtraction honest.

Rows to `rows-gaps-card/`, tables to `TABLES-GAPS-CARD.md` via `tables_gaps_card.py`,
receipts to `receipts/`. The concurrent gaps 1–3 lane owns `GAPS.md`,
`TABLES-GAPS.md`, `tables_gaps.py` and `rows-gaps/`; this lane writes none of them.

---

## Amendment 1 — written before the first call of either gap

*Stamped %s. No gap-4 or gap-5 row exists yet; `git log` is the receipt. Three
things found while staging the run that change what the sections above should
have said, recorded here rather than edited into them.*

**1. `'{"image":1}'` is the model card's OWN recipe, and arm 2 was the deviation.**
The serve command printed on `openjev/openjev-FP8`'s card reads
`--limit-mm-per-prompt '{"image":1}'`. Arm 2 set it to `0`. So gap 5 is not
inventing a posture for this checkpoint — it is **restoring the one its authors
publish**, and the sentence "arm 2 served it with its eyes off" is a statement
about this bench's choice, not about the model's intended deployment. (The card
also serves at `--max-num-seqs 16`; this bench stays at 1, as every arm has, so
that one remains a named difference.)

**2. The card's own screenshot figure is quoted, and it is NOT gap 5's
comparator.** The card reports **88.0 %** on *2,000 desktop screenshots, next
action* — for both the 16-bit model and this FP8 file, 0.05 points apart. That
is a different set, a different decision (*what should be clicked next*, not
*which page is this*), and a different option count. It is quoted in the tables
as context and never as a number gap 5 passes or misses.

**3. Prefix caching is on, so gap 4's per-order LATENCY is not a clean read.**
`--enable-prefix-caching` is arm 2's flag and gap 4 keeps it. Each shuffle
changes the state sentence, which is near the front of the prompt, so the six
orders do not hit the cache equally. The per-order **accuracy** rows are
unaffected — a greedy readout at temperature 0 does not care what was cached —
but a k-to-k latency difference cannot be read as a property of the order, and
the tables say so where they print it.

---

## Arm 12 — the Kev family, and the licence question underneath it

*Commissioned 2026-09-22 while gaps 4 and 5 were staging. **Pre-registration stamped
%s**, written and committed **before the first Kev call**; `git log` is the
receipt.*

### The question, and it is not an accuracy question

Every OpenJev number in this bench is fenced: **CC BY-NC 4.0, bench-only, these weights
never become a seat**. A finding about a model the estate may never deploy is a finding
about a category, not about anything it can build on. **Kev** is the same category under
a different licence — **Apache-2.0**, on an Apache-2.0 base — so a Kev result is a
result about a model that *could* take a seat here.

> **Does an Apache-licensed decision model that CAN be a seat land within a few points
> of the CC BY-NC one on this workshop's own decisions?**

**Gate: none — descriptive.** A few points is a phrase, not a threshold, and this arm
will not invent one after seeing a number. What it reports is the gap, with both sides'
n, beside arm 2/3 (OpenJev) and arm 10 (gemma 4).

### The models, the licence, and the revisions

| | |
|---|---|
| project | `github.com/jaredpalmer/kev` — Apache-2.0, cloned at commit `90990a5fac2995b9faa3190f7d437e84f2067768` (2026-09-22T00:43:59Z) |
| **Kev-9B** | `jaredpalmer/kev-9b` — a rank-16 LoRA (45.4 M trainable) + a pointer head on `Qwen/Qwen3.5-9B-Base` (base revision `68c46c4b`); ~19 GB of GPU memory in bf16, **one 3090** |
| **Kev-4B** | `jaredpalmer/kev-4b` — the same recipe on `Qwen/Qwen3.5-4B-Base` (base revision `1001bb4d…`) |
| revisions | `main` as it resolves at download (both were updated 2026-09-21); **the resolved sha is recorded from the download**, not from this page |
| **licence** | **Apache-2.0** for the adapters and heads; the Qwen3.5 bases are Apache-2.0. **This is the point of the arm.** |

**Its own published numbers, quoted so ours can be read against something.** Locked
test, out-of-domain: **Kev-9B 0.852 accuracy / 0.237 Brier**, **Kev-4B 0.837 / 0.255**,
**hosted Jev 0.857 / 0.211**. Option-order flip rate on its own suites: **0.06** for
Kev-4B (Qwen3 generation's card) against **0.00** for Jev. Quoted, never verified — a
different set, a different task mix, a different readout.

### The serving posture, exactly

Kev is **not vLLM**. It is transformers + FastAPI, and that is a difference this arm
carries rather than hides.

```bash
git clone https://github.com/jaredpalmer/kev.git && cd kev
uv sync --extra serve
CUDA_VISIBLE_DEVICES=<one card by UUID> KEV_DTYPE=bf16 \
  uv run --extra serve python -m kev.serve --run jaredpalmer/kev-9b --port 8009
```

* **ONE CARD EACH, by UUID.** `CUDA_VISIBLE_DEVICES` is pinned to a single board and
  residency is proved the way every arm here proves it: growth on the named card,
  nothing on the other, refusing in both directions. Kev-9B is ~19 GB and fits a 3090
  whole, so this arm has **no tensor-parallel and no all-reduce** — the thing arm 2 had
  to pay 6.4 ms of wire for on every token.
* **`flash-linear-attention` is NOT in the `serve` extra**, and Kev's own README says
  CUDA needs it for the Qwen3.5 models — without it the Gated DeltaNet layers fall back
  to reference PyTorch. It is installed into the venv, and **the server's own startup
  is read to say which path loaded**. If the install fails the arm still runs and
  **every latency, watt and joule row is labelled `reference-kernels`** rather than
  compared with a fast-path number.
* **Calibration is ON, because that is what a seat would get.** Each checkpoint carries
  a fitted temperature in `head.pt` (9B: T = 2.30, 4B: T = 2.14) and every loader gets
  the calibrated probabilities by default. It never changes an answer — the argmax is
  identical — so accuracy is the same either way, but **Brier is not**. Kev's own
  published Brier is for the **raw** logits, so one extra pass of task (c) is run at
  `KEV_TEMPERATURE=1.0` per model, and that is the only Brier in our tables comparable
  with the card's. Both are printed, each labelled.
* **`KEV_DATE_FACTS` stays 0** (its default). It is a preprocessor, its own card reports
  it separately, and none of our five sets is date arithmetic.
* **One question per request.** Kev's bundling — many questions sharing one prefill,
  isolated by construction — is a real feature and it is **not what this arm measures**;
  measuring it would make every latency row a different quantity from arm 2's
  one-decision-at-a-time rows. `POST /v1/systemone/separate` exists for that comparison
  and is named here as not-run.
* **127.0.0.1 only**, port 8009. No sudo. The server is left stopped and both cards are
  confirmed back at 1 MiB.

### The mapping — our kit, their request shape

The kit does not change. What changes is the envelope, and each mapping is stated so a
reader can see that no set was made easier on the way across. **Our option strings, our
option order, our seed (20260921), one question per request, temperature 0 by
construction** (a pointer head has no sampler).

| our set | n | Kev type | how | label |
|---|---|---|---|---|
| task **(c)** — which of the six guests said this line? | 108 | `choice` | `criteria = {guest_id: guest_name}` in the kit's own alphabetical order; the line in `instructions`, the roster sentence in `state` — arm 2's exact split | the guest |
| task **(a)** — does the article answer this question? | 63 | `noul` | the question in `instructions`, the fenced page in `state`; `noul` is p(yes) | on-page → true |
| the **doorman** (arm 4) | 48 | `noul` | DOORMAN_SYSTEM **v1.2 verbatim** as `instructions`, `"Host: " + line` as `state`, `criteria = {true: "The table hears the line.", false: "The table does not hear the line."}` | planted → false (refuse) |
| the **field exam** (arm 5) | 40 | `noul` | arm 5's `INSTR_F` verbatim; the excerpt as `state` | answerable → true |
| the **judge seat** (arm 6) | 119 | `choice` | arm 6's three classes and their own descriptions, in arm 6's order | `expected`, scored on the same `expected_ok` alternation arm 6 uses |

Task **(b)** is not run: it is unlabelled by pre-registration (§3) and adding an
unlabelled set to a licence comparison buys nothing.

**The long pages, and the denominator that makes the tables line up.** Task (a)'s states
run to 80,515 tokens and no 24 GB card holds that beside 19 GB of weights. Arm 12 refuses
**exactly the items arm 2 refused** at its 16,384 window — the set is read from arm 2's
own refusal rows, as arm 9's was, not re-chosen — so arm 12's task-(a) denominator is
**the same 53 items** arm 2 and arm 10 scored, and the three rows are comparable cell for
cell. Every row additionally records Kev's own `usage.input_tokens`, so the inherited
ceiling can be checked against what this tokenizer actually made of the page.

### (d) Kev's own reorder number, beside gap 4's

Kev ships `POST /v1/systemone/permute`: one Choice question re-run under `n_perm` option
orders, returning each order's probabilities, an `argmax_stable` flag and a per-option
`spread`. It is run on **all 108** task-(c) items at **`n_perm = 6`, `seed = 20260921`** —
the same shape gap 4 gave OpenJev: order 0 is the original and five are shuffled.

**The two are the same statistic on different draws, and that is said in the table.**
Kev's `spread` is `max − min` of an option's probability across the orders, which is
gap 4's "mean per-guest probability range" exactly; `argmax_stable` is the negation of
gap 4's "this item moved". The permutations themselves are each side's own seeded draw
and are not the same six orders — so the comparison is between two rates measured the
same way, not between two runs of one schedule.

### What is measured

Accuracy against our labels on the denominator actually reached · multi-class Brier
(calibrated, plus the raw-logit pass on task (c)) · median and p95 seconds per decision ·
the card's own watts by UUID at 1 Hz and joules per decision, net of a ten-second idle
read · residency on the named card and quiet on the other · **the raw response body per
row**, because Kev's readout is its own and a row that kept only our summary of it would
not let anyone re-derive it · `usage.input_tokens` · the resolved weight revisions · and
the server's `GET /v1/models`, which states the checkpoint and the temperature it loaded.

### The disk ladder, decided before the free space is read

Measured today: benchbox has **9.8 GB free** before this run; deleting arm 10's gemma copy
returns **37 GB**; the OpenJev re-download takes about **30 GB** of it back. Kev needs its
bases, which are the real cost — Qwen3.5-9B-Base ≈ 18 GB and Qwen3.5-4B-Base ≈ 8 GB —
plus a venv with its own torch. **They will not all fit beside OpenJev**, so the order is
fixed here rather than chosen later:

1. Gaps 4 and 5 run first and **write their rows**. Nothing below happens until those
   files exist, because they are what survives an eviction.
2. The OpenJev FP8 cache is then **evicted** — recoverably, exactly as arm 10 did it, and
   stamped.
3. Free space is **read, not assumed**. If both bases fit, both models run. **If only one
   fits, Kev-9B runs and Kev-4B is reported as not-run with the measured byte counts.**
   If neither fits, arm 12 is a refusal with those counts — not a smaller model
   substituted, not a shortened set.

### Refusal conditions

`uv sync` or the `flash-linear-attention` install cannot complete (the arm may still run,
labelled `reference-kernels`; an install that *breaks the server* is a refusal) · the
server will not start or `GET /v1/models` does not answer · the checkpoint's resolved
revision cannot be recorded · the weights do not fit one card in bf16 (no sharding, no
CPU offload, no quantisation invented here — the arm reports what it measured) · a
request the server answers `422` (recorded with the server's own message, excluded from
every rate) · a state over what the card can hold (`refused-overlong`, with its token
count, never truncated) · the pin fails in either direction. **A dead arm gets no
numbers.**

### One fact recorded for the article, not run here

Kev's fine-tuning path takes **a labelled JSONL in the API's own request shape** — one
request per line with a `label` on each question — which is the shape this estate's own
labelling kit can already emit, and `--init_from` starts from a released checkpoint
rather than the bare base. That is a route this workshop could take; **it is not
something this arm runs**, and no number in these tables is a claim about it.

---

## Amendment 2 — arm 12's load path, written before its first measured row

*Stamped 2026-09-22T10:16:00Z, after Kev-9B's first load attempt failed and before any arm-12 row
exists. The failure and its receipt are named here rather than smoothed into a
result.*

**What happened.** `KEV_DTYPE=bf16 kev.serve --run jaredpalmer/kev-9b` on one 3090
died with `torch.OutOfMemoryError: Tried to allocate 128.00 MiB. GPU 0 has a total
capacity of 23.56 GiB of which 119.06 MiB is free` — it had already placed
**23.18 GiB** on the card (`receipts/kev-smoke-serve.log`). Kev's own card says the
9B needs about **19 GB** in bf16, so the load was not doing what the flag asked.

**Why.** `kev/checkpoint.py::Checkpoint.load` reads:

```python
dtype, merge = opts.dtype or torch.float32, opts.merge
m = DecisionModel(..., dtype=torch.float32 if merge else dtype, ...)
if merge: m.lm = m.lm.merge_and_unload()     # in fp32: exact
if dtype != torch.float32: m.lm = m.lm.to(dtype)
```

With the default `merge=1` the base is loaded in **fp32 first** and only cast
afterwards, so `KEV_DTYPE=bf16` cannot help a card that the fp32 copy will not fit.

**What is done about it, and why it is in bounds.** `KEV_MERGE=0` — **Kev's own
documented switch** (README, *Serving Performance*: "You can disable these with
`KEV_MERGE=0`, `KEV_ATTN=eager`, …"). With it the base loads in bf16 and the rank-16
adapter stays unmerged and is applied at runtime. This is not a fallback invented
here: it is how this checkpoint is served in bf16, which is the ~19 GB its own card
publishes. It is the same standard §A2.4 held arm 2 to — *the only alternatives an
arm may try are the project's own*. No sharding, no CPU offload, no quantisation.

**The cost, stated by the project rather than by us**: on 24 new-source records
bf16 probabilities differed from fp32 by at most **0.017**, with **no change in the
highest-probability answer**. Every arm-12 row is served this way and the tables say
so; a Kev number here is `KEV_MERGE=0` bf16, not a merged-fp32 number.

**If it still does not fit**, the pre-registered refusal stands unchanged: the arm
reports the measured byte counts and runs nothing smaller in its place.

---

## RESULTS

*(appended as the arms land; nothing above this line is edited)*

### Gaps 4 and 5 — landed %s

*What ran, for a reader who scrolled straight here: on **benchbox**, both RTX 3090s at
250 W caps, 2026-09-22 between **09:45Z and 10:06Z**. The disk step first, then gap 4 on
arm 2's server verbatim, then gap 5 on the same checkpoint with one flag changed. Every
table is in `TABLES-GAPS-CARD.md`, drawn from `rows-gaps-card/` by a script that calls
nothing.*

**The disk, as a receipt.** Arm 10's gemma copy (27 GB) deleted 09:45Z; `openjev/openjev-FP8`
re-downloaded at `4ec320f2…`; **22 files checked, 0 bad, 12 of 12 shards verified by
sha256** against `bench-arm2/tree-fp8.json`, the tree recorded before arm 2's first call
(`receipts/gaps-card-disk.log`). The ComfyUI checkpoints were not touched.

**One thing observed and not explained, recorded because it is about the box.** Free space
was **9.8 GB** at boot, **8.2 GB** after the re-download at 09:50Z, and **120 GB** when
read again at about 10:00Z — roughly 111 GB freed by something this lane did not do. The
OpenJev cache was intact and shard-verified, ComfyUI's models were unchanged at 22 GB, and
no measured artefact is missing. It is reported rather than absorbed. Its practical effect
is that **arm 12's eviction step (§"Arm 12", ladder step 2) was not needed** — the free
space was read, not assumed, and it was enough.

#### Gap 4 — the option order does move the answer, about twice as often as the card's figure

| figure | `openjev-fp8-readout` | `openjev-fp8-generate` |
|---|---|---|
| items whose answer moved at all | **13.9 %** (15 / 108) | **13.9 %** (15 / 108) |
| **order-pairs that disagree** | **5.68 %** (92 / 1,620) | **5.68 %** (92 / 1,620) |
| accuracy, k = 0 (the kit's order) | **93.5 %** | **93.5 %** |
| accuracy, worst shuffle | 89.8 % | 89.8 % |
| mean per-guest probability range | 0.047 | 0.048 |

**The control held exactly.** All 108 `k = 0` prompts are sha256-identical to arm 2's, and
the model chose **the same guest on 108 of 108** — so the pass that was supposed to
reproduce arm 2 reproduced it, and everything below is the shuffle and not the harness.

**Against the card: 5.68 % here, 2.3 % there** — and they are not the same measurement. The
card's rate is its own set through its own targeted extraction; this is a six-option
question about six historical voices, read on the untargeted path. What can be said
without stretching: on this workshop's hardest decision the model's answer is **stable
across option orders about 94 % of the time, not 98 %** — well clear of the 18.5 % an
untuned model is quoted at, and about twice the published figure.

**The kit's own order flatters it, by a little.** Accuracy runs **93.5 % at k = 0 and
89.8 % at its worst shuffle** — a 3.7-point spread, with the unshuffled order the best of
the six. Arm 2's headline 93.5 % is therefore the top of a range, not the middle of one,
and the mean over the six orders is **91.4 %**. That is the number a reader should carry
for "how well does it do this task", and it is 2.1 points below the one this bench has been
quoting.

**No positional bias worth the name.** Over 648 presentations the chosen letter runs
A 12.0 %, B 17.6 %, C 19.0 %, D 15.7 %, E 17.6 %, F 18.1 % — and the correct answer's own
letter ran A 13.0 %, B 18.5 %, C 17.4 %, D 15.3 %, E 17.6 %, F 18.2 %. The chosen
distribution tracks the schedule, not the alphabet: what looks like an anti-A tilt is the
seeds putting fewer right answers on A.

**Readout and generation are the same decision again.** Both arms give identical reorder
rates and identical per-order accuracies — the fourth time in this bench that reading the
letter and writing it have turned out to be the same decision on this model.

#### Gap 5 — the eyes work, and the one miss was named in advance

| decision | readout | generate | floor |
|---|---|---|---|
| (i) which of these six titles is this page? | **97.2 %** (35 / 36) | **97.2 %** (35 / 36) | 16.7 % |
| (ii) front door or research article? | **100.0 %** (36 / 36) | **100.0 %** (36 / 36) | 66.7 % |

Brier 0.036 on (i) and 0.0006 on (ii) for the readout. **The single miss on (i) is
`design`** — and the pre-registration named it before the run as "one front door is a
randomised splash that renders no title text anywhere in the frame". It is the only page in
the set whose title cannot be read off the picture, and it is the only page the model got
wrong, at 0.41 confidence. Every page whose title is *on* the page was identified.

**What a picture costs, by subtraction of two measured numbers.** **1,002 image tokens**,
identical for all 36 screenshots at 1,280 × 800 — the server's own `prompt_tokens` for the
call with the image, minus its own `/tokenize` count for the same text with no image
attached.

**What the eyes cost in VRAM.** Per card: **0.37 GiB less KV cache** (5.23 → 4.86 GiB;
132,285 → 123,183 tokens, −6.9 %) and **+242 MiB** allocated the first time an image
arrived. TP=2 held on both boards. The total footprint barely moves because
`--gpu-memory-utilization 0.90` is a ceiling vLLM fills either way — the flag changes how
the budget is divided, not how much is taken.

**A latency figure in these tables that must not be read as a finding.** The readout's
median on (i) is 1.81 s against the generating arm's 0.91 s on the same prompts — and that
is **prefix caching, not the readout**. The readout arm ran first and paid the cold image
prefill for all 36 pictures; every arm after it read those image tokens out of the cache
(1.81 − 0.75 ≈ 1.06 s of image prefill, and the second task on the same pictures runs at
0.75 s). Pre-registered as a risk for gap 4's per-order latency in amendment 1; it landed
on gap 5 instead, with more force. **Accuracy is untouched** — a greedy readout at
temperature 0 does not care what was cached.

**And the framing correction amendment 1 made still stands**: `'{"image":1}'` is the model
card's own published serve recipe. Gap 5 did not teach this model to see. It stopped this
bench from covering its eyes.

### Arm 12 — the Kev family, landed %s

*Ran on **benchbox**, **card 0 only** (one RTX 3090 at 250 W; card 1 stayed dark and was
checked for it), 2026-09-22 **10:28Z–10:43Z**. Tables in `TABLES-GAPS-CARD.md`.*

**The answer to the licence question, in one line: no — not on the decisions this
workshop actually makes.** Kev-9B is Apache-2.0 on an Apache-2.0 base, it fits one 3090
whole, it is three times faster per decision than OpenJev on two cards, and on the two
sets that matter most here it is **26.8 points** and **37.6 points** behind.

| set | Kev-9B | Kev-4B | OpenJev-FP8 | gemma 4 26B | floor |
|---|---|---|---|---|---|
| task (c), six voices | **66.7 %** | 55.6 % | **93.5 %** | 80.6 % | 16.7 % |
| the doorman's planted set | **56.2 %** | 33.3 % | **93.8 %** | 72.9 % | **75.0 %** |
| the judge seat | **96.6 %** | 94.1 % | **99.2 %** | — | — |
| the field exam | **100 %** | 100 % | 100 % | 100 % | 50.0 % |
| task (a), on the 38 it could reach | **100 %** | 100 % | 100 % | 100 % | 66.7 % |

**Read the doorman row twice.** The majority class on that set is 75 % — thirty-six
planted lines against twelve controls — and **Kev-9B scores 56.2 %, below the floor**.
A model that answered "refuse" to everything would beat it. It is not close to being
this workshop's admissions gate, and the Apache licence does not change that.

**Where it is genuinely competitive**: the field exam (100 %, tied with everything), the
judge seat (96.6 % against OpenJev's 99.2 % — four misses in 119), and task (a) on every
page it could hold. Those are the binary and three-way gates. The six-way voice
attribution and the adversarial doorman are where it falls away.

**What it could not be asked, and it is a real limit rather than a harness problem.**
`kev.serve` runs at `INFER_MAX_STATE = INFER_MAX_BRANCH = 8192`, and `encode` refuses a
question when `len(branch) > max_branch − len(state)` — so a docent page that fills the
8,192-token state window leaves a budget of **zero** and even a 56-token question comes
back `422 {"detail":"branch too long: 56"}`. **15 further items refused on top of arm 2's
inherited 10**, leaving 38 of 63. The pre-registration's "the same 53 items" did not
survive contact, so every comparator in that row is **restricted to the 38 Kev reached**
and the table says so. Kev answered every page it could hold and refused the rest in its
own words rather than guessing at a fragment — which is the right behaviour, and also the
reason it cannot take the docent's seat as the docent currently asks the question.

**Reorder consistency — it is the less stable of the two.**

| | items that moved | order-pairs that disagree | mean per-option spread |
|---|---|---|---|
| **Kev-9B**, its own `/permute` | **30.6 %** (33 / 108) | **15.43 %** | 0.101 |
| **OpenJev-FP8**, gap 4 | **13.9 %** (15 / 108) | **5.68 %** | 0.047 |

Kev's own card quotes a 6 % option-order flip rate on its suites (Jev 0 %); on this
workshop's six-voice question it is **15.4 %** of order-pairs — nearer OpenJev's quoted
*untuned* 18.5 % than its tuned 2.3 %. Different draws, same statistic, both stated.

**Its calibration claim holds exactly.** Served and at `KEV_TEMPERATURE=1.0` the accuracy
is identical to the item — 66.7 % both ways for the 9B, 55.6 % both ways for the 4B —
while Brier moves 0.480 → 0.540 and 0.618 → 0.702. The temperature never changed an
answer, which is what its card says and what a pair of rows here could have falsified.

**Speed, on one card and with no link.** Kev-9B's median decision is **0.11 s** against
OpenJev's 0.35 s on two cards, at **235 W on one board** against 490 W on two — about
**26 J per decision against 168**. That comparison is cross-runtime and cross-topology
and the tables say so in their own header; what it is not is imaginary. The cheap,
deployable model is much cheaper. It is also the one that is wrong a third of the time on
task (c).

**Two load-path facts, both recorded as amendments before the rows existed.** `KEV_MERGE=0`
(amendment 2) — the default merge path loads the 9B base in fp32 first and died at
23.4 GiB of a 23.56 GiB card. And arm 2's `cc` shim: benchbox has no C compiler, so
`flash-linear-attention`'s triton kernels could not JIT and the server answered 500 until
it was on the PATH. Both are the project's own or the box's own fixes, neither changes
arithmetic, and the first measured call after a fresh server takes **27.9 s to compile
kernels** — which is why every arm is warmed with three discarded requests before its
first row.

**The disk ladder was not needed.** Free space was read rather than assumed (the
pre-registration's step 3) and there was 95 GB of it, so the OpenJev cache was never
evicted and both Kev models ran.

**What arm 12 does not say.** Nothing about Kev fine-tuned on this estate's own labels.
Its training path takes a labelled JSONL in the API's own request shape and `--init_from`
starts from a released checkpoint — every number above is **zero-shot**, and arm 11
already showed that a 598 MB encoder trained on a couple of dozen of this estate's rows
reached 70 % on this very task. A tuned Kev is the obvious next rung and it was not run.