## The gate

| gated task | base readout | adapter readout | delta (points) | floor (base − 2) | verdict |
|---|---|---|---|---|---|
| (a) does the article answer this question? | 100.0 % | 100.0 % | +0.0 | 98.0 % | **PASS** |
| (c) which of the six guests said this line? | 80.6 % | 83.3 % | +2.8 | 78.6 % | **PASS** |

Verdict: **ACCEPTED**

## Accuracy

| task | arm | accuracy | correct / labelled | Brier | unparsed | floored rows |
|---|---|---|---|---|---|---|
| (a) does the article answer this question? | `jev-readout` | 100.0 % | 63 / 63 | 0.0038 | 0 | 5 |
|  | `base-readout` | 100.0 % | 63 / 63 | 0.0001 | 0 | 19 |
|  | `jev-generate` | 100.0 % | 63 / 63 | 0.0001 | 0 | 0 |
|  | `base-generate` | 100.0 % | 63 / 63 | 0.0001 | 0 | 0 |
| (b) which section is the answer in? | `jev-readout` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 0.9461 | 0 | 7 |
|  | `base-readout` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 39 |
|  | `jev-generate` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 0 |
|  | `base-generate` | **not a rate** — 1 of 42 rows carry a label | 0 / 1 | 1.0000 | 0 | 0 |
| (c) which of the six guests said this line? | `jev-readout` | 83.3 % | 90 / 108 | 0.2875 | 0 | 0 |
|  | `base-readout` | 80.6 % | 87 / 108 | 0.3635 | 0 | 101 |
|  | `jev-generate` | 83.3 % | 90 / 108 | 0.3333 | 0 | 0 |
|  | `base-generate` | 82.4 % | 89 / 108 | 0.3596 | 1 | 0 |

_Brier here is the MULTI-CLASS form — the squared error summed over every option — so on a two-option task it is twice the familiar binary Brier and must not be read against a published binary number. Lower is better; 0 is perfect. A generating arm has no probabilities, so its Brier is just its error rate doubled and its reliability table is one bin at 1.0: that is a property of writing a letter instead of scoring one, and it is the point._

## Task (a), split — the abstain half is the half that matters

| arm | on-page answered yes (of 42) | off-page answered no (of 21) | accuracy | always-yes would score |
|---|---|---|---|---|
| `jev-readout` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % |
| `base-readout` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % |
| `jev-generate` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % |
| `base-generate` | 42 (100.0 %) | 21 (100.0 %) | 100.0 % | 66.7 % |

## Latency, tokens and energy

| task | arm | median s | p95 s | prompt tokens | written tokens | mean W | J / decision | J / decision net of idle | idle W |
|---|---|---|---|---|---|---|---|---|---|
| (a) does the article answer this question? | `jev-readout` | 3.527 | 12.182 | 825,919 | 63 | 214.4 | 1181.3 | 464.8 | 130.1 |
|  | `base-readout` | 3.746 | 12.943 | 825,919 | 63 | 211.1 | 1240.1 | 490.0 | 127.7 |
|  | `jev-generate` | 3.621 | 28.089 | 825,919 | 232 | 218.4 | 1362.7 | 657.4 | 113.0 |
|  | `base-generate` | 3.841 | 29.896 | 825,919 | 128 | 215.4 | 1429.8 | 646.6 | 118.0 |
| (b) which section is the answer in? | `jev-readout` | 3.766 | 12.210 | 538,426 | 42 | 217.1 | 1298.0 | 584.4 | 119.4 |
|  | `base-readout` | 4.010 | 12.969 | 538,426 | 42 | 214.4 | 1363.5 | 606.6 | 119.0 |
|  | `jev-generate` | 3.848 | 12.214 | 538,426 | 84 | 217.5 | 1305.0 | 567.4 | 122.9 |
|  | `base-generate` | 4.024 | 13.052 | 538,426 | 90 | 214.4 | 1368.6 | 601.6 | 120.2 |
| (c) which of the six guests said this line? | `jev-readout` | 1.017 | 1.093 | 30,905 | 108 | 148.0 | 150.7 | 25.8 | 122.6 |
|  | `base-readout` | 1.025 | 1.116 | 30,905 | 108 | 146.9 | 151.0 | 28.2 | 119.4 |
|  | `jev-generate` | 1.031 | 1.090 | 30,905 | 216 | 146.8 | 150.9 | 27.1 | 120.4 |
|  | `base-generate` | 1.060 | 1.116 | 30,905 | 398 | 149.9 | 158.3 | 30.6 | 120.9 |

## Agreement between arms

| task | pair | agree | of | rate |
|---|---|---|---|---|
| (a) does the article answer this question? | `jev-readout` vs `base-readout` | 63 | 63 | 100.0 % |
|  | `jev-readout` vs `jev-generate` | 63 | 63 | 100.0 % |
|  | `jev-readout` vs `base-generate` | 63 | 63 | 100.0 % |
|  | `base-readout` vs `jev-generate` | 63 | 63 | 100.0 % |
|  | `base-readout` vs `base-generate` | 63 | 63 | 100.0 % |
|  | `jev-generate` vs `base-generate` | 63 | 63 | 100.0 % |
| (b) which section is the answer in? | `jev-readout` vs `base-readout` | 31 | 42 | 73.8 % |
|  | `jev-readout` vs `jev-generate` | 42 | 42 | 100.0 % |
|  | `jev-readout` vs `base-generate` | 31 | 42 | 73.8 % |
|  | `base-readout` vs `jev-generate` | 31 | 42 | 73.8 % |
|  | `base-readout` vs `base-generate` | 41 | 42 | 97.6 % |
|  | `jev-generate` vs `base-generate` | 31 | 42 | 73.8 % |
| (c) which of the six guests said this line? | `jev-readout` vs `base-readout` | 89 | 108 | 82.4 % |
|  | `jev-readout` vs `jev-generate` | 108 | 108 | 100.0 % |
|  | `jev-readout` vs `base-generate` | 95 | 108 | 88.0 % |
|  | `base-readout` vs `jev-generate` | 89 | 108 | 82.4 % |
|  | `base-readout` vs `base-generate` | 102 | 108 | 94.4 % |
|  | `jev-generate` vs `base-generate` | 95 | 108 | 88.0 % |

## Task (c): the floor an arm has to beat

103 of 108 lines name at least one other guest; crossing those off and guessing among the rest is worth **24.0 %**, against a six-way chance of 16.7 %

### Reliability — `jev-readout`, task a

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.8–0.9 | 4 | 0.851 | 100.0 % | -0.149 |
| 0.9–1.0 | 59 | 0.985 | 100.0 % | -0.015 |

### Reliability — `jev-readout`, task c

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.2–0.3 | 3 | 0.289 | 33.3 % | -0.044 |
| 0.3–0.4 | 11 | 0.357 | 45.5 % | -0.097 |
| 0.4–0.5 | 11 | 0.458 | 81.8 % | -0.360 |
| 0.5–0.6 | 12 | 0.541 | 75.0 % | -0.209 |
| 0.6–0.7 | 5 | 0.650 | 80.0 % | -0.150 |
| 0.7–0.8 | 10 | 0.763 | 90.0 % | -0.137 |
| 0.8–0.9 | 15 | 0.853 | 100.0 % | -0.147 |
| 0.9–1.0 | 41 | 0.971 | 92.7 % | +0.044 |

### Reliability — `base-readout`, task a

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 |

### Reliability — `base-readout`, task c

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.4–0.5 | 2 | 0.432 | 50.0 % | -0.068 |
| 0.7–0.8 | 3 | 0.770 | 33.3 % | +0.437 |
| 0.8–0.9 | 6 | 0.863 | 50.0 % | +0.363 |
| 0.9–1.0 | 97 | 0.999 | 84.5 % | +0.153 |

### Reliability — `jev-generate`, task a

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 |

### Reliability — `jev-generate`, task c

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 108 | 1.000 | 83.3 % | +0.167 |

### Reliability — `base-generate`, task a

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.9–1.0 | 63 | 0.994 | 100.0 % | -0.006 |

### Reliability — `base-generate`, task c

| bin | n | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.1–0.2 | 1 | 0.167 | 100.0 % | -0.833 |
| 0.9–1.0 | 107 | 1.000 | 82.2 % | +0.178 |

### Task (c) confusion — `jev-readout`

| said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` |
|---|---|---|---|---|---|---|
| `darwin` | **17** | 0 | 0 | 1 | 0 | 0 |
| `einstein` | 1 | **13** | 1 | 0 | 3 | 0 |
| `hypatia` | 1 | 0 | **12** | 3 | 2 | 0 |
| `ibn_sina` | 0 | 0 | 0 | **18** | 0 | 0 |
| `sagan` | 1 | 1 | 0 | 0 | **16** | 0 |
| `socrates` | 0 | 0 | 0 | 2 | 2 | **14** |

### Task (c) confusion — `base-readout`

| said by ↓ / read as → | `darwin` | `einstein` | `hypatia` | `ibn_sina` | `sagan` | `socrates` |
|---|---|---|---|---|---|---|
| `darwin` | **13** | 0 | 1 | 2 | 2 | 0 |
| `einstein` | 0 | **14** | 0 | 0 | 3 | 1 |
| `hypatia` | 0 | 3 | **12** | 0 | 2 | 1 |
| `ibn_sina` | 0 | 0 | 2 | **14** | 1 | 1 |
| `sagan` | 1 | 0 | 0 | 0 | **17** | 0 |
| `socrates` | 0 | 0 | 0 | 0 | 1 | **17** |

## Residency, per arm

| arm | task | card 0 MiB after | named grew | others quiet | ok |
|---|---|---|---|---|---|
| `base-generate` | a | 21472.0 | False | True | True |
| `base-generate` | b | 21472.0 | False | True | True |
| `base-generate` | c | 21472.0 | False | True | True |
| `base-readout` | a | 21472.0 | False | True | True |
| `base-readout` | b | 21472.0 | False | True | True |
| `base-readout` | c | 21472.0 | False | True | True |
| `jev-generate` | a | 20336.0 | False | True | True |
| `jev-generate` | b | 20336.0 | False | True | True |
| `jev-generate` | c | 20336.0 | False | True | True |
| `jev-readout` | a | 20336.0 | False | True | True |
| `jev-readout` | b | 20336.0 | False | True | True |
| `jev-readout` | c | 20336.0 | False | True | True |