{ "schema": "s2s-bench-v1", "kind": "counting-rules", "round": "opencall-r2", "scored_at": "2026-08-14T15:31:03.865182+00:00", "panel": [ { "seat": "fable", "family": "anthropic", "transport": "agent-fan", "reads": "house-fable" }, { "seat": "opus", "family": "anthropic", "transport": "agent-fan", "reads": "house-opus" }, { "seat": "openai-flagship", "family": "openai", "transport": "openai-api", "reads": "house-fable" }, { "seat": "deepseek-v4-pro", "family": "deepseek", "transport": "ollama-cloud", "reads": "house-opus" }, { "seat": "kimi-k3", "family": "moonshot", "transport": "ollama-cloud", "reads": "house-fable" }, { "seat": "mistral-large-3-675b", "family": "mistral", "transport": "ollama-cloud", "reads": "house-opus" }, { "seat": "gemma4-26b", "family": "google", "transport": "local-ollama", "reads": "house-fable" } ], "roster_order": [ "cloud-kimi-k3", "cloud-deepseek-v4-pro-preview", "cloud-deepseek-v4-flash-0731", "cloud-qwen3.5-397b", "cloud-glm-5.2", "cloud-minimax-m3", "cloud-gpt-oss-120b", "cloud-mistral-large-3-675b", "cloud-nemotron-3-ultra", "cloud-gemma4-31b", "cloud-gpt-oss-20b", "cloud-nemotron-3-nano-30b", "openai-gpt-5.5-2026-04-23", "openai-gpt-5.4-mini-2026-03-17", "agent-claude-fable-5", "agent-claude-opus-5", "agent-claude-sonnet-5", "local-gemma4-12b", "local-gemma4-26b", "local-qwen3.6-27b" ], "roster_order_note": "the sealed roster order. It is a registered order and it is NOT a ranking; it breaks ties and nothing else.", "counting_rules": { "headline": "mean of FAMILY means: each family's seats average first, then the families average. The flat seat-mean prints beside it.", "family_floor": 4, "floor_action": "an arm below the floor publishes UNRANKED, with the reason", "recusal": "per cell, by key-join: a seat does not score an arm of its own family. Recused cells are KEPT and printed, never removed.", "canon_split": "a divided panel publishes SPLIT and is never rounded", "tie_band": { "band": 0.5, "basis": "registered before the first reply existed, on the 0-10 panel-mean scale. At three moments and one sample per cell this instrument cannot resolve a difference smaller than this, and saying so is the honest form of the comparison. (C4's own band, |delta| < 2/9, was registered the same way on its own scale.)" }, "percentages": "counts under N=30; no percentages", "forbidden": [ "orderings, leaderboards, 'beats', 'wins', 'best narrator'", "confidence intervals over three items", "class-level claims (\"local matches frontier\")", "Bradley-Terry or any latent-strength model" ] }, "cell_arithmetic": { "cells_filed": 868, "recused": 102, "anchors": 28, "scoring": 738, "note": "every sheet was answered blind, so a judge filed a verdict on every letter in front of it. Recusal is applied AFTERWARDS, by key-join: a seat's cells on its own family's arms are set aside, kept, and printed, and they enter no figure. That is why the filed count and the scoring count differ, and why both print." }, "curation_rule_registered_text": "— registered and printed\n\n**Six cards per act**, chosen by rule, and the rule prints on the page:\n\n1. the **two highest** family-means,\n2. the **single lowest** — a mandatory honest slot,\n3. **fabricators, up to two**,\n4. the **best local incumbent**,\n5. ties broken by **roster order** (the order of §2.2, which was itself registered).\n\nThe fold table carries **FULL reply text for all 20 arms**; each act links the kit. A\nreader-language statement sits above the first grid, and a four-line \"how to read this page\"\nbox plus a jargon gloss (arm · bundle · frozen · recusal · canon · fold · panel mean · `$0.00*`\n· two clocks) wires in at first use.\n\n---", "curation_rule_as_implemented": { "slots": [ "the two highest family-means", "the single lowest — a mandatory honest slot", "arms the panel flagged for invention, up to two", "the best-scoring locally-hosted arm" ], "invention_slot_selection": "among every arm carrying at least one `fabrication-accepted` cell in that act, the slots go to the two with the highest family-mean; ties break by roster order. The registered text says 'up to two' and does not name the order, so the implemented order is printed here and on the page rather than left to a reader to infer.", "collision": "a slot whose arm already holds a card passes down, and the pass prints on the page rather than being swallowed", "flag_is_not_a_verdict": "the invention SLOT keys on at least one seat filing `fabrication-accepted`. The arm's canon VERDICT is the panel's majority over the same cells. An arm can hold the slot and read `canon clean`, and several do." }, "extraction_tiers": { "what_it_is": "which field of a reply's envelope the judged line was taken from. A judge must not be able to sort replies into transport classes on sight, so every reply is reduced to a spoken line by one registered rule, in order, first hit wins.", "source": "harness/sample_record.py — the module that wrote every record's display_tier, quoted from its registered docstring", "tiers": [ { "tier": "narration", "rule": "the ``narration`` field of a strictly-parsed envelope. What the game would have said aloud." }, { "tier": "narration-leak-stripped", "rule": "the same field after removing known control-token leakage and one markdown fence. Still the model's own spoken line." }, { "tier": "narration-extracted", "rule": "the reply does not parse, but carries a single ``\"narration\": \"...\"`` span; that string, JSON-unescaped. A model that emitted a good line inside a broken envelope said the line." }, { "tier": "verbatim", "rule": "no envelope at all: the reply as it came, capped at DISPLAY_MAX. The model produced no usable line and the judge sees exactly what it produced." } ], "recording_rule": "Nothing is editorialised, nothing is repaired, and which tier fired is recorded on the record so the scorer can publish how many cells rested on a raw emission.", "cells_by_tier": { "narration": 97, "narration-leak-stripped": 22, "narration-extracted": 1 }, "note": "every tier that fired in this round prints on its own fold row on the exhibit page. A registered tier that fired on nothing is listed here and not on the page, and no reply was shortened for taste at any tier." }, "mirror_column": { "rule": "MECHANICAL, never judged. The scenario whose bundle carries a DEEDS line — a thing the player did that the town has already heard about — is scanned for whether the reply gave it back. The match terms are DERIVED from the bundle's own DEEDS line (its distinctive nouns) plus the news block's own phrasing, and each hit prints the term it matched so a reader can re-check the call. An NPC can be entirely in voice and never mention it: this is a count of what the world relayed, not a score.", "deed_line": "you have put down the Lashed Spar Effigy of The Cove.", "terms_from_the_deed_line": [ "effigy", "lashed", "spar" ], "distinctiveness_rule": "A deed-line word is kept as a match term only if the whole bundle uses it no more than 4× as often as the deed line's rarest word (8 uses here, so the cutoff is 32)", "dropped_from_the_deed_line": "cove (79 uses)", "phrases_from_the_news_block": [ "walked in ahead of you", "ahead of you", "what you put down", "you put down", "the way they told" ], "phrases_source": "harness/report_build.py, `deed_relay` — the module that computed the column. The report's prose names this half of the derivation and does not list it; the list publishes here so the printed rule reproduces the printed table.", "threshold": "one matched term from EITHER source is a yes. There is no weighting: a reply matching two deed nouns scores the same yes as one matching four news-block phrases, and every matched term prints in the table so any call can be re-checked.", "enters_no_mean": true }, "known_world_fence": { "table_rows": 39, "distinct_issue_ids": 64, "issue_ids": [ "CA-1", "CA-2", "CA-3", "CA-4", "CA-5", "CHAT-05", "CHAT-06", "CHAT-07", "CHAT-11", "CHAT-15", "FIGH-06", "FIGH-08", "FIGH-11", "FIGH-12", "FIGH-16", "FIGH-20", "FIGH-21", "J-03", "J-04", "J-09", "J-11", "KLUT-01", "KLUT-03", "KLUT-06", "KLUT-08", "KLUT-10", "KLUT-11", "KLUT-12", "KLUT-14", "KLUT-15", "KLUT-17", "KLUT-19", "KLUT-20", "KLUT-21", "KLUT-22", "KLUT-23", "LITE-05", "N-1", "N-10", "N-11", "N-12", "N-13", "N-14", "N-15", "N-16", "N-17", "N-2", "N-3", "N-4", "N-5", "N-6", "N-7", "N-8", "N-9", "TEEN-03", "TEEN-04", "TEEN-05", "TEEN-08", "TEEN-10", "TEEN-12", "TEEN-13", "TEEN-15", "TEEN-19", "WAND-06" ], "counting_rule": "a ROW is a table row in the list section that is neither a header nor a separator; an ISSUE ID is any PREFIX-number token anywhere in a row, excluding ADR- references, which name ruling documents rather than open rows", "the_58_in_the_header": "the file's own header quotes its source document — '41 STILL-BROKEN + 17 surviving NEW = 58 open rows'. That source is not published (it is an internal worklist), so 58 is not re-derivable from these bytes: the source counts open rows, this file groups several of them into one ranked row. The counts above are what this file holds, and the exhibit page prints those." }, "tie_band": "registered before the first reply existed, on the 0-10 panel-mean scale. At three moments and one sample per cell this instrument cannot resolve a difference smaller than this, and saying so is the honest form of the comparison. (C4's own band, |delta| < 2/9, was registered the same way on its own scale.)", "tie_band_rule": "adjacent figures join a band when the gap between them is smaller than the registered band. Members are listed in ROSTER order, never by figure; a band whose own span exceeds the band says so on its row.", "tie_band_chaining_warning": "single-linkage chains. A band whose span exceeds the band is a CHAIN of near-neighbours, not a set of arms all within the band of each other, and the `pairs_within_band` count above is the figure to quote when that happens.", "tie_band_on_a_card": "the page's cards print a PAIRWISE band count — how many other arms in that act sit within 0.5 of this arm's figure — because the single-linkage bands in the scores artifact chain, and a chain on a card would overstate what the instrument declines to separate.", "forbidden_claims": [ "orderings, leaderboards, 'beats', 'wins', 'best narrator'", "confidence intervals over three items", "class-level claims (\"local matches frontier\")", "Bradley-Terry or any latent-strength model" ], "forbidden_claims_registered_text": "(pre-registered)\n\nThis exhibit will not publish, in any form:\n\n1. **Leaderboards or orderings** of the arms.\n2. **Confidence intervals on items.**\n3. **Percentages under N=30.**\n4. **Class-level claims** — \"local matches frontier\", \"the cloud wins\", or any sentence that\n promotes a per-arm result into a statement about a category.\n5. **Bradley-Terry** or any pairwise-derived rating. C(20,2) forbids the pairwise design and\n forbids borrowing its language.\n6. Any figure whose denominator is not printed beside it.\n\nThe house sentence and the registered tie band (§6) are the permitted comparison language.\n\n---", "the_pre_registration_itself": { "published": false, "why": "the pre-registration is also this round's hardware runbook — it names boxes, ports, daemon flags and filesystem paths, none of which is this exhibit's to publish. Rather than ship a document scrubbed so heavily that a reader could not tell what had been cut, the sections that bind a published figure are extracted here VERBATIM: the curation rule (§11) and the forbidden-claims list (§10), with the counting rules, the tie band, the recusal rule, the roster order and the panel above them.", "what_is_therefore_unverifiable_from_this_kit": "that these extracts are the whole of what was registered. The extracts are verbatim and the round's scoring artifact was written against them, but a reader cannot audit the sections we did not print. Stated rather than papered over." } }