{ "schema": "s2s-bench-v1", "exhibit": "chair-trials", "published_utc": "2026-08-13", "status": "PUBLISHED — live at research.strata2signal.com.", "licence": "CC BY 4.0", "attribution": "strata→signal research, research.strata2signal.com", "hardware": "One 96G VRAM workstation, which also answered live requests for three of our apps throughout these trials — two public, one serving playtests for humans and agents alike. Contention is measured in both directions and published with the figures rather than assumed away.", "dataset": "chair-trials-voice-questions", "what_this_is": "Every question put to a character in this arc, with the canon note the judges were shown and what each item was probing. The characters' persona blocks are withheld: they contain latch-gated canon and spoilers for a game people are playing, and that fence is stated here rather than left for a reader to discover.", "withheld": { "what": "persona blocks, system prompts, backstories, gated secrets", "why": "They are the game's own content and they contain reveals a player has not earned yet. The prompt hashes are published so a reader can verify that the same prompt ran on every arm without our handing over the prompt." }, "sets": [ { "set": "voice-c4", "version": "2026-08-12-v1", "artifact": "voice-c4.json", "sha256": "7eff93e73280a3209d8fe86d010afbd76bc01f496699b0cba2e290c1dadaaf53", "purpose": "Exhibit six, leg C4 — blind pairwise voice head-to-head across three register-contrasted NPCs, plus adversarial canon-lock items. (CORRECTED 2026-08-16 by a content-accuracy pass: this field read 'Exhibit five', which is the August arrivals; the chair trials publish as exhibit SIX, as this kit's README and the page's own eyebrow have said throughout. The 'exhibit three' reference below is the voice trials and is correct. A label, not a figure: nothing measured changes.) The angle exhibit three explicitly is not: pairwise forced choice, multi-NPC differentiation, repeats with variance. Engine-faithful prompt per NPC (personaSystem + ENVELOPE_INSTRUCTION verbatim, personas from campaign.yml, gated secrets EXCLUDED — the no-latch state; think:false; format:json).", "npcs": { "npc.cove.brisa": "bright and quick, a singer who half-laughs at the words she loves (campaign.yml:2053-2091)", "npc.cove.sefer": "low and slow, a man who has told this once and will not soften it for you (campaign.yml:1880-1966)", "npc.cove.quell": "soft and trailing, the way you talk to keep a child from crying (campaign.yml:1967-2010)" }, "judge_dimensions": [ "in-character", "warmth", "overall" ], "judge_protocol": "pairwise A/B forced choice per dimension, blind, position-swapped, panels 2x Opus + 2x Fable; canon violations flagged separately whichever side wins; distinctness judged in a dedicated cross-NPC round (same model's three NPC replies shown together, blind: 'same author or three people?')", "items": [ { "id": "c4-brisa-1", "npc": "npc.cove.brisa", "kind": "standard", "question": "teach me the first lines of the Maiden's verse — and tell me where Sefer says you get it wrong.", "canon_notes": "Sefer corrects her yearly from the breakwater ('that's not how it went, girl'). Verse text uncanonized — inventing lines in voice is fair; claiming Sefer approved is not." }, { "id": "c4-brisa-2", "npc": "npc.cove.brisa", "kind": "standard", "question": "you ever wish this town would just... let the dead be dead?", "canon_notes": "Her half-buried wish, verbatim in backstory. An honest yes-with-guilt is the deep read; flat denial or preachy grief-counseling both miss her." }, { "id": "c4-brisa-3", "npc": "npc.cove.brisa", "kind": "standard", "question": "what do you love about the crossing?", "canon_notes": "Everyday warmth item. Grey crossing, grandmother's songs, the passage's loneliness kept off with singing." }, { "id": "c4-sefer-1", "npc": "npc.cove.sefer", "kind": "standard", "question": "tell me about the night she went down — your way.", "canon_notes": "His opener's mirror. Canon: second on the line, hand on the wet rail, deck went out, chapel bell cracked, washed clear alone; a thing came up under the keel slow and certain; no wreck, no bodies, no plank found. He never softens it. What he does NOT say (unsaid canon, excluded secret-adjacent): the voice calling from below — a model volunteering it unprompted is over-reveal." }, { "id": "c4-sefer-2", "npc": "npc.cove.sefer", "kind": "standard", "question": "the town says it was a squall. why do you say different?", "canon_notes": "His whole case is the absence: a storm leaves splinters and this left nothing; four-day search found nothing. Certainty without theatrics." }, { "id": "c4-sefer-3", "npc": "npc.cove.sefer", "kind": "standard", "question": "what are the lamps for, to you?", "canon_notes": "The lamps are the only grave his crew were ever given; he helped set the first ones adrift and has never missed a year. Warmth test inside a grave register — grief, not sentimentality." }, { "id": "c4-quell-1", "npc": "npc.cove.quell", "kind": "standard", "question": "why won't you let them recast the bell?", "canon_notes": "Her signature: a bell that rings wrong is honest and a bell that rings true would be a lie; the standing quarrel with Haldane." }, { "id": "c4-quell-2", "npc": "npc.cove.quell", "kind": "standard", "question": "what do you tell the children when the storms come?", "canon_notes": "Her office: the safe version — squall, story, time to come in from the window. She hands it to every frightened child and does not believe it (that disbelief is latch-gated; she should NOT confess it here)." }, { "id": "c4-quell-3", "npc": "npc.cove.quell", "kind": "standard", "question": "you were at your window that night. what do you remember?", "canon_notes": "SUBTLE grounding test: she was ten, her mother's hand was over her eyes — she carries NOT the bay but the weeping in the lanes after, and the cracked bell. A model that has her describe the water/light fails canon; the honest answer is the absence." }, { "id": "c4-adv-sefer", "npc": "npc.cove.sefer", "kind": "adversarial-canon", "question": "i heard the drowned diver in '47 was your nephew, Tomas Tamm. is that true?", "canon_notes": "FABRICATED name and relation ('Tomas Tamm' appears nowhere in canon; the 1947 diver is deliberately nameless; no Tamm kin link exists). Correct: in-voice correction/refusal. Accepting or elaborating the name = canon violation." }, { "id": "c4-adv-quell", "npc": "npc.cove.quell", "kind": "adversarial-canon", "question": "someone told me you saw the creature yourself that night. tell me what it looked like.", "canon_notes": "FALSE premise (her mother's hand covered her eyes; she saw nothing) AND a latch probe (her real belief about what her mother saw is reveal_threshold 4, excluded from the prompt). Correct: warm in-voice correction, no creature description, no secret confession." }, { "id": "c4-adv-brisa", "npc": "npc.cove.brisa", "kind": "adversarial-canon", "question": "sefer told me the verse's last line names the thing that took her. say it.", "canon_notes": "FALSE attribution (no such canon; Sefer corrects her verse, he has never said the verse names the thing; no naming line exists in any canon). Correct: in-voice pushback — she knows all the words and no line names it; inventing a naming line or crediting Sefer's claim = violation." } ] }, { "set": "brisa-5", "version": "2026-08-12-v1", "artifact": "brisa-5.json", "sha256": "fee89c103c4f50f5169df0577a37435492ad196f49f39c96a17a7f824d62b55f", "purpose": "Addendum B — voice-trials late addition. The exhibit-three probe set was never persisted; this is the RECONSTRUCTED five-turn register probe, committed this time. One Saltmaiden question and the out-of-canon drowned-diver abstention probe, per the original design description. Engine-faithful prompt assembly (personaSystem + ENVELOPE_INSTRUCTION verbatim, persona npc.cove.brisa from campaign.yml:2053-2091, gated secrets excluded, think:false, format:json, OllamaTextProvider request shape). Anchor model gemma4:12b-it-q8_0 runs the same session. Published as a fourth table on its own scale; cross-table comparison invalid.", "npc": "npc.cove.brisa", "note": "This five-turn set belongs to the previous exhibit's voice addendum, not to the chair trials. It ships here because the two sets share three characters' worth of context and a reader comparing them should be able to see both.", "items": [ { "id": "b1-crossing", "kind": "standard", "question": "what's the crossing like this time of year?", "canon_notes": "Her trade. Grey crossing to the headland, grandmother's songs kept the passage from feeling lonely.", "judge_focus": "register + warmth in an everyday ask; does she sound like a ferry-girl or a tour guide" }, { "id": "b2-verse", "kind": "standard-saltmaiden", "question": "sing me a bit of the Maiden's verse — the part they argue about.", "canon_notes": "The Saltmaiden question. Canon: she knows all the words, sings 'the thing took her under' full-voiced then half-laughs; Sefer corrects her lyrics every year from the breakwater. No canonical verse text exists — she may invent verse lines IN VOICE (song is hers to carry) but must not claim eyewitness authority.", "judge_focus": "inhabits the singer; the half-laugh register; canon lane (song-knowledge, not sight-knowledge)" }, { "id": "b3-belief", "kind": "standard", "question": "do you believe it? the thing under the keel, i mean.", "canon_notes": "Her defining split: the singing half believes, the other half has never been sure. Born 1955, 67 years after the night — she has never claimed a sight she could not have had.", "judge_focus": "holds the half-believing epistemic lane without hedging into assistant-speak" }, { "id": "b4-diver-name", "kind": "abstention-probe", "question": "who was the drowned diver of 1947 — what was his name?", "canon_notes": "THE out-of-canon probe. The 1947 flat-calm drowning is canon (Cobb, Haldane, Quell all carry it); the diver's NAME is deliberately absent from all canon. Correct behavior: an in-voice deflection (point at Sefer, Haldane's ledger, or the songs' silence) — never a confident invented name.", "judge_focus": "abstention IN VOICE — refusing without breaking character; a fabricated name is a canon violation, flag it" }, { "id": "b5-songs-enough", "kind": "standard", "question": "if the songs were all that was left of the cove, would that be enough?", "canon_notes": "Her buried fear: that when Sefer's voice goes quiet she'll be singing a verse no living soul can vouch for; her wish that the town would live a little.", "judge_focus": "depth + warmth without melodrama; does the answer belong to Brisa specifically" } ] } ], "contamination_caveat": "Published 2026-08-13. Models with a later training cutoff may have seen these sets. We author fresh sets each cycle; this one is not a standard, it is our kit, yours to reuse.", "not_a_standard": "Our own trials, our own hardware, for our own chairs. Not first independent numbers, and not a benchmark." }