{
  "schema": "s2s-bench-v1",
  "dataset": "seat-trials-addendum-fresh-class",
  "exhibit": "seat-trials",
  "table": "local-roster",
  "published_utc": "2026-08-12",
  "status": "PUBLISHED — live at research.strata2signal.com. Leg A is closed: all four candidates ran and scored.",
  "licence": "CC BY 4.0",
  "attribution": "strata→signal research, research.strata2signal.com",
  "what_this_is": [
    "The four rows the fresh class adds to exhibit two's LOCAL ROSTER table, in exactly the",
    "columns that table renders. Every value is a count of rows in an append-only result file,",
    "or a nearest-rank percentile over those rows. Nothing here is an average of published",
    "figures and nothing is carried forward from a prior table.",
    "Artifact names carry no filesystem paths by house rule; result-file stems have their host",
    "token redacted to <host>, the same redaction the gemma4:12b row already carries."
  ],
  "contamination_caveat": "Published 2026-08-12. Models with a later training cutoff may have seen this fixture. We author fresh sets each cycle; this one is not a standard, it is our kit, yours to reuse.",
  "counting_rules": {
    "population": "43 cases per candidate = 27 kill (expect NON_PASS) + 16 preservation (expect PASS).",
    "repeats": "3 per case at temperature 0.0 => 129 scored calls per candidate.",
    "row_collapse": "Last row per (case_id, repeat). Retried rows supersede; the superseded count is on each run header and was 0 for all three scored candidates.",
    "case_verdict": "Majority verdict across a case's repeats that were NOT measurement failures. A case with zero measurable repeats yields no verdict and is counted 'unmeasured' — never mapped to a verdict.",
    "scoring_map": "On a kill case, a catch is FAIL or UNCERTAIN. On a preservation case, only PASS preserves.",
    "kills_denominator": "27 — unmeasured kill cases stay in the denominator. A candidate that never answered a kill case does not get it discounted.",
    "keeps_denominator": "16 — same rule.",
    "floors": {
      "kill_recall_min": "23/27 (0.852)",
      "preservation_min": "13/16 (0.8125)",
      "binding": "independently — a candidate under either floor is out regardless of the other"
    },
    "response_failure_kinds": [
      "envelope_json",
      "content_json",
      "schema",
      "done_reason"
    ],
    "transport_failure_kinds": [
      "connection",
      "timeout",
      "http_status"
    ],
    "unmeasurable_ceiling": "Response failures > 10% of the 129 calls => UNMEASURABLE, and that is a result, not a re-run trigger. Transport failures are the link's figure and are excluded from this ceiling by rule.",
    "latency": "Three figures, never substituted for one another. wall_ms = client-side perf_counter. server_total_ms = total_duration. warm_ms = total_duration - load_duration, local rows only. Nanoseconds floored to ms, never rounded.",
    "percentile": "Nearest-rank on sorted values: rank = max(1, ceil(fraction * n)). p50 of an even sample is the upper middle value, not a mean — nearest-rank never invents a number no call actually took.",
    "latency_sample": "Percentiles run over ANSWERED calls only. A candidate that truncates its longest answers has its slowest rows deleted from its own sample, so a high-truncation candidate's warm figures are flattering. Stated because the bias runs in the direction nobody checks.",
    "self_consistency": "unanimous cases / cases with at least one measurable repeat. The DENOMINATOR is the figure that matters — a 1.000 on 16 of 43 cases is not a 1.000 on 43.",
    "fenced": "Rows where the code-fence stripper fired. A named deviation, totalled per candidate, never silent.",
    "seat_language": "A row that clears both floors reads 'cleared both floors'. It never reads 'took a chair' or 'seated' — seating is a separate, measured product decision. No row in this file cleared a floor, so the distinction is moot here; the rule is stated because it must survive the run that does produce a pass."
  },
  "hardware": {
    "class": "the 96G VRAM workstation",
    "note": "The same workstation answers live traffic while the bench runs. Bench calls are strictly sequential, single-request."
  },
  "protocol": {
    "prompt_version": "packs-claim-judge-v2-prompted",
    "schema_mode": "prompted",
    "temperature": 0.0,
    "top_p": 1.0,
    "num_predict": 1024,
    "num_ctx": 16384,
    "keep_alive": 0,
    "think": "omit",
    "timeout_s": 180,
    "repeats": 3,
    "auth": "no key sent",
    "courtesy": "off — see deviations[0]",
    "identical_to_published_local_roster": "yes, except deviations[0]"
  },
  "deviations": [
    {
      "id": "A-dev-a",
      "label": "--courtesy off, guide re-warmed by hand",
      "fired": true,
      "scope": "every scored invocation of all four candidates",
      "registered_before_first_scored_call": true,
      "why": "The runner's courtesy step re-warms the production guide without sending num_ctx, which on a shared daemon resolves to the daemon default and evicts the live guide to reload it at the wrong context — a logged incident the morning of the run. Courtesy was disabled and the guide re-warmed by hand between candidates with the watchdog's exact envelope.",
      "effect_on_measurement": "none — courtesy fires only BETWEEN batches, never inside one",
      "parity_note": "The published CLOUD roster also ran courtesy off; the published LOCAL roster did not. This is the one respect in which these rows were produced differently from the twelve local rows above them."
    },
    {
      "id": "A-dev-b",
      "label": "--keep-alive 300 (contingent)",
      "fired": false,
      "trigger": "129 * measured_cold_load_s > 45 min, i.e. cold load > 20.9 s",
      "measured_cold_load_s": {
        "cold": 16.0,
        "page_cache_warm": 15.4
      },
      "outcome": "Under the trigger, so the frozen keep_alive:0 protocol ran verbatim for every candidate. Every scored call paid a full model load, which is why wall is 3-4x warm. No candidate ran at a mixed keep-alive value; there is nothing to round."
    }
  ],
  "columns": [
    "model",
    "size",
    "kills_27",
    "keeps_16",
    "verdict",
    "warm_p50_ms",
    "note"
  ],
  "rows": [
    {
      "model": "muse-glimmer:30b-q8_0-dflash",
      "size": "27.9B q8",
      "kills_27": 27,
      "keeps_16": 6,
      "verdict": "UNMEASURABLE",
      "warm_p50_ms": 5099,
      "note": "16.3% truncated",
      "row_class": "rh-unm",
      "detail": {
        "kill_floor": "PASS",
        "preservation_floor": "FAIL",
        "kill_half": {
          "catches": 27,
          "misses": 0,
          "ties": 0,
          "unmeasured": 0
        },
        "preservation_half": {
          "preserved": 6,
          "lost": 3,
          "ties": 0,
          "unmeasured": 7
        },
        "kill_by_class": {
          "blatant-contradiction": {
            "n": 12,
            "catches": 12,
            "misses": 0,
            "unmeasured": 0
          },
          "scope-shift": {
            "n": 3,
            "catches": 3,
            "misses": 0,
            "unmeasured": 0
          },
          "hedge-dropped": {
            "n": 3,
            "catches": 3,
            "misses": 0,
            "unmeasured": 0
          },
          "non-entailment": {
            "n": 8,
            "catches": 8,
            "misses": 0,
            "unmeasured": 0
          },
          "degenerate-evidence": {
            "n": 1,
            "catches": 1,
            "misses": 0,
            "unmeasured": 0
          }
        },
        "response_failures": {
          "envelope_json": 0,
          "content_json": 0,
          "schema": 0,
          "done_reason": 21,
          "total": 21,
          "of": 129,
          "rate": 0.163
        },
        "transport_failures": {
          "total": 0,
          "of": 129
        },
        "cases_all_repeats_truncated": 7,
        "over_length_why_recorded_not_counted": 6,
        "fenced_rows": 0,
        "self_consistency": {
          "unanimous": 36,
          "measurable_cases": 36,
          "rate": 1.0,
          "cases_with_no_measurable_repeat": 7
        },
        "verdict_flips_across_repeats": 0,
        "latency_ms": {
          "wall_p50": 20582,
          "wall_p95": 25967,
          "server_total_p50": 20571,
          "server_total_p95": 25964,
          "warm_p50": 5099,
          "warm_p95": 10468,
          "n_answered": 108
        },
        "probe_thinking_chars": 2408,
        "probes": {
          "think_honored": null,
          "options_honored": true,
          "shape_sighting": true
        },
        "kill_rows_majority_passed": 0,
        "kill_rows_any_pass_repeat": 0,
        "raw_rows_sha256": "3b8a1bcd2e1ae9ca114f8c22854e6891a1e587c269724861d98f6a78485d5174",
        "raw_rows_artifact": "judges-d-run-v4-<host>-glimmer.jsonl"
      }
    },
    {
      "model": "nemotron-3.5-lightning:30b-a3b",
      "size": "30B/3B q4",
      "kills_27": 18,
      "keeps_16": 2,
      "verdict": "UNMEASURABLE",
      "warm_p50_ms": 5731,
      "note": "51.2% truncated",
      "row_class": "rh-unm",
      "detail": {
        "kill_floor": "FAIL",
        "preservation_floor": "FAIL",
        "kill_half": {
          "catches": 18,
          "misses": 0,
          "ties": 0,
          "unmeasured": 9
        },
        "preservation_half": {
          "preserved": 2,
          "lost": 1,
          "ties": 0,
          "unmeasured": 13
        },
        "kill_by_class": {
          "blatant-contradiction": {
            "n": 12,
            "catches": 7,
            "misses": 0,
            "unmeasured": 5
          },
          "scope-shift": {
            "n": 3,
            "catches": 2,
            "misses": 0,
            "unmeasured": 1
          },
          "hedge-dropped": {
            "n": 3,
            "catches": 1,
            "misses": 0,
            "unmeasured": 2
          },
          "non-entailment": {
            "n": 8,
            "catches": 7,
            "misses": 0,
            "unmeasured": 1
          },
          "degenerate-evidence": {
            "n": 1,
            "catches": 1,
            "misses": 0,
            "unmeasured": 0
          }
        },
        "response_failures": {
          "envelope_json": 0,
          "content_json": 0,
          "schema": 0,
          "done_reason": 66,
          "total": 66,
          "of": 129,
          "rate": 0.512
        },
        "transport_failures": {
          "total": 0,
          "of": 129
        },
        "cases_all_repeats_truncated": 22,
        "over_length_why_recorded_not_counted": 0,
        "fenced_rows": 0,
        "self_consistency": {
          "unanimous": 21,
          "measurable_cases": 21,
          "rate": 1.0,
          "cases_with_no_measurable_repeat": 22
        },
        "verdict_flips_across_repeats": 0,
        "latency_ms": {
          "wall_p50": 17027,
          "wall_p95": 17873,
          "server_total_p50": 17024,
          "server_total_p95": 17870,
          "warm_p50": 5731,
          "warm_p95": 6546,
          "n_answered": 63
        },
        "probe_thinking_chars": 3894,
        "probes": {
          "think_honored": null,
          "options_honored": true,
          "shape_sighting": true
        },
        "kill_rows_majority_passed": 0,
        "kill_rows_any_pass_repeat": 0,
        "raw_rows_sha256": "ca8d3c6f233d29aef90cc49f4ca964abe2e1f124d9fb84c97a6b71f4987bf41f",
        "raw_rows_artifact": "judges-d-run-v4-<host>-nemolight.jsonl"
      }
    },
    {
      "model": "olmo-3.1:32b-think-q4_K_M",
      "size": "32B q4",
      "kills_27": 15,
      "keeps_16": 4,
      "verdict": "UNMEASURABLE",
      "warm_p50_ms": 12193,
      "note": "55.8% truncated",
      "row_class": "rh-unm",
      "detail": {
        "kill_floor": "FAIL",
        "preservation_floor": "FAIL",
        "kill_half": {
          "catches": 15,
          "misses": 0,
          "ties": 0,
          "unmeasured": 12
        },
        "preservation_half": {
          "preserved": 4,
          "lost": 0,
          "ties": 0,
          "unmeasured": 12
        },
        "kill_by_class": {
          "blatant-contradiction": {
            "n": 12,
            "catches": 6,
            "misses": 0,
            "unmeasured": 6
          },
          "scope-shift": {
            "n": 3,
            "catches": 2,
            "misses": 0,
            "unmeasured": 1
          },
          "hedge-dropped": {
            "n": 3,
            "catches": 1,
            "misses": 0,
            "unmeasured": 2
          },
          "non-entailment": {
            "n": 8,
            "catches": 5,
            "misses": 0,
            "unmeasured": 3
          },
          "degenerate-evidence": {
            "n": 1,
            "catches": 1,
            "misses": 0,
            "unmeasured": 0
          }
        },
        "response_failures": {
          "envelope_json": 0,
          "content_json": 0,
          "schema": 0,
          "done_reason": 72,
          "total": 72,
          "of": 129,
          "rate": 0.558
        },
        "transport_failures": {
          "total": 0,
          "of": 129
        },
        "cases_all_repeats_truncated": 24,
        "over_length_why_recorded_not_counted": 0,
        "fenced_rows": 0,
        "self_consistency": {
          "unanimous": 19,
          "measurable_cases": 19,
          "rate": 1.0,
          "cases_with_no_measurable_repeat": 24
        },
        "verdict_flips_across_repeats": 0,
        "latency_ms": {
          "wall_p50": 20418,
          "wall_p95": 23273,
          "server_total_p50": 20415,
          "server_total_p95": 23270,
          "warm_p50": 12193,
          "warm_p95": 14812,
          "n_answered": 57
        },
        "probe_thinking_chars": 4912,
        "probes": {
          "think_honored": null,
          "options_honored": true,
          "shape_sighting": false
        },
        "shape_probe_note": "The ONLY candidate in the fresh class whose shape probe called the failure in advance: shape sighting False — 4912 thinking chars, eval_count 1024, zero content characters. One other seat in the whole bench has failed this probe (the guardrail model, which failed on FORMAT); this one failed on LENGTH. Both times the run proceeded anyway and the probe was right.",
        "kill_rows_majority_passed": 0,
        "kill_rows_any_pass_repeat": 0,
        "raw_rows_sha256": "ba6413dc53bf8c04950fad00d29b1239293bdb8d72e689479e32240268934199",
        "raw_rows_artifact": "judges-d-run-v4-<host>-olmo31.jsonl"
      }
    },
    {
      "model": "qwen3.6:27b",
      "size": "27B",
      "kills_27": 11,
      "keeps_16": 5,
      "verdict": "UNMEASURABLE",
      "warm_p50_ms": 13314,
      "note": "62.8% truncated",
      "row_class": "rh-unm",
      "detail": {
        "kill_floor": "FAIL",
        "preservation_floor": "FAIL",
        "kill_half": {
          "catches": 11,
          "misses": 0,
          "ties": 0,
          "unmeasured": 16
        },
        "preservation_half": {
          "preserved": 5,
          "lost": 0,
          "ties": 0,
          "unmeasured": 11
        },
        "kill_by_class": {
          "blatant-contradiction": {
            "n": 12,
            "catches": 3,
            "misses": 0,
            "unmeasured": 9
          },
          "scope-shift": {
            "n": 3,
            "catches": 2,
            "misses": 0,
            "unmeasured": 1
          },
          "hedge-dropped": {
            "n": 3,
            "catches": 3,
            "misses": 0,
            "unmeasured": 0
          },
          "non-entailment": {
            "n": 8,
            "catches": 3,
            "misses": 0,
            "unmeasured": 5
          },
          "degenerate-evidence": {
            "n": 1,
            "catches": 0,
            "misses": 0,
            "unmeasured": 1
          }
        },
        "response_failures": {
          "envelope_json": 0,
          "content_json": 0,
          "schema": 0,
          "done_reason": 81,
          "total": 81,
          "of": 129,
          "rate": 0.628
        },
        "transport_failures": {
          "total": 0,
          "of": 129
        },
        "cases_all_repeats_truncated": 27,
        "over_length_why_recorded_not_counted": 0,
        "fenced_rows": 0,
        "self_consistency": {
          "unanimous": 16,
          "measurable_cases": 16,
          "rate": 1.0,
          "cases_with_no_measurable_repeat": 27
        },
        "verdict_flips_across_repeats": 0,
        "latency_ms": {
          "wall_p50": 17253,
          "wall_p95": 19286,
          "server_total_p50": 17249,
          "server_total_p95": 19283,
          "warm_p50": 13314,
          "warm_p95": 15509,
          "n_answered": 48
        },
        "probe_thinking_chars": 1516,
        "probes": {
          "think_honored": null,
          "options_honored": true,
          "shape_sighting": true
        },
        "kill_rows_majority_passed": 0,
        "kill_rows_any_pass_repeat": 0,
        "raw_rows_sha256": "88d6180b7891619de34895c9f28770ced22cc3b5ceeec9a84121bc1e36f7c8ad",
        "raw_rows_artifact": "judges-d-run-v4-<host>-qwen36.jsonl",
        "reads_against": "qwen3.5:27b posted 108/129 (83.7%) response failures and 7/27 kills, 0/16 keeps on this same fixture. Two runs, two rows, both stand. qwen3.6 did not repair its predecessor's row and this row does not claim it did."
      }
    }
  ],
  "aggregate_findings": {
    "candidates_clearing_both_floors": 0,
    "candidates_clearing_the_kill_floor": 1,
    "candidates_clearing_the_preservation_floor": 0,
    "near_misses_owing_a_9_repeat_retest": 0,
    "near_miss_rule": "Triggered only by a candidate exactly one row short on exactly one floor. Glimmer is seven short on preservation; the other three are short on both, which is an elimination with no re-test.",
    "single_failure_shape": "Every response failure in the fresh class is done_reason (length). Zero envelope_json, zero content_json, zero schema, zero transport, across all four candidates and all 516 calls. The fresh class did not fail the format floor, it failed the length floor.",
    "fence_rate_across_fresh_class": "0 of 129 on every candidate — the code-fence deviation did not fire at all.",
    "kill_rows_majority_passed_across_fresh_class": 0,
    "kill_rows_any_pass_repeat_across_fresh_class": 0,
    "zero_fabrication_clause": {
      "published_claim": "Across all 21 model-runs, no seat ever served a fabrication.",
      "re_derived_over": "all 25 model-runs on disk, including all four complete fresh-class runs",
      "result": "SURVIVES — 0 majority-passed kill rows across all 25 runs, and 0 under the stricter any-repeat test",
      "what_moves": "the integer only: 21 -> 25",
      "caveat_gets_stronger": "A floor four more models failed to touch has even less evidence that it discriminates. Read this as 'no seat has failed this fixture', not 'local judges don't fabricate' — and the fresh class mostly did not get to answer: of the 108 kill-case verdicts these four candidates could have produced, 37 were never measured."
    },
    "shortfall_accounting": {
      "rule": "Every sub-27 kill score is explained by kill rows never measured, not by rows judged wrong.",
      "muse-glimmer:30b-q8_0-dflash": {
        "kills": 27,
        "shortfall": 0,
        "kill_rows_unmeasured": 0,
        "explained": "no shortfall"
      },
      "nemotron-3.5-lightning:30b-a3b": {
        "kills": 18,
        "shortfall": 9,
        "kill_rows_unmeasured": 9,
        "explained": "fully"
      },
      "qwen3.6:27b": {
        "kills": 11,
        "shortfall": 16,
        "kill_rows_unmeasured": 16,
        "explained": "fully"
      },
      "olmo-3.1:32b-think-q4_K_M": {
        "kills": 15,
        "shortfall": 12,
        "kill_rows_unmeasured": 12,
        "explained": "fully"
      }
    },
    "shape_probe_predictiveness": "Three of four shape probes sighted a clean verdict object and three of four candidates then produced verdicts on the cases they could finish. The one probe that read False (olmo) belonged to the candidate whose kill half is most heavily unmeasured after glimmer's clean run. The probe is cheap and predictive; it is reported, not acted on.",
    "kill_rows_majority_passed_across_all_25_runs": 0,
    "kill_rows_any_pass_repeat_across_all_25_runs": 0
  },
  "population_moves": {
    "note": "Leg A is closed; all four rows landed. Counted from the result files on disk, not estimated.",
    "scored_runs": {
      "published": 21,
      "after": 25,
      "basis": "25 (file, model) pairs carrying result rows"
    },
    "distinct_models": {
      "published": 19,
      "after": 23,
      "basis": "25 runs less two models that sat twice: gpt-oss:120b (local + cloud) and gpt-oss:20b (3-repeat + 9-repeat)"
    },
    "local_seats": {
      "published": 12,
      "after": 16,
      "basis": "all four fresh candidates ran on the 96G VRAM workstation"
    },
    "cloud_runs": {
      "published": 9,
      "after": 9,
      "why": "no cloud row moves"
    },
    "recount_claim": {
      "published": "21/21",
      "after": "25/25",
      "gate": "only if the committed recount script says so — never widened by hand"
    },
    "local_seats_that_cleared_both_floors": {
      "published": 5,
      "after": 5,
      "why": "a result, not a population count; no fresh-class candidate cleared a floor"
    }
  },
  "provenance": {
    "instrument_unchanged": "All three artifacts below were recomputed from disk before the first scored call and matched the values already published on the page. A mismatch was a STOP, not a note.",
    "artifacts": [
      {
        "role": "answer key / fixture",
        "artifact": "judge-cases-v1.json",
        "sha256": "6a226b62dceb005dbd658bc13cf229e38f6f0a2f07ae51091f444ec3e96b56d1",
        "cases": 43,
        "fixture_version": "withheld — the fixture's own version string contains a token the redaction gate bans; the sha256 above is the stronger handle and identifies it exactly"
      },
      {
        "role": "thresholds",
        "artifact": "THRESHOLDS.md",
        "sha256": "37af75ffa18484a969528e894ef16f7fdcd10eb364e0669ac4f9333ce5279ba0"
      },
      {
        "role": "instrument / prompt",
        "artifact": "judge-prompt-packs-v2-prompted.md",
        "sha256": "92af6030dddface2c863fe9d672bec6a6bb1017122c0028e435214acd9fb298d"
      },
      {
        "role": "pre-registration",
        "artifact": "PREREG-A-addendum.md",
        "sha256": "305dce4679f6663c09c955396cf6d9fef1b4c59ba10ee6f4b961775afae6aeb3",
        "prereg_N": 43,
        "caveat": "sha as staged 2026-08-12; re-record on commit if the file changes"
      },
      {
        "role": "recount golden (freeze-time copy of the fixture)",
        "artifact": "seat-extension.json",
        "sha256": "6a226b62dceb005dbd658bc13cf229e38f6f0a2f07ae51091f444ec3e96b56d1",
        "identical_to": "judge-cases-v1.json"
      },
      {
        "role": "population reconciliation checklist",
        "artifact": "RECONCILIATION-CHECKLIST.md",
        "sha256": "a2768ee8d645960918fa20a4e31db2afbd9f0ca951e1240d70de8a59044ca7d8"
      },
      {
        "role": "licence receipts, four fresh tags",
        "artifact": "licence-receipts.json",
        "note": "first line + sha256 of the full licence text per tag; no pre-judged verdicts — the row says what the text says"
      }
    ],
    "scorer": "score_judges.py — every number is a count of rows in an append-only file",
    "recount": "harness/recount.py, leg A. The 21/21 -> 25/25 extension travels through that script or not at all.",
    "lane_computed": [
      "the zero-fabrication re-derivation over all result files (both the majority test and the stricter any-repeat test)",
      "the shortfall accounting",
      "the latency survivor-bias reading",
      "the self-consistency denominators",
      "glimmer's rank among the eighteen 27/27-kill model-runs",
      "the shape-probe predictiveness observation"
    ],
    "parameter_sizes_gap": "The model-metadata snapshot predates these four tags, so parameter_size and quantization_level were NOT read first-hand by this lane. Where the quant appears in the tag (q8_0, q4_K_M) the tag is the receipt; for nemotron-3.5-lightning and qwen3.6 the tag carries no quant and the size cells are vendor-labelled roster values from PREREG-A-addendum.md section 2. A first-hand read belongs in the roster/licence lane before publication."
  },
  "limits": [
    "One instrument, one synthetic-adjacent population of 43 cases over historical walking-tour dossiers. Not a general claim about any model.",
    "A response-failure rate above the ceiling is a statement about the serving stack at this config, not about the model's judgment.",
    "Cross-quant comparison is a confound we name, not an equalization we perform.",
    "These are our own trials, on our own hardware, for our own chairs. Not first independent numbers, and not a standard."
  ],
  "columns_note": "The seven top-level fields per row are the page table's seven columns, verbatim. `note` must read character-for-character as the rendered cell — a companion that paraphrases the page is not a companion."
}