{
 "schema": "s2s-bench-v1",
 "kind": "data-kit-index",
 "exhibit": "the-same-sixteen",
 "exhibit_title": "The Same Sixteen",
 "exhibit_url": "https://research.strata2signal.com/the-same-sixteen/index.html",
 "licence": "CC BY 4.0",
 "attribution": "strata→signal research, research.strata2signal.com",
 "contact": "hello@strata2signal.com",
 "run_id": "rs-newkids-trials-2026-08-26",
 "run_window_utc": {
  "opened_utc": "2026-08-28T16:37:59Z",
  "closed_utc": "2026-08-28T16:49:43Z",
  "c3_utc": "2026-08-28T16:41:17Z - 2026-08-28T16:44:06Z",
  "field_exam_utc": "2026-08-28T16:47:49Z - 2026-08-28T16:48:57Z",
  "note": "The window opens at the first probe of the scored session and closes at the `format` enforcement probe, taken deliberately AFTER the last scored call so the receipt covers the whole of it. The morning's blocked attempt (three cloud tags, HTTP 401, 16:30:43Z) is outside this window and is kept verbatim in the gate receipt, because a row that was once blocked should be able to show why."
 },
 "arm": "glm-5.3-flash:cloud",
 "arm_class": "REFERENCE ARM · cloud · dated",
 "pins": {
  "task_file": {
   "name": "tasks.json",
   "sha256": "d172e9c326735df3848389e24f0a82d67b11d97df62805285b2a0b5d527ee140",
   "note": "the sha the page prints; the shipped file is byte-identical, so `sha256sum tasks.json` returns it"
  },
  "tool_manifest": {
   "name": "tools-manifest.json",
   "sha256": "37a3d20f99a366413a6be7b69be7d80ff2a7cb642c8abf8dcd90d7f81189941b",
   "note": "the second sha the page prints. The runner builds the tools array at run time and hashes its canonical form (sorted keys, no whitespace); the shipped file IS that canonical form, checked against the runner's own hash before it was written, so `sha256sum tools-manifest.json` returns it directly"
  },
  "system_prompt_sha256": "7b38278b7435b84b2e872b7f61281c24288ba8148ae2b18030d08e399486294c",
  "tool_count": 10,
  "task_count": 19
 },
 "counting_rules": {
  "ranked_against_nothing": "A reference arm carries no ordering, no margin and no comparison verdict, and no floor is applied to it. The scorer stamped the toolbench row RANKED mechanically; the class label overrides it. Registered in prereg-reference-arm.md (D-20260828-18), fence 2 of 4.",
  "dated_and_only_dated": "A hosted tag is not a fixed object. This is a reading of 2026-08-28, stamped at both ends, never carried forward as a ceiling. Fence 3.",
  "two_legs_read_seven_refused": "Only C3 and the field exam could honestly be read on this path. Every other leg carries a stated disposition and reason rather than a blank; see reference-arm-row.json.",
  "think_false_is_a_measurement_failure": "think:false does not switch this model's reasoning off on this path, only whether ollama parses it out of the reply, so the raw deliberation lands in the field every leg scores. Never run; every figure here is think:true.",
  "literal_checkers_make_floors": "Two of the nineteen toolbench tasks match string literals and can only ever reject a correct answer, so every row they touch is a floor rather than a reading — the arm's 16/19 and the seat's rows alike. The frozen numbers stay as scored; widening the checkers is a pre-registered revision applied to every row equally, and it is not applied here.",
  "resolution": "Against a 16/19 row this bench separates a comparator only at a gap of seven tasks or more (Fisher exact, alpha = 0.05: 16-vs-9 separates, 16-vs-10 does not). Wilson 95% intervals ride beside every rate; two rows on the same count carry the same interval by construction."
 },
 "not_a_standard": "One hosted model, on one day, on two of this house's own frozen instruments, sized to one question. Not a leaderboard, not a benchmark suite and not a verdict about any model — and, by its own registration, ranked against nothing.",
 "sanitisation": "Sanitised at the pen, never copied raw. provenance.json names every rule applied to every file with the count of times each fired, names what was left out and where it actually lives, and names the two things deliberately left unaltered.",
 "self_reference": "provenance.json, index.json and index.html are not listed among the files they describe: a file cannot carry its own sha256. index.json carries provenance.json's; index.html carries both of theirs.",
 "file_count": 19,
 "files": [
  {
   "file": "C3-RESULTS.md",
   "role": "Leg C3, the toolbench, in full: the four rows with their outcome states and the reason each one carries, then the PER-TASK detail for every row — all nineteen tasks × each model, with passes, conjunctive verdict, groundedness, whether the expected calls were made, the spurious count and the checker's own note. This is where the arm's 16/19 decomposes.",
   "media_type": "text/markdown",
   "bytes": 10514,
   "sha256": "656845a54367149a698eba67f9936eb167c3fc1ec8d5f5efc9e33afd12ca3689",
   "sha256_original": "656845a54367149a698eba67f9936eb167c3fc1ec8d5f5efc9e33afd12ca3689",
   "identical_to_original": true,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "README.md",
   "role": "What each file is, the two fingerprints to check first, what was converted and why, the one clock deliberately left as captured, what is NOT in this kit and where those files actually live, the one open item, and what was not touched.",
   "media_type": "text/markdown",
   "bytes": 10664,
   "sha256": "de11aed204dba81538b033e73f31232edb24ae237c06d8551ef040d614fa11ff",
   "sha256_original": null,
   "identical_to_original": null,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "REFERENCE-glm-5.3-flash-2026-08-28.gate.json",
   "role": "The gate receipt: the sign-in state and the morning's 401 history verbatim, the `think` posture receipt (all three postures, with the leak reproduced four times), the tool-emission gate with the tool_calls it returned, the `format` enforcement probe taken AFTER the last scored call, and the run windows. This is the exclusion evidence the page promises: every leg reported NOT-RUN is refused here rather than left blank.",
   "media_type": "application/json",
   "bytes": 7181,
   "sha256": "ebb4c82969338950bf8cad8541502e96e363b708341139d533f73b4a4463142d",
   "sha256_original": "41afc7c99d50f54901e220d2bfccdf631ee56357fc2923e993294647e9991c29",
   "identical_to_original": false,
   "stripped": [
    "absolute-paths-to-relative",
    "box-name-to-role",
    "private-host-to-role"
   ],
   "strip_counts": {
    "absolute-paths-to-relative": 4,
    "box-name-to-role": 5,
    "private-host-to-role": 1
   }
  },
  {
   "file": "REFERENCE-glm-5.3-flash-2026-08-28.md",
   "role": "The full results narrative for the reference arm, written the day it was measured: the three gates, the ledger of what could not be read and why, the toolbench and its five honesty traps item by item, the rows it sits beside, the field exam, the cost signals, what the reading does and does not license, and the receipts index. It ends with TWO dated corrections (both 2026-08-30) appended rather than folded in — the §5 cell that attached 19/20 to conformance instead of correctness, and three defects the round-two lenses found in §§3 and 7. The house convention is that a correction appends; the corrected sentences are left standing above so a reader can see what moved.",
   "media_type": "text/markdown",
   "bytes": 29411,
   "sha256": "c533dc67101364df745d69001c3620e292f977e9ebe996f1d57b9a4084c0ac91",
   "sha256_original": "804ee1af0e297c7b6f62d46699dda6c53d5a0216fccc0b070504baedd53603e2",
   "identical_to_original": false,
   "stripped": [
    "absolute-paths-to-relative",
    "box-name-to-role",
    "operator-name-to-the-operators",
    "private-host-to-role"
   ],
   "strip_counts": {
    "absolute-paths-to-relative": 13,
    "box-name-to-role": 4,
    "operator-name-to-the-operators": 1,
    "private-host-to-role": 1
   }
  },
  {
   "file": "RESULTS-TABLES.md",
   "role": "The whole battery per candidate × per instrument, the arm's row among them — labelled REFERENCE ARM · cloud · dated and set apart — plus the beside-the-seat comparison and the arm's cost signals. Every cell is copied from that leg's own scorer output; this view computes nothing.",
   "media_type": "text/markdown",
   "bytes": 9167,
   "sha256": "f677889953984b44cca621b08b6240ee95f0f27a57cc8f5a3f5613f90166ee26",
   "sha256_original": "31facba62e137f3997ad473423e81cb809d1bb9fe2e5f847c9470c582446397a",
   "identical_to_original": false,
   "stripped": [
    "box-name-to-role",
    "operator-name-to-the-operators",
    "private-host-to-role"
   ],
   "strip_counts": {
    "box-name-to-role": 3,
    "operator-name-to-the-operators": 1,
    "private-host-to-role": 1
   }
  },
  {
   "file": "c3-primary-rows-2026-08-12.json",
   "role": "The 2026-08-12 rows on this same frozen instrument — the local models the arm's count is set beside, as the chair trials published them: task success, grounded task success, honesty traps and the response-failure ledger, with the table's own column definitions. `gemma4:12b` and `qwen3.6:27b` are the two rows that land on 16/19. Taken on the shared production daemon, sixteen days before the arm and on a different path; the page prints no delta across that line.",
   "media_type": "application/json",
   "bytes": 2884,
   "sha256": "3d0e653d99349b0db129a1713ff6a037768555511ea82c835d3eec8f1c17957b",
   "sha256_original": "3d0e653d99349b0db129a1713ff6a037768555511ea82c835d3eec8f1c17957b",
   "identical_to_original": true,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "counting-rules.md",
   "role": "The five rules that govern every figure here and on the page — the reference arm's four fences in short, the two legs read and the seven refused, why think:false was never run, and the rule that makes any count a literal-string checker touches a FLOOR rather than a reading. Each names where it was registered.",
   "media_type": "text/markdown",
   "bytes": 4406,
   "sha256": "59bd601001a3d43919419ee5e930c0ab03dde3e1c08686e72276bfdad091626e",
   "sha256_original": null,
   "identical_to_original": null,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "field/glm-5.3-flash-cloud.json",
   "role": "The field exam per item, all 60: the question, the model's raw response, the wire counters and the scorer's own verdict with its matched keys. T1 grounded-qa 20/20 and T2 stay-grounded 19/20 are the arm's 39/40; T3 schema-extract 20 is the EXCLUDED task — its items ship because the exclusion is the evidence. `wire_options` records what actually rode the wire, `think: true` included.",
   "media_type": "application/json",
   "bytes": 45832,
   "sha256": "31c2255bb774192658da80bb6d6b02c288b786c2904570bd7249873b2f8fe569",
   "sha256_original": "3dbe1b203d787cf6ccd158cd421045b77e7f5978cd7716ebc64fd3e645cb09d6",
   "identical_to_original": false,
   "stripped": [
    "private-host-to-role"
   ],
   "strip_counts": {
    "private-host-to-role": 1
   }
  },
  {
   "file": "prereg-reference-arm.md",
   "role": "The reference-arm class as it was REGISTERED, extracted verbatim from the bench's own plan (its \"ROSTER ADDENDUM 2026-08-28\" section, operator ruling D-20260828-18) and dated before any scored call: what a reference arm is, the four binding fences (never a candidate, ranked against nothing, dated and only dated, no threshold ever derived from it), the two legs it runs and the reason each of the other four is NOT-RUN, the gate it had to clear first, and the rule that every published cell for this arm carries the label REFERENCE ARM · cloud · dated. Nothing else from that plan ships; the section is self-contained and says so in its own first paragraph.",
   "media_type": "text/markdown",
   "bytes": 6850,
   "sha256": "2641249b0a77415738894581d96c185a010b79e828579f42c2f60a8fe5fd2d51",
   "sha256_original": "1c5f1c2caf0b5e29d6b9bef5e4a70629951dd8fbfc672ecd077007fc797aa92c",
   "identical_to_original": false,
   "stripped": [
    "operator-name-to-the-operators"
   ],
   "strip_counts": {
    "operator-name-to-the-operators": 1
   }
  },
  {
   "file": "provenance.json",
   "role": "Every rule applied to every file with the number of times each fired, the pre-sanitisation sha256 for everything that is not byte-identical to what the harness wrote, the run window, the instrument, the counting rules, what was left out and what was deliberately left alone.",
   "media_type": "application/json",
   "bytes": 22939,
   "sha256": "c023a60c05e2b63a75952eb7d13e9e2f83e24b7318e0ba2a3fc595a6c84d62ce",
   "sha256_original": null,
   "identical_to_original": null,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "raw/c3/glm-5.3-flash_cloud/preprobe.json",
   "role": "The frozen C3 pre-probe, run before any scored call and probing BOTH postures for every model as a matter of course. It captured the think:false leak independently of the gate receipt, which is why the leak is two findings and not one.",
   "media_type": "application/json",
   "bytes": 6852,
   "sha256": "af5b8ea006f3b8fbad68113dc1239ef687d0b7ac818a201ff5ef5e9842c0a1b9",
   "sha256_original": "d007c98f2c79a61b76100c673b94fe104a59606d254ce6fc00bfb7d01735e620",
   "identical_to_original": false,
   "stripped": [
    "fixture-clock-left-as-captured",
    "local-timestamps-to-utc"
   ],
   "strip_counts": {
    "fixture-clock-left-as-captured": 2,
    "local-timestamps-to-utc": 1
   }
  },
  {
   "file": "raw/c3/glm-5.3-flash_cloud/think_true/calls.jsonl",
   "role": "Every individual tool call the arm made, one a line, 88 of them: the round it belongs to, the tool, the arguments, whether the call was expected by the task, whether the sandbox allowed it, and its result summary. This is the file the 44.3% spurious-call figure counts. One extra first line is this kit's own provenance record; skip records whose type is kit-provenance for the harness's stream unchanged.",
   "media_type": "application/x-ndjson",
   "bytes": 65765,
   "sha256": "539a15286ceef26a6df5a905739bb52da29f52e1cffe3e3143de45fd1c3eede9",
   "sha256_original": "b2a5fed44233f50fb5e01ecca91e8ba6d191b0fbeacb40cd4ed4716c1100327b",
   "identical_to_original": false,
   "stripped": [
    "fixture-clock-left-as-captured",
    "provenance-header-added"
   ],
   "strip_counts": {
    "fixture-clock-left-as-captured": 29,
    "provenance-header-added": 1
   }
  },
  {
   "file": "raw/c3/glm-5.3-flash_cloud/think_true/trace.jsonl",
   "role": "Every scored attempt, one record a line, 38 of them: the task, the posture, the prompt, the per-round wire exchange, every tool call with its arguments and the sandbox's outcome, the stop reason, and the final answer as scored — including the fortnight trap's answer the page quotes verbatim. One extra first line is this kit's own provenance record; skip records whose type is kit-provenance for the harness's stream unchanged.",
   "media_type": "application/x-ndjson",
   "bytes": 200086,
   "sha256": "5ef0d227959cdc5a7f7d97a93067adda2e6972adc2918e07447de5de7e7b7c77",
   "sha256_original": "338abb95fefba99b08d9800aed94afb780c7798ef8a5210a8beb50bf00a275ae",
   "identical_to_original": false,
   "stripped": [
    "fixture-clock-left-as-captured",
    "provenance-header-added"
   ],
   "strip_counts": {
    "fixture-clock-left-as-captured": 60,
    "provenance-header-added": 1
   }
  },
  {
   "file": "raw/c3/manifest-glm-5.3-flash-cloud-2026-08-28.json",
   "role": "This run's own manifest, kept under a dated name: the protocol (2 repeats, temperature 0, 8-round cap, 32k context), the posture assignment, the pins — task-file sha256, tools sha256, system-prompt sha256, the fixed fixture clock and the HTTP allowlist — and the per-posture attempt, call and round counts. The runner writes `manifest.json` at a fixed path and each run overwrites the last, which is a named defect in the narrative's §8; this copy is the one that survives.",
   "media_type": "application/json",
   "bytes": 5468,
   "sha256": "1f1bbbc95e642782ee62daafa016796905694fd11cab5a67d5279802abf7713d",
   "sha256_original": "1c9bfef24eae96e53b49c7924ef02a4ae19d92c082316eea1e0470026b4eca58",
   "identical_to_original": false,
   "stripped": [
    "absolute-paths-to-relative",
    "fixture-clock-left-as-captured",
    "local-timestamps-to-utc",
    "private-host-to-role"
   ],
   "strip_counts": {
    "absolute-paths-to-relative": 2,
    "fixture-clock-left-as-captured": 1,
    "local-timestamps-to-utc": 8,
    "private-host-to-role": 2
   }
  },
  {
   "file": "reference-arm-row.json",
   "role": "The arm as ONE row, with its class label attached to it rather than to the prose around it: the tag, the class, what it was read on, the vendor-stated figures, the posture and why, every leg with its disposition, the secondary counts, the cost signals, and the finding. Copied from RESULTS-TABLES.md and the narrative; nothing computed. The site's model roster reads its `class` field from here.",
   "media_type": "application/json",
   "bytes": 4771,
   "sha256": "7a8b5cc0fe57d526a6097f74648a78c0b6d49754d1feb0102f076d99047b376a",
   "sha256_original": null,
   "identical_to_original": null,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "tasks.json",
   "role": "The frozen tool-use task file, BYTE-IDENTICAL to the file the run pinned: sha256 d172e9c326735df3848389e24f0a82d67b11d97df62805285b2a0b5d527ee140, the sha the page prints. 19 tasks — 6 single-call, 5 chains, 2 distractor, 5 honesty traps, 1 error recovery — with each task's checker beside it. Run `sha256sum tasks.json` and you have the pin.",
   "media_type": "application/json",
   "bytes": 23053,
   "sha256": "d172e9c326735df3848389e24f0a82d67b11d97df62805285b2a0b5d527ee140",
   "sha256_original": "d172e9c326735df3848389e24f0a82d67b11d97df62805285b2a0b5d527ee140",
   "identical_to_original": true,
   "stripped": [],
   "strip_counts": {}
  },
  {
   "file": "tools-manifest.json",
   "role": "The ten tools offered on every task, as the `tools` array actually rode the wire, in the CANONICAL form the run hashes: JSON with sorted keys and no whitespace. Its sha256 is 37a3d20f99a366413a6be7b69be7d80ff2a7cb642c8abf8dcd90d7f81189941b, the second sha the page prints, and `sha256sum tools-manifest.json` returns it directly. The runner builds this array at run time from its own definitions rather than reading a file, so this is the array re-emitted through the same canonicaliser, checked against the runner's own hash before it was written.",
   "media_type": "application/json",
   "bytes": 5236,
   "sha256": "37a3d20f99a366413a6be7b69be7d80ff2a7cb642c8abf8dcd90d7f81189941b",
   "sha256_original": null,
   "identical_to_original": null,
   "stripped": [],
   "strip_counts": {}
  }
 ]
}