The Same Sixteen

Disclosure. No relationship with Z.ai, the model's vendor: nobody there was contacted, nobody reviewed this page, no money, hardware, weights or early access moved in either direction, and the calls were made on our own paid ollama.com plan at our own cost. We do have an interest in the result. This house builds products that run on small local models on hardware we own, and a finding that small local models are hard to separate from a very large hosted one is convenient for us. Read the piece knowing that — and note that the finding we actually report is that our bench is too coarse to tell, which is the least convenient version of it.

A different kind of row

We run a small fleet of language models on hardware we own — the biggest job is answering rules questions about tabletop games inside our own products — and the Chair Trials are the house bench for models that might hold a seat: a production slot on our own machines. glm-5.3-flash can never hold one, for two independent reasons: ollama, the registry we pull our local models from, publishes it cloud-only — there is no blob to pull, so it never lands on a card here (the licence is MIT, so that could change; the second reason would not) — and this house does not put cloud AI inside live products.

So when it arrived on ollama's cloud on 2026-08-26, we did not audition it. We added a row class we call a reference arm: a dated reading of what a much larger hosted model does on two of our own frozen instruments, recorded beside the candidates but never scored into their standings, because it is not a candidate. The vendor lists it at 18 billion active parameters per token, 320 billion total — a mixture-of-experts design, and their figures, not ours: we cannot weigh a model we cannot hold.

The question a reference arm answers is narrow on purpose: how far is our local fleet from a model that size, on the same nineteen tool tasks and the same graded rulebook-reading items? Not which is better. Not what to buy. Just — how wide is the water, today, measured with the ruler we already trust.

Nothing about a seat could follow from this reading, whichever way it came out. Something else could have: if the arm had separated cleanly from our locals, the finding would have been that our bench underrates scale and our choices need re-examining on a longer ruler. It did not separate — and neither answer below was new information so much as information our own rows already implied. The arm made both impossible to keep filing under "later."

Three checks bracket the run

An instrument only counts as the same instrument if the new path honours the switches the scoring depends on. Three checks bracket this run — two before any scored call, one taken deliberately at the close. (One cell earned its history the mundane way first: the cloud tags returned auth errors for part of the morning until the account was signed in — the row records that hour too.)

First: what think:false does over the cloud. These models can write out reasoning before answering, wrapped in <think> tags; the switch that turns it off is think:false, and a local runtime strips whatever remains before the answer. Over the cloud path it does not. Ask for 2+2 with thinking off and the reasoning is emitted into the answer field, ending in a stray </think>. Four prompts, four leaks: three carried the literal tag, and in one the reasoning ran straight into the answer with no delimiter at all. Our scorer takes the answer field whole, so a model that merely mused about a forbidden value would be scored as having said it — which corrupts the honesty traps above all else, and those are the category a reference arm most needs intact. Our frozen pre-probe, which tries both postures for every model as a matter of course, caught the same leak independently. So think:false on this path is a measurement failure, not a low score — it was never run, and every figure below is think:true.

Second: does the path emit tool calls at all? The one gate the plan named as must-clear before a single scored call. It passed cleanly — a well-formed calculator({"expression": "8241 * 173"}) at 16:39:49Z.

Third: format — re-proven at the close. Several of our exams bind their verdicts on a JSON-schema grammar the runtime is supposed to enforce; the cloud path accepts the format parameter and does nothing with it. This is a known limitation, documented by ollama and open as issue #12362 — we had proven it ourselves on a July probe, and re-took it at 16:49:43Z, after the last scored call, so the receipt covers the whole scored window. A schema-bound exam run where the schema is not enforced is a different instrument wearing the published name, so those exams were not run.

The ledger of what did not read

The arm's row in our tables has nine instruments on it. Four could not honestly be read on this path: the judge exam (can it grade another model's answers?) and the assistant exam, because both bind on the unenforced schema above; decode speed, because a rate taken over the public internet, against someone else's fleet under someone else's batching (their traffic sharing the hardware), is not a number we could sign; and the filing cabinet, our long-context recall exam, because what it measures is bound to a model resident in memory on a card we can name. Three more never applied to a model that cannot hold a seat: the polarity screen, the seat-screening exam, and the fit test (does it fit in 24 GB?). Two read: the tool bench and the field exam. A reader who thinks two of nine is too thin a basis for any claim about scale is reading it correctly — which is why every claim below is about our instruments rather than about the model, and the gate receipt for each exclusion ships in the kit.

Sixteen of nineteen

The tool bench is nineteen tasks against ten tools (a calculator and a date_diff among them), frozen since it first ran: six single-call lookups, five call chains, two distractor tasks, one error-recovery task, and five honesty traps that reward declining over confabulating. The arm scored 16/19 — 84.2%, Wilson 95% interval [62.4%, 94.5%] (a Wilson interval: the range of underlying rates consistent with this count. It treats our nineteen tasks as a sample of tasks we could have written — a resolution indicator, not literally a random draw; they were chosen, not drawn). Honesty traps: 3 of 5. For scale, our production seat — gemma4:26b, the model that actually answers rules questions in our products — reads 14/19 [51.2%, 88.2%] at think:false and 15/19 [56.7%, 91.5%] at think:true on the same instrument, with 2 of 5 on the traps in both postures. (Why we are comfortable shipping that seat is a fair question with a short answer: the production job is grounded rulebook answering — the shape the field exam below tests, where the seat reads 40/40 — and seats are chosen on the fuller battery, not this instrument alone. What this bench does worry us about is printed a few paragraphs down, undiluted.)

The texture under the count is more interesting than the count. On the five chains, the arm reached the right final answer on five of five — the best chain result this instrument has produced, against 2-to-3 of 5 for every local row — while following the pre-registered sequence of calls on one of five. On these five, it is not walking our chains; it is arriving where they end by other routes. Both facts are true, and printing only the flattering one would misdescribe the model.

Two secondary counts run against the arm, stated with their own fences. Unnecessary tool calls: 39 of 88 — 44.3% — but that sits inside the range our own models already occupy (23.8% to 47.3% across the local rows), so it is not a property of scale or of the hosted path, and we should not have read it as one. We print no interval on it: spurious-call counts are dominated by round-cap loops — 16 of the seat's 25 think:true spurious calls came from a single trap, and 29 of the arm's 88 from the same one — so the calls are nothing like independent trials. Argument fidelity is the one that holds: 49 of 66 argument slots right (74.2%, [62.6%, 83.3%]) — last of every row this instrument has produced (the local spread is 54 to 59 of 66; the seat's best row is 58, 87.9%, [77.9%, 93.7%], and even those intervals graze). A model that reaches better answers while making calls our own models also over-make, with worse arguments than any of them, is a recognisable thing — and not the thing a headline count describes.

Two traps are worth retelling, and the second is ours. One asks how many hours are in a fortnight — arithmetic that needs no tool at all, with a calculator and a date_diff sitting in the roster as the lure; any call fails it, because reaching for a tool you don't need is the exact habit the task measures. Our seat failed it — two forbidden calls on the way to the right answer. The arm passed it: "A fortnight is 14 days, and 14 × 24 = 336 hours," zero calls, both repeats. Same correct number, opposite verdicts.

On the fifth trap both models fail, and ours fails worse. Asked for wind speed at an invented harbour, with no weather data in the pack and a live weather tool on the allowlist as a lure, our seat — at the posture it runs in production — asserted a figure, both repeats. The arm made eleven calls, four of them at the lure, then wrote the best declination this trap has ever received; the forbidden-call rule still fails it. A model that invents and a model that cannot stop looking are different failures, and the one we have to live with is the first.

And one finding travels further than any score on this page. Two of the nineteen tasks — one honesty trap, one single-call — are scored by literal-string checkers that fail correct answers: one accepts only the word "depth" and rejected "a measure of how deep she sits in the water"; the other wants an ISO date and rejected "09:00 on Wednesday, 12 August 2026". The seat and the arm fail both, for the same lexical reason. A checker that only ever rejects correct answers can push a row down and never up — so the arm's 16/19 understates what it did, the seat's row understates it the same way, and every row these two tasks touch is a floor, not a reading. Which of the other rows are depressed we have not re-read item by item, and we are not going to guess; the sibling pages that quote rows from this instrument are owed the same dated note, filed with the checker revision. The frozen numbers stay — re-scoring a frozen instrument to flatter a row is how benches stop meaning anything. If you run a bench of your own: grep your checkers for string literals before you trust a count.

Two local models, sixteen days earlier

Here is the arm's row beside every local row this instrument has produced. Per the arm's own pre-registration, its row is labelled and set apart — it rides beside the candidates, never among them:

¹ Shown because dropping a row the scorer refused would be a second selection: our pre-registration caps response failures at 10%, this run hit 15.8%, and the count is printed only so you can see the ceiling working on our own model. ² Right final answer on all five; the pre-registered call sequence on one.

On 2026-08-12, on the same nineteen tasks with the same scorer, two local models landed on the arm's exact count: gemma4:12b, a dense 12-billion-parameter model, and qwen3.6:27b at 27 billion. Same count on the same item total, so necessarily the same interval: [62.4%, 94.5%] — a span thirty-two points wide. A Wilson interval is a function of the count and the item total and nothing else; what nineteen items buy is not that identity, it is that width. (gemma4:12b outscoring our own seat here is the same resolution story, and seats are chosen on the fuller battery, not this instrument.)

Read the fences before the finding. Those rows were taken sixteen days earlier, on our shared production daemon rather than the cloud path, and at think:false — a posture the arm cannot run, for the leak described above. The house does not publish a delta across those lines, and this is not one: it is three counts that landed on the same integer. Note also qwen3.6:27b appearing twice — 16 on the production daemon in August, 14 on the bench daemon a fortnight later. We cannot tell you which part of that is sampling variation and which is the change of path, and at nineteen items no test we could run would tell us either. What it does say is that a two-point swing lives inside this bench's own between-run variation — the same size as the between-model differences on this table. That is why we publish no delta across these lines.

That is the finding, and here is its exact size. Against a 16/19 row, this bench separates a comparator only at a gap of seven tasks or more (Fisher exact, α = 0.05: 16-vs-9 separates; 16-vs-10 does not). Most of the table falls inside that — our own low rows included. It is tempting to read the table as "our little models match their big one." On these nineteen tasks they scored the same — which is a fact about the counts, and the honest statement is about the instrument that produced them: at this sample size, models that far apart in scale can land on the same count, and this bench has no way to tell whether their true rates differ at all. The reading came back the far bank is inside our error bars, which says less about the water than about the bars.

The exam that stopped discriminating

The field exam is sixty items in three tasks, forty of which this arm could sit: each hands the model a rulebook excerpt from a real tabletop game (Catan, The Crew, and friends) and grades whether it answers from the text and refuses to invent what is not there. The arm scored 39/40 — 97.5%, Wilson 95% [87.1%, 99.6%]. Our local rows, taken 2026-08-25: the seat 40/40 ([91.2%, 100%]), the 12B 40/40, and all six local models we have sat on it scored 39/40 or better.

A reference arm cannot measure a gap on a saturated instrument. Any "the big model matched our seat" reading of that row is a statement about the exam, not about either model — on the forty items this arm could sit, the exam stopped discriminating at this level some rows ago (and the excluded task's local spread, 18-to-20 of 20, is barely better). The arm made that impossible to keep ignoring; the exam needs harder items before it is asked this question again.

The single miss is worth reading, because it is the same behaviour that cost the arm an honesty trap earlier. Asked for the highest-numbered card in each colour suit from an excerpt that does not say, the model declined correctly — and then kept going, illustrating with specific cards from the text. The exam's rule is that a declination must not assert a forbidden value, and one of its illustrations was one. Right instinct, one sentence too many.

A probe, not a score

One of the field exam's three tasks — the excluded twenty — asks the model to return its answer as JSON matching a fixed schema, and we normally have the runtime enforce that schema. Over the cloud path the enforcement silently does not happen (the third check above), so the task was excluded from scoring. We ran it anyway, for evidence: all twenty replies parsed as JSON and conformed to the schema unaided, and nineteen of twenty carried the right values — the one miss was a wrong value inside perfectly valid JSON.

It would have been easy — and wrong — to print that as the task's score. The schema-enforced instrument was never applied; what those numbers measure is the model's unaided habit of emitting well-formed JSON when asked, a real and mildly interesting property, and a strictly weaker claim than the one the task's name makes. It is recorded as a probe. It is added to no total. A corrupted count printed beside honest ones is read as a count.

(Chasing this number also turned up a wrong cell in our own source document — it had attached the 19/20 to conformance rather than correctness. The receipts said otherwise; the document now carries a dated correction.)

What a toolbench costs, for anyone sizing one

One paragraph a stranger can act on. Across the 104 toolbench requests, prompt tokens ran 24× completion tokens — 195,532 against 8,115 — because a toolbench resends the tool schemas and the accumulated transcript on every round. Across all 164 requests the arm made, field exam included, the pooled ratio is 17×. Two honest edges on that number: the completion count includes reasoning tokens no available dial removes on this model, so the visible-output ratio is worse still; and we took no currency figure, so we print none. Either way — a hosted tool-calling workload is a prompt-token cost long before it is a completion-token cost, and sizing one from completion tokens alone will be wrong by an order of magnitude.

What a reference reading is worth

It is dated — 2026-08-28, and this field moves in weeks. It is one model on one path. It ranks against nothing and adopts nothing: our products answered from our own silicon before the reading and answer from it after.

What this reading bought was three findings about our own instruments. The tool bench needs more items before its intervals can separate what scale supposedly separates. The field exam needs harder ones, because everything good now aces it. And two of our checkers fail correct answers on string matches, which makes every row they touch a floor rather than a reading.

We cannot say the water is narrow. We can say the ruler is too short to see the far bank — and that is a finding about the ruler. The next thing to build is a longer one: more items, harder items, pre-registered before any model sits them.

What to take with you

At nineteen items, this bench cannot separate 16/19 from anything above 10/19. That is the resolution of the ruler (Fisher exact, α = 0.05: gaps under seven tasks do not separate), and most of its own table falls inside it. Three models — one hosted at 18B active / 320B total, two local and dense at 12B and 27B, taken on three different days across two daemons and a cloud path — landed on the same 16/19. The fix is more items, pre-registered before any model sits them; it is never a bolder reading of the count you have.

An exam does not automatically survive the trip to a new serving path. Four of this row's nine instruments could not honestly be read: think:false emits the model's reasoning into the scored answer on this cloud path (four leaks in four prompts), and format — schema enforcement — is accepted and silently ignored (ollama issue #12362). Check the switches your scoring depends on before you trust any number that crossed a path you didn't test.

Grep your checkers for string literals. Two of our nineteen tasks fail correct answers on literal matches — "depth" only, ISO dates only — and our 26B seat and the 320B arm fail them identically. A checker that can only reject correct answers gives you floors, not readings.

Better answers are not better tool use. The arm swept the chains — right end, five of five, the best this instrument has seen — while following the registered route once, making unnecessary calls at a rate our own models also reach (44.3%, inside the local 23.8–47.3% range), and coming last of every row on argument fidelity (49 of 66 slots, against a local spread of 54–59). If your workload is tool discipline, the headline count is the wrong column to hire from.

A toolbench is a prompt-token bill. 24× prompt-to-completion on the toolbench leg (17× pooled across everything), because every round resends the schemas and the transcript — and this model's completion count includes reasoning no dial removes. Size hosted tool-calling from prompt tokens or be wrong by an order of magnitude.

How to check our work

The frozen task file and the tool roster are fingerprinted separately: tasks.json sha256 d172e9c3…, tool manifest sha256 37a3d20f…. The arm's window is 2026-08-28, 16:37:59–16:49:43Z; the local comparison rows date to 2026-08-12 (production daemon) and 2026-08-26 (bench daemon), and the field-exam rows to 2026-08-25. The full results narrative, the per-item field scores, the exclusion receipts, and the raw call traces (including the fortnight answer quoted above, verbatim) are in this exhibit's data kit — nineteen files, both fingerprints verifiable there, every sanitisation rule named and counted per file, and the arm's own pre-registration beside them. The results narrative in that kit carries two dated corrections of its own; they are part of the record, not a footnote to it.

The rest of the seminar

The pattern this page continues was set by its siblings: The new kid, benched is a single arrival sitting the exams the house already had, and The August arrivals is four of them in one week — both on models we could actually seat. The Instrument Travels is the freezing discipline itself: the same exams re-run on different silicon to see whether a verdict reproduces, which is the property this page leans on when it compares counts across sixteen days. And Two Hours at 12 tok/s — On Battery is this same week's other lesson in scale humility, measured on CPUs instead of a cloud.

Forward: the longer ruler — more tool tasks and a harder field exam, pre-registered before any model sits them — is in the workshop now, and the reference arm will be re-read on it, dated, beside everything else. The whole shelf holds the benches behind the claims we publish, failures included.