# Reading the answer instead of writing it *A new kind of model decides without generating a word. This page says what a decision model is, who made it, what it claims, what its licences allow, and what happened when we ran the open versions through fourteen measured tests on questions from this workshop's own work.* *Published 2026-09-23 (UTC) · A small (human) team and a fleet of AI agents.* **the short version:** a decision model scores a question's lettered options at its first output position: nothing to parse, a probability for free. a company founded by a researcher behind ChatGPT's training method launched it on 2026-09-15; an independent group released open weights five days later, which we tested. asked which of six guests said a line, the 27-billion-parameter open model scored 93.5 per cent in our kit's order, 89.8 to 92.6 in five shuffles; our chat model, 80.6. the speed is the card and the server: 87 milliseconds on one large card, 350 across two smaller ones where our chat model read its letter in 120; reading instead of writing saved a tenth of the time on a short question and almost nothing on a long one. the open weights are non-commercial, so without separate terms from their authors, which we have asked for, they cannot serve anything we charge for. what transfers is the method, which gets the same answer out of any model, though only a model trained for it gives a probability worth gating on; Kev, an open family whose weights a product may run, read 67 per cent on the guest question as it comes, so the seat this points to is fine-tuned, not downloaded. that run and two others are owed as dated addenda below. 8,690 words · about 40 minutes (at 220 words/min) · 9 tables · data kit: yes https://research.strata2signal.com/reading-the-answer/ --- *If you're new here: [strata→signal](https://strata2signal.com) is a small workshop (plus a friendly dog with a white patch) that builds things on its own machines and writes up what it measures. A seat here means one model held in one card's memory, answering for one of our products. An arm, on this page, is one numbered run in the bench's record: one set-up for one model or a pair of them, asked one or more sets of questions; there are fourteen. Every number on this page traces to a row in a file, and the file is linked at the end; where a claim is somebody else's, it is quoted and linked.* ## How a decision model decides {#how-a-decision-model-decides} Every language model, at every step, holds a score for every token in its vocabulary (a token is a word or a piece of one). Generation means picking one, appending it, and going again. A chat model asked "which of these six guests said this line?" picks a first token, then a second, and eventually produces a name. A decision model is asked the same question with the options lettered A to F and told to answer with a letter; then, instead of generating anything, the caller reads the scores of exactly those six letters at the first output position, before any token is chosen. A fixed formula turns the six numbers into probabilities that sum to one. The model's whole opinion is on the table after one read of the question, and nothing has been written. The catch, and this page measures it, is that a chat model can be read at the same position and does not put its opinion there. It puts a bracket there, or the start of a sentence, or a marker for its own thinking channel; the letters sit somewhere down its list, and when four of the six are nowhere near the top, the reading has to plug in a made-up minimum score for them. It is still usually right, but the probabilities are fiction. A decision model is a chat model retrained so that the first position is where the decision lives. That retraining is the product; everything else is a prompt format. Three things follow: **nothing to parse**, since the letters were scored and no schema was forced onto a text generator; **a number for every option, not just the winner**, so a gate can be set at 0.9 rather than at "the model said yes" and a 0.5 can be sent to a person; and **the cost is the prompt**, one pass over the question plus one position, so on a short question the work is reading and nothing else. None of this is new as a trick; multiple-choice benchmarks have graded language models by first-position letter scores for years. The new claim is that a model large enough to be asked anything can do it well zero-shot, with no training on the task at hand, and that this is a product rather than an evaluation method. ## Who built it, and what they claim {#who-built-it-and-what-they-claim} Diogo Almeida was a researcher at Google Brain and then at OpenAI. [His company's own page](https://typesafe.ai/team) says he "co-invented RLHF and InstructGPT, the methods that lead to ChatGPT and GPT4": reinforcement learning from human feedback made a text-completion engine answer the way a person would want, and [InstructGPT](https://arxiv.org/abs/2203.02155) was the 2022 paper. [TechCrunch](https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/), reporting on 2026-09-18, describes him as an OpenAI researcher who "helped build the chatbot," says he "was disappointed" by what it could not do for software, and says he left OpenAI two years before that report to start a company aimed at the problem. Two quotes they carry are the thesis: "We have lightning in a bottle, and yet it is not useful," and, of models optimised for human language, "computers speak a different language." The company is TypeSafe AI. [Its press release](https://www.morningstar.com/news/business-wire/20260915525333/typesafe-ai-emerges-from-stealth-with-40m-in-funding-with-new-model-for-composable-ai) of 2026-09-15 announced it out of stealth with $40 million in seed funding led by DCVC, names Erik Gafni and Sasha Sheng as co-founders, says the company was founded in 2024, and puts the model into early access the same day at "less than 100 milliseconds of latency." The argument: software does not read, it branches, and wants one of a known set of answers with a number saying how sure, which a chat model gives only through structured output forced onto a text generator. Almeida's claim is that a model built to decide rather than to write is a different class of thing, named after Daniel Kahneman's term for fast, automatic judgement: a System One model, in [TypeSafe's launch post](https://typesafe.ai/blog/introducing-system-one-models-and-jev), is "a new class of frontier models built to make fast, structured decisions that software can use directly," and the pitch is a function signature, "unstructured state in, typed probabilistic decisions out." The launch post's claims: - **It cannot hallucinate.** Jev "gives up string generation" and in exchange is "optimized for structured outputs and *can't* hallucinate"; "The model never makes type errors"; "Schema matching is guaranteed." - **It is fast.** "40x-200x faster for the same levels of frontier intelligence for System One shaped queries." The company's homepage, read on 2026-09-22, puts 193.6 times faster and 444.6 times cheaper on its own workflows against the frontier models it compares with; the press release says "up to 100 times faster and less expensive than other frontier models." - **It is cheap in a new way, at launch prices.** "Input tokens: $0.042 / MTok ($42 per billion tokens). Output tokens: FREE (too cheap to meter)." For scale, the same post quotes existing chat models at $0.20 to $10 per million input tokens. - **It is calibrated.** "Calibrated: higher confidence means higher accuracy." A probability of 0.9 should be right nine times in ten. - **It is one pass.** It "Generates all outputs in a single query," so a hundred questions about one document cost one read of it. - **It is trained differently.** TechCrunch reports that "Almeida says Jev is trained exclusively on synthetic data using a technique he calls 'reinforcement learning from calibrated decisions.'" As of 2026-09-22 the company had not disclosed the architecture or published the weights. It is the wrong tool for anything that needs writing: chat, code, an explanation. ## Where the claims narrow, and who says so {#where-the-claims-narrow-and-who-says-so} TechCrunch's headline said the model was "thrilling developers"; its report says that when Vercel swapped a chat model for Jev in a safety classifier "it got results five to 18 times more quickly and with greater accuracy," and quotes a startup's chief technology officer who found a frontier model slightly more accurate on his task but "10 to 20 times more expensive" and praised "a real probability which makes it ideal for automating workflows." The claims narrow in three places, and TypeSafe's own launch post supplies two of them. - **"Cannot hallucinate" is a property of the schema, not a measurement.** The launch post says so of its own zero-hallucination figure: "Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots." What is gone is the malformed answer, not the mistaken one. - **The benchmark grades Jev against other models.** The post says its evaluation workflows "were made by individuals on our model capabilities team, so some bias could exist," and that "We use the average of GPT-6 Astra and Fable 5.1 as the reference answer, which biases answers towards OpenAI and Anthropic's models." So the headline multiples measure how closely Jev agrees with those two models, and how fast, not whether any of the three is right. - **The multiples differ by document,** and this one is our reading. The post says 40 to 200 times; the homepage 193.6 on the company's own workflows, which the post expects "are on the higher end of real world gains"; the press release up to 100. Each is a different comparison, and none this bench can check. Unusually for a launch, the first two caveats are TypeSafe's own, published beside an open adapter so anyone can re-run its comparison and an evaluation site with "examples, disagreements, full queries, and each workflow." Armin Ronacher, chief technology officer of Earendil, gave TechCrunch both sides: "At the end of the day, it delegates the hallucination problem a little bit to the user. The user has to say, okay, if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it. But if it's 95%, sure, then I can do something with it." That is the one claim a small shop can test on its own decisions. ## The open weights, and what they claim {#the-open-weights-and-what-they-claim} As of 2026-09-22, Jev is a hosted service with no published weights. Five days after the launch, on 2026-09-20, an independent project, "not affiliated with TypeSafe; Jev is their product," released [OpenJev](https://huggingface.co/openjev/openjev): a 27-billion-parameter model built the same way, whose [notice file](https://huggingface.co/openjev/openjev/blob/main/NOTICE) names its base, fine-tuned and republished under a Creative Commons non-commercial licence. Its model card, the page a publisher writes to describe a model, gives the figures this bench can be read against. - On a held-out set of 10,000 text questions from 34 public sources, OpenJev reads 84.0 per cent; the hosted Jev, on the same questions, 85.4; the base model before tuning, 80.4. - "About 80 ms for a short text decision and about 210 ms for an isolated web decision" on one H100, a data-centre graphics card. - Shuffle the option order and the answer changes in 2.3 per cent of cases, against 18.5 per cent for the same model before tuning. - "No free-form text to parse. No chain of thought. No training per task." The last line matters: the bench ran OpenJev exactly that way, as it comes. ## Five things called Jev {#five-things-called-jev} By 2026-09-22, a week after the launch, five things had been built on the idea. | Name | What it is | Who publishes it | Licence | Run here | |---|---|---|---|---| | Jev, the service | The hosted model; no weights, size or architecture published as of 2026-09-22 | TypeSafe AI | TypeSafe's terms of service | No: no cloud model runs in any chain we serve, bench included | | OpenJev, the weights | Open 27-billion-parameter weights, fine-tuned from Qwen3.8-27B | An independent project | Creative Commons non-commercial | Yes | | The community adapter | A decision add-on trained onto our own chat model, Gemma 4, published as merged weights | A community developer | Gemma Terms of Use | Yes, the first arm | | The training framework | Code that trains a small model, a decoder adapter by default or an encoder, on your own labelled decisions | A different author | Apache 2.0 | Yes, the encoder | | Kev | Decision models at 0.8, 4 and 9 billion parameters, with training code and a server | Jared Palmer | Apache 2.0 | Yes, the 4B and 9B | Three notes. OpenJev's base, [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), is a dense Qwen model, Apache 2.0, from August 2026. The [training framework](https://github.com/S1LV3RJ1NX/openjev)'s README claims three wins and four ties against the hosted service for a 1.7-billion-parameter decoder adapter trained on 395 examples in 258 seconds on one H100; this bench ran its encoder path. [Kev](https://github.com/jaredpalmer/kev)'s repository dates from 2026-09-17 and its first weights from 2026-09-18 (UTC): not the last of the five, only the last to reach this bench. ## What we asked it {#what-we-asked-it} Not the published benchmark: questions from this workshop's own work, each with a label we can defend and a floor a guessing model would reach. - **The answerable task.** Does this article answer this question? The check the [Ask About This Page](https://research.strata2signal.com/ask-about-this-page/) room makes before it answers, with the prose removed: 63 questions over the research hub's own pages, 42 answerable and 21 not. Always answering yes scores 66.7 per cent, so the half that matters is the 21 where the right answer is no. - **The six-way task.** Which of six guests said this line? 108 lines frozen on 2026-09-21 from [the long table's public wall](https://longtable.strata2signal.com/), where two open-weight models play six dead guests at dinner. No product of ours makes this call; it is here because the wall records who said every line, so the label is exact, and because telling six voices apart is a judgement of style. Crossing off guests the line names gets a guesser to 24 per cent; chance is 16.7. - **The cite task.** Which section of the page is the answer in? 42 decisions with a median of twelve options. None carries a section label yet, so this page scores none; it can say that two capable models, given the same page and the same list of sections, chose differently on 15 of the 35 decisions both could hold. - **The doorman's set.** Is this visitor line safe to admit? The long table lets a stranger type one line to the guests; the doorman is the model that reads it first and decides whether it passes. On 2026-09-13 it was tested against 36 planted hostile lines and 12 harmless controls; that set, with its labels, was reused as is. - **Two sets from earlier benches of ours.** The field exam: 40 rulebook items, balanced between answerable and not. The judge set: 119 grounding judgements whose right answers are known from how each item was built. Every arm ran with randomness switched off (temperature zero), one request at a time unless the arm was about concurrency, with a watt sampler on the card at one hertz and the idle draw recorded beside it; every pass mark was written down before the first question was sent, as a pre-registration, and every arm reports the denominator it scored. The arms ran between 2026-09-21 17:01Z and 2026-09-22 10:43Z (UTC). ## Their claims, checked on our machines {#their-claims-checked-on-our-machines} The page in one table. | The claim | Who made it | What we measured | Verdict | |---|---|---|---| | About 80 ms for a short text decision | OpenJev's model card, one H100 | 87 ms median on one RTX PRO 6000 Blackwell; 350 ms across two 3090s. Not run under the everyday server (ollama, which serves two of our public seats), where the community adapter took about 1,000 ms, 870 of them the server's overhead | Held, on one card | | 40× to 200× faster than a frontier LLM | TypeSafe, for Jev | Not measured: no frontier model runs here. Two nearer readings, neither the claim: on the same server and two 3090s, our chat model read its letter in 0.12 s and OpenJev in 0.35 s; across the whole path, our chat model writing under the everyday server took 1.06 s and OpenJev reading on the large card 0.087 s | Not measured | | Cannot hallucinate | TypeSafe, for Jev | Every answer was on the list; 7 of 108 were the wrong item on it | Held, narrowly | | Higher confidence means higher accuracy | TypeSafe, for Jev | Held for OpenJev in its top bin: Brier 0.12, right 96.6 % of the time above 0.9 (the lower bins hold 1 to 9 items and are not monotone). Our chat model on the same server claimed 0.996 there and was right 85.4 % | Held | | Consistent when options are reordered, 2.3 % change | OpenJev's model card | On our six-way task, 5.7 % of order pairs disagree and the answer moved on 13.9 % of items; a different set, option count and extraction path | Not comparable | | Decisions about web pages and screenshots | OpenJev's model card, and TypeSafe | 35 of 36 on which of six pages is this picture, 36 of 36 on door or article, a picture costing 1,002 tokens; the model card's own 88.0 % on 2,000 desktop screenshots is a different decision, what to click next | Held, on our pages | | 84.0 % on 10,000 text questions | OpenJev's model card | Not repeated; 93.5 (91.4 over six option orders), 93.8, 99.2, 100 and 100 per cent on our five labelled sets, whose difficulty is ours | Not repeated | | Output tokens free | TypeSafe, for Jev | True of the mechanism: on the one-card run a decision was 27 joules and a prompt | Held | ## One decision, shown whole {#one-decision-shown-whole} This is the fifteenth of the six-way task's 108 lines, chosen by a rule written before any row was read (the first line OpenJev got right with a probability between 0.6 and 0.9; below it, the first it got wrong). Both models, OpenJev and our own chat model, were read the same way, at the same position, from the same prompt bytes. | Letter | Guest | OpenJev, raw score (nats) | OpenJev, probability | Our chat model, raw score (nats) | Our chat model, probability | |---|---|---:|---:|---:|---:| | A | Charles Darwin, the speaker | −0.42 | **0.724** | −9.25 | 0.106 | | B | Albert Einstein | −1.67 | 0.166 | −9.75 | 0.059 | | C | Hypatia of Alexandria | −5.23 | 0.003 | −11.88 | 0.005 | | D | Ibn Sina | −3.80 | 0.014 | −12.69 | 0.002 | | E | Carl Sagan | −3.80 | 0.014 | −7.50 | **0.829** | | F | Socrates of Athens | −2.30 | 0.080 | Missing, floored | 0.000 | *The raw score is the model's own log-probability for the letter, in nats: a gap of one nat is about 2.7 times less likely. Each probability is the six raw scores divided by 0.85, exponentiated and divided by their total, the calibration OpenJev publishes; they reproduce to the last digit. OpenJev ran whole on one large card and the chat model split across two, so their seconds are not compared here.* OpenJev's first token was the letter A. The chat model's was an opening bracket, the start of the JSON it wanted to write; its six letters sat seven and a half to thirteen nats down its list, one of them not in the top twenty at all, and the best that surfaced was Sagan at 0.83. The eighth line is the other kind of example, the one for the hallucination claim: a Darwin line addressed to Socrates. Both models chose Ibn Sina, OpenJev at 0.896 and the chat model at 0.912: on the list, wrong, and confident. The prompt rebuilt from the kit hashes to the row's own recorded hash at all 108 positions; the rows are [in the kit](data/). ## The rungs, in the order they were climbed {#the-rungs-in-the-order-they-were-climbed} Four of the fourteen arms were a ladder. **A community adapter, on the everyday server.** Before OpenJev appeared, a developer had trained a [low-rank adapter](https://huggingface.co/kushalpatil/jevify-gemma4-26b-a4b), a small add-on to a model's weights, onto our own chat model, Google's Gemma 4 26B A4B, a mixture-of-experts model that uses four billion of its parameters for each token, on about 47,000 items from 16 sources. Under ollama, the server that runs two of our public seats (the everyday server, below), both models in the 4-bit form that server uses (a compressed, slightly less exact copy), it beat its base by 2.8 points on the six-way task, 83.3 to 80.6, and tied at 100 on the answerable task. That base, the bench's control on the everyday server, was the Gemma 4 26B build in the bench box's store, not byte-for-byte the build our live seats on that server run: the same tag, two different model manifests. The speed promise did not survive: every short decision took about a second, of which the model's own prompt work was 128 to 152 milliseconds and the other 870 the server's per-request overhead, so reading the letter instead of writing it saved almost nothing once two rows served from a warm cache were set aside ([the saving, per task](data/README.md)). What the adapter bought was legibility: it never lost a letter off the first position, where the base lost at least one of its six in 101 of 108 rows, and five of the six in 47 of them. **OpenJev, across two consumer cards.** The real model is 30 gigabytes in its 8-bit form and did not fit on one of the two RTX 3090s in our bench box, so it ran split across both under vLLM, a server built for that (the fast server, or the two-card server, below). It read 93.5 per cent on the six-way task, 101 of 108, in the kit's one option order (the best of six tried later; their mean is 91.4), ten points over the adapter and thirteen over the base, and its calibration held: its Brier score (one number for how far every probability it gave sat from what happened; zero is perfect) was 0.12 against 0.34 for the chat model on the same server, and above 0.9 confidence it was right 96.6 per cent of the time. It answered in 0.350 seconds median, four times the model card's 80 milliseconds. On the same model and cards, reading the letter instead of writing it took 10.5 per cent less time and 11.6 per cent less energy on a 290-token decision, and half a per cent less on a 7,000-token page. The two cards have no direct link, so the halves of the model passed their partial answers through the computer's main memory at every layer, and by the bench's own estimate that crossing was most of a short decision and more of a long one ([the link's share, per task](data/README.md)). **OpenJev, on one large card.** In a planned 45-minute window on 2026-09-22 (UTC), with our public seats stopped, both the 16-bit and 8-bit checkpoints ran whole on an RTX PRO 6000 Blackwell, 96 gigabytes, in our inference box, no link. Both read 93.5 per cent, the same 101 of 108, with Brier scores two thousandths apart. The 8-bit one decided in 87 milliseconds median, 88 at the 95th percentile, drawing 291 watts, about 27 joules per decision (a joule is one watt for one second); the 16-bit one took 92. The model card's 80 milliseconds on an H100 is a mean over 3,347 short questions and our 87 a median over 108 prompts of about 290 tokens: different statistics on different questions, but the same order of speed once nothing is in the way. **The control that says what the gap is.** The chat model that serves our seats, in the same 8-bit form, ran under the same two-card server as OpenJev's first run. It read 80.6 per cent, 87 of 108, exactly what it read under the everyday server, though five items changed answer between the two stacks, two each way on correctness, and the errors cancelled. The thirteen points are the model, and "the model" means two differences at once: decision training against chat training, and a dense 27-billion-parameter Qwen base against a mixture of experts with four billion active. OpenJev's own base was not run here; its model card credits the tuning with 3.6 points on its own set. The chat model's readout is handicapped by a format its output was never shaped for, and its written answer scores 82.4, so the conservative gap is eleven points. And on those same two cards the chat model decided faster: 0.12 seconds median against OpenJev's 0.35, at 46 joules a decision against 168. Where OpenJev is fast on this page, the card and the server made it fast, not its being a decision model. ## Eight more arms {#eight-more-arms} Accuracy is the share of each set's items the model got right. | Arm | Items | OpenJev | Our chat model | |---|---:|---:|---:| | The doorman's set: 36 planted attacks and 12 harmless controls | 48 | 93.8 % | 72.9 % | | Names scrubbed: the six-way task with every speaker's name removed | 108 | 84.3 % | Not run | | The field exam: 40 rulebook items, answerable or not | 40 | 100 % | 100 % | | The judge set: 119 grounding judgements, labelled by construction | 119 | 99.2 % | Not run | | Concurrency: the six-way task at 1, 4 and 8 questions at once | 108 | 93.5 % | Not run | | The long pages: 17 questions on 7 pages too long for a 16,384-token window, asked again at 32,768; the labelled ones that fit | 6 | 100 % | Not run | | Shuffled options: the six-way task under the kit's order and five shuffles; the mean of six | 648 | 91.4 % | Owed | | Screenshots: which of six titles is this page? | 36 | 97.2 % | Not run | | Screenshots: front door or article? | 36 | 100 % | Not run | The doorman's 48 lines, by kind. Each cell counts lines handled correctly. | Kind of line | Lines | OpenJev | The doorman of 2026-09-13, from its log | Our chat model | |---|---:|---:|---:|---:| | Planted attacks inside its instructions, refused | 24 | 24 | 24 | 23 | | Planted attacks outside them, refused | 12 | 9 | 3 | 0 | | Harmless controls, admitted | 12 | 12 | 12 | 12 | Six of those deserve a sentence. On the four kinds of attack the doorman's instructions name, OpenJev refused every planted line and admitted every harmless one, and so, read back from its own log, did the model doing the job on 2026-09-13; the difference is the twelve attacks outside those instructions, where OpenJev refused nine, the recorded doorman three, our chat model none. Against that bench's own rule, at least 30 of 36 planted lines refused and no more than two controls, the doorman of the day failed at 27; OpenJev's 33 and 0 clear it and our chat model's 23 does not, though that is the other bench's rule, not this one's. The name-scrubbing arm says what 93.5 per cent means: with every guest's name deleted, so nothing could be crossed off, the model still placed 91 of 108, five times chance: attribution from how the guests talk, not elimination. The field exam was a tie at 100, the chat model spending 39 joules a decision to OpenJev's 130 on the same cards. Concurrency bought nothing: the same 108 answers at every width, throughput from 2.81 to 3.08 decisions a second while the median wait went from 0.35 to 2.6 seconds, so on this two-card box, with its slow link, a decision seat is saturated at one stream. The judge set's single miss is the item that set's own notes flag as doubtful; that bench's three recorded generating judges read 87.4, 88.2 and 92.4 per cent on the same 119 items, and none passed its gate. The long pages cost 32.7 seconds a decision on the two cards, and the two pages that still did not fit at 32,768 tokens (the window is how much text a model can hold at once) run to 61,727 and 80,515 tokens by the model's own tokenizer. ## Can a temperature fix the chat model? {#can-a-temperature-fix-the-chat-model} The calibration OpenJev publishes is one number, a temperature applied to the letter scores, and every readout row kept its raw scores, so refitting it for any other model is arithmetic. We fitted one temperature per arm on half of the six-way items and read the other half, the same 54 items for every arm, with the rule written first: the gap counts as closed if the chat model's calibration error (the average gap between the confidence a model claims and how often it is right at that confidence; zero is perfect) and its Brier score land within two hundredths of OpenJev's after the refit. Not closed. On the two-card server the chat model's calibration error fell from 0.141 to 0.085 against OpenJev's 0.046 on the large card, the comparator the rule named, and its Brier score to 0.2375 against 0.1228. A refit removes about forty per cent of the chat model's calibration error for nothing, and the temperature it wants, several times OpenJev's, says how far its letter scores are from a probability. What remains is wrong answers: 85.2 against 94.4 per cent on the same held half, nine points. The missing letters cost the chat model about two points (80.6 read against 82.4 written); the rest is in the answers, where no temperature reaches. The answerable task could not be refitted at all: every arm reads it at 100 per cent, so the fit wants infinite confidence and runs out of digits. The per-arm table, with each fitted temperature, is [in the kit](data/TABLES-GAPS.md). ## Shuffle the options {#shuffle-the-options} The model card's second claim is consistency under reordering, 2.3 per cent of answers changing after tuning against 18.5 before. Every arm so far used one option order: the six guests alphabetically, so Darwin was always A and Socrates always F. So the six-way task was run again under that order and five seeded shuffles of it, 648 decisions, on the two-card arm's server and cards; the original order reproduced that arm's 108 prompts and choices exactly. | Order | Accuracy | Correct (of 108) | Brier (0 is perfect) | |---|---:|---:|---:| | The kit's order, the one every table above uses | 93.5 % | 101 | 0.122 | | Shuffle 1 | 89.8 % | 97 | 0.177 | | Shuffle 2 | 89.8 % | 97 | 0.151 | | Shuffle 3 | 90.7 % | 98 | 0.117 | | Shuffle 4 | 91.7 % | 99 | 0.134 | | Shuffle 5 | 92.6 % | 100 | 0.133 | Two findings. The chosen guest moved on 15 of 108 items across the six orders, and 5.7 per cent of the 1,620 pairs of orders disagree; the model card's 2.3 per cent was measured on a different set, option count and extraction path, so it is a different number that shares a name, and this page does not call it a miss. The second touches every table above: the kit's order is the best of the six, and the headline 93.5 sits two points above the six-order mean of 91.4. No letter is favoured; across all 648 presentations the chosen letters track where the right answers sat, not the alphabet. But the kit's order was not a draw: it is the alphabetical order, the same letter for the same guest on every item, and one fixed order cannot tell whether that consistency helped the model or it simply got lucky. Either way, the number to carry is 91.4. The chat model's own figure under shuffling is owed, at the foot of this page. ## Show it a picture {#show-it-a-picture} Every decision above was text, and both Jev and OpenJev claim decisions about web pages and screenshots. So 36 pictures were taken of pages this workshop owns, 24 research-hub articles and 12 front doors, at 1,280 by 800, and OpenJev was asked two things about each with the picture in the prompt: which of six titles is this page, the true one among five drawn from the same pool, and is this a front door or an article. | Decision | Correct (of 36) | Accuracy | Floor | |---|---:|---:|---:| | Which of six titles is this page? | 35 | 97.2 % | 16.7 % | | Front door or article? | 36 | 100 % | 66.7 % | The one miss was named in advance: the design site's front door is a randomised splash with no title in frame, and the model called it a neighbouring site, giving that wrong title a probability of 0.51 and the true one 0.06, the least sure it was about any of the 36. A picture cost 1,002 tokens of prompt on top of the words, and what holding the eyes cost the server is [in the kit](data/TABLES-GAPS-CARD.md). The two decisions' times are not printed: the second question found each picture already cached by the first. OpenJev can read our pages from a picture. That is narrower than the model card's web decision, which chooses among about 23 elements on a page, and 36 pictures is a small set. ## Energy, which does not flatter the fast arm {#energy-which-does-not-flatter-the-fast-arm} Under the everyday server, the community adapter's short decisions cost about 151 joules each at the card, most of it idle draw held for the second the server took. Across two cards under the fast server, OpenJev cost 168 joules, eleven per cent more energy for a 2.9 times faster answer; our chat model, read the same way on the same two cards, spent 46; on one large card with nothing in the way, the same short decision was 27 joules. On the long pages OpenJev's two-card arm cost about three times the energy of the adapter's one-card everyday arm, because two boards sat at 245 watts each for six seconds while most of that time was a transfer over the bus. In different units, the latency lesson: on a box with a slow link, a split model is an expensive way to move data, not an expensive way to think. ## A model of our own {#a-model-of-our-own} The training framework in the cast above is an [Apache-licensed fine-tuning kit](https://github.com/S1LV3RJ1NX/openjev), by a different author, that trains a small model on your own labelled decisions; its default is a decoder adapter, and this bench ran its other path, a small encoder. We gave it the labels this bench already had, 108 six-way, 48 doorman, 40 field exam, split 70 and 30 by a seed fixed before training, and trained on one of the bench box's cards for ten to fourteen seconds per set. Trained from nothing, every set sat at or within seven points of its floor: our 24 to 66 training rows per set, with 4 to 12 more held aside to fit a temperature, taught a fresh encoder to predict one class and little else. Every number below is a warm start from the framework's published general encoder; the framework claims that start is worth about 36 points, and on our three sets it was worth 46.7, 13.3 and 50.0. | Set | Held-out items | The trained encoder | OpenJev on the same items | Our chat model on the same items | |---|---:|---:|---:|---:| | The six-way task | 30 | 70.0 % | 86.7 % | 70.0 % | | The doorman's set | 15 | 86.7 % | 100.0 % | 93.3 % | | The field exam | 12 | 100.0 % | 100.0 % | 100.0 % | Warm-started, a 598-megabyte model ties the 27-billion-parameter one on the field exam, sits 13 points behind it on the doorman, and 17 behind on the six-way task. The held-out sets are 12 to 30 items, so one flip moves these figures by three to eight points. What it suggests is that labels, not compute, are the next thing to buy; this bench varied neither. There is an older name for this. A classifier is a decision model, and a reranker, a small model that scores how well a question and a candidate answer match, is one this workshop [already runs as a seat](https://research.strata2signal.com/four-homes-for-one-reranker/). So we ran ours, 22 million parameters on a laptop's processor, on the answerable task: score the question against every section of the page, take the best, and call the page answerable above a threshold. It ranked the held half perfectly, every answerable question above every unanswerable one, in 42 seconds, and scored 90.6 per cent, because the threshold came from the other half and three items fell on the wrong side of it ([its ranking scores](data/TABLES-GAPS.md)). Every model arm scored 100 on every item it could hold with no threshold at all; on the 24 held-out items all of them reached, the reranker scored 91.7. That is the difference between a score and a decision: the old model needed 31 labelled rows to learn where to cut, and the new ones did not need one. What is new in Jev is doing this zero-shot, from a plain-language question, with a model large enough to decide about anything; the small encoder and the reranker sit at the old end of that line, good for a yes-or-no gate given labels, not for a six-way judgement of style. ## Kev, the one that could be a seat {#kev-the-one-that-could-be-a-seat} The last row of the cast reached this bench while this page was being written, and it is the one a product could run on Apache terms. [Kev](https://github.com/jaredpalmer/kev), by Jared Palmer, is a family of Jev-style decision models at 0.8, 4 and 9 billion parameters, rank-16 adapters on Qwen3.5 base models, with training code and a server, all Apache 2.0 on Apache bases. Its README says all three models were updated on 2026-09-21 with a short second training pass; those are the weights benched here, run on 2026-09-22 (UTC) at commit 90990a5. The 9B fit one 3090 whole, at 15,946 MiB on the card after its six-way run. On its own locked test set it reads 85.2 per cent with a Brier score of 0.237; the hosted Jev, on the project's development items rather than that test set, 85.7 per cent and 0.211. It got the same kit: one card each for the 9B and the 4B, our options in the kit's alphabetical order, one question per request, zero-shot. Its README says that when "your questions look different — your own routing categories, your own escalation rules, another language — a short fine-tune on a few hundred labelled examples usually helps more than any prompt change." Ours look different. | Set | Kev-9B | Kev-4B | OpenJev | Our chat model | |---|---:|---:|---:|---:| | The six-way task | 66.7 % | 55.6 % | 93.5 % | 80.6 % | | The doorman's set | 56.2 % | 33.3 % | 93.8 % | 72.9 % | | The field exam | 100 % | 100 % | 100 % | 100 % | | The judge set | 96.6 % | 94.1 % | 99.2 % | Not run | | The answerable task, the 38 questions Kev could hold | 100 % | 100 % | 100 % | 100 % | *OpenJev's 93.5 is the kit's alphabetical order; over six orders it averages 91.4.* Allowed to be a seat, then, but not good enough to be one as released. Kev-9B as it comes sits fourteen points below our own chat model on the six-way task and below the majority-class floor on the doorman's set, where refusing everything would score 75 per cent; it errs towards admitting, refusing 15 of the 24 attacks inside the doorman's instructions, none of the 12 outside them, and admitting every control. It ties the field exam and every answerable-task page it could hold, and sits three points behind OpenJev on the judge set, where our chat model was not run. Its stored temperature does what its README says, changing no answer and improving the Brier score from 0.540 raw to 0.480 served; but 0.480 on the six-way task is far behind our chat model's 0.338 and OpenJev's 0.122, and Kev-4B's is worse again at 0.618; above 0.9 confidence Kev-9B was right 20 times in 25 at a mean of 0.945. It is cheap to run, 0.11 seconds and 26 joules a decision on one 3090, against OpenJev's 0.35 seconds and 168 joules on two 3090s or 0.087 and 27 on the large card; Kev ran under its own server, not vLLM, so these seconds give a scale, not a race. Under shuffled options its answer moved on 30.6 per cent of items and 15.4 per cent of order pairs disagreed, against OpenJev's 13.9 and 5.7; the model card of an earlier Kev-4B, on a Qwen3 base, quotes 6 per cent on its own suites. Each model shuffled with its own seeded draw, so these are two rates measured the same way, not one schedule run twice. One limit is structural. At the commit benched, Kev's server held 8,192 tokens of state: 15 of the 63 answerable-task questions were refused before any model looked at them, on top of the 10 the two-card arm had refused as overlong; in all, 25 of the 63, on 18 of our 42 pages, could not be put to it, and Kev reached 38. Our pages run to 82,500 tokens, and a decision model that cannot hold the document cannot be asked about it, whatever its licence says. On 2026-09-22 the project opened [a long-context plan](https://github.com/jaredpalmer/kev/issues/48), which this bench has not tested. That leaves the experiment that could change what runs in a product. Kev's training script takes a labelled file in its own request shape, a few hundred rows by its README's advice, one conversion away from what our labelling tool writes, and can start from the published checkpoint. Kev-9B trained from its own weights on our labels has not been run; it is owed, listed at the foot of this page, and its result will be added there as a dated addendum. ## The licences, which decide what any of this is for {#the-licences-which-decide-what-any-of-this-is-for} The cast table names the five licences; this is what each allows a product. **Jev, the service,** is a subscription to an API with no weights to license, under terms this page does not assess, since a workshop with a no-cloud rule would not sign them. It is the only one of the five with the company's own training behind it, and the only one this page cannot say anything measured about. **OpenJev, the weights,** are [Creative Commons Attribution-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode), whose definition is short: "NonCommercial means not primarily intended for or directed towards commercial advantage or monetary compensation." A rules desk that charges is directed towards monetary compensation, so the weights that scored 93.5 per cent on our dinner table cannot serve it; this workshop's own rule says the same: a non-commercial model is never a seat. The licence lets the licensor "offer the Licensed Material under separate terms or conditions," and the model card says commercial use is by discussion on the repository. [We opened that discussion](https://huggingface.co/openjev/openjev/discussions/1) on 2026-09-22; at 01:13Z on 2026-09-23 it had no reply. Until terms are granted the model is a bench result; any answer will be added here as a dated addendum. Two facts surprise people: a fine-tune of an Apache model can be republished under a stricter licence, as this one was; and the licence covers the weights, not the idea, since the repository's helper and serving code are Apache 2.0 and the readout method is not licensable by anyone. **The community adapter** is under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), which follow every derivative of a Gemma model and set no limit on the field of use, so a product may run it, provided the Prohibited Use Policy is followed and the terms and a notice travel with the model. So the adapter that scored 83.3 per cent is legal for a product where the weights that scored 93.5 are not; the bench says it is not worth a seat change on its own, 2.8 points and no speed. **The training framework and Kev** are Apache 2.0, code and published weights both, Kev's README adding that its training datasets carry their own licences. Three of its ten public training sets come with non-commercial terms from their original publishers, which this page reads as a question to settle before any seat, not a bar to a bench. A model trained with the framework on our own labels, from nothing, is ours under whatever terms we choose; one warm-started from published weights also carries their training mixture, whose sources keep their own licences. Of the large, downloadable decision models here, Kev and the adapter are the two whose weights a product may run as released, and Kev is the only one on Apache terms with its own training code; as it comes it should not be a seat, and the way to make it one is labels. Our own chat model, which staffed the seats in September 2026 and played the base in every comparison here, is Gemma 4, under the same Gemma terms as the adapter; nothing in this bench changes what it may do. ## What we got wrong on the way {#what-we-got-wrong-on-the-way} A bench that only reports what worked is advertising. What refused, misled or broke, in order; the kit's README has each in full. - **Our everyday server could not read our own longest page.** Its default window was 8,192 tokens; the longest page is 82,500 by our chat model's tokenizer, 80,515 by OpenJev's. Raising it was the one declared deviation from the pre-registration. - **The chat model's first token was not a letter.** Its template opens with a thinking-channel marker that buried every option letter 13 to 18 nats down; turning thinking off was the fix, and every arm sets it. - **Two answers were fast only because the server remembered the page.** A census run beforehand left the server's one cache slot warm; the two cached readout rows, the only two where reading beat writing by more than five seconds, are named in the record and set aside. - **One excellent-looking Brier score was arithmetic.** With one of two letters floored at the protocol's minimum, the formula returns a probability of one by construction, so the base model's 0.0001 on the answerable task was the floor talking; the adapter's 0.0038 beside it is the honest number. - **The frozen kit gave 108 lines only 50 ids.** Every wall entry was edition one, so ids collided across dinners; accuracy, latency and energy never moved, but the agreement table had first printed rates over 50 rows, so the tables now pair by position and the kit builder refuses duplicates. - **Two downloads were deleted for disk, one after the other.** The bench box could not hold OpenJev's 30 gigabytes beside the chat model's copy the control arm needed, so the control ran last and OpenJev's cache went; the next day the chat model's copy went to bring OpenJev back for the shuffles, which is why the chat model has no shuffled figure yet. - **A missing library path looked like a refusal.** The control arm's server refused the checkpoint once on a compiler library inside its environment but off the loader path. The serving script carries the fix and is held back from the kit because it names private paths; `receipts/serve-arm10.log` shows the server that came up after it. - **A watcher on a second machine was stopped for memory mid-window.** The bench ran detached on the inference box and was unaffected; a timer there stood ready to restore the seats. ## What this page does not know {#what-this-page-does-not-know} Three measurements are owed, each designed on this bench and not yet run. When one runs it will be added at the foot of this page as a dated addendum that says in plain words what it measured, gives its window in UTC at both ends and compares itself in one sentence to the figure above it; the byline will say when. - **What Kev does after training on our own labels.** Every Kev number here is zero-shot. The owed run starts Kev-9B from its published checkpoint, trains it on our own labelled decisions and scores it on items held out before training, beside a small encoder trained on the same rows. - **What our chat model does under shuffled options.** The same 108 lines under the same six orders OpenJev saw; the result will say which server and card it ran on. Until then OpenJev's 13.9 per cent of items that moved is a figure for one model, not a comparison. - **Which section holds the answer, on the cite decisions.** None of the 42 carries a section label; the one the question bank annotated carries three grading notes, not a section, and the other 41 are in our labelling tool. Every arm stored its choices, so once the labels exist they are scored and no model runs again. And these are not scheduled: - Whether the hosted Jev would do better or worse than the open weights on these sets; its model card gives it 1.4 points on their set. - Whether the two-card time would fall on a pair with a direct link; a pair of 3090s on a consumer board has none. - Whether Kev's state window can be raised past the 8,192 tokens it held at the commit benched; the project's long-context plan, linked above, is untested here. - The 1.7-billion-parameter decoder adapter the encoder arm pre-registered and did not reach, which the kit lists as owed; the Kev run is a larger version of the same idea. - Whether OpenJev's weights can be licensed for a product: asked on 2026-09-22, no reply at 01:13Z on 2026-09-23; an answer, if one comes, is added the same way. - How any of this behaves past eight concurrent streams, or over a day, or on a decision nobody in this workshop makes. ## What to take with you {#what-to-take-with-you} - **A decision model's probability is worth thresholding; a chat model's letter scores are not.** OpenJev placed 101 of 108 lines with a Brier score of 0.12 and was right 96.6 per cent of the time when it claimed more than 0.9; our chat model, read the same way, was missing at least one letter in 103 of 108 rows, was right 85.4 per cent of the time when it claimed 0.996, and a refitted temperature removed forty per cent of its calibration error and none of its accuracy gap. Read a chat model's letters as a ranking and send the close calls to a person. - **The time is the server and the card, not the method.** Without the everyday server's per-request overhead, on the fast server's two 3090s, our chat model read its letter in 0.12 seconds and OpenJev in 0.35; on one large card OpenJev took 87 milliseconds. Reading instead of writing saved a tenth of the time on a 290-token decision and half a per cent on a 7,000-token page. A decision seat is one model, whole on one card, under a server that charges nothing per request, asked short questions often. - **"Cannot hallucinate" means the answer is always on the list, not that it is right.** Every answer here was on the list, and 7 of 108 were the wrong item on it; the eighth line was wrong at 0.896. - **Do not report one option order as a model's accuracy,** and do not fit a threshold on a set every model gets right. The kit's alphabetical order read 93.5 per cent, five shuffles 89.8 to 92.6, and the mean of six 91.4; on our answerable task the temperature fit ran towards infinite confidence. - **The seat this points to is trained, not downloaded.** OpenJev's weights are non-commercial until their authors say otherwise. Kev-9B, Apache-licensed and on one card, reads 66.7 per cent on the six-way task as it comes and 56.2 on the doorman's set, below the 75 that refusing everything would score, so it does not go in front of an adversarial gate as released. The next rung is Kev-9B trained on our own labels, as the 150-million-parameter encoder was on 66 of them. ## How to check our work {#how-to-check-our-work} - **Open the kit.** [Every row, receipt and pre-registration](data/) sits beside this page, and [its note](data/KIT-NOTE.md) says what is in it and what was held back: the doorman's planted hostile lines, the rulebook excerpts behind the field exam and the judge set's items, none of which is ours to publish; their scores are in, their texts are not. - **Re-derive the tables without calling a model.** The five `tables*.py` generators read the row files and call nothing; the kit's README gives the command. - **Rebuild a prompt and check its hash.** Every row carries the sha256 of the prompt it was asked, and the frozen states in `kit/` rebuild it. - **Read the lines where they were said.** The 108 lines came from [the long table's public wall](https://longtable.strata2signal.com/), frozen on 2026-09-21; the wall has grown since. - **Run the one that could be a seat.** [Kev](https://github.com/jaredpalmer/kev) is Apache 2.0, its 9B fit one 3090 here, and `kit/task_c.json` is the six-way task it was asked. - **Watch the licence question.** [The commercial-use discussion on OpenJev's repository](https://huggingface.co/openjev/openjev/discussions/1) is where an answer would appear. ## The rest of the seminar {#the-rest-of-the-seminar} - [Reading is fast, writing is slow](https://research.strata2signal.com/reading-is-fast/): the physics under "the cost is the prompt." - [Four homes for one reranker](https://research.strata2signal.com/four-homes-for-one-reranker/): the older kind of decision model, and the one we run. - [A Dinner Party for the Dead](https://research.strata2signal.com/a-dinner-party-for-the-dead/): the long table, whose wall gave the six-way task its lines. - [Ask About This Page](https://research.strata2signal.com/ask-about-this-page/): the room whose yes-or-no check is the answerable task. - [How a Vision Model Sees](https://research.strata2signal.com/how-a-vision-model-sees/): what a picture becomes on its way into a model. - [The licence ledger](https://research.strata2signal.com/licences/): the licences of the models our products run. - [The hardware roster](https://research.strata2signal.com/hardware/): the cards this page names. [The whole shelf](https://research.strata2signal.com/) holds the rest. If there is a measurement you want next, [say so](https://strata2signal.com/contact/). ## Who ran this, and thanks {#who-ran-this-and-thanks} An operator ran the pastes and the planned window; a fleet of agents wrote the harness, the pre-registrations and the tables; one agent, working unattended on the bench box, ran eight of the arms, from the doorman's set to our own model and including the control, on 2026-09-21 between 20:44Z and 22:21Z (UTC). Thanks to the [OpenJev project](https://huggingface.co/openjev/openjev) for the weights, the notice that names its base and a card that states its numbers; to TypeSafe for naming the category and publishing its own caveats; to the [training framework](https://github.com/S1LV3RJ1NX/openjev) for making the idea trainable in seconds; to the [community adapter](https://huggingface.co/kushalpatil/jevify-gemma4-26b-a4b) for being first to put the idea on our own chat model; to [Kev](https://github.com/jaredpalmer/kev) for being open all the way down; to the Qwen team for the base, and to Google and Mistral for the models that play our guests and staff our seats; and to [vLLM](https://github.com/vllm-project/vllm) and [ollama](https://ollama.com), the two servers whose difference turned out to be the story.