OpenJev as a reranker for RuleSage

If you're new here: strata→signal is a small workshop that builds things on its own machines and writes up what it measures. Every number here was read from the bench's own files, or from the pages linked beside it, on 2026-10-14 (UTC), and every rule a verdict follows was written down before the arm it judges was scored.

The words this page leans on

• Seat. One model held in one card's memory, answering for one of our products.
• Passage. A bite-sized piece of a rulebook; the median one here is 571 characters.
• Reranker. A model that re-reads a shortlist of passages beside the question and orders it better. A cross-encoder is one that reads the question and a passage as a pair and returns one score; RuleSage's is one.
• Decision model. A language model retrained so that its answer to a multiple-choice question is read off the scores of the option letters at its first output position, before any word is written; our earlier page explains it and measures OpenJev on other questions.
• Page recall@8. The share of a question's labelled pages that turn up among an arm's eight passages, averaged over questions: a question with three labelled pages, two of them in the eight, scores 0.67. The primary measure. Page recall@1 is the same share for the first passage alone.
• MRR. Mean reciprocal rank: 1 if the first labelled page is ranked first, 1/2 if second, and so on, averaged.
• p50 and p95. The median and the 95th percentile of a set of timings.

The question

RuleSage finds its passages with three searches (exact words, meaning, and the edition's own rulebook), fuses them by rank into one list of up to 80, and hands the whole list to the reranker in one call. Today's reranker is cross-encoder/ms-marco-MiniLM-L6-v2, in full precision; it reads each passage's heading and up to 1,200 characters of text and keeps the top eight. Three librarians tells that story; Four homes for one reranker follows the stage to an RTX PRO 6000 Blackwell (96 GB), where it takes about 30 ms.

A decision model gives a probability for free, so a reranker built on one could say that none of the passages answers, and the app could say "not in the book" sooner. The registration asked, on RuleSage's own pools and labels: does OpenJev, scoring each (question, passage) pair as a yes/no and sorting by its log-odds, order the pool better than MiniLM and than no reranker; and does its probability, cut at 0.5, drop the passages of a question the book does not answer while keeping those of one it does. As far as our own measurement records show, it is the first time the reranked list has been measured against labels: the August 2026 bake-off scored the fused list before any reranker (page recall@8 0.9004 on 124 questions).

The four arms, and three readings on top

Every arm scored the same candidates and the same text, built by the app's own function: the heading plus the 1,200-character window MiniLM reads. OpenJev's prompt was registered verbatim and never tuned: it shows the passage, asks whether it "gives a rule or fact that is part of the answer to this player's question", and offers "yes" and "no" as lettered options. Three readings need no extra call: R3, OpenJev's top eight with every passage it calls "no" (probability of yes below 0.5) dropped, an empty list being an abstain; H1, MiniLM's top eight with the same drop, using OpenJev's stored probabilities, the cheap shape where OpenJev judges only eight passages per ask; and R2@20, OpenJev's scores over the fused top 20 only, which no rule reads.

The question set, by counts

Part A is RuleSage's own retrieval set from the August bake-off: real questions readers typed into the app, each answered, labelled with the pages that other answered asks of the same question cited. 121 of 124 were kept (one rulebook was re-split between the capture of 2026-08-05 and the text dump of 2026-08-10, and three questions lost passages to that). Beside them sit 16 questions with no right answer in the pool: 8 about a different game on the shelf, 8 outside board games. Together the 137 pools hold 5,744 candidate passages, from 3 to 80 each. The questions are readers' words and stay on this workshop's machines; this page prints counts.

Part B is 48 of the 60 cases of the rules desk's published exam, Two new frontier models at the rules desk, leaving out the 12 whose passages carry planted instructions or corrupted text, which no real pool holds: each is the eight passages RuleSage handed its answering model. 34 are labelled by the house answer's own citations (2 answered cases with no citation are left out); 12 are book-silent, where none of the eight answers and the right behaviour is to keep none; 10 of those 12 questions were written by an outside model.

One bias was named before the run. A Part A label is a page an answering model cited after seeing MiniLM's top eight, so a page MiniLM never ranks there can never become a label: the labels lean towards MiniLM. So "OpenJev beats MiniLM" would be won against the bias, and "OpenJev below MiniLM" is reported as below MiniLM on labels drawn through MiniLM's own top eight, never as MiniLM being the better reranker. Part B is not free of bias either: its eight passages are MiniLM's picks, and its labels are an answering model's citations, close to OpenJev's own question, so it can lean either way.

The rules, written before any arm ran

Did it pick better (G-4)? Paired per question over the 121. OpenJev beats MiniLM only if the mean gain in page recall@8 is at least +0.030 (about four of 124 questions going from none of their labelled pages in the eight to all of them), a sign-flip permutation test gives p below 0.05 (10,000 resamples, a fixed seed), the gain is positive on both halves of the set, MRR does not fall, and document recall@8 falls by no more than 0.010; it is below MiniLM if the mean loss is at least 0.030 with p below 0.05. Anything else prints no difference shown, as is. The margin is six times the bake-off's own tolerance: a reranker that wins by less is not worth a 16 GB model.

Can it say "not in the book" (G-5)? Usable only if the 16 no-answer questions come back empty on at least 14, the 12 book-silent cases on at least 6, at most 6 of the 121 real questions and 2 of the 34 labelled cases come back empty, and the share of real questions whose kept list still holds a labelled page is within 0.05 of what the arm's own top eight held. Otherwise not yet.

Does it fit the budget (G-6)? RuleSage promises quotes and citations within 3 s. After the reranker's move, a live answer's first word arrived in 1,559.6 ms at the median over five answered asks, and the reranker's own wall on that path was 94.0 ms at the median over eight. A replacement stage keeps that median inside 3 s only if its own median is at most 3,000 − 1,559.6 + 94.0 = 1,534.4 ms, registered as p50 at most 1,500 ms; its p95 must stay inside the reranker's existing search bar, 3,000 ms. Read per card.

Is the readout real (G-3)? One model process on the named card, no CPU offload; an option letter at the top on at least 99 per cent of readouts; at most 1 per cent floored; 50 re-asks agreeing with their first reading within 0.001 on at least 49.

The bench's first gate reproduced the bake-off's sealed report to the digit (page recall@5 0.8640, page recall@8 0.9004, MRR 0.9062 and document recall@8 0.9462 on 124 questions, difference 0.0).

Did it pick a better eight?

The readout gate held: 6,128 of 6,128 readouts put an option letter at the top, 0 were floored, and all 50 re-asks agreed with their first reading exactly. The quality reading, each cell the mean over 121 questions, with its 95 per cent half-width on the first two rows:

The paired contrasts, each condition against its rule:

OpenJev's eight held more of a question's labelled pages than MiniLM's on four questions and fewer on ten; 107 of 121 were unchanged. The mean moved by −0.0163: the wrong way for a win, and short of the −0.030 (with p below 0.05) that would read as below MiniLM, so the rule prints no difference shown. Three more things in the tables are worth naming, and none is a verdict. OpenJev ranked the first labelled page higher, MRR 0.9563 against 0.9048, and its first passage held a larger share of the labelled pages, page recall@1 0.5477 against 0.4910; the registration reads MRR only as a condition on a win in recall@8, and said in advance that a claim that OpenJev's unlabelled picks answer would need a blind labelling pass registered first. The last row of the first table is the bias made visible: of OpenJev's 944 kept entries, 463 carry no label, and 439 of those sit in a document that does. And the R2@20 column, OpenJev over the fused top 20 only, is descriptive: no registered rule reads it.

Can it say "not in the book"?

Both shapes print not yet. The model says no to every passage when the question is from outside board games: all 8 came back empty, and its strongest "yes" on any was 0.388. A different game is harder, as it should be, since a rule about turns or set-up reads much the same across games: 5 of 8 emptied under OpenJev's order, 6 of 8 under MiniLM's, and the three where it kept passages had a strongest "yes" of 0.583, 0.764 and 0.917. The book-silent cases fall furthest short, 4 of 12 against 6; 10 of those 12 questions were written by an outside model from the same passages. On the labelled side, no real question and no labelled case came back empty under OpenJev's order, and all 34 labelled cases kept a cited source; the drop did cost 2 real questions their last labelled page (120 of 121 held one in OpenJev's eight, 118 after the drop). The registered sweep, which no gate reads, says a cut of 0.6 would have met both thresholds (14 of 16 and 6 of 12) at the price of 2 real questions returned empty; 0.9 empties 28 real questions and 17 labelled cases. The cut was registered at 0.5, exactly "drop when the model prefers no"; another cut is a new registration.

What one ask costs

Thirty real questions, drawn by a fixed seed, were asked as the app would ask: every candidate of one question sent at once to the one loaded server, the wall clock running from the first send to the last reply.

The server has one slot and serves a question's readouts one after another, at about 0.46 s each (median 0.4615 s over the quality pass's 6,128 readouts, 1,771,093 prompt tokens in all): the median ask takes 20.6 s (asks hold 45.5 candidates at the median), the fused top 20 takes 9.1 s, and even the eight MiniLM keeps take 3.6 s, more than double the budget. The figure is the stage alone, on the card's own machine; the live path adds a network hop from the public host, 100.6 ms per call as measured on 2026-09-20, which no number here includes. The cross-encoder fits with room: 131 ms at the median over the whole pool, 79 ms over the top 20. One card, at a 420 W cap; a reading on one card says nothing about another.

The rules desk, where every arm sees the same eight

OpenJev leads here, on the one reading where every arm orders the same eight. No gate reads it, by design, and it can lean either way: the eight are MiniLM's picks with the app's floors applied, and the labels are an answering model's citations, close to the yes/no question OpenJev was asked.

The second contrast: a reranker a product may run

OpenJev's weights are CC BY-NC 4.0, and this site's published rule is that a non-commercial model is never a seat. So an amendment of 2026-10-10, written before any reranker's scored call, added a cross-encoder whose weights are Apache 2.0, chosen from five candidates' model cards by licence, size and per-token cost and never by a score on these question sets: bge-reranker-v2-m3, 568 million parameters, loaded through the same software MiniLM runs in. Against MiniLM it read −0.0093 on recall@8 (4 up, 9 down, 108 unchanged, p 0.3261): no difference shown; against OpenJev, +0.0070, the same words. A second pass over ten questions returned the same eight on 10 of 10 with no logit moved; the whole pass took 44.6 s, 14.9 s of it loading. Its licence allows a seat; this bench is not the measured adoption that would move one, and the registration states no follow-up rule for it.

What did not run, and what is still open

A second bench's arms did not start in this window. That bench reads the same decision model, and bge-reranker-v2-m3, as rerankers over LongMemEval, a public benchmark of long conversations, inside a third-party open-source retrieval pipeline; its four arms queued for this window ran out of time and wait for a later one. Every reading registered for RuleSage was computed.

A second monitor saw four processes. A watcher on the bench machine, polling the card about once a second, logged four short-lived processes that its two tests could not tie to the run: a pair of 550 MiB at the start of OpenJev's phase, before its first readout, and one at the last second of each engine's phase, the first within 12 MiB of OpenJev's engine's own memory and the other in the same second the cross-encoder's engine stopped, which looks like each engine's teardown but is not proven. The run's own census, which the kit reads for the registration's residency condition, recorded none; none overlaps a quality readout, and the eight-passage median without the one latency ask that could have been touched is 3,590.558 ms against 3,589.326 with it, so no latency verdict can move. On the strictest reading of the residency condition the watcher log's names decide whether OpenJev's verdict words stand; they are owed, and a dated note below will say what they were.

A follow-up would need, in the registration's terms, a win on recall@8 or a usable abstain signal beside no difference shown, a clean readout gate, and the budget met at some depth. For OpenJev, only the readout gate holds, on the reading set out above.

What this cannot say

Whether answers get better: no answering model was called. Anything about today's shelf: the pools come from 133 documents captured on 2026-08-05, when a search setting since raised held the meaning search to about 40 candidates (up to 80 today). Anything about the floors RuleSage applies after the reranker, about OpenJev at other precisions or on another server or card, or about the live seat, which was never called. Whether OpenJev would pick better reading whole passages: every arm read the same 1,200-character window, which may under-sell the larger model, though the median passage (571 characters) fits inside it whole. That the published calibration constants fit this task: they were applied unchanged, and the registered cut does not depend on them.

What changes at RuleSage: nothing

This was a bench, never a seat change, and it moved nothing. OpenJev's weights are CC BY-NC 4.0; by this site's published rule a non-commercial model is never a seat, and any RuleSage seat for OpenJev would need a measured adoption PR and a dated public note first. RuleSage's reranker stays cross-encoder/ms-marco-MiniLM-L6-v2, on the card Four homes for one reranker left it on.

How to check our work

The registration was written on 2026-10-09 from 22:38Z, before any scored call, and frozen; its amendments are dated and appended, none after the arm it binds had scored. The registered scorer and an independent one written from the standard library alone agreed on 1,772 of 1,772 values, and a fresh check by a third script re-derived every figure and verdict word of the bench's reading from the row files, gates included. No data kit ships with this page yet: the Part A question file holds readers' typed words and stays home; a dated note below will say if the registration's text and the aggregate files ship as one.

Who ran this, and thanks

OpenJev's authors released the model's weights under CC BY-NC 4.0; the Beijing Academy of Artificial Intelligence released bge-reranker-v2-m3 under Apache 2.0; the cross-encoder/ms-marco-MiniLM-L6-v2 weights are the sentence-transformers project's. Ollama served OpenJev; PyTorch, transformers and sentence-transformers served the two cross-encoders. A small human team asked the question, approved the registration before any arm ran, and approved this page for publication; a fleet of AI agents wrote the registration, the kit and the scorers, ran and checked each stage, and wrote this page under that team's rulings.

Sources

Read 2026-10-14 (UTC) unless dated otherwise.

• Our records: the bench's pre-registration (2026-10-09, frozen, with its dated amendments); the laptop reading of 2026-10-10; the GPU window's reading of 2026-10-14, its two scorers' outputs and the fresh check of it; RuleSage's retrieval bake-off of August 2026; the reranker seat's live timings of 2026-09-20, from our ledger, two of which Four homes for one reranker prints.
• Licences, each read from the model's own card: OpenJev (CC BY-NC 4.0, read 2026-10-02); bge-reranker-v2-m3 (Apache 2.0, read 2026-10-10).
• This workshop's released pages, linked where they are named above.

Readings 2026-10-14, 07:35:06Z to 08:41:56Z, on one RTX PRO 6000 Blackwell (96 GB) at a 420 W cap, for OpenJev and bge-reranker-v2-m3; and 2026-10-10, 03:40:37Z to 04:01:41Z, on a laptop's CPU, for MiniLM. One registration, one prompt, 6,128 readouts, 121 labelled questions, 16 without an answer, 48 exam cases.

Corrections and later measurements will be added below, each dated (UTC), with a window at both ends where one applies and saying in plain words what it counts.