{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "openjev-as-a-reranker",
 "title": "OpenJev as a reranker for RuleSage: measured",
 "dek": "We benched OpenJev, a 27-billion-parameter open-weight decision model read as a yes/no judge, in place of the small model that keeps eight of up to 80 rulebook passages before RuleSage, our board-game rules helper, answers: on RuleSage's own labelled questions, under rules written before the run, its eight showed no difference, its yes/no missed both \"not in the book\" thresholds, and one ask took 20.6 s at the median against a 1.5 s budget, so nothing at RuleSage changes.",
 "published": "2026-10-16",
 "series": [
  "bench"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/openjev-as-a-reranker/",
  "md_url": "https://research.strata2signal.com/openjev-as-a-reranker/index.md",
  "md_sha": "4a25e58520bddb3517962e0f8971a5d65d22d5ae7e8d4f7435f0c4148a0aa094",
  "html_sha": "b0b8b9b72fbd338e86fc4a86303ed7e50795f0e940b5b10453c5d27b3abdd1ed",
  "anchors_sha": "985e6fa42527460a7701de2be237272323e85c7538d9956436b09c9d3313b6a6"
 },
 "short": {
  "paragraph": "A reranker is the model that re-reads a shortlist of rulebook passages and decides which eight the answering model gets to see. RuleSage's is a small cross-encoder, MiniLM. We benched OpenJev, a 27-billion-parameter decision model, in its place, asking each passage one yes/no question and sorting by how strongly it said yes. On 121 real questions with labelled pages, MiniLM's eight held 0.9271 of a question's labelled pages on average and OpenJev's 0.9108 (MiniLM's held at least one on 119 of the 121, OpenJev's on 120); the rule written before the run needed a gain of 0.030 to call a win, and reads this as no difference shown. OpenJev ranked the first labelled page higher (MRR 0.9563 against 0.9048) and led the rules desk's exam 32 to 28 on first picks, but no registered rule turns either into a verdict, and the real questions' labels lean towards MiniLM by construction. Its yes/no emptied the shortlist on 8 of 8 off-topic questions, 5 of 8 wrong-game questions and 4 of 12 questions the book does not answer; the thresholds were 14 of 16 and 6 of 12. One ask took 20.6 s at the median on an RTX PRO 6000 Blackwell (96 GB), and 3.6 s even when judging only eight passages, against a 1.5 s budget. A permissively licensed cross-encoder, bge-reranker-v2-m3, read page recall@8 0.9178 (no difference shown against MiniLM) and 131 ms at the median over the whole shortlist. OpenJev's weights are non-commercial and this was a bench; RuleSage's reranker stays as it is.",
  "from_dek": false,
  "counts": {
   "words": 3720,
   "minutes": 17,
   "tables": 6,
   "kit": false
  },
  "bullets": []
 },
 "sections": [
  {
   "id": "the-words-this-page-leans-on",
   "heading": "The words this page leans on",
   "level": 2,
   "span": [
    3230,
    4502
   ],
   "chunks": [
    [
     3230,
     4502
    ]
   ],
   "chars": 1272,
   "digest": "this section defines technical terms used in the article, including seat, passage, reranker, decision model, page recall@8, page recall@1, MRR, and p50 and p95.",
   "digest_skipped": null
  },
  {
   "id": "the-question",
   "heading": "The question",
   "level": 2,
   "span": [
    4502,
    5913
   ],
   "chunks": [
    [
     4502,
     5913
    ]
   ],
   "chars": 1411,
   "digest": "RuleSage finds passages using three searches and a reranker, currently using cross-encoder/ms-marco-MiniLM-L6-v2. the article asks if OpenJev can order a pool better than MiniLM and if its probability can identify unanswerable questions.",
   "digest_skipped": null
  },
  {
   "id": "the-four-arms-and-three-readings-on-top",
   "heading": "The four arms, and three readings on top",
   "level": 2,
   "span": [
    5913,
    7630
   ],
   "chunks": [
    [
     5913,
     7630
    ]
   ],
   "chars": 1717,
   "digest": "the article describes four arms (R0, R1, R2, R4) ranking candidates using different methods and three readings (R3, H1, R2@20) that use OpenJev's results or probabilities without extra calls.",
   "digest_skipped": null
  },
  {
   "id": "the-question-set-by-counts",
   "heading": "The question set, by counts",
   "level": 2,
   "span": [
    7630,
    9560
   ],
   "chunks": [
    [
     7630,
     9560
    ]
   ],
   "chars": 1930,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-rules-written-before-any-arm-ran",
   "heading": "The rules, written before any arm ran",
   "level": 2,
   "span": [
    9560,
    11692
   ],
   "chunks": [
    [
     9560,
     11692
    ]
   ],
   "chars": 2132,
   "digest": "the article outlines the rules for determining if OpenJev picks better (G-4), says \"not in the book\" (G-5), fits the budget (G-6), and provides a real readout (G-3).",
   "digest_skipped": null
  },
  {
   "id": "did-it-pick-a-better-eight",
   "heading": "Did it pick a better eight?",
   "level": 2,
   "span": [
    11692,
    14623
   ],
   "chunks": [
    [
     11692,
     14623
    ]
   ],
   "chars": 2931,
   "digest": "the readout gate held, but the quality reading showed no difference shown for OpenJev vs MiniLM or other comparisons, despite OpenJev having higher MRR and page recall@1.",
   "digest_skipped": null
  },
  {
   "id": "can-it-say-not-in-the-book",
   "heading": "Can it say \"not in the book\"?",
   "level": 2,
   "span": [
    14623,
    16622
   ],
   "chunks": [
    [
     14623,
     16622
    ]
   ],
   "chars": 1999,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-one-ask-costs",
   "heading": "What one ask costs",
   "level": 2,
   "span": [
    16622,
    18281
   ],
   "chunks": [
    [
     16622,
     18281
    ]
   ],
   "chars": 1659,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-rules-desk-where-every-arm-sees-the-same-eight",
   "heading": "The rules desk, where every arm sees the same eight",
   "level": 2,
   "span": [
    18281,
    19043
   ],
   "chunks": [
    [
     18281,
     19043
    ]
   ],
   "chars": 762,
   "digest": "on the 34 labelled cases where every arm orders the same eight, OpenJev leads with 32 of 34 first picks being a cited source.",
   "digest_skipped": null
  },
  {
   "id": "the-second-contrast-a-reranker-a-product-may-run",
   "heading": "The second contrast: a reranker a product may run",
   "level": 2,
   "span": [
    19043,
    20041
   ],
   "chunks": [
    [
     19043,
     20041
    ]
   ],
   "chars": 998,
   "digest": "an amendment added bge-reranker-v2-m3, an Apache 2.0 cross-encoder, which showed no difference shown against MiniLM or OpenJev.",
   "digest_skipped": null
  },
  {
   "id": "what-did-not-run-and-what-is-still-open",
   "heading": "What did not run, and what is still open",
   "level": 2,
   "span": [
    20041,
    21752
   ],
   "chunks": [
    [
     20041,
     21752
    ]
   ],
   "chars": 1711,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-this-cannot-say",
   "heading": "What this cannot say",
   "level": 2,
   "span": [
    21752,
    22567
   ],
   "chunks": [
    [
     21752,
     22567
    ]
   ],
   "chars": 815,
   "digest": "the article cannot say whether answers get better, anything about today's shelf, the floors RuleSage applies, or if OpenJev would pick better reading whole passages.",
   "digest_skipped": null
  },
  {
   "id": "what-changes-at-rulesage-nothing",
   "heading": "What changes at RuleSage: nothing",
   "level": 2,
   "span": [
    22567,
    23092
   ],
   "chunks": [
    [
     22567,
     23092
    ]
   ],
   "chars": 525,
   "digest": "this was a bench and moved nothing; RuleSage's reranker stays cross-encoder/ms-marco-MiniLM-L6-v2.",
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work",
   "heading": "How to check our work",
   "level": 2,
   "span": [
    23092,
    23767
   ],
   "chunks": [
    [
     23092,
     23767
    ]
   ],
   "chars": 675,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    23767,
    24449
   ],
   "chunks": [
    [
     23767,
     24449
    ]
   ],
   "chars": 682,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "sources",
   "heading": "Sources",
   "level": 2,
   "span": [
    24449,
    25666
   ],
   "chunks": [
    [
     24449,
     25666
    ]
   ],
   "chars": 1217,
   "digest": "the sources include the bench's pre-registration, laptop and GPU readings, RuleSage's retrieval bake-off, and various model licences.",
   "digest_skipped": null
  }
 ],
 "chips": [
  {
   "id": "c-861f73a3",
   "q": "how does the reranking process function?",
   "a": "RuleSage finds passages through three searches, fuses them into a list of up to 80, and uses a reranker to keep the top eight. The reranker reads each passage's heading and up to 1,200 characters of text.",
   "cites": [
    "the-question"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-95461cb7",
   "q": "what models and methods were compared in the arms?",
   "a": "The comparison included R0 (no reranker), R1 (MiniLM), R2 (OpenJev 27B), and R4 (bge-reranker-v2-m3). Three additional readings, R3, H1, and R2@20, were also evaluated based on OpenJev's probabilities and different passage depths.",
   "cites": [
    "the-four-arms-and-three-readings-on-top"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-413cd6c4",
   "q": "how is the question set structured?",
   "a": "Part A consists of 121 real questions from a retrieval set, plus 16 questions with no right answer. Part B includes 48 cases from a published exam, including 34 labelled cases and 12 book-silent cases.",
   "cites": [
    "the-question-set-by-counts"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-ce167449",
   "q": "what criteria determine if a model picks better?",
   "a": "To win, OpenJev must show a mean gain in page recall@8 of at least +0.030, pass a sign-flip permutation test with p below 0.05, maintain positive gains on both halves of the set, ensure MRR does not fall, and limit document recall@8 loss to 0.010.",
   "cites": [
    "the-rules-written-before-any-arm-ran"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "reading-the-answer",
   "why": "shares ground with § The rules desk, where every arm sees the same eight · § Eight more arms"
  },
  {
   "slug": "four-homes-for-one-reranker",
   "why": "shares ground with § The question · § What a reranker does at a rules desk"
  },
  {
   "slug": "the-new-kid",
   "why": "shares ground with § The four arms, and three readings on top · § The four answers"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nOpenJev's authors released the model's weights under CC BY-NC 4.0; the Beijing Academy of Artificial Intelligence released `bge-reranker-v2-m3` under Apache 2.0; the `cross-encoder/ms-marco-MiniLM-L6-v2` weights are the sentence-transformers project's. Ollama served OpenJev; PyTorch, transformers and sentence-transformers served the two cross-encoders. A small human team asked the question, approved the registration before any arm ran, and approved this page for publication; a fleet of AI agents wrote the registration, the kit and the scorers, ran and checked each stage, and wrote this page under that team's rulings.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 0,
  "digests_written": 14,
  "digests_kept": 10,
  "chips_written": 8,
  "chips_kept": 4,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 1,
   "figure": 1,
   "length": 1,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 1,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 7
  },
  "new_noun_tokens_checked": [
   "0.010",
   "0.0163",
   "0.030",
   "0.05",
   "0.9108",
   "0.9178",
   "0.9271",
   "1",
   "12",
   "121",
   "13",
   "130.861",
   "137",
   "14",
   "16",
   "2.0",
   "20",
   "200",
   "27B",
   "3",
   "30",
   "32",
   "34",
   "48",
   "5",
   "568",
   "571",
   "589.326",
   "744",
   "79.440",
   "8",
   "80",
   "Apache",
   "B",
   "G-3",
   "G-4",
   "G-5",
   "G-6",
   "GPU",
   "H1",
   "MRR",
   "MiniLM",
   "OpenJev",
   "OpenJev's",
   "Part",
   "R0",
   "R1",
   "R2",
   "R3",
   "R4",
   "RuleSage",
   "RuleSage's",
   "bge-reranker-v2-m3",
   "cross-encoder/ms-marco-MiniLM-L6-v2",
   "p50",
   "p95"
  ],
  "new_noun_tokens_withheld": 1,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "bullet",
    "reason": "bare_figure",
    "item": "the median passage length is 571",
    "detail": "the cited section writes this figure as '571 characters' and the bullet prints '571' alone"
   },
   {
    "kind": "chip",
    "reason": "new_noun",
    "item": null,
    "item_chars": 7,
    "withheld": "a string naming something the article does not",
    "detail": ""
   },
   {
    "kind": "chip",
    "reason": "figure",
    "item": "MiniLM achieved a page recall@8 of 0.9271, while OpenJev scored 0.9108 and bge-reranker-v2-m3 scored 0.9178. The comparison between OpenJev and MiniLM resulted in no difference shown because the mean ",
    "detail": "figures absent from cited spans: ['3']"
   },
   {
    "kind": "chip",
    "reason": "length",
    "item": "can the models identify when an answer is missing?",
    "detail": "yes/no opener: can"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 33,
    "text_sha": "8f362a67aa85",
    "figure": "568",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "loses its unit and its name: the article gives bge-reranker-v2-m3 568 million parameters, and the line ends on a bare 568 million and says only the cross-encoder, which on this page is also MiniLM, the incumbent",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: loses its unit and its name: the article gives bge-reranker-v2-m3 568 million parameters, and the line ends on a bare 568 million and says only the cross-encoder, which on this page is also MiniLM, the incumbent. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 30,
    "text_sha": "d045e65994e3",
    "figure": "30",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "drops its condition: about 30 ms is the incumbent MiniLM's stage on the page's 96 GB card, quoted from Four homes for one reranker; the line gives it to the reranker, which on this page is also OpenJev, at 20.6 s for a median ask",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: drops its condition: about 30 ms is the incumbent MiniLM's stage on the page's 96 GB card, quoted from Four homes for one reranker; the line gives it to the reranker, which on this page is also OpenJev, at 20.6 s for a median ask. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 164,
    "text_sha": "fae979f7355c",
    "section": "the-question-set-by-counts",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "wrong as written: the 137 pools are the 121 real questions and the 16 questions with no right answer in the pool beside them, and the line says all 137 come from real questions",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: wrong as written: the 137 pools are the 121 real questions and the 16 questions with no right answer in the pool beside them, and the line says all 137 come from real questions. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 164,
    "text_sha": "57c2f7b3a3d1",
    "section": "can-it-say-not-in-the-book",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "wrong as written for MiniLM's eight with the drop (H1): it met the no-answer threshold, 14 of 16, and fell short only on the book-silent cases, 4 of 12; OpenJev's eight (R3) fell short on both",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: wrong as written for MiniLM's eight with the drop (H1): it met the no-answer threshold, 14 of 16, and fell short only on the book-silent cases, 4 of 12; OpenJev's eight (R3) fell short on both. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 133,
    "text_sha": "5f11fa116a28",
    "section": "what-one-ask-costs",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "drops its condition: every timing and both budget verdicts are one 96 GB card's at a 420 W cap, and the section says a reading on one card says nothing about another",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: drops its condition: every timing and both budget verdicts are one 96 GB card's at a 420 W cap, and the section says a reading on one card says nothing about another. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 154,
    "text_sha": "1afb407f3643",
    "section": "what-did-not-run-and-what-is-still-open",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "understates the section: a follow-up also needs a clean readout gate and the budget met at some depth beside a win on recall@8 or a usable abstain signal, and the line names only the win and the abstain signal, as if either were enough",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: understates the section: a follow-up also needs a clean readout gate and the budget met at some depth beside a win on recall@8 or a usable abstain signal, and the line names only the win and the abstain signal, as if either were enough. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 265,
    "text_sha": "20e92de5eea0",
    "chip_id": "c-3f150fb8",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-14",
    "why": "drops its conditions: 3,589.326 ms is OpenJev's median over MiniLM's eight passages only, and every figure in the line is one 96 GB card's at a 420 W cap, and the section says a reading on one card says nothing about another card",
    "detail": "removed from this pack by hand on 2026-10-14 (UTC), after the writer ran and after the gates had counted: drops its conditions: 3,589.326 ms is OpenJev's median over MiniLM's eight passages only, and every figure in the line is one 96 GB card's at a 420 W cap, and the section says a reading on one card says nothing about another card. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-10-14T21:01:05Z"
}