{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "nemotron-rerank-8b-on-our-own-workloads",
 "title": "Nemotron 3.5 Rerank 8B, measured on our own workloads",
 "dek": "An 8-billion-parameter reranker released on 2026-10-20, run on an RTX PRO 6000 Blackwell (96 GB) over RuleSage's own labelled questions and a 500-question memory benchmark: \"exploratory: no material difference\" on RuleSage at about four times the cost per pair, and an \"exploratory lead\" of 457 to 409 on LongMemEval that is not a gated win.",
 "published": "2026-10-26",
 "series": [
  "bench"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/nemotron-rerank-8b-on-our-own-workloads/",
  "md_url": "https://research.strata2signal.com/nemotron-rerank-8b-on-our-own-workloads/index.md",
  "md_sha": "91a6561d9efe1119bc82e4d59c161e75f9486e1ce46fb6278c9a36af476a0c8b",
  "html_sha": "f3e66711566ce1cae38e3631a759e6ec901d4efee60813e57b253a9c4e3dc9ba",
  "anchors_sha": "687902f2858cf5ce0bc83d2b15490228a9000a30429e973d3052ce8a0a359e95"
 },
 "short": {
  "paragraph": "A reranker re-reads a shortlist of passages beside a question and decides which few the answering model gets to see. RuleSage's is a small cross-encoder, MiniLM; an earlier page, published 2026-10-16, benched a 27-billion-parameter decision model and a 568-million-parameter cross-encoder in its place, and both read \"no difference shown\" against it. This page benches a third, Nemotron 3.5 Rerank 8B, under the same rule and the same labels. On 121 real questions, its eight held 0.9098 of a question's labelled pages on average, against 0.9178 for the 568-million-parameter cross-encoder and 0.9271 for MiniLM; the rule written before the run needed a gain of 0.030 to call a lead, and prints \"exploratory: no material difference\" on every pair. It ranked the first labelled page higher than any arm (MRR 0.9758), which no registered rule reads as a win, and one ask took 524.0 ms at the median over the whole shortlist on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB): in budget, and about four times what the 568-million-parameter cross-encoder took on the same card. On LongMemEval it put every evidence session in the top five on 457 of 470 answerable questions, where the hybrid search it reranked had read 409: an \"exploratory lead\" by the registered words, outside the gated family of contrasts, carrying a caveat that its training data may overlap public question sets, and standing beside two contrasts that were not read because their other arms have no registered reading yet. The weights are under the OpenMDW-1.1 licence; a seat moves only by a measured adoption, after a read of those terms; RuleSage's reranker stays as it is.",
  "from_dek": false,
  "counts": {
   "words": 4526,
   "minutes": 21,
   "tables": 8,
   "kit": false
  },
  "bullets": []
 },
 "sections": [
  {
   "id": "the-words-this-page-leans-on",
   "heading": "The words this page leans on",
   "level": 2,
   "span": [
    3644,
    5395
   ],
   "chunks": [
    [
     3644,
     5395
    ]
   ],
   "chars": 1751,
   "digest": "this section defines technical terms used throughout the article, including reranker, seat, page recall@8, mrr, every@5, any@5, mcnemar's exact test, p50, p95, bf16, and exploratory.",
   "digest_skipped": null
  },
  {
   "id": "the-model-and-the-question",
   "heading": "The model, and the question",
   "level": 2,
   "span": [
    5395,
    7679
   ],
   "chunks": [
    [
     5395,
     7679
    ]
   ],
   "chars": 2284,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-two-workloads-by-counts",
   "heading": "The two workloads, by counts",
   "level": 2,
   "span": [
    7679,
    10450
   ],
   "chunks": [
    [
     7679,
     10450
    ]
   ],
   "chars": 2771,
   "digest": "the article details two workloads: RuleSage, consisting of Part A with 121 questions and Part B with 34 cases, and LongMemEval, which includes 500 questions. it describes the various arms being compared and the potential for bias in the labels.",
   "digest_skipped": null
  },
  {
   "id": "the-rules",
   "heading": "The rules, written before the model was called",
   "level": 2,
   "span": [
    10450,
    12854
   ],
   "chunks": [
    [
     10450,
     12854
    ]
   ],
   "chars": 2404,
   "digest": "this section outlines the pre-registered rules for determining an exploratory lead, exploratory trail, or no material difference for both RuleSage and LongMemEval. it also notes changes in precision, runtime, and input compared to earlier arms.",
   "digest_skipped": null
  },
  {
   "id": "did-it-pick-a-better-eight",
   "heading": "Did it pick a better eight for RuleSage?",
   "level": 2,
   "span": [
    12854,
    16102
   ],
   "chunks": [
    [
     12854,
     16102
    ]
   ],
   "chars": 3248,
   "digest": "the results for RuleSage show that the new model, R5, achieved an mrr of 0.9758. however, the primary measure, page recall@8, resulted in an exploratory: no material difference reading against all compared arms.",
   "digest_skipped": null
  },
  {
   "id": "the-rules-desk",
   "heading": "The rules desk, where every arm sees the same eight",
   "level": 2,
   "span": [
    16102,
    16881
   ],
   "chunks": [
    [
     16102,
     16881
    ]
   ],
   "chars": 779,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-one-ask-costs",
   "heading": "What one ask costs",
   "level": 2,
   "span": [
    16881,
    18417
   ],
   "chunks": [
    [
     16881,
     18417
    ]
   ],
   "chars": 1536,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "longmemeval",
   "heading": "LongMemEval: the exploratory lead, and what sits beside it",
   "level": 2,
   "span": [
    18417,
    22333
   ],
   "chunks": [
    [
     18417,
     22333
    ]
   ],
   "chars": 3916,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "not-read",
   "heading": "Two contrasts were not read",
   "level": 2,
   "span": [
    22333,
    24227
   ],
   "chunks": [
    [
     22333,
     24227
    ]
   ],
   "chars": 1894,
   "digest": "two specific contrasts, A7 vs A6 and A7 vs A3, were not read. the section also lists five predictions made before the model was called, noting which occurred or did not occur.",
   "digest_skipped": null
  },
  {
   "id": "what-changes-at-rulesage",
   "heading": "What changes at RuleSage: nothing",
   "level": 2,
   "span": [
    24227,
    24914
   ],
   "chunks": [
    [
     24227,
     24914
    ]
   ],
   "chars": 687,
   "digest": "nothing changes at RuleSage; the reranker remains cross-encoder/ms-marco-minilm-l6-v2 and this bench was not a seat change.",
   "digest_skipped": null
  },
  {
   "id": "what-this-cannot-say",
   "heading": "What this cannot say",
   "level": 2,
   "span": [
    24914,
    25717
   ],
   "chunks": [
    [
     24914,
     25717
    ]
   ],
   "chars": 803,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-to-take-with-you",
   "heading": "What to take with you",
   "level": 2,
   "span": [
    25717,
    26814
   ],
   "chunks": [
    [
     25717,
     26814
    ]
   ],
   "chars": 1097,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work",
   "heading": "How to check our work, and see it live",
   "level": 2,
   "span": [
    26814,
    28100
   ],
   "chunks": [
    [
     26814,
     28100
    ]
   ],
   "chars": 1286,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "the-rest-of-the-seminar",
   "heading": "The rest of the seminar",
   "level": 2,
   "span": [
    28100,
    28838
   ],
   "chunks": [
    [
     28100,
     28838
    ]
   ],
   "chars": 738,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    28838,
    29916
   ],
   "chunks": [
    [
     28838,
     29916
    ]
   ],
   "chars": 1078,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "sources",
   "heading": "Sources",
   "level": 2,
   "span": [
    29916,
    31725
   ],
   "chunks": [
    [
     29916,
     31725
    ]
   ],
   "chars": 1809,
   "digest": "this section lists the sources for the records, the model, LongMemEval, deep-recall, and the licences of the earlier arms used in the benchmarks.",
   "digest_skipped": null
  }
 ],
 "chips": [
  {
   "id": "c-634c2c06",
   "q": "what terminology defines the metrics used in this study?",
   "a": "The article defines several key terms, including Reranker, Seat, Page recall@8, MRR, every@5, any@5, McNemar's exact test, p50 and p95, BF16, and Exploratory.",
   "cites": [
    "the-words-this-page-leans-on"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-b300d2c8",
   "q": "how is the nemotron-3.5-rerank-8b-bf16 model described?",
   "a": "It is an 8.38-billion-parameter cross-encoder based on Mistral 3, with weights held in BF16 and a scoring head computing in FP32. It was selected as the newest reranker release in its license class for the benchmark.",
   "cites": [
    "the-model-and-the-question"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-b29bf2ae",
   "q": "how are the rulesage workloads structured?",
   "a": "RuleSage Part A consists of 121 real questions with associated page labels, while Part B consists of 34 labelled cases from a published exam. Both workloads are compared against several existing reranking arms.",
   "cites": [
    "the-two-workloads-by-counts"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-f990bfa6",
   "q": "what criteria determine an exploratory lead for rulesage?",
   "a": "An exploratory lead requires a mean gain in page recall@8 of at least +0.030, a sign-flip permutation test p below 0.05, a positive gain on both halves of the set, MRR not falling, and document recall@8 falling by no more than 0.010.",
   "cites": [
    "the-rules"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-b92feccd",
   "q": "how did the new model perform on rulesage recall metrics?",
   "a": "The model achieved a page recall@8 of 0.9098. While it reached the highest MRR of 0.9758 and the highest page recall@1 of 0.5618, the paired contrasts against other arms resulted in exploratory: no material difference.",
   "cites": [
    "did-it-pick-a-better-eight"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "openjev-as-a-reranker",
   "why": "shares ground with § Did it pick a better eight for RuleSage? · § Did it pick a better eight?"
  },
  {
   "slug": "voice-trials",
   "why": "shares ground with § Did it pick a better eight for RuleSage? · § The method, limits first"
  },
  {
   "slug": "four-homes-for-one-reranker",
   "why": "shares ground with § Did it pick a better eight for RuleSage? · § What a reranker does at a rules desk"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nNVIDIA released the Nemotron 3.5 Rerank 8B weights under the OpenMDW-1.1 licence, naming Ministral-3-8B-Instruct-2512-BF16 (Apache-2.0) as the base model in the repository's NOTICE; the Beijing Academy of Artificial Intelligence released `bge-reranker-v2-m3` under Apache 2.0; the `cross-encoder/ms-marco-MiniLM-L6-v2` weights are the sentence-transformers project's; Nomic's nomic-embed-text supplied the hybrid search's vectors. LongMemEval is the work of Wu et al. (2024), its cleaned release under MIT; the retrieval pipeline follows the shape of deep-recall, an open-source memory tool under MIT. PyTorch, transformers and sentence-transformers served the cross-encoders, and Python's standard library alone wrote and checked every figure. A small human team asked the question, approved the registration before the model was called, and approved this page for publication; a fleet of AI agents wrote the amendment, the kit and the scorers, ran and checked each stage, and wrote this page under that team's rulings.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 0,
  "digests_written": 13,
  "digests_kept": 7,
  "chips_written": 8,
  "chips_kept": 5,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 0,
   "figure": 1,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 1,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 0,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 10
  },
  "new_noun_tokens_checked": [
   "0.010",
   "0.030",
   "0.05",
   "0.5618",
   "0.9098",
   "0.9758",
   "1",
   "121",
   "20",
   "27.7",
   "3",
   "300.437",
   "327.218",
   "34",
   "409",
   "457",
   "466.972",
   "470",
   "5",
   "500",
   "506",
   "512-token",
   "523.989",
   "6000",
   "67",
   "8",
   "8.38-billion-parameter",
   "8b",
   "95th",
   "96",
   "A2",
   "A3",
   "A6",
   "A7",
   "B",
   "BF16",
   "Exploratory",
   "FP32",
   "LongMemEval",
   "MRR",
   "McNemar's",
   "Mistral",
   "Page",
   "Part",
   "R1",
   "R2",
   "R4",
   "R5",
   "Reranker",
   "RuleSage",
   "Seat",
   "bf16",
   "cross-encoder/ms-marco-minilm-l6-v2",
   "fp32",
   "nemotron-3.5-rerank-8b-bf16",
   "nvidia/nemotron-3.5-rerank-8b-bf16",
   "p50",
   "p95"
  ],
  "new_noun_tokens_withheld": 0,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "digest",
    "reason": "redaction",
    "item": null,
    "item_chars": 254,
    "withheld": "a string carrying a fenced literal",
    "detail": ""
   },
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the median time for the rerank 8b, whole pool arm is 523.989 ms",
    "detail": "terminal token not verbatim in the cited span"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 37,
    "text_sha": "880914986231",
    "figure": "16",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "its figure is matched inside the model repository's name, not in the sentence that says what the weights are held in, and the line leaves out that the head computes its score in FP32, which the page says beside it",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: its figure is matched inside the model repository's name, not in the sentence that says what the weights are held in, and the line leaves out that the head computes its score in FP32, which the page says beside it. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 80,
    "text_sha": "3bd7d7e79b8d",
    "figure": "27.7",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "it welds two sentences into one claim: the page says the model loaded once, for all three phases, and says separately which library's implementation it loaded through; the line drops the once-for-all-three-phases condition its figure carries",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: it welds two sentences into one claim: the page says the model loaded once, for all three phases, and says separately which library's implementation it loaded through; the line drops the once-for-all-three-phases condition its figure carries. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 148,
    "text_sha": "5a2ef24acb6f",
    "section": "the-rules-desk",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "it calls the 34 labelled cases ones where the first pick is a cited source, which is one of the table's three readings and not what the cases are, and it leaves out that no gate reads this table",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: it calls the 34 labelled cases ones where the first pick is a cited source, which is one of the table's three readings and not what the cases are, and it leaves out that no gate reads this table. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 155,
    "text_sha": "f105d016d96a",
    "section": "what-one-ask-costs",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "it gives the medians without the one card, at a 420 W cap, they were read on, which the page says speaks for no other card, and without the 30 timed asks or the budget's own numbers",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: it gives the medians without the one card, at a 420 W cap, they were read on, which the page says speaks for no other card, and without the 30 timed asks or the budget's own numbers. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 149,
    "text_sha": "6718eec1e189",
    "section": "longmemeval",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "it gives the exploratory lead without what the page sets beside it every time: a context line outside the gated family, and the registered caveat that the model's training data may overlap the benchmark, unchecked",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: it gives the exploratory lead without what the page sets beside it every time: a context line outside the gated family, and the registered caveat that the model's training data may overlap the benchmark, unchecked. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 149,
    "text_sha": "3132aa26b9f5",
    "section": "what-this-cannot-say",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "the page says it cannot speak for this model run in FP32; the digest turns that into a comparison with other models, and it leaves out the section's main limits, the two unread LongMemEval contrasts and the unchecked training overlap",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: the page says it cannot speak for this model run in FP32; the digest turns that into a comparison with other models, and it leaves out the section's main limits, the two unread LongMemEval contrasts and the unchecked training overlap. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 176,
    "text_sha": "ca523edef20d",
    "section": "what-to-take-with-you",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "it calls the MRR high without the page's condition that no registered rule reads it as a win, and gives the LongMemEval lead without its context-line and training-overlap caveats",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: it calls the MRR high without the page's condition that no registered rule reads it as a win, and gives the LongMemEval lead without its context-line and training-overlap caveats. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 267,
    "text_sha": "ad3cba1bc381",
    "chip_id": "c-9a289a56",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "the latencies are given without the one card, at a 420 W cap, they were read on, which the page says speaks for no other card",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: the latencies are given without the one card, at a 420 W cap, they were read on, which the page says speaks for no other card. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 275,
    "text_sha": "948cea6c8d1c",
    "chip_id": "c-abee232e",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "the answer drops the registered caveat the page sets beside the 457 every time: the model's training data may overlap the benchmark, and that was not checked",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: the answer drops the registered caveat the page sets beside the 457 every time: the model's training data may overlap the benchmark, and that was not checked. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 268,
    "text_sha": "6a57ae46738b",
    "chip_id": "c-df814588",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-26",
    "why": "the page says the LongMemEval number was not checked against the model's training data; the answer turns that into results under different training data, which the page does not discuss",
    "detail": "removed from this pack by hand on 2026-10-26 (UTC), after the writer ran and after the gates had counted: the page says the LongMemEval number was not checked against the model's training data; the answer turns that into results under different training data, which the page does not discuss. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-10-26T17:24:50Z"
}