Nemotron 3.5 Rerank 8B, measured on our own workloads

If you're new here: strata→signal is a small workshop that builds things on its own machines and writes up what it measures. Every measured figure here was read from the bench's own files on 2026-10-26 (UTC) by a scorer written from the standard library alone, and a second scorer written apart from it recomputed 449 of them, all equal; the model's size and dates come from its registration and the pages linked beside them, and every verdict word follows a rule written down before the model was called.

The words this page leans on

• Reranker. A model that re-reads a shortlist of passages beside the question and orders it better. A cross-encoder is one that reads the question and a passage as a pair and returns one score. RuleSage's reranker, MiniLM, is one; so is the model on this page.
• Seat. One model held in one card's memory, answering for one of our products.
• Page recall@8. The share of a question's labelled pages that turn up among an arm's eight passages, averaged over questions: a question with three labelled pages, two of them in the eight, scores 0.67. RuleSage's primary measure. Page recall@1 is the same share for the first passage alone.
• MRR. Mean reciprocal rank: 1 if the first labelled page is ranked first, 1/2 if second, and so on, averaged.
• every@5 and any@5. On LongMemEval, each question names the chat sessions that hold its evidence. every@5 counts the questions where every one of those sessions is in an arm's top five; any@5 counts those where at least one is.
• McNemar's exact test. A paired test for two arms scored on the same questions. It looks only at the questions the arms disagree on (one hit, the other missed) and asks how surprising the split would be if the two were really even.
• p50 and p95. The median and the 95th percentile of a set of timings.
• BF16. A 16-bit number format for a model's weights, half the size of the full 32-bit format (FP32) the 568-million-parameter cross-encoder ran in on both benchmarks.
• Exploratory. The registered word for every reading on this page. The rule prints "exploratory lead", "exploratory trail" or "exploratory: no material difference", and never "beats" or "equal".

The model, and the question

The model is nvidia/Nemotron-3.5-Rerank-8B-BF16, an 8.38-billion-parameter cross-encoder, pinned at one revision of its repository. Its NOTICE names its base model, Ministral-3-8B-Instruct-2512-BF16, as Apache-2.0; its LICENSE reads "All binary model files and source code files are licensed under the OpenMDW-1.1 License". The body is a Mistral 3 model with a scoring head on top; every parameter, the head's included, is held in BF16, the head computes its score in FP32, and no code from the repository ran: the model loaded through the transformers library's own implementation.

It was picked by the bench's queue as the newest reranker release in its licence class, before any call of it on either question set, and its settings (the batch sizes, the 512-token length, the widen threshold on LongMemEval) are the registered values of the arms it is compared with, not tuned. Two questions were written down on 2026-10-22 and registered at 05:32:34Z on 2026-10-23, in one amendment appended to both of the bench's registrations, with every reading marked exploratory:

1. On RuleSage's own retrieval, does it order the shortlist better than bge-reranker-v2-m3, the 568-million-parameter cross-encoder we measured on 2026-10-14, and than MiniLM, the seat's own model, and does it fit the latency budget? The setup, the labels, the rule and the earlier arms are on OpenJev as a reranker for RuleSage; this page runs one more arm through them.
2. On LongMemEval, inside a retrieval pipeline we built in the shape of an open-source memory tool, does it lift a hybrid search's ranking of the sessions that hold a question's evidence?

Both halves ran in one night, 2026-10-26 from 07:34:45Z to 08:25:49Z, on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) at a 420 W cap, the model loaded once, in 27.7 s, for all three phases. The RuleSage quality pass took 138.3 s including that load and the repeat pass; the latency leg 32.6 s; the LongMemEval pass 2,887.8 s, 5.736 s a question. One model process held the card (its memory grew by 16,866 MiB at the load, against a floor of 15,000), with no other process on it during any phase.

The two workloads, by counts

RuleSage, Part A is the retrieval set of the earlier page: 121 real questions readers typed into the app, each labelled with the pages that other answered asks of the same question cited, and for each the shortlist of up to 80 passages RuleSage's three searches fused. The questions are readers' words and stay on this workshop's machines; this page prints counts. Part B is 34 labelled cases of the rules desk's published exam, each the eight passages RuleSage handed its answering model, labelled by the house answer's own citations.

The earlier arms are read beside the new one, from the same files: R0, no reranker, the fused list as it stands; R1, MiniLM, the seat; R2, OpenJev 27B read as a yes/no judge; R4, bge-reranker-v2-m3. The new arm is R5. Every reranking arm scored the same 6,128 (question, passage) pairs; R5's texts were checked against the earlier arms' before any score was read, 6,128 of 6,128 equal.

One bias was named before the run, and it binds this page as it bound the last. A Part A label is a page an answering model cited after seeing MiniLM's top eight, so the labels lean towards MiniLM, and a new reranker's "exploratory lead" over MiniLM would be read against that lean. Against bge-reranker-v2-m3, the registration added, the lean favours whichever of the two ranks more like MiniLM, a direction not known in advance. Part B is not free of bias either: its eight are MiniLM's picks.

LongMemEval (Wu et al., 2024, arXiv:2410.10813), the S variant in its cleaned release, is 500 questions, each asked over its own history of chat sessions and each naming the one to six sessions that hold its evidence. 470 are answerable; 30 are abstention questions, which every@5 leaves out and any@5 counts, as the benchmark's own published figures do. The pipeline is ours, built from local pieces in the shape of deep-recall, an open-source memory tool: A2 is a hybrid search that fuses a word-match search with a meaning search (vectors from nomic-embed-text) by reciprocal rank, and it was read and registered on 2026-10-17. A reranked arm takes A2's top 20 sessions plus a small leg of sessions matched on the question's rare terms, cuts each into windows of 300 words, keeps each session's three most question-like windows, scores them, and ranks sessions by their best window. If the best score is below 0.5, the pool widens once to A2's 150-deep list, in practice the whole history. A3 is that procedure with MiniLM, on a laptop's CPU; A6 is it with bge-reranker-v2-m3; A7, the new arm, is it with Nemotron 3.5 Rerank 8B, over the very same A2 lists A6 reranked, checked session for session before any contrast was read.

The rules, written before the model was called

RuleSage, paired per question over the 121 (the earlier page's G-4, in its exploratory form). "Exploratory lead" needs all five: a mean gain in page recall@8 of at least +0.030; a sign-flip permutation test giving p below 0.05 (10,000 resamples, a fixed seed); the gain positive on both halves of the set; MRR not falling; document recall@8 falling by no more than 0.010. "Exploratory trail" is a mean loss of at least 0.030 with p below 0.05. Anything else prints "exploratory: no material difference", as is. The budget rule is the earlier page's: p50 at most 1,500 ms and p95 at most 3,000 ms, read per card.

LongMemEval, on every@5 over the 470. "Exploratory lead" needs a gain of at least +10 questions and McNemar's exact two-sided p at or below 0.05, unadjusted; the mirror prints "exploratory trail"; anything else "exploratory: no material difference". On any@5 over all 500, a secondary line, the margin is +5. Three contrasts were registered: A7 vs A6, two cross-encoders over the same sessions, windows and widen rule, the model contrast (read on the no-widen line, each arm ranked inside the same initial pool), read only after A6's own reading is registered; A7 vs A3, once A3 has scored; and A7 vs A2, the search it reranks, as context. All three sit outside the bench's gated family of four contrasts, which was fixed on 2026-10-09, before this model's release.

What changed besides the model, said in the registration and printed beside every line: the precision (BF16, with the head's score computed in FP32, where R4 and A6 ran FP32); the runtime (transformers 5.20.0 and sentence-transformers 6.2.0, where R4 and A6 ran 5.15.0 and 5.7.0); the input (the checkpoint's own query: / passage: rendering, where the earlier arms read the bare pair); and the cut at 512 tokens, which falls elsewhere under another tokenizer. Not controlled: the model's training data may overlap public question and retrieval sets, and neither LongMemEval nor RuleSage's labels were checked against it.

Did the readout hold? A second pass over the first ten questions returned the same eight on 10 of 10 with no logit moved (max |Δ| 0.0). Every parameter loaded as BF16 with the head computing in FP32, no weight was missing, and the model's rendering of a pair matched its own template on 42 of 42 ids.

Did it pick a better eight for RuleSage?

Each cell is the mean over 121 questions, with its 95 per cent half-width on the first two rows. The R0, R1, R2 and R4 columns equal the earlier reading's stored values to 1e-12, re-derived here from the same files.

The paired contrasts, each condition against its rule:

Against the 568-million-parameter cross-encoder, the new model's eight held more of a question's labelled pages on six questions and fewer on twelve; 103 of 121 were unchanged. The mean moved by −0.0079: the wrong way for a lead, and far short of the −0.030 with p below 0.05 that would read as a trail, so the rule prints "exploratory: no material difference". The same words print against MiniLM, against no reranker and against OpenJev.

Three things in the tables are worth naming, and none is a verdict. The new model ranked the first labelled page higher than any arm, MRR 0.9758, and its first passage held a larger share of the labelled pages, page recall@1 0.5618, the highest of the five; the registration reads MRR only as a condition on a lead in recall@8, and the labels were drawn through MiniLM's own top eight. The last row of the first table is the bias made visible: of the new model's 944 kept entries, 465 carry no label, and 450 of those sit in a document that does. A claim that those unlabelled pages answer would need a blind labelling pass registered first. And the 512-token cut fell on 6 of the 6,128 pairs, all in five of the real questions, one of them inside the model's top eight; the longest pair ran 761 tokens, the median 164. The 568-million-parameter cross-encoder's rows carry no token count, so its cut count is not known.

The rules desk, where every arm sees the same eight

No gate reads this table, by design: the eight are MiniLM's picks and the labels are an answering model's citations, so it can lean either way. The new model has no "not in the book" reading: a relevance score has no registered "no", as was true of the 568-million-parameter cross-encoder.

What one ask costs

Thirty real questions, drawn by a fixed seed, the same thirty the earlier page timed, were asked as the app would ask: every candidate of one question sent at once to the one loaded model, the wall clock running from the send to the reply.

The new model fits the budget at both depths, with room. It runs 4.00× the 568-million-parameter cross-encoder's median over the whole pool (4.03× at the 95th percentile), and 4.12× and 4.05× over the top 20. Read per pair over the whole quality pass, 100,885.6 ms of model time over 6,128 pairs, it took 16.463 ms a pair against 3.961 ms for the smaller model read the same way: 4.16×. The latency leg is deterministic: its whole-pool asks re-scored the same pairs with logits equal to the quality pass's on 31 of 31 asks. One card, at a 420 W cap, the stage alone on the card's own machine; a reading on one card says nothing about another, about the seat's own hardware, or about the network hop the live path adds.

LongMemEval: the exploratory lead, and what sits beside it

Hits for each arm, with the misses in brackets; the any@5 (500) column counts all 500 questions, the others the 470 answerable:

The contrasts, each in the registered form, exact McNemar, two-sided, unadjusted:

The new model put every evidence session in the top five on 457 of 470 questions, where the hybrid it reranked had read 409: 51 questions only it hits, 3 only the hybrid hits. By the registered words that is an "exploratory lead". Three things stand beside the number.

It is a context line, not a gated win. A7 vs A2 was registered "as context", outside the bench's gated family of contrasts. The registered model contrast is A7 vs A6, the same procedure with the 568-million-parameter cross-encoder, and it was not read (the next section says why). And the registration's caveat applies to this number first of all: the model's training data may overlap public question and retrieval sets, and LongMemEval's labels were not checked against it.

It compares a procedure as much as a model. The model's raw scores (logits) are all negative: no window anywhere scored a logit above −0.585 (a sigmoid of 0.358), under the 0.5 widen value, which the registration foresaw from the vendor's own example scores. So the pool widened on all 500 questions, and the model scored the three most question-like windows of every session in every history: 67,506 windows in all, exactly the registered whole-history bound (29,363 in the initial pools and 38,143 added by widening), 6,117 of them (9.1 %) cut at 512 tokens. The hybrid has no widen step.

The descriptive no-widen line keeps the model inside the initial pool, A2's top 20 plus the rare-term leg (non-empty on 119 questions), ranked by the same scores: it reads +44, with 46 questions only it hits, and on every one of those 46 every evidence session was already in A2's top 20. No registered rule gives that line a word. Widening itself changed the outcome on eight questions, 6 up and 2 down, a net +4 (p 0.289), also descriptive.

A run count, never a ratio of scores. The earlier cross-encoder's pass, in an earlier window on the same card, scored 47,853 windows and widened on 247 questions in 500.9 s; this one scored 67,506 and widened on 500 in 2,887.8 s. The two ran on the same card under different runtimes and precisions, and times are never compared as a verdict.

By question type, every@5 over the 470:

On any@5 the new model read 500 of 500, every type included; the 13 questions it misses on every@5 are ones where some, not all, of the evidence sessions made the top five.

Two contrasts were not read

A7 vs A6 is NOT READ. It is the model contrast, and the registration reads it only after the 568-million-parameter cross-encoder's own LongMemEval reading is registered; none is, in any results file or ledger row as of 2026-10-26. The procedure is ready for it, from the session ids alone: both arms opened the same initial pool and the same initial windows on 500 of 500 questions; on the 247 questions both arms widened, the widened windows are the same on 247 of 247; 253 questions were widened by one arm only, the new one. No A6 score was read to say that.

A7 vs A3 is NOT READ. The MiniLM arm on LongMemEval has no scored call yet.

The five predictions, written in the registration before any call, each given a confidence and no direction, and read by no gate:

The budget prediction took the smaller model's times times fifteen; the measured ratio is about four. This reading does not separate the causes: the precision, the attention path and the runtime all changed with the model.

What changes at RuleSage: nothing

This was a bench, not a seat change, and it moved nothing. RuleSage's reranker stays cross-encoder/ms-marco-MiniLM-L6-v2, on the card Four homes for one reranker left it on. The weights of the model on this page are under the OpenMDW-1.1 licence; by the registration, a seat moves only by a measured adoption after a read of those terms, and this bench is not that. The bench copied the weights to a staging folder and to the card's machine, redistributed nothing, and removed the card's copy and its runtime environment by name when the night ended.

What this cannot say

Whether answers get better: no answering model was called on either workload. Whether this model orders LongMemEval's sessions better than the 568-million-parameter cross-encoder or than MiniLM, the two contrasts the registration wrote for that question: neither was read. Whether the LongMemEval number would survive a check of the model's training data against the benchmark: not checked. Anything about this model in FP32, under another server, at another revision, on another card, on the seat's own hardware, or on image documents (text only here). Whether the 512-token cut, which fell on 6 of RuleSage's 6,128 pairs and 9.1 % of LongMemEval's windows, cost it anything: the cut is registered, its count printed, and nothing more is read from it.

What to take with you

• On RuleSage's own questions, an 8-billion-parameter reranker read "exploratory: no material difference" against every arm, page recall@8 0.9098 against 0.9178 for bge-reranker-v2-m3 and 0.9271 for MiniLM over 121 questions, with the labels leaning towards MiniLM by construction, and against bge-reranker-v2-m3 in a direction not known in advance.
• It ranked the first labelled page highest of the five arms, MRR 0.9758, and no registered rule reads that as a win.
• It costs about four times the smaller cross-encoder per pair, 16.463 ms against 3.961 ms on the same card, and still fits the budget: 524.0 ms at the median over the whole shortlist against 1,500 ms allowed.
• On LongMemEval it read an "exploratory lead" of 457 to 409 on every@5 over the search it reranked, a context line outside the gated family, with a training-overlap caveat, every question widened to the whole history, and the registered model contrast not yet read.
• Nothing at RuleSage changes. This was a bench, and the seat stays MiniLM.

How to check our work, and see it live

• Ask RuleSage a rules question at rulesage.strata2signal.com: the shortlist it re-reads is MiniLM's, as it was before this page.
• The registration. The amendment binding this model was written on 2026-10-22 from 22:08:10Z, before any scored call, appended byte for byte to both of the bench's registrations, and frozen by hash; its predictions are the table above, as written. The bench's first registration and its rule are described on OpenJev as a reranker for RuleSage.
• Two scorers. The registered scorer read every figure from the run's own files after checking each file's gates; a second scorer, written apart from it and importing none of its code, recomputed 449 values from the raw files, 449 equal and 0 different, and a check of the reading folded three must-fix findings and six smaller ones, all on words and records, moving no number.
• No data kit ships with this page yet. RuleSage's Part A holds readers' typed words and stays home; whether the registration's text, the LongMemEval rows and the aggregate files ship as one kit is a later ruling, and a dated note below will say.

The rest of the seminar

• OpenJev as a reranker for RuleSage: the bench this page adds an arm to, with the question set, the rule and the first two challengers.
• Four homes for one reranker: how MiniLM came to sit on the card it answers from, and what it costs there.
• Three librarians: the three searches whose fused list every reranker on this page re-orders.
• Reading the answer: the decision model that was the first challenger, and how its yes/no is read.

Who ran this, and thanks

NVIDIA released the Nemotron 3.5 Rerank 8B weights under the OpenMDW-1.1 licence, naming Ministral-3-8B-Instruct-2512-BF16 (Apache-2.0) as the base model in the repository's NOTICE; the Beijing Academy of Artificial Intelligence released bge-reranker-v2-m3 under Apache 2.0; the cross-encoder/ms-marco-MiniLM-L6-v2 weights are the sentence-transformers project's; Nomic's nomic-embed-text supplied the hybrid search's vectors. LongMemEval is the work of Wu et al. (2024), its cleaned release under MIT; the retrieval pipeline follows the shape of deep-recall, an open-source memory tool under MIT. PyTorch, transformers and sentence-transformers served the cross-encoders, and Python's standard library alone wrote and checked every figure. A small human team asked the question, approved the registration before the model was called, and approved this page for publication; a fleet of AI agents wrote the amendment, the kit and the scorers, ran and checked each stage, and wrote this page under that team's rulings.

Sources

Read 2026-10-26 (UTC) unless dated otherwise.

• Our records: the bench's two pre-registrations (2026-10-09, frozen, with their dated amendments; the amendment for this model written 2026-10-22); the registered reading of the night of 2026-10-26, its two scorers' outputs and the check of it; the hybrid search's registered LongMemEval reading of 2026-10-17; the earlier rerankers' readings of 2026-10-10 and 2026-10-14, as OpenJev as a reranker for RuleSage prints them.
• The model, its card, LICENSE and NOTICE, read at the pinned revision on 2026-10-22: https://huggingface.co/nvidia/Nemotron-3.5-Rerank-8B-BF16
• LongMemEval: Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory", arXiv, 2024. https://arxiv.org/abs/2410.10813 · the cleaned release, https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned
• deep-recall, the memory tool whose pipeline shape the LongMemEval half follows (MIT). https://github.com/turlockmike/deep-recall
• Licences of the earlier arms, each read from the model's own card: bge-reranker-v2-m3 (Apache 2.0, read 2026-10-10); OpenJev (CC BY-NC 4.0, read 2026-10-02).
• This workshop's released pages, linked where they are named above.

Readings 2026-10-26, 07:34:45Z to 08:25:49Z, on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) at a 420 W cap: 6,128 RuleSage pairs, 121 labelled questions, 34 exam cases, 60 timed asks and 2 warm-ups, and 500 LongMemEval questions over 67,506 windows. One registration amendment, every reading exploratory.

Corrections and later measurements will be added below, each dated (UTC), with a window at both ends where one applies and saying in plain words what it counts.