# Reciprocal rank fusion: the data kit

This folder holds the records behind the LongMemEval figures in *Reciprocal rank fusion*
(https://research.strata2signal.com/reciprocal-rank-fusion/). LongMemEval (Wu et al., 2024, arXiv:2410.10813) is a
public benchmark of 500 questions asked over long chat histories; in its S version each question comes with 38 to 62
past chat sessions, and the dataset labels the sessions that hold the answer. Each list here ranks one question's
sessions, and the page counts **every@5**: the answerable questions (470 of the 500) whose every answer-holding
session is in a list's top 5.

Two folders, one for each run:

- `longmemeval-2026-10/`: the October bench, registered before its first score. **A0** is BM25 (keyword search)
  alone, run 2026-10-09. **A1** is nomic-embed-text (search by meaning) alone, and **A2** their reciprocal rank
  fusion (RRF) at k = 60, with weight 1.0 on nomic-embed-text and 0.5 on BM25: the fusion with a test registered
  against A0. Both ran 2026-10-17. **A5** is the RRF of BM25 and BGE-small (`BAAI/bge-small-en-v1.5`) at the same k
  and weights, as the open-source memory tool deep-recall (https://github.com/turlockmike/deep-recall) ships it, run
  2026-10-10 with no registered test.
- `learned-sparse-2026-11-07/`: the second registered run, on 2026-11-07. **S1** is a learned-sparse list from
  opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill (Apache-2.0). Its fusions are **F1** (S1 and
  BM25), **F2** (S1 and nomic-embed-text) and **F3** (all three), each at k = 60 with weight 1.0 on each model's list
  and 0.5 on BM25. Beside them: three exploratory fusions the registration named and never gated, with S1 at
  weight 0.5 (**F1h**, **F2h**, **F3h**; not on the page), a second, independently written scorer's recount of each
  list, and the five comparisons registered before the run.

Inside each folder the files keep the bench's own layout, so every path a record prints names a file here. Every
file was copied from the bench's own record and rewritten only by the named rules in `provenance.json`. `index.json`
lists every file but itself, `provenance.json` and this folder's index page, each with its size and sha256.
`SHA256SUMS` carries the same digests for the 34 record files, so `sha256sum -c SHA256SUMS`, run in this folder,
checks them.

## Where each figure on the page comes from

`L` is `longmemeval-2026-10/` and `N` is `learned-sparse-2026-11-07/full-s1/`.

| On the page | File here | Fields |
|---|---|---|
| BM25 alone: 400 of 470 (every@5); 485 of 500 (any@5) | `L/a0/run/rows.ndjson` | `search`, `gold`, `abstention`, `score` |
| nomic-embed-text alone: 403 of 470 | `L/w7/rows/a1/legs.ndjson`, `L/w7/score-registered-w7.json` | the first 10 of `leg50`; `A1` |
| The fusion with a registered test: 409 of 470; +9 (23 questions only the fusion hit, 14 only BM25), p 0.1877, no material difference | `L/w7/rows/a2/a2.ndjson`, `L/w7/score-registered-w7.json`, `L/w7/w7-per-question.tsv` | `top10`, `score`; `G-WIN A2 vs A0 (every@5, 470)`; the `A2:all@5` and `A0:all@5` columns |
| The fusion deep-recall ships, untested: 417 of 470; 487 of 500 (any@5) | `L/a5/run/rows.ndjson` | `top10`, `score` |
| The table by question type, and the BGE-small fusion's net gain over BM25 alone by type | the three row files above | `type`, `abstention`, `score.all@5` |
| The learned-sparse list alone: 416 of 470 | `N/run/rows.ndjson`, `N/score/rescore-s1.json` | `search`, `score`; `counts.answerable.all@5` |
| The three fusions: 417, 422 and 425 of 470 | `N/fuse/F1.rows.ndjson`, `F2`, `F3`; `N/score/rescore-F1.json`, `-F2`, `-F3`; `N/fuse/fuse.json` | `top10`, `score`; `counts.answerable.all@5`; `blends.F1.every@5` and the same for F2 and F3 |
| The five registered comparisons: the two counts, the gain, the questions only one side hit, p unadjusted and Holm-adjusted, the registered words | `N/score/family.json` | `primary[]`: `arm_hits`, `ref_hits`, `delta`, `b_arm_only`, `c_ref_only`, `p`, `p_holm`, `words` |
| The same five, recomputed with the registered harness's own test: equal | `N/score/reading.json` | `family_crosscheck` |
| F2's exact ties across ranks 5 and 6: 10 of the 500 questions; the order reversed changed no question's every@5 outcome | `N/fuse/fuse.json`, `N/fuse/F2.rows.ndjson`, `N/score/reading.json` | `blends.F2.questions_with_an_exact_tie_across_ranks_5_6`; `tie_at_5`; `f2_ties_reversed_exploratory` |
| The integrity checks: the fusion code rebuilt A2's stored top 10 on 500 of 500 questions; each recount found no problem | `N/fuse/fuse.json`, `N/score/rescore-*.json` | `ctrl_a2_rebuild`; `g0_problems`, `row_diffs`, `summary_diffs` |
| Texts longer than 510 tokens: 54,802 of 125,667, encoded in 510-token windows | `N/score/reading.json`, `N/run/run.json` | `s1_run.texts_windowed`, `s1_run.texts`; `settings.window` |
| The model, its revision, a CPU run | `N/run/run.json` | `settings.model_id`, `settings.revision`, `settings.device` |
| naver/splade-v3, skipped before any download | `learned-sparse-2026-11-07/s2-skip.json` | `gated`, `license`, `has_safetensors`, `pickled_weights`, `verdict` |

## How the figures were counted

- **every@5 and any@5.** every@5 (the files call it `all@5`) counts an answerable question, one of the 470, when
  every session in its `gold` list is in the list's top 5. any@5 counts any of the 500, the 30 abstention questions
  included, when at least one is. `any@1` and `all@10` are in the files too; the page does not use them.
- **The scored list.** A0's and S1's is `search` (their top 10), A1's is the first 10 of `leg50`, and A2's, A5's and
  each fusion's is `top10`. Every count above can be recounted from the row files alone; the stored `score` fields
  and the recount files say the same.
- **Blind names.** Sessions are named by blind file names (`s`, ten hex digits, `.md`), never by the dataset's own
  session ids, in every list and in `gold`. Questions keep the dataset's own ids.
- **The fusion.** Reciprocal rank fusion gives each session, from each list it is in, that list's weight divided by
  (k + its rank there), and adds the points: at weight 1.0, the first place is worth 1/61. k is 60 throughout. An
  exact tie goes to the list named first in the fusion's `spec` in `fuse.json` (for F2, the learned-sparse list).
  Each of the six fusions here, re-fused by that rule from S1's and A0's `ranked` lists and A1's `leg50`, gives its
  stored top 10 on all 500 questions.
- **The comparisons.** Each counts the questions only one side hit and reads McNemar's exact two-sided test on those
  two counts. A win needed a gain of at least 10 questions and p at or below 0.05 after a Holm adjustment: across
  the four comparisons the October bench registered, and across the learned-sparse run's five. The words each
  comparison would print were fixed before the run. Each reading is the registered one and stands for this set of
  questions; a second test on the same 500 questions would not be a replication.

## What was rewritten, by rule

`provenance.json` names every rule and counts every firing, file by file:

- **`record-path-to-kit-path`.** The learned-sparse run's scorer, fuser and reading printed the absolute paths of
  the files they read, into a private folder. Each path now names the same file in this kit: that run's folder
  became `learned-sparse-2026-11-07/` and the October bench's `longmemeval-2026-10/`. The rest of each path is
  unchanged.
- **`box-name-to-role`.** A1's lock named the machine its embedding calls were sent to, in its `route` field; it
  reads `desktop`, the role that machine played. The card it names is as the lock recorded it.
- **`digest-withheld`.** Where `family.json` and `reading.json` printed the sha256 of a file the path rule rewrote
  (the seven recounts, `family.json` and `fuse.json`, as the run wrote them), the digest reads `withheld-2026-09-29`:
  beside the rewritten copy, it would check a guess at the private path. Every other digest these files print, of
  the row files, S1's lock and summary, the control and the October files, checks against the copy here.

Nothing else moved: no row, list, score, count, p-value or registered word.

## What is not here, and why

- **The registrations and the results records.** The October bench's registration, the learned-sparse run's, and
  the bench's internal results records are not here: they name machines, people and private paths. The page states
  in words each rule it uses, and each lock here pins the harness by sha256.
- **The question text.** No question, answer or chat session from LongMemEval is in any file here. The dataset is
  public: the cleaned release, https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned, file
  `longmemeval_s_cleaned.json`, sha256 `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`, as
  every lock here pins it.
- **The embedding vectors.** A1's stored vectors (1,000 files of the dataset's text, embedded) are not here; no
  figure on the page reads them.
- **The rest of the October bench.** The OpenJev reranker arm (cut at 44 of 50 questions, never scored), the
  cross-encoder rerank (432 of 470 on the page, exploratory), the run of deep-recall's own code (its top 10 identical
  on 499 of 500 questions, and the two fused scores at rank 10) and the logs are not here; those page figures rest on
  the bench's records, which are not published.
- **The rest of the learned-sparse run.** The scorer's self-attack, a per-question table of S1's outcomes (the same
  outcomes are in `rescore-s1.json`), the smoke runs, the control run's own folder and the step stamps are not here;
  no record in this kit prints a path to any of them.

Every time in this folder is UTC. Licence: CC BY 4.0 for this kit's records and this README; name the source and
link to it. The question ids, type labels, abstention flags and answer-holding session lists are LongMemEval-S's own
labels (Wu et al., 2024; the cleaned release is under the MIT licence, read from its own card on 2026-11-09), carried
as identifiers.