{
  "schema": "s2s-bench-v1",
  "kind": "data-kit-index",
  "exhibit": "reciprocal-rank-fusion",
  "exhibit_title": "Reciprocal rank fusion",
  "exhibit_url": "https://research.strata2signal.com/reciprocal-rank-fusion/",
  "published_utc": "2026-11-09",
  "licence": "CC BY 4.0",
  "licence_note": "CC BY 4.0 covers this kit's records and README. The question ids, type labels, abstention flags and answer-holding session lists are the LongMemEval-S dataset's own labels (Wu et al., 2024, arXiv:2410.10813; the cleaned release, https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned, under the MIT licence, read from its own card on 2026-11-09), carried as identifiers, with every session named by a blind file name. No question, answer or chat text is here.",
  "attribution": "strata→signal research, research.strata2signal.com",
  "contact": "hello@strata2signal.com",
  "run_id": "longmemeval-s-retrieval-2026-10-09-to-2026-11-07",
  "run_window_utc": {
    "a0_bm25_started_utc": "2026-10-09T23:22:41Z",
    "a5_bge_small_fusion_started_utc": "2026-10-10T13:25:40Z",
    "a1_nomic_embed_text_started_utc": "2026-10-17T07:34:58Z",
    "a2_nomic_embed_text_fusion_started_utc": "2026-10-17T07:57:04Z",
    "a1_a2_registered_scorer_utc": "2026-10-17T10:15:42Z",
    "learned_sparse_registration_frozen_utc": "2026-11-07T03:56:44Z",
    "s1_learned_sparse_utc": [
      "2026-11-07T03:57:08Z",
      "2026-11-07T06:25:43Z"
    ],
    "note": "Each start is its lock's own `started_utc`, the registered scorer's time its output's `utc`, and the learned-sparse run's end its summary's `finished_utc`. That run's registration was frozen 24 seconds before its first question, by the registration's own stamp (the registration is not in this kit). The fusions F1, F2 and F3 were computed from the stored lists after S1 finished."
  },
  "card_class": "A1's embedding calls ran on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB); every other list ran on a CPU or was fused from stored lists",
  "counting_rules": "every@5 (the files' `all@5`): an answerable question, 470 of the 500, counts when every answer-holding session (`gold`) is in the list's top 5. any@5: any of the 500 counts when at least one is. A0's and S1's scored list is `search`, A1's the first 10 of `leg50`, and A2's, A5's and each fusion's `top10`. A comparison counts the questions only one side hit and reads McNemar's exact two-sided test on those two counts: a win needed a gain of at least 10 and p at or below 0.05 after a Holm adjustment, across four registered comparisons in October and five in the learned-sparse run.",
  "sanitisation": "Machine readers start at `index.json`, which lists every file in this kit but three, each with its byte count and its sha256; the three it leaves out are itself, `provenance.json` and this directory's index page. `provenance.json` is the other half: it names every rule applied to every file, with the count of times each one fired, and says whether any rule touched each file.",
  "files_note": "`index.json` lists every file in this kit but itself, `provenance.json` and this directory's index page, each with its sha256; `provenance.json` names every rule and how many times it fired on each file. `SHA256SUMS` carries the same digests for the 34 record files, in the form `sha256sum -c` reads.",
  "file_count": 39,
  "files": [
    {
      "file": "README.md",
      "bytes": 10097,
      "sha256": "e76ceba1010534efaa406c460b2ac0f81ba51446144a359b4e9d428571a0f1bb",
      "role": "The kit's guide: the arms in the page's words, which file answers which figure on the page, the counting rules, what was rewritten and what was left out, and why."
    },
    {
      "file": "SHA256SUMS",
      "bytes": 3889,
      "sha256": "216044e9fe6d1f37931165f7ce5cf7926f7818016ed53e128e4b3f5b5b885094",
      "role": "The sha256 of each of the 34 record files, as served, in the form `sha256sum -c SHA256SUMS` reads from this folder."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/control-card.json",
      "bytes": 1386,
      "sha256": "b5356f0e5ad05e7fa216e91db09b9dab93f3d4762ea05ea6291f520d009ef0d0",
      "role": "The activation's positive control, run in the scored run's environment before the first question: the model card's own example scored with the activation the run used (`card-v3`) and with a second one (`st-log1p`), beside the values the card prints (`card`), with the per-word weights each gave; and the sha256 of each model file."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F1.rows.ndjson",
      "bytes": 278919,
      "sha256": "becbdca73b15fd048e1da267aa57e98d9bcbc2816dc321e17c1343f62ccc0f80",
      "role": "F1, the RRF of S1 (weight 1.0) and BM25 (0.5): one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F1h.rows.ndjson",
      "bytes": 279243,
      "sha256": "edf226e9c74bd47f2f0e80f43f8e8a971b80e29331db47e1248cb04b2829d9d2",
      "role": "F1h, an exploratory fusion the registration named and never gated (S1 at 0.5, in BM25's seat): the RRF of S1 (0.5) and BM25 (0.5). Not on the page: one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F2.rows.ndjson",
      "bytes": 279067,
      "sha256": "65595bad63ee4c831950539e264af93d7de2f9580cfff3782617bbc24159434e",
      "role": "F2, the RRF of S1 (1.0) and nomic-embed-text (1.0), S1 named first: one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F2h.rows.ndjson",
      "bytes": 278858,
      "sha256": "380b7b203b21d9848808bad92265f742020c9283d689342ac0133a38bc565941",
      "role": "F2h, exploratory and never gated: the RRF of nomic-embed-text (1.0) and S1 (0.5), nomic-embed-text named first. Not on the page: one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F3.rows.ndjson",
      "bytes": 279115,
      "sha256": "abeade6c724cbe8cfd50b4377b26c459e0b86925b5a73446344507ba7ee3538b",
      "role": "F3, the RRF of S1 (1.0), nomic-embed-text (1.0) and BM25 (0.5): one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/F3h.rows.ndjson",
      "bytes": 278932,
      "sha256": "28d17f889f2423994c678ac5b804e59b63f06ff80240afbbaa6830124f82aa5f",
      "role": "F3h, exploratory and never gated: the RRF of nomic-embed-text (1.0), S1 (0.5) and BM25 (0.5). Not on the page: one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), whether an exact tie fell across ranks 5 and 6 or 10 and 11 (`tie_at_5`, `tie_at_10`), and the four outcomes (`score`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/fuse/fuse.json",
      "bytes": 3300,
      "sha256": "8593f3a5c1c802617255172a3e8cbb510a5f68f2850d2c4c1764841ef787e8e8",
      "role": "The fusions: for each of the three registered (F1, F2, F3) and the three exploratory (F1h, F2h, F3h), its lists and weights (`spec`), every@5 and any@5, the questions with an exact tie across ranks 5 and 6 and across 10 and 11, and the sha256 of its rows file; the input lists' paths and sha256s; and the control that rebuilt A2's stored top 10 from its two lists (`ctrl_a2_rebuild`: 500 of 500 equal)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/run/rows.ndjson",
      "bytes": 983184,
      "sha256": "724a352b8417129bb059fbfdbc28eaba64c061da25d9a1edbf7ab08819887432",
      "role": "S1, the learned-sparse list alone (2026-11-07): one row per question, with `qid`, `type`, `abstention`, `gold`, the 50 files kept (`ranked`) and their scores (`s1`), the top 10 scored (`search`), the four outcomes (`score`), and per-question counts (texts, windows, texts windowed, tokens, query terms) and seconds."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/run/run.json",
      "bytes": 2133,
      "sha256": "02a152f7f182d79ef43970c2922baf06a1737cb2b6e67330f5154330aa982961",
      "role": "S1's lock, written before the first question: the model (opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) at its pinned revision and the sha256 of each of its files, the activation, the 510-token window, the dataset's and the question list's sha256, n 500, 50 kept, threads and batch, the device (CPU), the harness files' sha256s, library versions, the start time (UTC) and the run's environment switches."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/run/summary.json",
      "bytes": 3663,
      "sha256": "8d1559929ec7185d583edb8f6b37f253af36c2b12ce337add36976ff7a449b79",
      "role": "S1's own summary: every@5 and any@5, the four outcomes over the answerable questions, over all 500 and by question type, the timing and the run's end (UTC)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/family.json",
      "bytes": 4312,
      "sha256": "e4476f210e53d5e8a57d9e508344c5fc4a66770619a51d8543bc2f212190390a",
      "role": "The five registered comparisons (`primary`) and their five secondary forms on any@5 (`secondary`): for each, the two counts, the questions only one side hit, the gain, the margin, McNemar's exact two-sided p, the Holm-adjusted p across the five (primary) and the registered words; and the file each list was read from, with its sha256 (`sources`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/reading.json",
      "bytes": 24607,
      "sha256": "9a3d9b7deb9fbd90d57954aac8cf7317d750f5e33752bfa7d686863143e267d4",
      "role": "The reading: every list's counts overall and by type; the five comparisons recomputed with the registered harness's own McNemar and Holm (`family_crosscheck`, all equal); the question ids behind each comparison's one-sided counts; the three exploratory comparisons; the tie counts and the exploratory reversal of F2's tie order (`f2_ties_reversed_exploratory`); the run's integrity extras; S1's run counts (`s1_run`, the texts windowed among them); a breakdown of S1's misses; and the files read, with their sha256s (`sources`)."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F1.json",
      "bytes": 76068,
      "sha256": "08d723c9dab755e265f886266a8162ed156a4a5b6ae0c96f726e231cd41dcbdf",
      "role": "The second, independently written scorer's recount of F1's fused list: its checks (`g0_problems`, `row_diffs` and `summary_diffs`, each empty), the counts over the 470 answerable questions and over all 500, and each question's four outcomes (`per_question`). The family of comparisons reads its counts from this file."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F1h.json",
      "bytes": 76061,
      "sha256": "ff2b85d2bd5e340f9c60be4d6a1816cff78d79ffaba7b7f63c48a4c320b7d643",
      "role": "The second scorer's recount of the exploratory fusion F1h's list, in the form of the recounts above. The reading's three exploratory comparisons read it; no registered comparison does."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F2.json",
      "bytes": 76062,
      "sha256": "f11e04de8a97edd3217a8708d17d05e445e5b931629f925a01b27279c60c3145",
      "role": "The second, independently written scorer's recount of F2's fused list: its checks (`g0_problems`, `row_diffs` and `summary_diffs`, each empty), the counts over the 470 answerable questions and over all 500, and each question's four outcomes (`per_question`). The family of comparisons reads its counts from this file."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F2h.json",
      "bytes": 76076,
      "sha256": "ea35451c54b928ddadaa26e72368fd4b3f015fb18d61707e4a1a5a50f90db40f",
      "role": "The second scorer's recount of the exploratory fusion F2h's list, in the form of the recounts above. The reading's three exploratory comparisons read it; no registered comparison does."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F3.json",
      "bytes": 76043,
      "sha256": "f549b99fbec3bd5b7f9312861c622e81c3bfd73f56850bfbfaff6ce7276a6089",
      "role": "The second, independently written scorer's recount of F3's fused list: its checks (`g0_problems`, `row_diffs` and `summary_diffs`, each empty), the counts over the 470 answerable questions and over all 500, and each question's four outcomes (`per_question`). The family of comparisons reads its counts from this file."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-F3h.json",
      "bytes": 76060,
      "sha256": "6ca29c4a8aa9d0ba258c6f76456c2e6a2dc530b414b95b42f7bbdd71f9f82fa5",
      "role": "The second scorer's recount of the exploratory fusion F3h's list, in the form of the recounts above. The reading's three exploratory comparisons read it; no registered comparison does."
    },
    {
      "file": "learned-sparse-2026-11-07/full-s1/score/rescore-s1.json",
      "bytes": 76080,
      "sha256": "72d7bf476dfbf128ae88c7eb7f0ce18ca4da1e1f93c61d3c0a82b86990a3872d",
      "role": "The second, independently written scorer's recount of S1's list: its checks (`g0_problems`, `row_diffs` and `summary_diffs`, each empty), the counts over the 470 answerable questions and over all 500, and each question's four outcomes (`per_question`). The family of comparisons reads its counts from this file."
    },
    {
      "file": "learned-sparse-2026-11-07/s2-skip.json",
      "bytes": 836,
      "sha256": "533b8965472f7617be2f3d0a24567498f85885ce8fe16f52e3ca1c063fb9b7af",
      "role": "The reference model the learned-sparse run was to include, naver/splade-v3, as its host described it on 2026-11-07 (UTC): gated behind a licence acceptance, `cc-by-nc-sa-4.0`, its weights only as a Python pickle (`pytorch_model.bin`), and the verdict: skipped, no file downloaded."
    },
    {
      "file": "longmemeval-2026-10/a0/a0-per-question.tsv",
      "bytes": 22487,
      "sha256": "83584b3f7520a5cce1e03359e370f20d32705d71fafa759db353253bc3f3de9e",
      "role": "A0's outcomes per question: id, type, abstention, the number of answer-holding sessions, their ranks among the 50 kept, and 0/1 for any@1, any@5, all@5 (every@5) and all@10. The learned-sparse run's comparisons read A0 from this file."
    },
    {
      "file": "longmemeval-2026-10/a0/run/rows.ndjson",
      "bytes": 904485,
      "sha256": "4d545fdcd364ce86f688250b8b46f8f36db782aba683fde49497425c6f421e55",
      "role": "A0, BM25 alone (2026-10-09): one row per question, with `qid`, `type`, `abstention`, `gold` (the answer-holding sessions, by blind file name), the 50 files BM25 kept (`ranked`) and their scores (`bm25`), the top 10 scored (`search`), the four outcomes (`score`), counts and seconds."
    },
    {
      "file": "longmemeval-2026-10/a0/run/run.json",
      "bytes": 1178,
      "sha256": "825284c97d9b4443e0ebc8db21502b61ac25aea46bbc923863e99c99621fea47",
      "role": "A0's lock, written before its first question: the arm, the dataset's file name and sha256, n 500, 50 kept, blind file names, deep-recall's commit, the harness files' sha256s, the Python and SQLite versions, the start time (UTC) and the run's environment switches."
    },
    {
      "file": "longmemeval-2026-10/a5/a5-per-question.tsv",
      "bytes": 22462,
      "sha256": "34dc22ad11181b4bba43f2c6d5e8875dccde9b13d35a347578b1f0d01725c8cc",
      "role": "A5's outcomes per question, in A0's file's form (the ranks are in the 150-deep fused list). The learned-sparse run's comparisons read A5 from this file."
    },
    {
      "file": "longmemeval-2026-10/a5/run/legs.ndjson",
      "bytes": 1552000,
      "sha256": "de7cf181265a937790b61077d50fbb7d1a191e0134fbfab402e5ed7374e16453",
      "role": "A5's BGE-small embedding list for each question, at 50 and 150 (blind names and similarities), with the embedding calls' counts, so A5 can be re-fused from A0's BM25 lists."
    },
    {
      "file": "longmemeval-2026-10/a5/run/rows.ndjson",
      "bytes": 981098,
      "sha256": "11b778721600031baf8fca0d5b83b85a9b49a8aed840b80354f58dfb5705c534",
      "role": "A5, the RRF of BM25 and BGE-small that deep-recall ships, untested (2026-10-10): as A2's rows, plus the arm, the number of files, per-question seconds, a stand-in flag and the machine's one-minute load average at each question's start."
    },
    {
      "file": "longmemeval-2026-10/a5/run/run.json",
      "bytes": 3114,
      "sha256": "55f33084c3dde80671ca1bce9988bf31f989145ea03aafb1555746432c6481f2",
      "role": "A5's lock: BGE-small and the sha256 of each of its ONNX files (Hugging Face repository `qdrant/bge-small-en-v1.5-onnx-q`, revision pinned), deep-recall's commit and file pins, the harness pins, the driver's sha256, library versions, k 60, weight 1.0 on BGE-small and 0.5 on BM25, the pools and the start time (UTC)."
    },
    {
      "file": "longmemeval-2026-10/w7/rows/a1/legs.ndjson",
      "bytes": 1411814,
      "sha256": "6a4299d76c89e38818fe6d69062c3fb6b9b9bbc2836915793374bdc889bf7994",
      "role": "A1, nomic-embed-text alone (2026-10-17): one row per question, with `qid`, `n_files`, the embedding list cut at 50 (`leg50`) and at 150 (`leg150`), by blind file name, and their similarities (`sim50`, `sim150`). A1's scored list is the first 10 of `leg50`."
    },
    {
      "file": "longmemeval-2026-10/w7/rows/a1/run.json",
      "bytes": 400,
      "sha256": "521e7b653926e4dfbddfe26cf721813ba89801a50da3b50edf69c3b1c9d2fc22",
      "role": "A1's lock: the arm, the batch size, the card the model ran on, the dataset's sha256, the two prefixes the model's card asks for, the model tag, n 500, the machine the calls were sent to (by the role it played; see `provenance.json`) and the start time (UTC)."
    },
    {
      "file": "longmemeval-2026-10/w7/rows/a2/a2.ndjson",
      "bytes": 911249,
      "sha256": "eb489cdba861b7795d07bd2ec187eb585ed0e9d5a90ea97d939e41fa3bcdd8a6",
      "role": "A2, the fusion with the registered test, RRF of BM25 and nomic-embed-text at k 60, weight 1.0 on nomic-embed-text and 0.5 on BM25 (2026-10-17): one row per question, with `qid`, `type`, `abstention`, `gold`, the fused top 10 (`top10`) and their fused scores (`rrf10`), the top 20 and the 150-deep fused list (`pool20`, `wide150`), the four outcomes (`score`) and each list's kept counts."
    },
    {
      "file": "longmemeval-2026-10/w7/rows/a2/run.json",
      "bytes": 229,
      "sha256": "b42f34bf70e7ea98d6a42317b29169a4ec05d2650bc285547ea2abead18512b7",
      "role": "A2's lock: the arm, the dataset's sha256, the keyword weight 0.5, n 500, RRF's k 60 and the start time (UTC)."
    },
    {
      "file": "longmemeval-2026-10/w7/score-registered-w7.json",
      "bytes": 8623,
      "sha256": "8f3da50a4267e4bf71ecbfc0a6dc63c0642eadef8d231a54223660ea80f5ccf0",
      "role": "The registered scorer's own output for A1 and A2: each list's counts overall and by type, A0 recomputed, and the registered test of A2 against A0 on every@5 (23 and 14, McNemar's exact two-sided p, the margin, the verdict) and on any@5, with its note on the Holm adjustment."
    },
    {
      "file": "longmemeval-2026-10/w7/w7-per-question.tsv",
      "bytes": 27581,
      "sha256": "612af3d11aa62e71b92fd01a2c6d25141306bcd750c7c76732c2884cfdea8c20",
      "role": "Per question: id, type, abstention and 0/1 outcomes for A1, A2 and A0 on any@1, any@5, all@5 (every@5) and all@10. The paired input for the page's 23 and 14, and the file the learned-sparse run's comparisons read A1 and A2 from."
    }
  ]
}