{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "four-homes-for-one-reranker",
 "title": "Four homes for one reranker: what RuleSage's slowest stage cost at each stop",
 "dek": "On the morning of 2026-09-17 an operator at a game table waited nearly twenty seconds for the first word of a rules answer and thought the app was broken. It was not. One small stage of the answer, a cross-encoder that re-reads the shortlist before anything is written, had moved to a rented server whose processors lack the instruction set its weights were built for. This page follows that stage across four machines, three of them in five days: two rented VPSes, a mini PC, and a graphics card that was already in the house. Every millisecond we measured at each stop is here, with the gate we ran before the last move, the first word a player waits for after it, the traps that bit on the way, and what is still unmeasured.",
 "published": "2026-09-21",
 "series": [
  "bench"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/four-homes-for-one-reranker/",
  "md_url": "https://research.strata2signal.com/four-homes-for-one-reranker/index.md",
  "md_sha": "c091814337ff37ec2c56bdafaa838fed40352ee6ce4c168481449235c7627abc",
  "html_sha": "5a88946438fc86a80a75a708f19b8cf4daf1baa518aca154b6e4c10aa2232e8d",
  "anchors_sha": "d90d4dacda95a1965bb76d00811a907e4116b245133b53157ead66dfbb7b219e"
 },
 "short": {
  "paragraph": "On the morning of 2026-09-17 a rules answer took nearly twenty seconds to start, and an operator at a game table thought RuleSage, our board-game rules helper, was broken. It was not. One small stage of the answer - the reranker, a cross-encoder that re-reads the shortlist before the answer is written - had moved to a rented server whose processors could not do that stage's arithmetic the fast way: 1,792 ms to re-read a shortlist of 32 at one thread, against 329 ms on the machine it came from. It has since moved to an RTX PRO 6000 Blackwell (96 GB) already in the house, where the same stage takes 30.1 ms. Measured on eight asks of our own, sent down the public path half an hour after the move, the first word now arrives in about a second and a half (1.56 s median, on the five that answered). Four homes in six weeks; every millisecond, with the bench file it came from, is on the page.",
  "from_dek": false,
  "counts": {
   "words": 11479,
   "minutes": 52,
   "tables": 19,
   "kit": false
  },
  "bullets": [
   {
    "text": "the card's power under a 60 s saturated burst reached a median of 168.61 W",
    "figure": "168.61 W",
    "cite": "what-the-seat-costs-where-we-can-say",
    "quote": "Seat What a rerank at the long shape holds Settled idle Under a 60 s burst The card, the seat 67.2 ms of a card shared with other seats; 79.5 ms a call when saturated, because the seat queues behind itself 17.78 W (n = 19, range 17.35-22.92) 168.61 W (p95 177.06) The first VPS's CPU 4 vCPUs of 4 for 734.5 ms, all of the box - (no watt reading on a rented vCPU) - The public host's CPU 4 vCPUs of 8 for 2,690.6 ms, half the box - (no watt reading on a rented vCPU) - The mini PC's CPU 4 vCPUs of 16 for 962.0 ms, a quarter of the box - (not read during its bench) - The same-night pass, Tables 5 and 6: whole-card board power, every tenant resident; the burst is n = 70 samples over 775 calls at 12.92 calls a second, VRAM flat.",
    "span": [
     54218,
     54229
    ]
   }
  ]
 },
 "sections": [
  {
   "id": "a-question-on-game-night",
   "heading": "A question on game night",
   "level": 2,
   "span": [
    3204,
    4889
   ],
   "chunks": [
    [
     3204,
     4889
    ]
   ],
   "chars": 1685,
   "digest": "on 2026-09-17, a question asked to RuleSage did not receive an answer. while the app was not broken and the model and search speeds remained unchanged, the reranker stage on a new rented server took 1,792 ms for a batch of 32 short candidates, compared to 329 ms on the old server.",
   "digest_skipped": null
  },
  {
   "id": "four-homes-in-six-weeks-three-of-them-in-five-days",
   "heading": "Four homes in six weeks, three of them in five days",
   "level": 2,
   "span": [
    4889,
    7323
   ],
   "chunks": [
    [
     4889,
     7323
    ]
   ],
   "chars": 2434,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-a-reranker-does-at-a-rules-desk",
   "heading": "What a reranker does at a rules desk",
   "level": 2,
   "span": [
    7323,
    10580
   ],
   "chunks": [
    [
     7323,
     10580
    ]
   ],
   "chars": 3257,
   "digest": "RuleSage uses three searches to find candidate passages and a cross-encoder to score them. the cross-encoder reads the question and shortlisted passages together to decide which eight reach the answering model. this stage is the expensive part of the process by design.",
   "digest_skipped": null
  },
  {
   "id": "where-should-your-reranker-live",
   "heading": "Where should your reranker live?",
   "level": 2,
   "span": [
    10580,
    14558
   ],
   "chunks": [
    [
     10580,
     14558
    ]
   ],
   "chars": 3978,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "four-seats-one-night-the-same-passages",
   "heading": "Four seats, one night, the same passages",
   "level": 2,
   "span": [
    14558,
    19922
   ],
   "chunks": [
    [
     14558,
     19922
    ]
   ],
   "chars": 5364,
   "digest": "on 2026-09-20, the same cross-encoder was timed on four different seats using the same passages. the card, running fp32, achieved 28.9 ms for the short shape, while the public host using int8 took 386.4 ms at four threads.",
   "digest_skipped": null
  },
  {
   "id": "the-model-and-the-two-shapes-behind-every-latency-table",
   "heading": "The model and the two shapes behind every latency table",
   "level": 2,
   "span": [
    19922,
    23784
   ],
   "chunks": [
    [
     19922,
     23784
    ]
   ],
   "chars": 3862,
   "digest": "the cross-encoder model uses two shapes: a short shape of 32 candidates at about 133 tokens and a long shape of 32 candidates at 512 tokens. the engine trims passages to a 1,200-character budget before the model truncates them at 512 tokens.",
   "digest_skipped": null
  },
  {
   "id": "the-move-to-the-public-host-and-the-slowness",
   "heading": "The move to the public host and the slowness",
   "level": 2,
   "span": [
    23784,
    25666
   ],
   "chunks": [
    [
     23784,
     25666
    ]
   ],
   "chars": 1882,
   "digest": "moving RuleSage to a new rented VPS on 2026-09-15 caused slowness. the new host's vCPUs are Haswell-class with AVX2 but no VNNI, while the shipped model graph had been quantized for avx512_vnni.",
   "digest_skipped": null
  },
  {
   "id": "one-thread-was-the-whole-free-lever",
   "heading": "One thread was the whole free lever",
   "level": 2,
   "span": [
    25666,
    27674
   ],
   "chunks": [
    [
     25666,
     27674
    ]
   ],
   "chars": 2008,
   "digest": "setting the thread count from 1 to 4 provided a speed-up for the reranker on the public host. for the 32 × 512 tok shape, the speed-up was 3.1×, moving from 9,589.7 ms to 3,063.3 ms.",
   "digest_skipped": null
  },
  {
   "id": "a-quantized-graph-targets-an-instruction-set",
   "heading": "A quantized graph targets an instruction set",
   "level": 2,
   "span": [
    27674,
    32708
   ],
   "chunks": [
    [
     27674,
     32708
    ]
   ],
   "chars": 5034,
   "digest": "benchmarking showed that re-exporting the graph for AVX2 was slower in five cells of six on the public host. the int8 graph on the first VPS was 3.0× faster than fp32 at the short shape.",
   "digest_skipped": null
  },
  {
   "id": "the-mini-pc-measured-and-not-chosen",
   "heading": "The mini PC, measured and not chosen",
   "level": 2,
   "span": [
    32708,
    34323
   ],
   "chunks": [
    [
     32708,
     34323
    ]
   ],
   "chars": 1615,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "the-same-model-on-a-card-already-in-the-house",
   "heading": "The same model on a card already in the house",
   "level": 2,
   "span": [
    34323,
    38983
   ],
   "chunks": [
    [
     34323,
     38983
    ]
   ],
   "chars": 4660,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "why-a-door-stands-in-front-of-the-seat",
   "heading": "Why a door stands in front of the seat",
   "level": 2,
   "span": [
    38983,
    42929
   ],
   "chunks": [
    [
     38983,
     42929
    ]
   ],
   "chars": 3946,
   "digest": "the door, or quartermaster, acts as a gateway between the app and the seat. it relays calls to the role and handles failures by serving the fused shortlist unranked. the door's budget is 9,000 ms with an attempt count of 1.",
   "digest_skipped": null
  },
  {
   "id": "the-gate-before-the-flip",
   "heading": "The gate before the flip",
   "level": 2,
   "span": [
    42929,
    47363
   ],
   "chunks": [
    [
     42929,
     47363
    ]
   ],
   "chars": 4434,
   "digest": "the gate tested the parity between the int8 graph on a CPU and the fp32 model on a card. the gate passed with an overlap@8 mean of 0.9625, though this was under the expected band of 0.97.",
   "digest_skipped": null
  },
  {
   "id": "the-flip",
   "heading": "The flip",
   "level": 2,
   "span": [
    47363,
    49403
   ],
   "chunks": [
    [
     47363,
     49403
    ]
   ],
   "chars": 2040,
   "digest": "the flip on 2026-09-20 initially resulted in an outage where RuleSage's calls were dropped by a cgroup filter. after an operator applied a second drop-in, the first live row landed at 21:02:51.055Z.",
   "digest_skipped": null
  },
  {
   "id": "before-you-move-a-seat",
   "heading": "Before you move a seat",
   "level": 2,
   "span": [
    49403,
    53315
   ],
   "chunks": [
    [
     49403,
     53315
    ]
   ],
   "chars": 3912,
   "digest": "builders should check CPU flags, thread arithmetic, and free memory before moving a seat. an avx512_vnni graph on a box with AVX2 and no VNNI was 5.4× slower at one thread.",
   "digest_skipped": null
  },
  {
   "id": "what-the-seat-costs-where-we-can-say",
   "heading": "What the seat costs, where we can say",
   "level": 2,
   "span": [
    53315,
    56374
   ],
   "chunks": [
    [
     53315,
     56374
    ]
   ],
   "chars": 3059,
   "digest": "the card's power was sampled during a 60 s saturated burst, reaching a median of 168.61 W. the reranker's share of the card's cost is not separable from the other tenants.",
   "digest_skipped": null
  },
  {
   "id": "what-is-still-unmeasured",
   "heading": "What is still unmeasured",
   "level": 2,
   "span": [
    56374,
    60357
   ],
   "chunks": [
    [
     56374,
     60357
    ]
   ],
   "chars": 3983,
   "digest": "several items remain unmeasured, including the server-side first token after the flip and whether anything pools connections between RuleSage and the door. the live unit's per-candidate budget is also unread.",
   "digest_skipped": null
  },
  {
   "id": "friday-night-after-the-move",
   "heading": "Friday night, after the move",
   "level": 2,
   "span": [
    60357,
    61630
   ],
   "chunks": [
    [
     60357,
     61630
    ]
   ],
   "chars": 1273,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-to-take-with-you",
   "heading": "What to take with you",
   "level": 2,
   "span": [
    61630,
    62931
   ],
   "chunks": [
    [
     61630,
     62931
    ]
   ],
   "chars": 1301,
   "digest": "key lessons include checking CPU flags for VNNI, re-doing thread arithmetic, and using a card already in the house. builders should also score the old seat against the new one before a flip.",
   "digest_skipped": null
  },
  {
   "id": "how-to-check-our-work",
   "heading": "How to check our work",
   "level": 2,
   "span": [
    62931,
    66484
   ],
   "chunks": [
    [
     62931,
     66484
    ]
   ],
   "chars": 3553,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "the-rest-of-the-seminar",
   "heading": "The rest of the seminar",
   "level": 2,
   "span": [
    66484,
    67549
   ],
   "chunks": [
    [
     66484,
     67549
    ]
   ],
   "chars": 1065,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    67549,
    68736
   ],
   "chunks": [
    [
     67549,
     68736
    ]
   ],
   "chars": 1187,
   "digest": null,
   "digest_skipped": "credits"
  }
 ],
 "chips": [
  {
   "id": "c-c7637224",
   "q": "what role does a cross-encoder play in the pipeline?",
   "a": "A cross-encoder reads the question and each shortlisted passage together to score how well the passage answers the question. It is the most expensive stage by design because it is the only one that reads pairs.",
   "cites": [
    "what-a-reranker-does-at-a-rules-desk"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-e118d8e7",
   "q": "what happens when a reranker seat is unreachable?",
   "a": "The system is designed to fail open. When a seat is unreachable, the pipeline catches a RemoteRerankUnavailable exception and serves the fused top eight candidates in fusion order, though this can cause a delay of up to nine seconds.",
   "cites": [
    "why-a-door-stands-in-front-of-the-seat"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-d97233f9",
   "q": "what are the primary risks when moving a reranker?",
   "a": "Risks include running a graph compiled for a specific instruction set on incompatible hardware, miscalculating thread arithmetic for new core counts, and failing to account for how a system handles a dead seat during a fail-open event.",
   "cites": [
    "before-you-move-a-seat"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-5cabd02e",
   "q": "what is the impact of a reranker move on user wait times?",
   "a": "After the final move, the median time for the first word to arrive is 1.56 s. This includes the search and the network trip, and is comparable to the time it takes to pick up dice.",
   "cites": [
    "friday-night-after-the-move"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "two-hours-on-battery",
   "why": "shares ground with § One thread was the whole free lever · § Set the thread count — the one free lever, and the exception that proves the rule"
  },
  {
   "slug": "three-librarians",
   "why": "shares ground with § What a reranker does at a rules desk · § The careful reader"
  },
  {
   "slug": "the-new-kid",
   "why": "shares ground with § Four seats, one night, the same passages · § The classifier seat — five gates, all pass, faster"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nThe model is `cross-encoder/ms-marco-MiniLM-L6-v2`, published on Hugging Face by the **Sentence-Transformers** project, at a pinned revision. On the CPU seats it ran as an int8 graph exported through **Hugging Face Optimum** and executed by **ONNX Runtime**. On the card it runs in full precision through **sentence-transformers** over **PyTorch** and **CUDA** (NVIDIA, proprietary, under the CUDA Toolkit EULA), the one closed piece in the stack. The open pieces' licences were not re-read for this page; the ones this shelf has read are on its [licences](https://research.strata2signal.com/licences/) page.\n\nA small human team asked for this page, chose what it would and would not claim, and signed the numbers. A fleet of AI agents read every bench file, ledger row and live record listed above, re-derived every ratio from two cells of the same table and drafted it under that team's rulings.\n\nCorrections and later measurements will be added below, each dated (UTC) with a window at both ends where one applies, each saying in plain words what it counts, and each comparing itself in one sentence to the reading it replaces.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 1,
  "digests_written": 19,
  "digests_kept": 14,
  "chips_written": 8,
  "chips_kept": 4,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 2,
   "figure": 5,
   "length": 0,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 3,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 0,
   "house_voice": 0,
   "hand-edit": 1
  },
  "new_noun_tokens_checked": [
   "0.75",
   "0.9625",
   "0.97",
   "000",
   "02",
   "02Z",
   "063.3",
   "1",
   "1.56",
   "133",
   "150",
   "168.61",
   "17.78",
   "19",
   "195",
   "2.1",
   "2.5",
   "20",
   "200-character",
   "2026-08-10",
   "2026-09-15",
   "2026-09-17",
   "2026-09-19",
   "2026-09-20",
   "21",
   "28.9",
   "3",
   "3.0×",
   "3.1×",
   "30.1",
   "32",
   "329",
   "386.4",
   "4",
   "5.4×",
   "51.055Z",
   "512",
   "512-token",
   "56Z",
   "589.7",
   "60",
   "6000",
   "792",
   "8",
   "9",
   "910",
   "96",
   "963.6",
   "AVX-512",
   "AVX2",
   "CPU",
   "Core",
   "GB",
   "GPU",
   "Haswell-class",
   "IPAddressDeny",
   "Intel",
   "MiB",
   "PC",
   "PRO",
   "RTX",
   "RemoteRerankUnavailable",
   "RuleSage",
   "RuleSage's",
   "VNNI",
   "VPS",
   "W",
   "avx512vnni",
   "fp32",
   "i5-12600H",
   "int8"
  ],
  "new_noun_tokens_withheld": 5,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "digest",
    "reason": "redaction",
    "item": null,
    "item_chars": 251,
    "withheld": "a string carrying a fenced literal",
    "detail": ""
   },
   {
    "kind": "digest",
    "reason": "redaction",
    "item": null,
    "item_chars": 231,
    "withheld": "a string carrying a fenced literal",
    "detail": ""
   },
   {
    "kind": "digest",
    "reason": "figure",
    "item": "a mini PC with an Intel Core i5-12600H was measured on 2026-09-19. while it was faster than the public host, it was not chosen because its 512-token window reading of 963.6 ms failed a rule that requi",
    "detail": "figures absent: ['5-12600']"
   },
   {
    "kind": "digest",
    "reason": "redaction",
    "item": null,
    "item_chars": 166,
    "withheld": "a string carrying a fenced literal",
    "detail": ""
   },
   {
    "kind": "digest",
    "reason": "figure",
    "item": "after the move, the rerank leg for a visitor is about 195 ms. on the evening of 2026-09-20, the median time for the first word to arrive was 1.56 s.",
    "detail": "figures absent: ['195']"
   },
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the reranker's share of the first token fell from seconds to tens of milliseconds",
    "detail": "terminal token carries no figure"
   },
   {
    "kind": "chip",
    "reason": "figure",
    "item": "Moving the reranker to a new server caused the time to re-read a shortlist of 32 candidates to increase from 329 ms to 1,792 ms. This change resulted in the first token arriving between 2.5 and 19 sec",
    "detail": "figures absent from cited spans: ['2.5', '19', '0.75', '2.1']"
   },
   {
    "kind": "chip",
    "reason": "new_noun",
    "item": null,
    "item_chars": 4,
    "withheld": "a string naming something the article does not",
    "detail": ""
   },
   {
    "kind": "chip",
    "reason": "new_noun",
    "item": null,
    "item_chars": 4,
    "withheld": "a string naming something the article does not",
    "detail": ""
   },
   {
    "kind": "chip",
    "reason": "figure",
    "item": "An IPAddressDeny configuration can cause the app to drop all connections to the gateway, resulting in a slow answer rather than an error because the client fails open. This caused an eleven-minute per",
    "detail": "figures absent from cited spans: ['20:56', '21:02']"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "count": 1,
    "item": null,
    "withheld": "one bullet removed from this pack by hand",
    "detail": "removed from this pack by hand on 2026-09-21 at the release cut for exhibit fifty-six, after the writer ran and after the gates had counted. The line gave the section's operator a verb the section does not: \"an operator saw that the reranker on the new server took 1,792 ms\" (the article's own words are \"On the new one it took 1,792 ms: 5.4 times longer at one thread\"); in that section the operator waited and stopped waiting, and the 1,792 ms is a bench reading stated two sentences later. The figure is verbatim in its cited section and gate 1 kept it; the compression is not something a gate on this shelf checks yet."
   }
  ]
 },
 "generated_utc": "2026-09-21T15:45:01Z"
}