# Three librarians and a careful reader — how RuleSage finds the right page *Before RuleSage answers a rules question, something has to find the right page in a book the size of a small novel. Finding it takes three searchers, one referee, and one careful reader — and none of them is the model that writes your answer. This page walks that journey in plain english, with the real numbers from the shipped app. At the end there is a live answer you can open yourself.* *Published 2026-08-18 · updated 2026-08-21 · a plain-english companion to [Where your question goes](https://research.strata2signal.com/where-your-question-goes/) · a small (human) team and a fleet of AI agents* **the short version:** Before RuleSage answers, something has to find the right page in a book the size of a small novel — three searchers, one referee, one careful reader, and none of them is the model that writes your answer. The whole journey, in plain english, with the shipped numbers. 3,977 words · about 18 minutes (at 220 words/min) · 1 table · data kit: no https://research.strata2signal.com/three-librarians/ --- *Four words this page leans on.* A **passage** is a bite-sized piece of a rulebook, usually a rule or two long. The **corpus** is every passage from every ingested book, held in a database. An **arm** is one of the search methods feeding the hunt — and, loosely, the ranked list of candidates it brings back. A **ruling** is what RuleSage publishes when it answers: the answer, its cited pages, and a debug panel — the wrench — showing where it all came from. ## Start with a bad idea {#start-with-a-bad-idea} First, the app this page walks through. [RuleSage](https://rulesage-live.strata2signal.com/) is our board-game rules helper: pick a game from a shelf of about 750, ask a rules question in your own words, and it answers by quoting the actual rulebook — page numbers attached, and "not in the book" when the book is silent, rather than a confident guess. It's free, it's live, and no account stands between you and it. Now suppose you ask it a question about Catan: *"can the robber go back to the desert?"* The search runs inside the game you asked from — its base rulebook, its expansions, whatever has been ingested for that family — which is still a lot of ground: hundreds of passages per book, and a game with expansions runs into the thousands. One of those passages answers you. The bad idea is the obvious one: hand everything to the language model and let it read. Models have a reading budget (a context window), a shelf of rulebooks strains it, and a model wading through thousands of passages to use a handful is a model you're paying to be lost — slower, costlier, and likelier to drift. Everything below exists to hand the model a *shortlist* instead: the few passages most likely to hold the answer, in the best order we can manage. So: who builds the shortlist? ## The first librarian searches for your exact words {#the-first-librarian-searches-for-your-exact-words} Imagine a librarian with an index of every word on every page. Ask her about the robber and the desert and she pulls every passage containing *robber* and *desert*. This is **full-text search**, the same technology behind ctrl-F, grown up. In RuleSage it runs inside Postgres, the database that holds the corpus. She is hard to beat when your words are the book's words. Ask about "Longest Road" and she goes straight there, because that exact phrase lives on exactly those pages. She is literal about vocabulary, but not about word endings: she'll forgive "setup" for "setting up," and never "thief" for "robber." Her rough edge shows when a word is everywhere — fifty passages mention the robber, and ranking which of the fifty answers *you* is not really her skill. Her real blindness is meaning. Ask *"can I move the thief back to the sand tile?"* and she finds nothing much, because Catan's book never says *thief* or *sand tile*. It says *robber* and *desert*. She has no idea those are the same thing. Players ask questions in their own words, and their own words are the one thing she cannot see past. ## The second librarian searches by meaning {#the-second-librarian-searches-by-meaning} The second librarian never checks an index. She has read everything, once, and she remembers what each passage is *about*. Under the hood this is **semantic search**, built on **embeddings** — and embeddings are less mysterious than they sound. When a book is ingested, a model reads each passage and assigns it a location: a long list of numbers that works like coordinates on a map of meaning. (Ours is a model called nomic-embed-text-v1.5, and its map has 768 directions — more than we can picture, and not needing to picture it is the point.) Passages about moving the robber land near each other on that map. Passages about victory points settle somewhere else, together. The map is drawn once, ahead of time; when you ask, only your *question* needs fresh coordinates, and "search" becomes "what's parked nearby?" Nearby means *similar in meaning*, whether or not a single word is shared. Vibes, with coordinates. Ask her about the thief and the sand tile and she walks straight to the robber-and-desert passages, unbothered: the question and the passages sit close together because they *mean* close together. She survives you not knowing the book's vocabulary, and that is her whole superpower. Her weakness is the mirror of the first librarian's. Precision blurs at close range: to her, the passage about *moving* the robber and the passage about *placing* the robber at setup are near neighbours, and an exact name like "Longest Road" is just another phrase in the fog. The first librarian nails it; the second waves at the district it lives in. She can also return something thematically adjacent and confidently wrong, which for a rules judge is the worst failure there is. Exact words, or meaning. Each one is brilliant precisely where the other is blind. So RuleSage refuses to choose. ## The third librarian never left your aisle {#the-third-librarian-never-left-your-aisle} There is one more searcher, easy to miss because she doesn't roam at all. You asked *while standing in front of Catan*, and that is evidence. The third librarian's whole existence depends on the app knowing which game you asked from and having that game's books ingested; when it does, she works your aisle only. She picks up the edition's own rulebook, plus any correction documents matched to your question (publisher errata — the fixes rulebooks accumulate), and ranks each book's passages the way the second librarian ranks the world: by the map of meaning, just never leaving the aisle. Each of her books sends its eight best passages into the room — up to four documents, corrections taking priority if the slots run short, thirty-two candidates at most, standing for election *because of where you're standing*, even if neither roaming librarian shortlisted them. (The wrench calls her arm *doc-priority boost* — a name that outlived the double weighting it once described, as you're about to read.) One law keeps her honest, and it was learned by measurement, not principle. An early prototype turned her up on two knobs at once — double voting weight and a bigger candidate budget — and the measurements from that prototype show what happened: her books flooded the pool and crowded out better passages the two roaming librarians had found. The cure came in pieces: one fair vote, a small budget, and later a guarantee that each of her books gets its *own* eight slots so one big book can't starve the others. Her ballot counts exactly like everyone else's. ## The election {#the-election} Every arm that runs searches the aisle, and each brings back a ranked list: the two roaming librarians up to eighty candidates each, the aisle librarian up to thirty-two. And here a real problem appears. The arms *score* in different currencies — one in word-match points, one in map-distance — and averaging those would be like averaging a temperature with a price. The referee, **reciprocal-rank fusion** (RRF, the least glamorous hero in this story), solves it by throwing every score away and keeping only positions. Each arm casts one ballot per passage it ranked, worth: ``` 1 / (60 + rank) ``` The 60 is called *k*, the fusion constant. It comes from the original RRF paper, and we did not retune it. Rank 1 on an arm's list is worth 1/61, rank 2 is worth 1/62, rank 10 is 1/70, and a passage's fused score is just its ballots added up. That small formula quietly encodes a philosophy: - **Consensus beats enthusiasm.** A passage ranked #3 by both roaming librarians scores 2/63 ≈ 0.0317. A passage ranked #1 by one and missed entirely by the other scores 1/61 ≈ 0.0164. The both-#3 passage wins, comfortably. Two independent skills agreeing at "pretty good" outranks one skill shouting "perfect." - **And consensus has a known cost.** When one arm is right and the other is systematically blind — that exact name only the word-matcher can see — the lone correct hit gets outvoted by mediocre agreements. That failure is one of the reasons a careful reader sits downstream of this election rather than trusting it. - **The 60 is a humility dial.** Without it, rank 1 would be worth 1 and rank 2 only ½, and a single arm's top pick would steamroll everything. Adding 60 flattens the curve so first place barely beats third, and agreement across arms becomes the main way to climb. - **Nomination is real power, bounded by arithmetic.** The aisle librarian can put a passage in the room that nobody else shortlisted — but standing alone it tops out at 1/61, which still loses to anything two arms agreed on at #3. Getting on the ballot is her gift; winning still takes votes. Two honest wrinkles before moving on. The election is not always three arms: the aisle librarian only works when the app knows your game and holds its books, so some asks fuse two arms and some fuse three — and the most a passage can possibly score depends on how many arms voted. Hold that thought; this page owes you a confession about it. And the fused list is cut to its top eighty before anyone deeper ever sees it, so the funnel narrows even when the arms brought more. ## The careful reader {#the-careful-reader} Everything so far is fast because it is shallow. Neither roaming librarian ever read your question and a passage *together* — one matched word forms, the other measured map-distance, and the referee is pure arithmetic that doesn't know what a robber is. So the fused shortlist goes to the character this article is named for: the **re-ranker**. Ours is a cross-encoder, a small model named MiniLM that runs on the same rented server that handles your question — trimmed to about 22 megabytes, tiny in a world of multi-gigabyte models, because its job is narrow and it does nothing else. Where the map-maker judged your question and each passage separately, as two dots whose distance it measured, the cross-encoder reads them **as a pair**, one eye on each, weighing the passage's words against your words as it goes. That lets it catch what distance cannot: a passage soaked in robber-vocabulary that is actually about *placing* the robber during setup scores poorly against a question about *moving* it later, because read side by side, the mismatch is plain. Depth costs time, which is why the machine is shaped like a funnel. You could never afford the careful reader on the whole corpus; you don't have to. The cheap, broad arms and their referee boil thousands of passages down to at most eighty, and the expensive reader spends its attention there. For each candidate it reads a fixed budget of 1,200 characters: about 595 from the opening, the final 600, and a five-character joiner between them. The tail matters, and we know because it once didn't exist. An earlier version read openings only, and a rule that began deep inside a long passage was invisible to it — Monopoly's turn rules start 1,812 characters into a 2,052-character passage, squarely inside the final 600 once a tail existed, and unreachable before. Three live rulings came back "not in the book" — the app's word for that is an *abstain* — on a question the book plainly answers. Measured, fixed, published; the shape it left behind is *read both ends*. The careful reader keeps the best **eight**. Two small guarantees can override its judgment, both in the game's favour: on an ask where the game's books rode along, the top passage from the *rulebook itself* always keeps its place, and the top passage from a matched correction document is never dropped — a fix the publisher shipped outranks any ranker's opinion of it. The wrench on every ruling says whether a guarantee fired. ## The order is dealt on purpose {#the-order-is-dealt-on-purpose} The eight survivors go to the answering model with their full text, in a deliberately strange order. Language models reading a long context are like people skimming a stack of papers: sharpest on what sits at the top and the bottom, foggiest in the middle. The research literature calls it *lost in the middle*. So RuleSage deals the strongest passages to the two ends of the stack and buries the weakest inside. With five passages the deal runs 1st, 3rd, 5th, 4th, 2nd; with the live eight, the same pattern — ends first, middle last. From there, the answering model — a larger model, one more hop away on a machine we own, exactly as [Where your question goes](https://research.strata2signal.com/where-your-question-goes/) tells it — writes the answer, quoting the rulebook pages it drew from. If the book is silent, it says "not in the book" rather than improvising. That discipline is a different article's story. ## The doorman stands off this road entirely {#the-doorman-stands-off-this-road-entirely} One more model touches your question, and it's worth naming precisely because readers assume it sits on the road above. It doesn't. Questions asked on RuleSage can appear on a public page — a wall of past rulings the next visitor can browse — and before yours does, this model reads it once and makes a single call: clear it to render publicly, or hold it for a person to look at. That is the whole job. It cannot touch your answer, and when it fails, it fails one way: your question waits for the human queue instead of rendering. Nothing is ever published *because* it failed. The role is newly filled. Three candidate models sat a pre-registered exam; all three passed all five gates, so the exam qualified everyone and separated no one, and the tie broke, on the record, on measured speed plus a stated preference for the newer generation. The model holding the role as of publication is qwen3.8:27b, and [the full audit is its own exhibit](https://research.strata2signal.com/the-new-kid/), including the different exam the same model failed the same day. Roles are held on evidence here, and the evidence publishes either way. Count the cast, then. Of everything this page introduced, only four are models, doing four different jobs: a map-maker placing passages and questions among each other, a careful reader scoring the shortlist, an answering model writing from what survived, and a doorman deciding only whether the question itself stands in public. Beside them: one librarian who is a database index, one who is an aisle's worth of privilege, and one referee who is arithmetic. Nothing here is a general intelligence doing everything. It is a workshop of specialists, each holding exactly the role it earned. ## The nerd shelf — the numbers, from the shipped configuration {#the-nerd-shelf-the-numbers-from-the-shipped-configuration} Settings are per-deployment. If a wrench shows you a different figure, you're looking at a differently-configured install, not at this page being wrong. | thing | value | what it is | | --- | --- | --- | | arms fused | 2 or 3 | exact words (Postgres FTS) · meaning (embeddings) · the game's book + errata, when armed | | roaming arms' lists | up to 80 each | the two roaming librarians' ranked candidates | | aisle arm's budget | 4 docs × 8 passages, ≤ 32 | the edition's rulebook + matched corrections | | fused shortlist | cut to top 80 | what the careful reader receives | | RRF constant k | 60 | from the original RRF paper; not retuned | | fused ceiling | arms/(k+1): 2/61 ≈ 0.03279 · 3/61 ≈ 0.04918 | every voting arm ranking one passage #1 | | embedder | nomic-embed-text-v1.5 · 768 dimensions | the 768 directions above; draws the map once per passage, at ingestion | | re-ranker | MiniLM cross-encoder · ~22 MB | stored at reduced precision (the trimming); runs on the rented server | | scored window | 1,200 chars: ~595 opening + 600 tail + joiner | what the re-ranker reads per candidate | | survivors | top 8 | what the answering model receives, full text | | final order | lost-in-the-middle | strongest at both ends of the context | ## A confession, with its receipt {#a-confession-with-its-receipt} For a while, the wrench on every ruling — the panel a reader opens to see these scores — printed the fused-score ceiling as `0.0328`, hardcoded. That *is* the ceiling… for two arms. By then the app could fuse three, whose ceiling is 0.04918, and 149 live rulings already carried scores above the number the panel called impossible. Real hits, labelled off-the-scale, by a constant nobody re-derived when the third arm joined. What a reader saw was quiet nonsense: a score sitting above a line described as unreachable. An audit of the wrench's own copy caught it. The cure was to derive the number instead of stating it: the panel now computes `arms/(k+1)` from each ruling's own arm count and prints the formula beside the result — which also repaired all 149 old rulings at once, since the ceiling is computed fresh each time a page renders. When a number can be derived, derive it; a hardcoded truth is a lie with a delay. And because a confession should name what it *hasn't* fixed, with what each item costs you: a fourth arm at equal weight can't re-break the ceiling, but a differently-weighted arm still could. The same 60 is written down by hand in two codebases, so the two could drift apart. And one health light on the same panel still assumes two arms, wearing a caveat instead of a fix. The list is shorter than it was. It is not zero. One footnote for the sharp-eyed: fused scores cross the wire rounded to six decimals, so a passage two arms ranked #1 can print a hair *above* the exact sum — 0.032787 against 2/61 = 0.0327868… The wrench compares with a tolerance for exactly this reason, and you will meet a live specimen of that number in the last section of this page. ## What this machinery gets wrong {#what-this-machinery-gets-wrong} Every character above got its weakness named; the pipeline as a whole deserves the same. - **An abstain is not proof the book is silent.** When every arm misses, the answering model sees nothing useful and honestly says "not in the book" — indistinguishable, from the outside, from the book truly being silent. The Monopoly turn-rules case above is the house specimen: the book answered; the window was blind; three rulings abstained. The cure was a wider window, and the class of failure is permanent — only its instances get fixed. - **Passages are cut by machine.** A rule and its exception can land on opposite sides of a cut, and then no single passage reads as complete. Oversized passages split hardest, and the split shows up as mush on the map of meaning. - **The map is drawn at ingestion.** A corrected book redraws it only when re-ingested; in between, the aisle librarian's arm and its guaranteed place are the compensation. - **The careful reader can be wrong too.** It reads 1,200 characters, not the whole passage, and it ranks — it does not verify. Everything downstream that checks quotes against pages exists because no ranker, however careful, is a guarantee. ## What to take with you {#what-to-take-with-you} - **Finding the right page is three skills, not one.** Exact-words search nails names and misses meaning; meaning search survives your vocabulary and blurs up close; the aisle arm brings the game’s own books. Each is blind exactly where another sees. - **The election counts positions, never scores.** One ballot per arm per passage, worth 1/(60+rank) — so two arms agreeing at “pretty good” beats one shouting “perfect,” and the most any passage can score is arms/61. - **The careful reader earns its keep at the end** — a 22 MB cross-encoder reading question and passage as a pair, over a 1,200-character window that covers both ends of a passage because a head-only window once went blind to a rule that started 1,812 characters into a 2,052-character passage. - **An abstain is not proof the book is silent** — it can also mean the shortlist missed. The machinery lists its own failure modes, and the confession about the hardcoded ceiling is on this page because we shipped it. ## How to check our work — and see it live {#how-to-check-our-work-and-see-it-live} The best way to understand this page is to use the machine it describes. [Ask RuleSage a real rules question](https://rulesage-live.strata2signal.com/) — it’s free, no account, about 750 games — then open your ruling’s wrench and read the journey you just took: the arms that voted, the fused score, the ceiling with its formula, the cited pages. Decode the score the way this page taught you (0.032787 is two arms voting #1, to the wire’s rounding). While drafting we asked it our own opening question — *“can the robber go back to the desert?”* — and published the result like any other ask: [ruling 4615](https://rulesage-live.strata2signal.com/ruling/4615), answered in 2.7 seconds, 88 milliseconds of which was everything this page described. One honesty note about the wrench: it shows the fused score, the arm count, the ceiling and the pages — but not each arm’s individual rank, because fusion happens on a machine upstream of the app and only the fused result crosses the wire. The wrench says so itself rather than leaving the gap silent. The librarians, the election, and the careful reader all leave receipts — that is the house rule for everything: if a page tells you how the machine works, the machine should be willing to show you, live. ## The rest of the seminar {#the-rest-of-the-seminar} This page walked one road: a question’s journey to the right page. Its companion, [Where your question goes](../where-your-question-goes/), walks the same journey for *you* — privacy, in plain english, door to door. The doorman’s hiring exam is [its own exhibit](../the-new-kid/), floor-miss and all, and [Reading is fast, writing is slow](../reading-is-fast/) explains why the answer then arrives word by word once the right page is found. Three more roads run through this workshop, and the field guide intends to walk them all, receipts in hand, no dates promised — a register we keep of owed work tracks them: how a rulebook becomes searchable passages and then a checklist that refuses to say what it cannot prove; what a bench is and how model judges are kept honest; and what a token costs and why. (The Q4\_K\_M road is walked: [The compressed photograph](https://research.strata2signal.com/the-compressed-photograph/).) [The whole shelf](../) holds every bench behind every claim — and if one of those roads is the piece you want next, [say so](https://strata2signal.com/contact/): the suggestion box is read. *Licence: CC BY 4.0, the whole page — name the source and link to it.*