Three librarians and a careful reader — how RuleSage finds the right page

Four words this page leans on. A passage is a bite-sized piece of a rulebook, usually a rule or two long. The corpus is every passage from every ingested book, held in a database. An arm is one of the search methods feeding the hunt — and, loosely, the ranked list of candidates it brings back. A ruling is what RuleSage publishes when it answers: the answer, its cited pages, and a debug panel — the wrench — showing where it all came from.

Start with a bad idea

First, the app this page walks through. RuleSage is our board-game rules helper: pick a game from a shelf of about 750, ask a rules question in your own words, and it answers by quoting the actual rulebook — page numbers attached, and "not in the book" when the book is silent, rather than a confident guess. It's free, it's live, and no account stands between you and it.

Now suppose you ask it a question about Catan: "can the robber go back to the desert?" The search runs inside the game you asked from — its base rulebook, its expansions, whatever has been ingested for that family — which is still a lot of ground: hundreds of passages per book, and a game with expansions runs into the thousands. One of those passages answers you.

The bad idea is the obvious one: hand everything to the language model and let it read. Models have a reading budget (a context window), a shelf of rulebooks strains it, and a model wading through thousands of passages to use a handful is a model you're paying to be lost — slower, costlier, and likelier to drift. Everything below exists to hand the model a shortlist instead: the few passages most likely to hold the answer, in the best order we can manage.

So: who builds the shortlist?

The first librarian searches for your exact words

Imagine a librarian with an index of every word on every page. Ask her about the robber and the desert and she pulls every passage containing robber and desert. This is full-text search, the same technology behind ctrl-F, grown up. In RuleSage it runs inside Postgres, the database that holds the corpus.

She is hard to beat when your words are the book's words. Ask about "Longest Road" and she goes straight there, because that exact phrase lives on exactly those pages. She is literal about vocabulary, but not about word endings: she'll forgive "setup" for "setting up," and never "thief" for "robber." Her rough edge shows when a word is everywhere — fifty passages mention the robber, and ranking which of the fifty answers you is not really her skill.

Her real blindness is meaning. Ask "can I move the thief back to the sand tile?" and she finds nothing much, because Catan's book never says thief or sand tile. It says robber and desert. She has no idea those are the same thing. Players ask questions in their own words, and their own words are the one thing she cannot see past.

The second librarian searches by meaning

The second librarian never checks an index. She has read everything, once, and she remembers what each passage is about.

Under the hood this is semantic search, built on embeddings — and embeddings are less mysterious than they sound. When a book is ingested, a model reads each passage and assigns it a location: a long list of numbers that works like coordinates on a map of meaning. (Ours is a model called nomic-embed-text-v1.5, and its map has 768 directions — more than we can picture, and not needing to picture it is the point.) Passages about moving the robber land near each other on that map. Passages about victory points settle somewhere else, together. The map is drawn once, ahead of time; when you ask, only your question needs fresh coordinates, and "search" becomes "what's parked nearby?" Nearby means similar in meaning, whether or not a single word is shared. Vibes, with coordinates.

Ask her about the thief and the sand tile and she walks straight to the robber-and-desert passages, unbothered: the question and the passages sit close together because they mean close together. She survives you not knowing the book's vocabulary, and that is her whole superpower.

Her weakness is the mirror of the first librarian's. Precision blurs at close range: to her, the passage about moving the robber and the passage about placing the robber at setup are near neighbours, and an exact name like "Longest Road" is just another phrase in the fog. The first librarian nails it; the second waves at the district it lives in. She can also return something thematically adjacent and confidently wrong, which for a rules judge is the worst failure there is.

Exact words, or meaning. Each one is brilliant precisely where the other is blind. So RuleSage refuses to choose.

The third librarian never left your aisle

There is one more searcher, easy to miss because she doesn't roam at all. You asked while standing in front of Catan, and that is evidence. The third librarian's whole existence depends on the app knowing which game you asked from and having that game's books ingested; when it does, she works your aisle only. She picks up the edition's own rulebook, plus any correction documents matched to your question (publisher errata — the fixes rulebooks accumulate), and ranks each book's passages the way the second librarian ranks the world: by the map of meaning, just never leaving the aisle. Each of her books sends its eight best passages into the room — up to four documents, corrections taking priority if the slots run short, thirty-two candidates at most, standing for election because of where you're standing, even if neither roaming librarian shortlisted them. (The wrench calls her arm doc-priority boost — a name that outlived the double weighting it once described, as you're about to read.)

One law keeps her honest, and it was learned by measurement, not principle. An early prototype turned her up on two knobs at once — double voting weight and a bigger candidate budget — and the measurements from that prototype show what happened: her books flooded the pool and crowded out better passages the two roaming librarians had found. The cure came in pieces: one fair vote, a small budget, and later a guarantee that each of her books gets its own eight slots so one big book can't starve the others. Her ballot counts exactly like everyone else's.

The election

Every arm that runs searches the aisle, and each brings back a ranked list: the two roaming librarians up to eighty candidates each, the aisle librarian up to thirty-two. And here a real problem appears. The arms score in different currencies — one in word-match points, one in map-distance — and averaging those would be like averaging a temperature with a price.

The referee, reciprocal-rank fusion (RRF, the least glamorous hero in this story), solves it by throwing every score away and keeping only positions. Each arm casts one ballot per passage it ranked, worth:

1 / (60 + rank)

The 60 is called k, the fusion constant. It comes from the original RRF paper, and we did not retune it. Rank 1 on an arm's list is worth 1/61, rank 2 is worth 1/62, rank 10 is 1/70, and a passage's fused score is just its ballots added up. That small formula quietly encodes a philosophy:

• Consensus beats enthusiasm. A passage ranked #3 by both roaming librarians scores 2/63 ≈ 0.0317. A passage ranked #1 by one and missed entirely by the other scores 1/61 ≈ 0.0164. The both-#3 passage wins, comfortably. Two independent skills agreeing at "pretty good" outranks one skill shouting "perfect."
• And consensus has a known cost. When one arm is right and the other is systematically blind — that exact name only the word-matcher can see — the lone correct hit gets outvoted by mediocre agreements. That failure is one of the reasons a careful reader sits downstream of this election rather than trusting it.
• The 60 is a humility dial. Without it, rank 1 would be worth 1 and rank 2 only ½, and a single arm's top pick would steamroll everything. Adding 60 flattens the curve so first place barely beats third, and agreement across arms becomes the main way to climb.
• Nomination is real power, bounded by arithmetic. The aisle librarian can put a passage in the room that nobody else shortlisted — but standing alone it tops out at 1/61, which still loses to anything two arms agreed on at #3. Getting on the ballot is her gift; winning still takes votes.

Two honest wrinkles before moving on. The election is not always three arms: the aisle librarian only works when the app knows your game and holds its books, so some asks fuse two arms and some fuse three — and the most a passage can possibly score depends on how many arms voted. Hold that thought; this page owes you a confession about it. And the fused list is cut to its top eighty before anyone deeper ever sees it, so the funnel narrows even when the arms brought more.

The careful reader

Everything so far is fast because it is shallow. Neither roaming librarian ever read your question and a passage together — one matched word forms, the other measured map-distance, and the referee is pure arithmetic that doesn't know what a robber is.

So the fused shortlist goes to the character this article is named for: the re-ranker. Ours is a cross-encoder, a small model named MiniLM that runs on the same rented server that handles your question — trimmed to about 22 megabytes, tiny in a world of multi-gigabyte models, because its job is narrow and it does nothing else. Where the map-maker judged your question and each passage separately, as two dots whose distance it measured, the cross-encoder reads them as a pair, one eye on each, weighing the passage's words against your words as it goes. That lets it catch what distance cannot: a passage soaked in robber-vocabulary that is actually about placing the robber during setup scores poorly against a question about moving it later, because read side by side, the mismatch is plain.

Depth costs time, which is why the machine is shaped like a funnel. You could never afford the careful reader on the whole corpus; you don't have to. The cheap, broad arms and their referee boil thousands of passages down to at most eighty, and the expensive reader spends its attention there. For each candidate it reads a fixed budget of 1,200 characters: about 595 from the opening, the final 600, and a five-character joiner between them. The tail matters, and we know because it once didn't exist. An earlier version read openings only, and a rule that began deep inside a long passage was invisible to it — Monopoly's turn rules start 1,812 characters into a 2,052-character passage, squarely inside the final 600 once a tail existed, and unreachable before. Three live rulings came back "not in the book" — the app's word for that is an abstain — on a question the book plainly answers. Measured, fixed, published; the shape it left behind is read both ends.

The careful reader keeps the best eight. Two small guarantees can override its judgment, both in the game's favour: on an ask where the game's books rode along, the top passage from the rulebook itself always keeps its place, and the top passage from a matched correction document is never dropped — a fix the publisher shipped outranks any ranker's opinion of it. The wrench on every ruling says whether a guarantee fired.

The order is dealt on purpose

The eight survivors go to the answering model with their full text, in a deliberately strange order. Language models reading a long context are like people skimming a stack of papers: sharpest on what sits at the top and the bottom, foggiest in the middle. The research literature calls it lost in the middle. So RuleSage deals the strongest passages to the two ends of the stack and buries the weakest inside. With five passages the deal runs 1st, 3rd, 5th, 4th, 2nd; with the live eight, the same pattern — ends first, middle last.

From there, the answering model — a larger model, one more hop away on a machine we own, exactly as Where your question goes tells it — writes the answer, quoting the rulebook pages it drew from. If the book is silent, it says "not in the book" rather than improvising. That discipline is a different article's story.

The doorman stands off this road entirely

One more model touches your question, and it's worth naming precisely because readers assume it sits on the road above. It doesn't. Questions asked on RuleSage can appear on a public page — a wall of past rulings the next visitor can browse — and before yours does, this model reads it once and makes a single call: clear it to render publicly, or hold it for a person to look at. That is the whole job. It cannot touch your answer, and when it fails, it fails one way: your question waits for the human queue instead of rendering. Nothing is ever published because it failed.

The role is newly filled. Three candidate models sat a pre-registered exam; all three passed all five gates, so the exam qualified everyone and separated no one, and the tie broke, on the record, on measured speed plus a stated preference for the newer generation. The model holding the role as of publication is qwen3.8:27b, and the full audit is its own exhibit, including the different exam the same model failed the same day. Roles are held on evidence here, and the evidence publishes either way.

Count the cast, then. Of everything this page introduced, only four are models, doing four different jobs: a map-maker placing passages and questions among each other, a careful reader scoring the shortlist, an answering model writing from what survived, and a doorman deciding only whether the question itself stands in public. Beside them: one librarian who is a database index, one who is an aisle's worth of privilege, and one referee who is arithmetic. Nothing here is a general intelligence doing everything. It is a workshop of specialists, each holding exactly the role it earned.

The nerd shelf — the numbers, from the shipped configuration

Settings are per-deployment. If a wrench shows you a different figure, you're looking at a differently-configured install, not at this page being wrong.

A confession, with its receipt

For a while, the wrench on every ruling — the panel a reader opens to see these scores — printed the fused-score ceiling as 0.0328, hardcoded. That is the ceiling… for two arms. By then the app could fuse three, whose ceiling is 0.04918, and 149 live rulings already carried scores above the number the panel called impossible. Real hits, labelled off-the-scale, by a constant nobody re-derived when the third arm joined. What a reader saw was quiet nonsense: a score sitting above a line described as unreachable.

An audit of the wrench's own copy caught it. The cure was to derive the number instead of stating it: the panel now computes arms/(k+1) from each ruling's own arm count and prints the formula beside the result — which also repaired all 149 old rulings at once, since the ceiling is computed fresh each time a page renders. When a number can be derived, derive it; a hardcoded truth is a lie with a delay.

And because a confession should name what it hasn't fixed, with what each item costs you: a fourth arm at equal weight can't re-break the ceiling, but a differently-weighted arm still could. The same 60 is written down by hand in two codebases, so the two could drift apart. And one health light on the same panel still assumes two arms, wearing a caveat instead of a fix. The list is shorter than it was. It is not zero.

One footnote for the sharp-eyed: fused scores cross the wire rounded to six decimals, so a passage two arms ranked #1 can print a hair above the exact sum — 0.032787 against 2/61 = 0.0327868… The wrench compares with a tolerance for exactly this reason, and you will meet a live specimen of that number in the last section of this page.

What this machinery gets wrong

Every character above got its weakness named; the pipeline as a whole deserves the same.

• An abstain is not proof the book is silent. When every arm misses, the answering model sees nothing useful and honestly says "not in the book" — indistinguishable, from the outside, from the book truly being silent. The Monopoly turn-rules case above is the house specimen: the book answered; the window was blind; three rulings abstained. The cure was a wider window, and the class of failure is permanent — only its instances get fixed.
• Passages are cut by machine. A rule and its exception can land on opposite sides of a cut, and then no single passage reads as complete. Oversized passages split hardest, and the split shows up as mush on the map of meaning.
• The map is drawn at ingestion. A corrected book redraws it only when re-ingested; in between, the aisle librarian's arm and its guaranteed place are the compensation.
• The careful reader can be wrong too. It reads 1,200 characters, not the whole passage, and it ranks — it does not verify. Everything downstream that checks quotes against pages exists because no ranker, however careful, is a guarantee.

The rest of the seminar

This page walked one road: a question's journey to the right page. Three more roads run through this workshop, and the field guide intends to walk them all, receipts in hand, no dates promised — a register we keep of owed work tracks them. How a rulebook becomes searchable passages, and then a checklist that refuses to say what it cannot prove. What a bench is, why exams freeze before any model sits them, and how model judges are kept honest. And what the machines themselves are: what "runs on our own machine" means, what those tags like Q4_K_M mean, and why reading costs money. If one of those is the piece you want next, say so — the suggestion box is read.

Why you can check any of this

While drafting this page we asked the live app its own opening question — "can the robber go back to the desert?" — and published the result like any other ask. The answer came back in 2.7 seconds, of which 88 milliseconds was the search call: everything upstream of the answering model, arms and election and careful reader together. The funnel is why the deep read is affordable at all; the remaining 2.6 seconds is the answering model writing. The result — "Yes, the robber can be moved to the desert hex," standing on three cited rulebook pages — is ruling 4615, and its wrench shows the journey: three arms fused, ceiling 0.04918 printed with its formula, and a top passage scored 0.032787. You now know how to read that number. It is 1/61 + 1/61 to the wire's rounding — the arithmetic says two of the three arms put that passage first while the third left it unranked. An inference from the formula, not something the wrench displays: per-arm ranks never cross the wire, fusion happens on a machine upstream of the app, and the wrench says so itself rather than leaving the gap silent.

Ask your own question, open the ruling, and read where it came from. The librarians, the election, and the careful reader all leave receipts.

Licence: CC BY 4.0, the whole page — name the source and link to it.