{
 "schema_version": "1.0",
 "page_kind": "article",
 "pack_kind": "generate",
 "slug": "elo-ratings-explained",
 "title": "Elo ratings, from chess to image models",
 "dek": "An Elo rating turns head-to-head results into one number per player, and only the gap means anything: Black Forest Labs' “gains 91 Elo” for its new image model predicts about 63 votes in 100, from the company's own test, which published no vote counts, voters or prompts.",
 "published": "2026-10-21",
 "series": [
  "guide"
 ],
 "licence": "CC BY 4.0",
 "status": "pending-judge",
 "pack_note": "no judge is seated, so every chip is held",
 "notice": "items with published:false are held and are not this page's published words",
 "source": {
  "url": "https://research.strata2signal.com/elo-ratings-explained/",
  "md_url": "https://research.strata2signal.com/elo-ratings-explained/index.md",
  "md_sha": "27e388de1aa5668e22ba9dbccafd42b137524d8e6cff0948205495d2f64dd098",
  "html_sha": "356b120d57f1459d69922aa8ca53bc86863833933005fe8c992f51825b5dc022",
  "anchors_sha": "965fb182578c980953bb2c0c6c458d1de4f0d71efbc2eaeb1e0cf6a214ddf40b"
 },
 "short": {
  "paragraph": "An Elo rating is a prediction. When two ratings differ by D points, 1 / (1 + 10^(-D/400)) is the higher-rated side's expected share of the points: 50 % at equal ratings, 64 % at 100 points, 76 % at 200. American chess adopted Arpad Elo's system in 1960 and the world chess federation in 1970. AI leaderboards fit the same scale to blind votes between two outputs made from one prompt. Black Forest Labs says its new image model \"gains 91 Elo\" on its predecessor: a gap that predicts about 63 votes in 100 for the new model, from the company's own test, which published no vote counts, voters or prompts.",
  "from_dek": false,
  "counts": {
   "words": 2055,
   "minutes": 9,
   "tables": 2,
   "kit": false
  },
  "bullets": []
 },
 "sections": [
  {
   "id": "what-the-number-means",
   "heading": "What the number on a launch post means",
   "level": 2,
   "span": [
    1633,
    2607
   ],
   "chunks": [
    [
     1633,
     2607
    ]
   ],
   "chars": 974,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "where-it-came-from",
   "heading": "Where it came from",
   "level": 2,
   "span": [
    2607,
    3973
   ],
   "chunks": [
    [
     2607,
     3973
    ]
   ],
   "chars": 1366,
   "digest": "Arpad Elo was a physics professor and chess player who developed a more objective rating scale. The method was adopted by the USCF in 1960 and FIDE in 1970. The mathematics relates to earlier work by Zermelo, R. A. Bradley, and M. E. Terry.",
   "digest_skipped": null
  },
  {
   "id": "how-a-rating-moves",
   "heading": "How a rating moves",
   "level": 2,
   "span": [
    3973,
    5304
   ],
   "chunks": [
    [
     3973,
     5304
    ]
   ],
   "chars": 1331,
   "digest": "A rating moves based on a formula using an expected score and a K factor. The K factor determines how far a result moves a rating. In a worked example, a favourite gains less for a win than it loses for a loss.",
   "digest_skipped": null
  },
  {
   "id": "the-one-formula",
   "heading": "The one formula",
   "level": 2,
   "span": [
    5304,
    6293
   ],
   "chunks": [
    [
     5304,
     6293
    ]
   ],
   "chars": 989,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "how-ai-leaderboards-use-it",
   "heading": "How AI leaderboards use it",
   "level": 2,
   "span": [
    6293,
    8602
   ],
   "chunks": [
    [
     6293,
     8602
    ]
   ],
   "chars": 2309,
   "digest": "AI leaderboards like Chatbot Arena use human votes to rank models. LMSYS moved from an online Elo system to the Bradley-Terry model for more stability. Artificial Analysis also uses a Bradley-Terry model to report results on an Elo scale for image models.",
   "digest_skipped": null
  },
  {
   "id": "what-a-rating-cannot-tell-you",
   "heading": "What a rating cannot tell you",
   "level": 2,
   "span": [
    8602,
    11394
   ],
   "chunks": [
    [
     8602,
     11394
    ]
   ],
   "chars": 2792,
   "digest": "A rating only applies to its specific pool and does not account for the certainty of the result. It depends on who voted and what they saw. An average rating may not reflect performance in a specific use case.",
   "digest_skipped": null
  },
  {
   "id": "our-bake-offs",
   "heading": "Our bake-offs, at a six-table scale",
   "level": 2,
   "span": [
    11394,
    12273
   ],
   "chunks": [
    [
     11394,
     12273
    ]
   ],
   "chars": 879,
   "digest": null,
   "digest_skipped": null
  },
  {
   "id": "what-this-page-does-not-say",
   "heading": "What this page does not say",
   "level": 2,
   "span": [
    12273,
    12631
   ],
   "chunks": [
    [
     12273,
     12631
    ]
   ],
   "chars": 358,
   "digest": "This page does not state whether FLUX 3 Image Klein beats FLUX.2 [klein] 9B, whether BFL used Elo or Bradley-Terry, or if these methods are the right way to rank models for any use.",
   "digest_skipped": null
  },
  {
   "id": "who-ran-this-and-thanks",
   "heading": "Who ran this, and thanks",
   "level": 2,
   "span": [
    12631,
    12910
   ],
   "chunks": [
    [
     12631,
     12910
    ]
   ],
   "chars": 279,
   "digest": null,
   "digest_skipped": "credits"
  },
  {
   "id": "sources",
   "heading": "Sources",
   "level": 2,
   "span": [
    12910,
    14361
   ],
   "chunks": [
    [
     12910,
     14361
    ]
   ],
   "chars": 1451,
   "digest": "The article cites various sources including Black Forest Labs, the World Chess Hall of Fame, Glickman and Jones, FIDE, LMSYS, Arena, and Artificial Analysis, all read on 2026-10-15 or 2026-10-21.",
   "digest_skipped": null
  }
 ],
 "chips": [
  {
   "id": "c-397303bc",
   "q": "how did the elo rating system originate?",
   "a": "Arpad Elo, a physics professor and chess player, developed the scale to make chess ratings more objective and mathematical, building upon previous systems used by the U.S. Chess Federation.",
   "cites": [
    "where-it-came-from"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-a8170632",
   "q": "how does a player's rating change after a game?",
   "a": "A new rating is calculated by adding a product of K and the difference between the actual score and the expected score to the old rating. The K factor determines how much a single result moves the rating.",
   "cites": [
    "how-a-rating-moves"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-e0d7abe6",
   "q": "how is the expected score calculated for a rating gap?",
   "a": "The expected score for the higher-rated side is determined by the formula 1 / (1 + 10^(-D/400)), where D is the point gap. For example, a 400-point gap means the favourite expects to score 10 points for every 1 the opponent scores.",
   "cites": [
    "the-one-formula"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-d0b9935f",
   "q": "how do ai leaderboards transition from elo to bradley-terry?",
   "a": "Some platforms, like Chatbot Arena, moved to the Bradley-Terry model to achieve more stable ratings and precise confidence intervals. This model finds the set of ratings that makes the entire history of votes most likely.",
   "cites": [
    "how-ai-leaderboards-use-it"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-7061da7c",
   "q": "why can elo ratings not be compared across different boards?",
   "a": "Ratings only exist relative to the specific pool of rivals, voters, and prompts used in that evaluation. Different pools of taste and different sets of competitors make the numbers incomparable between different leaderboards.",
   "cites": [
    "what-a-rating-cannot-tell-you"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  },
  {
   "id": "c-bd9065de",
   "q": "what results did the small scale image bake-off show?",
   "a": "The small scale bake-off could not produce an elo rating because a single voter is not a statistical pool. In one specific test, a model ranked above a rival on 3 of 6 subjects and below it on the other 3.",
   "cites": [
    "our-bake-offs"
   ],
   "published": false,
   "publish_state": "held",
   "quote": "",
   "quote_cite": ""
  }
 ],
 "related": [
  {
   "slug": "three-librarians",
   "why": "shares ground with § How AI leaderboards use it · § The election"
  },
  {
   "slug": "ten-minutes-with-living-artists",
   "why": "shares ground with § What a rating cannot tell you · § What we did NOT measure"
  },
  {
   "slug": "where-new-knowledge-comes-from",
   "why": "shares ground with § Where it came from · § The second engine: the rooms the rules imply"
  }
 ],
 "thanks": "## Who ran this, and thanks {#who-ran-this-and-thanks}\n\nThanks to Arpad Elo's work, Mark Glickman's writing about it and the arena teams who publish their methods. A small human team asked for this page; a fleet of AI agents read the sources, did the arithmetic and drafted it.",
 "kit": null,
 "seat_class": {
  "writer": "gemma-class",
  "judge": null,
  "writer_runtime": "vllm"
 },
 "bench": {
  "bullets_written": 3,
  "bullets_kept": 0,
  "digests_written": 9,
  "digests_kept": 6,
  "chips_written": 8,
  "chips_kept": 6,
  "chips_grounded": 0,
  "chips_published": 0,
  "dropped_by": {
   "new_noun": 0,
   "figure": 1,
   "length": 1,
   "cite": 0,
   "judge": 0,
   "quote": 0,
   "directive": 0,
   "redaction": 0,
   "profanity": 0,
   "identifier": 0,
   "bare_figure": 0,
   "house_voice": 0,
   "editorial": 0,
   "hand-edit": 6
  },
  "new_noun_tokens_checked": [
   "1",
   "10",
   "100",
   "1960",
   "1970",
   "2026-10-15",
   "2026-10-21",
   "3",
   "3090",
   "38%",
   "400-point",
   "5.6-billion-parameter",
   "6",
   "63",
   "70",
   "91",
   "91-point",
   "9B",
   "Analysis",
   "Arena",
   "Artificial",
   "BFL",
   "Black",
   "Bradley-Terry",
   "Chatbot",
   "Chess",
   "D",
   "D/400",
   "Elo",
   "FIDE",
   "FLUX",
   "FLUX.1",
   "FLUX.2",
   "Fame",
   "Federation",
   "Forest",
   "GeForce",
   "Glickman",
   "Hall",
   "Image",
   "Jones",
   "K",
   "Klein",
   "LMSYS",
   "Labs",
   "M",
   "NVIDIA",
   "R",
   "RTX",
   "U.S",
   "USCF",
   "World",
   "Zermelo"
  ],
  "new_noun_tokens_withheld": 0,
  "figure_definition": "v3",
  "figure_boundary": "guarded",
  "drops": [
   {
    "kind": "bullet",
    "reason": "figure",
    "item": "the rating system was adopted by the USCF in 1960",
    "detail": "terminal token carries no figure"
   },
   {
    "kind": "bullet",
    "reason": "length",
    "item": "a bake-off was performed on an NVIDIA GeForce RTX 3090 3090",
    "detail": "the prefix already ends in the figure"
   },
   {
    "kind": "bullet",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 21,
    "text_sha": "8c0f7ce7c3b7",
    "figure": "38",
    "withheld": "one bullet removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "a bare 38% with no 'fewer parameters': the figure is Black Forest Labs' own claim, quoted by the page, and the page's own arithmetic gives 37.8 %",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: a bare 38% with no 'fewer parameters': the figure is Black Forest Labs' own claim, quoted by the page, and the page's own arithmetic gives 37.8 %. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 282,
    "text_sha": "7a7f8da8e10a",
    "section": "what-the-number-means",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "'In internal evaluations' is lost from the ranking claim and from the 91 and 70 point gains, so the company's own test reads as a fact; the section's meaning of the gap (about 63 votes in 100) and 'the post does not say who voted' are lost too",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: 'In internal evaluations' is lost from the ranking claim and from the 91 and 70 point gains, so the company's own test reads as a fact; the section's meaning of the gap (about 63 votes in 100) and 'the post does not say who voted' are lost too. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 245,
    "text_sha": "eaaa7601f2c2",
    "section": "the-one-formula",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "'to within about a hundredth' is lost: FIDE's table and the formula differ (at 400 points the table prints .92 against the formula's 0.909)",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: 'to within about a hundredth' is lost: FIDE's table and the formula differ (at 400 points the table prints .92 against the formula's 0.909). Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "digest",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 231,
    "text_sha": "98bd06d68497",
    "section": "our-bake-offs",
    "withheld": "one section digest removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "the section's own condition is lost: one viewer on six subjects cannot produce an Elo rating, and a split of 3 above and 3 below is one that six tables cannot call either way",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: the section's own condition is lost: one viewer on six subjects cannot produce an Elo rating, and a split of 3 above and 3 below is one that six tables cannot call either way. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 239,
    "text_sha": "b35ad82cbef5",
    "chip_id": "c-3db29cb3",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "'blind evaluations' is the phrase of Black Forest Labs' page summary, quoted by the page, and the post does not say who voted or how the images were shown; the chip states it as fact",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: 'blind evaluations' is the phrase of Black Forest Labs' page summary, quoted by the page, and the post does not say who voted or how the images were shown; the chip states it as fact. Not a gate drop, and recorded here so the ledger adds up."
   },
   {
    "kind": "chip",
    "reason": "hand-edit",
    "item": null,
    "item_chars": 225,
    "text_sha": "af93481bb573",
    "chip_id": "c-0da56631",
    "withheld": "one chip removed from this pack by hand",
    "removed_utc": "2026-10-21",
    "why": "'we have not reproduced BFL's evaluation' is lost, and it is the company's own post, not this page, that does not say which kind of fit it ran",
    "detail": "removed from this pack by hand on 2026-10-21 (UTC), after the writer ran and after the gates had counted: 'we have not reproduced BFL's evaluation' is lost, and it is the company's own post, not this page, that does not say which kind of fit it ran. Not a gate drop, and recorded here so the ledger adds up."
   }
  ]
 },
 "generated_utc": "2026-10-21T01:29:50Z"
}