Elo ratings, from chess to image models

If you're new here: strata→signal is a small workshop that builds on its own machines and writes up what it measures. Every outside claim below is quoted from, or linked to, the page or paper named beside it, all read on 2026-10-15 (UTC); the launch post and the Arena and Artificial Analysis pages were read again on 2026-10-21 (UTC). Every number we computed is shown with its arithmetic.

What the number on a launch post means

On 2026-10-15 Black Forest Labs (BFL) released FLUX 3 Image Klein, a 5.6-billion-parameter open-weights image model (5.6 billion counts the image generator only, as BFL reports it). Its launch post says: "In internal evaluations, FLUX 3 Image Klein ranks first in text-to-image, multi-reference generation and image editing, ahead of larger open weights models," and "Compared with FLUX.2 [klein] 9B, it gains 91 Elo in text-to-image and 70 in image editing, with 38% fewer parameters." (5.6 billion is 37.8 % fewer than 9 billion.)

Elo is a surname, not an acronym. A 91-point gain is a claim about preference: in what BFL's page summary calls "blind evaluations", the new model was preferred often enough to rate 91 points higher, a gap that predicts it winning about 63 head-to-head votes in 100. The post does not say who voted or how the images were shown.

Where it came from

Arpad Elo (1903–1992) was a physics professor at Marquette University and a master-level chess player (World Chess Hall of Fame). In the 1950s American chess used the Harkness system; Elo set out to make rating more objective and mathematical (Hall of Fame). He "based his scale on one previously used by the U.S. Chess Federation (USCF)" and added "considerable statistical sophistication" (Glickman and Jones, 1999).

"The method Elo laid out for adjusting ratings was adopted by the USCF in 1960," and FIDE, the world chess federation, adopted it in 1970 (Glickman and Jones, 1999; the Hall of Fame gives the same years). Elo did the calculations himself until the mid-1980s (Hall of Fame) and set the system out in a 1978 book, The Rating of Chess Players, Past and Present (Glickman and Jones call it "his well-known 1978 monograph").

The mathematics is older than the name. Zermelo estimated chess players' strengths from an unfinished round-robin tournament in 1929, and R. A. Bradley and M. E. Terry published a model for paired comparisons in 1952, which Glickman and Jones say "has the strongest connection to the USCF's implementation of the Elo rating system."

How a rating moves

Before a game the system predicts each player's expected score (1 for a win, ½ for a draw, 0 for a loss, averaged over the long run) from the rating gap, by the formula in the next section. After it:

new rating = old rating + K × (actual score − expected score)

K sets how far one result moves a rating. FIDE uses 40 for a player new to its list (until 30 games) and for juniors under 2300, 20 below 2400, and 10 once a player has reached 2400 (FIDE Rating Regulations, 8.3.3). US Chess used 32, 24 or 16, by rating, before its current formula (The US Chess Rating system, 4.2).

A worked update. Player A is rated 1050, player B 1000, and K = 32. A's expected score is 1 / (1 + 10^(−50/400)) = 1 / (1 + 0.7499) = 0.5715; B's is 0.4285.

The favourite gains less for a win than it loses for a loss, and a draw costs it points. With one K for both, what one gains the other loses: ratings measure players only against each other.

The one formula

For a gap of D points, the higher-rated side's expected score is:

expected score = 1 / (1 + 10^(−D/400))

The 400 is a choice of scale: every 400 points multiplies the odds by ten, so a 400-point favourite expects to score 10 points for every 1 its opponent scores (10/11 = 0.909). FIDE converts gaps with a printed table (8.1.2) that agrees with the formula to within about a hundredth:

So BFL's 70 points in editing is an expected score of 0.599 for the new model, and its 91 points in text-to-image 0.628: about 63 votes in every 100, a tie counting as half. The older model would still win about 37.

How AI leaderboards use it

Many AI outputs have no answer key, so leaderboards ask people. On 2023-05-03 the LMSYS research group announced Chatbot Arena, open since about a week before: a visitor chats with two anonymous models side by side and votes for the better one, and "we only use the votes when the model names are hidden" (LMSYS, 2023-05-03). Its first table rated nine models from 4.7 thousand votes with the chess formula and update.

On 2023-12-07, at "over 130,000 votes", the arena moved "from the online Elo system to the Bradley-Terry model, which gives us significantly more stable ratings and precise confidence intervals" (LMSYS, 2023-12-07). The classic update had shown "considerable variability", and unlike chess the arena holds "the entire history of all games for all models", while "most models are static". Bradley-Terry, "the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate", finds the one set of ratings under which all the votes together are most likely, so their order stops mattering.

In the classic update, order matters. Two models start at 1000, K = 32, and each wins once. If A wins first, A goes to 1016 and B to 984; B's win then earns 32 × (1 − 0.4541) = 17.47, ending B on 1001.47 and A on 998.53. Reverse the order and A finishes ahead by the same margin.

The platform, "previously known as Chatbot Arena", is now "Arena (formerly LMArena)" (About) and still ranks models "through the Bradley-Terry rating system", "similar to the Elo rating system" (FAQ).

Image models are rated the same way. The image arena run by the benchmarking site Artificial Analysis has evaluators "see two outputs generated from the same prompt by two different models, without model identities, and select the one they prefer"; its votes come from "validated private evaluators", and "We fit all votes at once with a Bradley-Terry model and report the result on an Elo scale" (methodology). An "Elo" on an AI leaderboard today usually means that. BFL's post does not say which kind it computed.

What a rating cannot tell you

What it means outside its pool. A rating exists only against the others rated with it. Artificial Analysis's text-to-image board holds FLUX.2 [dev] "Fixed at 1000 as the reference point for this board's Elo scale" (leaderboard); there, read on 2026-10-15, FLUX.2 [klein] 9B is 932 and the 4B 840. Those numbers cannot go on BFL's chart, nor BFL's 91 onto the 932: different voters, prompts and rivals make different pools. FLUX 3 Image Klein was not on that board when we read it; the full-size FLUX 3 Image was, at 1114. On 2026-10-21 (UTC) FLUX 3 Image Klein was still not on the board, and FLUX 3 Image stood at 1117.

How sure it is. The same board prints each rating with a 95 % interval and a sample count: 932 ±10 from 10,344 samples for the 9B, 840 ±13 from 6,060 for the 4B, read on 2026-10-15. Three models at 1172, 1166 and 1164 shared the rank range 4 to 6 that day: any of them could be fourth. On 2026-10-21 (UTC) the 9B read 929 ±10 from 10,730 samples and the 4B 837 ±12 from 6,418, each inside its earlier interval, and the models ranked fourth to sixth read 1176 (rank range 3 to 6), 1166 and 1166 (4 to 6 each). Its intervals "may be slightly narrow if votes from the same evaluator or prompt are correlated," the methodology adds. The launch post gives no interval, so no outside reader can say how far 70 or 91 points stands clear of noise.

Who voted, on what. A crowd typing its own prompts (the 2023 arena found "most user prompts are in English") and a screened panel judging a curated set are different pools of taste. A gap measures the people who voted, on the prompts they saw.

Whether anyone can check it. A public arena publishes its method, counts and intervals, and rates every rival by one set of rules. An internal evaluation lets the maker choose prompts, voters, rivals and what to publish: here the larger 9B sibling (the new model is 40 % bigger than the 4B, and no gap is given for that pair). None of that makes the numbers wrong. It leaves them unchecked until someone else measures.

Whether it fits your use. "One model may lead on text rendering, another on complex layouts or photorealistic human characters," says Artificial Analysis's methodology, and "Each use case and capability contributes equal weight to the overall leaderboard rating." An average can rise while the one style you care about falls. Even in chess the curve is only a model: in 225,621 US Chess tournament games between established players, January to October 1997, favourites scored less than it predicted, and the best fit replaced the 400 with 561 (Glickman and Jones), which turns 76.0 % at 200 points into 69.4 %.

Our bake-offs, at a six-table scale

The workshop's image bake-offs are the same idea at a tiny scale: images from the same prompt, shown side by side without their model names, and one viewer picks or ranks them, on six real subjects. That cannot produce an Elo rating. One voter is a pool of one, and six tables cannot separate preference from chance: one given model placed above one given rival on all six tables would happen by chance 1 time in 64, and on five or more 7 times in 64. What a bake-off can do is put the pictures in front of you. Our own blind bake-off of FLUX 3 Image Klein, on one NVIDIA GeForce RTX 3090, came out level: one viewer ranked it above FLUX.1 [dev] on 3 of the 6 subjects and below it on 3, a split that six tables cannot call either way (the bake-off).

What this page does not say

• Whether FLUX 3 Image Klein beats FLUX.2 [klein] 9B: we have not reproduced BFL's evaluation, and a bake-off is not a rating.
• Whether BFL ran game-by-game Elo updates or a Bradley-Terry fit: its post does not say.
• That Elo or Bradley-Terry is the right way to rank models for any one use.

Who ran this, and thanks

Thanks to Arpad Elo's work, Mark Glickman's writing about it and the arena teams who publish their methods. A small human team asked for this page; a fleet of AI agents read the sources, did the arithmetic and drafted it.

Sources

All read on 2026-10-15 (UTC); the Black Forest Labs post, Arena's two pages and Artificial Analysis's two read again on 2026-10-21 (UTC).

• Black Forest Labs, "FLUX 3 Image Klein", 2026-10-15: https://bfl.ai/blog/flux-3-image-klein
• World Chess Hall of Fame, "Arpad Elo": https://worldchesshof.org/hof-inductee/arpad-elo
• M. E. Glickman, A. C. Jones, "Rating the Chess Rating System", Chance 12(2), 1999, pp. 21–28; the authors' copy: https://www.glicko.net/research/chance.pdf
• M. E. Glickman, "Parameter estimation in large dynamic paired comparison experiments", Applied Statistics 48(3), 1999, pp. 377–394: https://www.glicko.net/research/glicko.pdf
• M. E. Glickman, T. Doan, "The US Chess Rating system", 2026-04-06: https://www.glicko.net/ratings/rating.system.pdf
• FIDE Handbook, "FIDE Rating Regulations effective from 1 March 2024", 8.1.2 and 8.3.3: https://handbook.fide.com/chapter/B022024
• LMSYS, "Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings", 2023-05-03: https://lmsys.org/blog/2023-05-03-arena/
• LMSYS, "Chatbot Arena: New models & Elo system update", 2023-12-07: https://lmsys.org/blog/2023-12-07-leaderboard/
• Arena: About (https://arena.ai/about) and FAQ (https://arena.ai/faq)
• Artificial Analysis: image benchmarking methodology (https://artificialanalysis.ai/methodology/image) and the AA-Image-T2I v2.1 leaderboard (https://artificialanalysis.ai/image/leaderboard/text-to-image)