A new account has no results, so ranked games guess — and the first few matches are miserable for everyone while they do. You could instead read skill from how somebody plays. But to learn that reading you need labelled players, and a game that hasn't launched has none. So: train a ladder of bots from hopeless to expert, learn the mapping from their behaviour, and point it at people. The idea works. The hard part was that four separate measurements each told me it didn't — or that it did, for a reason that turned out to be false. Every one of them is playable below.
I built two games as simulators I could train agents inside: a Tetris sprint (clear 40 lines as fast as you can — a ranked mode on TETR.IO, a big online Tetris platform) and Minesweeper. Both are real games with real public leaderboards, which matters — the whole point is to check the bots against actual humans at the end.
The recipe is simple enough to state in one line. Make agents of varying skill, record how they play — how fast, how efficiently, how often they set up the big clears — then fit a model from behaviour to skill. Apply it to a human and you have a rank estimate from their very first game, with no match history at all.
Then you check it against 396 real Tetris players whose ranks you already know. That check is where things got strange.
Here is the scoreboard I got. Each method predicts an ordering of 396 ranked players from their play alone, and the score is a rank correlation: 1.0 means the predicted order matches their real ranks exactly, 0 means it might as well be random.
| method | score |
|---|---|
| just sort players by pieces-per-second | 0.932 |
| sort by keystrokes-per-piece | 0.773 |
| a model fitted on the humans themselves, scored on players it never saw | 0.923 |
| my agent-trained model | 0.660 |
That conclusion was wrong, and the reason is easy to miss.
Which is exactly the thing you don't have at a cold start. I had let the baseline read the answer key and then scored the honest method against it.
And look at what the agent-trained model actually learned, from zero human labels: put all the weight on pieces-per-second, ignore the other four features. It found the right answer on its own. That's not losing to the baseline — that is the baseline, derived without the labels the baseline needs.
So the honest question isn't "does it beat the best predictor?" It's how much labelled data would you need to do this well without it? Drag the slider.
At five labelled games the simulator is worth about 0.10 of correlation. At three hundred it's still ahead. That's the actual result, and it only appears once you stop asking the wrong question.
Before you compare bots to humans, you have to confirm the bots are even in the same neighbourhood — that they produce the range of behaviour people produce. For each bot I measure five behaviours: speed, keystroke efficiency, quad rate, use of the hold slot, and back-to-back streaks (quads chained without a lesser clear between them). My check asked whether each bot's numbers fell inside the human range. It came back 100% inside on three of the five.
All three were false. Here's the one that gave it away: quad rate, the fraction of your lines cleared four-at-once. It's the signature move of a good Tetris player.
Human quad rates span nearly the whole 0-to-1 range, so a bot sitting at 0.01 is technically "inside the range" — and simultaneously below every human alive. A range check cannot see the difference between covering a distribution and huddling at the bottom of it.
Asking "which human rank does this look like?" instead exposed it in one line. And the fix wasn't subtle either: my bot's scoring function paid it for any line clear, so it cashed in singles the moment it could and never stacked four rows. Teaching it to hold out for quads moved its quad rate from 0.01 to 0.61.
Next metric. For one bot, do all five of its measured features agree about how good it is? If its speed says "expert" and its stacking says "beginner", it isn't imitating any real person.
I scored my ladder: eight ranks of disagreement, on an 18-rank scale. Damning. I started rewriting the bot.
Then I ran the identical test on the humans.
The number wasn't a grade. It was a thermometer I'd never calibrated, and I nearly rebuilt a working system because of it.
What did survive the comparison was narrower and much more useful: four of the five features sat inside the human spread, and exactly one didn't. My bot used the hold slot ("save this piece for later") like a beginner while playing like an expert everywhere else. Fixing that made it better at the actual game, not just better at resembling a person.
To build a ladder of bots you need a knob that makes them worse by degrees. An obvious one: train a bot to copy an expert, and stop training early. A half-trained copy should play half as well.
It doesn't. Drag the accuracy slider — this is a perfect player whose decisions I corrupt at a known rate, so nothing else varies.
A ladder needs a knob whose middle settings produce middle play. Imitation accuracy isn't one, because a sprint is a hundred decisions in a row and one bad one ends it. The knob that did work changes what the bot is trying to do — expert bots hold out for quads, beginner bots grab any clear going — which degrades smoothly all the way down.
A footnote I enjoyed: a trained-but-imperfect bot is worse than a randomly corrupted one at the same error rate. Random mistakes cancel out. A model's mistakes are systematic — it's wrong the same way on similar boards, so its errors compound.
Here's why my agent-trained model got confused in the first place. I built the Tetris ladder with a single skill knob, so everything moved together: faster bots were also better stackers, better planners, tidier with their keystrokes. Real players don't work like that — there are fast sloppy players and slow careful ones.
When every feature moves in lockstep, a model can't tell which one matters. Mine gave pieces-per-second a negative weight, in a population where speed is the strongest predictor of skill there is.
The second game was built with three independent knobs instead. Same plot, both games:
Left: a diagonal smear. Speed and judgement are the same number in different clothes, so a model fitted on it learns a relationship that falls apart on humans. Right: a cloud. You can be fast and careless, or slow and careful, independently — like people.
The deep problem with the Tetris sprint is that it's a time trial. Rank correlates 0.93 with raw speed, so there was never room for a cleverer method to add anything. I picked Minesweeper next for one reason: it has a skill measure that logic can settle.
After any position, a solver can determine which hidden squares are provably safe. So for every click a player makes you can ask: was there a guaranteed-safe square available, and did you take it? That's a judgement score, it has nothing to do with speed, and it's computable for bots and humans identically.
Play a few clicks. Hold the hint button to see what the solver knows.
My best Minesweeper bot reveals a provably-safe square 83% of the time and clicks a known mine 1% of the time. Turn its "lapse rate" up and blunders climb to 19% while its speed doesn't change at all — two independent axes, which is exactly what the Tetris ladder failed to produce.
I spent two million steps of reinforcement learning on Minesweeper and got a bot that never won a single game. It learned something, though: it learned to never click anything. With 99 mines hidden among 480 squares, an uninformed click is bad enough that flagging squares forever is genuinely the better strategy. It found the optimum of the reward I gave it, and the optimum was cowardice.
What worked instead was changing not the reward but the amount of information per step. The simulator knows where its mines are. So instead of one thumbs-up-or-down per click, give the network the answer for all 480 squares at once and ask it to predict them.
The rule I came out of Tetris with was: before building a simulator for a game, check how much of rank one obvious statistic already explains. The next candidate was Tetris versus mode — two players, garbage attacks (your line clears dump junk rows on the opponent), and a rank that comes from a rating fed by match outcomes rather than any single number. Exactly the property the sprint lacked. I already had the data.
So I ran the screen before writing any of it.
A rating built from wins and losses didn't help, because the thing that rating measures is still mostly speed. Building the versus simulator would have inherited the exact ceiling that made the first one uninformative.
That one sentence disqualifies whole genres without downloading anything. Typing sites rank you by words-per-minute, so rank and WPM are the same thing wearing different hats. Rhythm games compute your rating as a formula over accuracy and chart difficulty. The entire speed-and-accuracy family fails identically.
It also lands on my own remaining hope. Minesweeper's leaderboard sorts by time — and its speed stat, 3BV/s, is just board value (the minimum clicks a board needs) divided by time. Same trap, third time. The label with real room in it is win rate: surviving an expert board is mostly about not guessing, while time is mostly about clicking fast. So the thing to ask a data holder for isn't the leaderboard — it's win rates, and the games people lost.
I want to be straight about what this does and doesn't show, because the temptation to round it up is strong.
What holds up: a model trained purely on bots was worth more than 300 labelled human games on the one test I could run. That's a real result and it survives the corrected comparison.
What doesn't: that test ran on a single game, and that game turned out to be nearly the worst possible choice — a time trial whose ranking is 93% explained by one obvious number. There was no headroom for any method to fill. I picked it because I could get the data, which is not the same as picking it because it could answer the question.
Minesweeper is the game where the question can actually be answered, and everything on the bot side is finished and waiting. The human half is stuck: the site's data isn't reachable without hammering their servers in ways I'm not willing to, and my email asking nicely hasn't been answered yet.
Stack the three requirements together and the squeeze is obvious. A game has to be simulatable to build bots for. It has to have headroom above its own obvious statistic. And its human data has to be reachable. Public sources fail at least one every single time: games rich enough to have interesting telemetry — shooters, MOBAs, card games — are far too complex to simulate honestly, while games simple enough to simulate are simple because their skill is thin.
A game studio dissolves all three at once. They own the simulator. Their ranks come from matchmaking ratings rather than leaderboard times, so the labels are latent. They record losses, not just records. And access is a conversation instead of a scrape. They also have the motive, since cold start is their problem — new players quitting in the first week is a number they already track.
So the honest framing isn't "here is a solved method". It's: here is the method, the toolkit, and — most usefully — the one-script screen that tells you in minutes whether your game has any room for it.
It's a number that looks decisive and points the wrong way. Every trap on this page produced a confident, plausible, entirely wrong conclusion — and each one was caught by asking what the measurement reads when nothing is wrong.