A player who is far better than their rating suggests, in a game where matches are played for money, has an obvious move: lose on purpose often enough that the rating never catches up, and keep farming beginners. Every rating system in wide use scores you on wins and losses, and there is a rate of losing on purpose - a tank rate - that makes that record exactly what an honest player at that rating would produce - not approximately, not eventually, exactly. No test looking at results alone can ever separate the two. But they are not there for the rating. They are there for the money, and every match they have to decide how much to stake before the result is known - and unlike everyone else at the table, they have already decided whether they are trying. This page builds a detector out of that, breaks it twice on purpose, and then stops guessing what a cheater looks like and learns it from the population. No statistics beyond what the page explains as it goes; the simulations run live in your browser, and the two figures that needed an offline model fit say so.
A smurf is an experienced player on a fresh account, stomping beginners for fun. It's annoying, matchmaking eventually sorts it out, and the cost is a bad hour.
Change one thing - make the matches wagered, for real money - and the same behaviour becomes a business. Now the incentive is not to enjoy easy games but to stay in easy games, indefinitely, because the low bracket is where the profitable opponents are. That means actively managing your rating downward - losing on purpose, tanking, at whatever rate keeps the number where the money is. The industry word for this player is a shark, and unlike a smurf they have every reason to be patient and unremarkable.
To see why they get away with it, you need ninety seconds on how the scoreboard works.
Nearly every competitive rating in use descends from Elo - invented for chess by Arpad Elo in 1960, and quietly running ranked ladders, esports and even AI leaderboards ever since. It does exactly two things.
Before a match, it converts the gap between two ratings into a predicted win probability: E = 1 / (1 + 10^(-gap/400)). Equal ratings predict 50%. A 400-point favourite is predicted to win 91% of the time. The curve flattens toward the extremes but never quite reaches 0% or 100% - no rating gap makes an upset impossible.
After the match, it pays out surprise. Each rating moves by K · (result − E): a heavy favourite who wins collects almost nothing, because the result was already priced in, while an underdog who wins collects a lot. K is just the volatility dial - how hard one result yanks the number.
The property everyone trusts is that this is self-correcting. Rated too low, you keep beating the prediction and climb; rated too high, you keep disappointing it and fall. But notice what the correction audits: how often you win, against how often you should. Nothing else. So it self-corrects only against players who are trying to win - a player who controls how often they win controls their rating, and the correction machinery becomes their parking brake. (I've written a longer interactive guide to Elo if you want to play with the curve itself.)
Drag the tank rate - the fraction of matches deliberately thrown. Watch the rating go flat.
At a tank rate of zero the rating runs away and the shark loses their farm within fifty matches. Push the slider and the climb flattens. Press Solve for invisible and it stops dead.
That setting isn't found by search. Write e for the win probability the shown rating predicts and e_delta for the one the player's real strength predicts. Throwing a fraction tau of matches gives an observed win rate of (1 - tau) · e_delta + tau · 0.02. Set that equal to e - the record an honest player would produce - and solve for tau. The answer, which we'll call tau*, the pinning rate, is one line:
# sharkhunt/elo.py # expected: the win probability Elo assigns from the two shown ratings # e_delta: that probability re-priced as if the player were delta stronger # tank_win_prob: even a thrown match gets won 2% of the time - # a loss can be made very likely, never guaranteed def pinning_tank_rate(expected, delta, tank_win_prob=0.02): e_delta = shift_expectation(expected, delta) return (e_delta - expected) / (e_delta - tank_win_prob) # +400 Elo of hidden skill, even matchup: tau* = 0.460 # At tau* the observed win rate is 0.500 - the honest number, exactly.
This is worth being precise about, because it is stronger than "the test is slow". Any test built on results watches them arrive one at a time and accumulates evidence match by match. At tau* the evidence per match is zero, so a million matches accumulate a million zeros. There is no sample size at which the outcome record separates these two players, because there is nothing in it to separate.
The textbook tool is a sequential test: rather than gathering a fixed pile of data and analysing it once, you score each match as it arrives and stop the moment the evidence is decisive, whichever way it points. The classic version is Wald's sequential probability ratio test, from 1945 - the same machinery computer-chess engine testing uses to decide whether a new engine is genuinely stronger, rather than just ahead so far.
It needs the accusation and the innocence stated precisely enough to price any result. H0, innocent: their skill is exactly what their rating says. H1, guilty: they are 400 Elo stronger and throwing 40% of their matches to hide it. Because both are fully specified, each can say precisely how likely any given result was.
That makes the scoring mechanical. After every match, ask both hypotheses how probable that result was, and add log( P(result | H1) / P(result | H0) ) - the log-likelihood ratio - to a running total. Concretely, take an even matchup: H0 prices a win at 50%, while H1 - stronger, but throwing 40% of the time - prices it at 55%. A win contributes log(0.55 / 0.50) = +0.10, a nudge toward guilt; a loss contributes log(0.45 / 0.50) = −0.11, a nudge toward innocence. Every match nudges the total one way or the other, and the total is the case file.
The total wanders like a random walk, and two horizontal lines bound it: cross the top and you accuse; cross the bottom and you clear the player and stop watching. Wald's result is that you can place those lines directly from the two mistakes you're willing to tolerate - α, the chance of ever accusing an honest player, and β, the chance of clearing a real shark. Accepting 1% of each puts the lines at ±4.60 (the general recipe is log((1-β)/α) and log(β/(1-α))).
Press Play. Twenty honest players, one shark who doesn't bother hiding, one who does.
The unconcealed shark gets caught every time, in a median of 56 matches. That is the test working. The honest players wander and mostly settle out the bottom, correctly cleared.
Now look at the amber line. It doesn't climb, and it doesn't hover either - it drifts down, the same direction as the honest players, and eventually exits through the "this player is fine" boundary. Over 2,000 simulated players, an outcome-only test caught 0.0% of sharks tanking at tau*, and actively exonerated most of them.
That's the worst possible failure. Not an alarm that doesn't ring - an all-clear.
Everything above only looks at who won. But in a wagered game there is a second number attached to every match, and the shark has to choose it before the result.
Not the result. This matters, and it is easy to skate past: being better is not the same as winning. Hiding 400 Elo means winning about 91% of the matches you genuinely try to win - not all of them. And a match you are actively throwing still gets won 2% of the time, because your opponent can blunder badly enough that you win despite yourself. Over a hundred matches, our shark takes about five losses they never intended and backs into one win they didn't want.
What they do control is intent - try, or throw - fixed before the bet is placed. So the stake never reveals the outcome. It reveals the intent, and the intent only leans on the outcome.
Stated properly, then, the asymmetry is this. An honest player's stake cannot correlate with their result at all, because nothing they know beforehand predicts it. A shark's stake correlates with their result imperfectly but relentlessly. That gap is the entire signal, and its imperfection is why no single match ever proves anything - a big bet that loses is genuinely something a shark does, roughly one time in eleven. It is why this has to be a test that accumulates, and why it takes a couple of dozen matches rather than one.
Before using it, the ground rules for the money on this page - every profit number later depends on them. A bet is one of three sizes, tiers min, mid and max, at 10%, 40% and 100% of the table cap. Both players post the same stake, the winner takes the pot, and the house keeps a 5% cut of it. The cut means a fair match is slightly worse than break-even: an honest player, matched fairly, loses money slowly. That slow loss is the baseline any cheat has to beat before the cheating pays for itself.
Now the obvious way to use the stakes: keep the same test, and scale each match's evidence by the size of the bet. Big bet, big evidence. Tiny bet, ignore it. Call it the wager-weighted score.
# sharkhunt/detectors.py - the intuitive first attempt class WeightedLLR(OutcomeSPRT): def increment(self, obs): return super().increment(obs) * STAKES[obs.tier] / max(STAKES)
obs.tier is which of the three bet sizes the player posted; the multiplier rescales that match's outcome evidence by the stake.It works, in the sense that it fires. The shark who was invisible a moment ago now gets caught about 55% of the time. That feels like progress, and it is where a lot of real systems stop.
It shouldn't be. There are two things wrong with it, and one of them is not a matter of taste.
Multiply the terms of a log-likelihood ratio by an arbitrary weight and the result is no longer the log of any probability ratio. Wald's thresholds were derived for a quantity this isn't, so ±4.60 no longer buys a 1% error rate - it buys an unknown rate that depends on how the player bets.
That sounds like pedantry until you measure it. Below, "resolved" means the test reached a verdict either way within 400 matches, on players who are all honest. One of them is a whale - the casino word for a player who always bets the table maximum, honestly, because money simply isn't their constraint.
A minimum-bet loss is exactly what a shark throwing a match cheaply looks like. The wager-weighted score multiplies that match's evidence by 0.1, then by an outcome term that is itself near zero at tau*. It scores the single most suspicious event in the dataset as nothing at all.
The weighting was a patch bolted onto the old test. The real fix is to change what one observation is. Until now the detector saw one thing per match - won or lost - and the bet came in from the side as a multiplier. Instead, make the observation the pair: which of the three tiers they bet, and whether they won. Three tiers times two results gives six possible observations, and each hypothesis now has to put a probability on all six. That is what a joint distribution is - "joint" because it prices the bet and the outcome together, so it can express a relationship between them, not just their separate frequencies.
And the relationship is exactly where the two hypotheses disagree. Under H0 the stake can't know something the player doesn't, so bet and outcome are independent: the probability of any pair is just how often they post that tier, times how often they win. Under H1 the pair shares a hidden cause - the shark's private decision, made before betting, about whether this is a match they intend to win. That one decision pushes the bet and the result in the same direction, which is the correlation an honest player has no way to produce.
Writing it down takes three ingredients, each just a short list of probabilities: pi, how often this player posts each tier when nothing is going on; phi_play, how a shark bets on matches they mean to win; phi_tank, how they bet on matches they mean to throw. For now every one of these is assumed - plausible values I wrote down by hand. Keep that discomfort warm; it becomes the final act.
# Bernoulli(y ; p) just means: y is a win with probability p, a loss otherwise # H0: skill equals rating, so the bet says nothing about the result P(t, y | H0) = pi[t] * Bernoulli(y ; e) # H1: a private choice drives both the bet and the result P(t, y | H1) = tau * phi_tank[t] * Bernoulli(y ; 0.02) + (1-tau) * phi_play[t] * Bernoulli(y ; e_delta) # Both sum to 1 over the six cells, so the ratio is a real # likelihood ratio and the thresholds mean what they say again.
The consequence is worth looking at directly. Below is the evidence each of the six observable events contributes, under this model and under the weighted score. The values are in the same units the running total accumulates - and the accuse line sits at 4.60, so an event worth +0.67 carries about a seventh of a conviction on its own.
In an even matchup against a shark hiding 400 Elo, a minimum-bet loss is worth +0.67 - the second most incriminating thing a player can do. The weighted score values that same event at 0.00.
And a minimum-bet win is worth −1.72, strongly exonerating, because sharks don't win the matches they staked nothing on. There is no way to express that with a multiplier. A multiplier can shrink evidence toward zero; it can't flip its sign.
The model above needs pi, the player's distribution over bet tiers. The easy thing is to use the field's average. Do that and the detector catches 100% of sharks in a median of 13 matches, which looks like a triumph until you check who else it caught.
A whale - honest, just rich, always bets the table maximum - never makes the small bets the field constantly makes. Match after match they look very slightly more like H1 than H0. A very slight bias, accumulated sequentially, convicts 100% of them.
The fix is to stop assuming everyone bets alike and learn each player's pi from their own bets. Done naively, that breaks at both ends: a brand-new player has no history to learn from, and a patient shark can manufacture a convenient one. So the estimate is stabilised the standard Bayesian way - start every player with a handful of phantom matches whose bets follow the population average, then let their real matches gradually outvote the phantoms. The formal name for this is a Dirichlet prior; the arithmetic is one line:
# the bet profile the detector holds for one player, n matches into their career P(next bet = tier t) = (count[t] + strength · pop[t]) / (n + strength) # count[t]: how often THIS player has posted tier t so far # pop[t]: how often the whole field posts tier t # strength: how many phantom matches the population average is worth
Switch between players. Every detector, same match stream, side by side. The tilter is the interesting honest case - a player who chases losses, betting bigger after a defeat. Their stakes genuinely do correlate with results, just with the previous one rather than the coming one: real correlation, innocent direction.
The population-profile detector and the per-player one behave identically on the shark. On the whale they could not be more different: one convicts, the other clears them decisively.
The number of phantom matches - the prior's strength - is now the thing to get right, and both extremes fail in opposite directions.
Trust each player's own history too much and their few early matches become their whole standard, so ordinary honest players get accused on noise - 12.5% of them at the far left. Trust the population too much and you are back to judging the whale by other people's habits, up to 53% at the far right. Everything from about 4 to 16 holds sharks at 100% with neither failure mode.
That is a comfortable window, and it is also a number somebody chose by looking at a chart. Hold that thought.
Everything so far fixes H1 at one specific shark: 400 Elo hidden, throwing 40% of matches. Point that detector at somebody who hides 400 Elo and throws none - the crude version from the very first widget - and it catches only 86% of them, because their bets carry no trace of the tanking the hypothesis is looking for.
The outcome-only test catches that player every time. So the two detectors have complementary blind spots, which is an uncomfortable place to leave a system whose whole job is not having blind spots.
Stop assuming one shark. Make H1 a mixture over a grid of them, and average:
# "Some kind of shark, weights w over the kinds" - average the fifteen # six-cell tables. The weights sum to 1 and each table sums to 1, so the # average still sums to 1: the ratio remains a genuine likelihood ratio. P(t, y | H1) = sum_k w[k] * P(t, y | H1 ; delta[k], tau[k])
That closes the hole. Both shark styles are now caught at 100%, false positives are unchanged, and on the blatant shark the mixture is not merely as good as the outcome test but four times faster - a median of 13 matches against 56 - because it gets to use the bets as well. It costs a little sharpness on the awkward middle cases, which the table below reports rather than hides.
The tank rates. The hidden-skill grid. What a shark's bets look like when they mean to win. The prior strength I picked off a chart two sections ago. All chosen by hand, all defensible, none of them measured.
They don't have to be, because all of them can be learned from the population's match records - with no labels, since nobody knows who the cheaters are. This is the part the results hang on, so it is worth slowing down for. The trick is to write a model that tells the story of how the whole season's data came to be, with every number I was choosing by hand promoted to an unknown:
Each player is secretly one of two kinds, decided by a weighted coin flip. The weight - the fraction of the population that is cheating, prevalence - is unknown. An honest player gets personal betting habits pi drawn from around a population average (unknown), held together by a phantom-match strength (unknown - the very dial I was setting by eye two sections ago), and their matches follow the honest half of the six-cell model. A shark instead draws a strategy - how much hidden skill, what tank rate - from the fifteen-cell grid with unknown weights, plus the betting habits phi_play and phi_tank (both unknown), and their matches follow the shark half.
Nothing per-player survives into the fitting. The secret coin and the personal habits would be thousands of parameters, but both can be summed away exactly. The coin has only two outcomes, so each player contributes one quantity: the probability I am honest and produced this record, plus the probability I am a shark and produced it. And averaging over every possible betting habit consistent with a player's bet counts has a closed form. What remains is roughly twenty numbers describing the whole population.
Those twenty are fitted by MCMC - a guided random walk through parameter settings that lingers wherever the data is well explained, so time spent at a setting is proportional to how plausible it is (the specific walker is NUTS, PyMC's default). What comes back is therefore not one best answer but a posterior: a cloud of parameter settings, each weighted by how consistent it is with everything observed. The width of the cloud is the honesty - it says how much the data actually pins down.
# sharkhunt/hierarchical.py - the story above, as PyMC sees it z[p] ~ Bernoulli(prevalence) # the secret coin: shark or not pi[p] ~ Dirichlet(strength * pop_profile) # honest habits, near the average # neither z nor pi is ever sampled - each is summed out exactly, # leaving one term per player that covers both possible identities: pm.Potential("population", logsumexp( [log1p(-prevalence) + ll_honest, # "I'm honest and produced this record" log(prevalence) + ll_shark], # "I'm a shark and produced this record" axis=1).sum())
Nobody tells this model what a shark looks like. No labels, no confirmed cases, no examples - its only foothold is structural: one subpopulation's bets are independent of their results, and another's are not. From that alone it has to discover, in an undifferentiated pile of match records, that the second group stakes heavily on the games it means to win and minimally on the ones it means to throw.
The bars below are posterior means - the centre of the fitted cloud - drawn over the true values the simulation was actually run with. Being able to check against the answer key is the one luxury of synthetic data.
The prior strength is the one to check against yourself. I picked 4 to 16 off a chart a few sections ago; the posterior puts it at 3.1, just outside the window I'd talked myself into. Not a dramatic miss, but it is the kind of thing that stays wrong forever when the number is a constant in a config file rather than a parameter with data behind it.
A new title has fifty players and three weeks of history. Fit the model there from scratch and something worse than imprecision happens.
An unlabelled mixture doesn't know which of its two components is supposed to be the cheaters. Nothing in the likelihood says "the shark group is the guilty one" - the maths is perfectly happy to decide that the cautious players are the anomalous subpopulation and score everyone accordingly. With a large, rich dataset the honest explanation wins on sheer weight of evidence. With a small one, it often doesn't.
Below, the new game has a deliberately different crowd - far more cautious, almost no whales - subtler sharks who hide less skill and let their bets track their intent only loosely, and just 40 matches per player. The score is AUC: draw one shark and one honest player at random, and ask how often the model ranks the shark as the more suspicious of the two. 1.0 is a perfect ranking, 0.5 is a coin flip - and below 0.5 the ranking is actively backwards. The prior carried over from the established title is doing something more fundamental than adding data.
The cold-start line doesn't climb, it thrashes. At two of the four sizes it lands below a coin flip - it found the mirror-image answer and ranked the cheaters as the safest players on the platform. More players don't fix it, because the problem isn't a shortage of evidence; it's that two very different explanations fit the same data about equally well and nothing tells the sampler which one we meant.
The transferred prior settles that. It is not extra data, it is the statement this is what a cheat looks like, carried over from a game where we had enough evidence to learn it. The local data then adjusts the details.
Transfer isn't the only way to break that symmetry - an ordering constraint on the components would do it, and so would a handful of confirmed cases to anchor the labels. It is the one that doesn't require you to already have caught somebody.
A shark reading this far knows the counter. The detector keys on the correlation between stake and outcome, so break the correlation: bet the same way regardless of intent. The tell disappears and detection falls to the false-positive floor.
It also means posting full stake on the matches they are about to throw on purpose. And tanking is not optional - it is the whole mechanism keeping them in the soft bracket. At the pinning rate they must throw 46% of their matches. Do that at full stake and the arithmetic stops working.
Fully correlated, the shark clears +0.223 per match and is caught every time. Fully decorrelated they are invisible - and make −0.017, against the −0.022 an honest player pays to the house cut. They have spent a great deal of effort to earn half a percent of a stake more than someone who isn't cheating at all.
Bet discipline isn't their only lever. They could instead hide less skill - sit 100 Elo above their rating rather than 400 - and let chance do the concealing for them. It genuinely works: at 100 Elo of hidden skill they win only 64% of the matches they try to win, so a big bet that loses stops being surprising, and the evidence per match gets much weaker.
Hiding less buys time, not immunity. It stretches survival by about two and a half times - and costs roughly two thirds of the edge to do it. Below about 200 Elo of concealed skill the arithmetic has already stopped being worth the trouble, and that is before counting the effort of maintaining a second identity and throwing every other match on purpose.
So there are two dials, and both are priced. There is no setting on either curve that is both profitable and safe. That's the actual deliverable: not a ban hammer, a tax that scales with how hard you press.
The wager evidence resolves a case in a median of fourteen matches. That is far quicker than a rating converges, and it is not zero - which leaves the shark one obvious move. Burn the account and open a new one. New accounts start in the middle of the distribution because nothing in the system has any reason to think otherwise, and the fourteen-match clock restarts in a fresh set of beginner lobbies.
I don't think that's unfixable, and it isn't fixable with anything on this page. It needs a completely different kind of evidence: not what happened in the match, but how the player moved - which is visible immediately, and which a rating system throws away.
That turns out to be its own project, with its own uncomfortable questions, so it gets its own writeup rather than a hurried final act here. The short version: you can train reinforcement-learning agents to a measured ladder of skill before your game has any players at all, and use them as the labelled data a behavioural classifier needs. Whether an agent's blunders look anything like a beginner's is exactly the part worth being sceptical about, and that's the experiment.
You need wager data joined to match data, which most skill-gaming platforms already have and most traditional games don't. Without stakes attached to matches, none of this applies - the entire method is built on a number that only exists when there is money on the table.
You need somewhere to put the accusations. A detector that produces a ranked list and no review process is worse than no detector, because it launders a statistical judgement into an operational one.
The population fit is genuinely offline: minutes of MCMC over the whole player base, re-run occasionally. The serving path is a running sum per player per match, which costs nothing. The expensive part is not compute, it's the review queue and the honesty to treat scoring your own players as the surveillance question it is.
A cheat can control what the scoreboard says about them. They cannot control what they need from it. Point the test at the thing they came for and their discipline stops helping.
Shark hunt · Orbitope