Do Crypto Scanners Actually Work? We Tested Ours Out of Sample

For almost every crypto scanner on the market, the honest answer is that nobody knows — including the people who built it. Not because the tools are fraudulent, but because the evidence normally offered cannot tell a working scanner apart from a rising market. This post is about what real evidence would look like, and what happened when we held our own scanner to it.

The number that proves nothing

Ask a scanner whether it works and you will usually be shown a win rate. Seventy per cent of signals went up. Sometimes a grid of green screenshots comes with it.

A win rate cannot answer the question, for two separate reasons.

It says nothing about size. A setup that gains 2% when it works and loses 10% when it doesn't is a losing strategy at an 80% win rate. A setup that gains 10% and loses 5% is a winning one at 40%. Until you know what the wins and losses are worth, the frequency of wins is decoration. The measure that survives this is expectancy: the average outcome per signal, wins and losses together, with costs already deducted.

It says nothing about the market. In a rising market almost everything has a high win rate. If a random coin picked by throwing a dart would have reached your target 17% of the time and the scanner's picks reached it 18% of the time, the scanner contributed roughly nothing — and it can still advertise a win rate that sounds impressive. Any number without a baseline beside it is a claim about the market, not about the tool.

This is not a hypothetical failure mode for us. An earlier version of our own scoring weighted setup types by their measured win rate. When those setups were later checked against actual outcomes, the weighting turned out to be inverted — it was rewarding the types that did worse. We zeroed the weights rather than guess at new ones, and they are still zero today.

Four tests a scanner's claim has to survive

If a win rate cannot settle it, what can? Four things, none of them exotic. They are standard practice in any field that has to distinguish a real effect from a lucky sample, and they are almost never applied to retail trading tools.

1. The settings were chosen before the test, not after

This is the one that quietly invalidates most published backtests. If you try fifty combinations of threshold, target and stop, then report the best one, you have not measured a strategy — you have measured how well the data can be fitted after the fact. Run the same procedure on random noise and it will also produce a winner, and that winner will also look convincing.

The fix is to split the history. Choose everything — every threshold, every parameter — using only the earlier stretch. Then run the single chosen configuration once on the later stretch that played no part in the choice. One number comes out. Whatever it says, that is the result.

2. Costs are deducted

Fees and slippage sound trivial until you notice how often a scanner fires. A round trip costing 0.30% is nothing against a 40% winner and fatal to a strategy whose real edge was 0.2% a trade. Any test that reports gross returns is reporting a number nobody could have earned.

3. There is a fair baseline next to it

The right comparison is not zero, it is what would have happened anyway: entering the same universe of coins at random times over the same window. And it needs one more control, because a scanner that fires mostly on days when the whole market is running will look brilliant by accident. Comparing its picks against a random entry on the same day separates "this tool picks the right coins" from "this tool notices when everything is going up", which are very different products.

4. The error bars respect the fact that coins move together

Three hundred altcoin signals in one afternoon are not three hundred independent pieces of evidence. They are close to one, repeated three hundred times, because altcoins move as a herd. Statistics that treat each signal as its own observation produce confidence intervals that are far too narrow, and a result that looks decisive when it is not. Resampling whole days rather than individual trades gives intervals that are wider, uglier, and honest.

A fifth practice belongs here for anything meant seriously: write down the exact test — the metric, the threshold, the cutoff date — before running it, and then run it once. A test you can re-run until it agrees with you is not a test.

What happened when we ran all four on ours

We built the harness and pointed it at our own scanner: 315 Binance USDC pairs on hourly candles, a barrier test on every signal, costs of 0.30% per round trip deducted, parameters chosen on an earlier period and run once on a later one.

On tradeability, it failed. Of the 75 configurations that had enough trades to evaluate, 0 made money on the training period. The best of a bad set was then run on the unseen stretch and returned +1.605% per trade — which sounds like vindication until you notice the confidence interval crosses zero, and that entering at random over the same weeks returned +0.508%. Verdict: no demonstrated edge. We published it.

On finding movers, it did better. Coins the scanner flagged at a high score reached +10% within 24 hours 18.5% of the time, against 8.4% for random entries — more than double. We then tried to explain that away, on the theory that the score simply picks volatile coins and volatile coins hit any target more often. Matching every flagged coin against unflagged coins that looked nearly identical on volatility, momentum and liquidity in the same hour absorbed about half the advantage. The other half stayed. A single pre-registered holdout test of that remainder came back positive but too small to settle it, and the next run of it is scheduled rather than continuous, for the reason in test five.

The full workings, including the parts that came back against us, are on our track record page.

So what is a scanner actually for?

The results above point at a narrower and more useful job than the one scanners usually advertise. There are over 200 USDC pairs on Binance. Nobody reads 200 charts, and by the time you have read forty the first ten have moved. A scanner is a triage function: it decides what you look at. It does not decide what you buy.

Judged as a decision system, ours failed. Judged as an attention allocator — the thing it is actually used for — it beats scrolling a list of gainers, and now there is a measured comparison saying so rather than a testimonial. Those are compatible findings, and a tool that tells you which one it is has told you something worth knowing.

What none of this can tell you

Honesty about method is worth little without honesty about its limits, and there are three that no amount of testing removes.

A test covers the market it was run on. Ours spans about five months of one regime. An effect that shows up in a stretch that short can be one rally wearing a disguise, and no statistical treatment fixes a sample that hasn't lived through a proper bear market yet. Anyone quoting a backtest without telling you which market it covers is telling you half of it.

Reaching a target is not the same as making money. Our discovery result measures whether a coin's price touched +10% within a day. A coin can touch +10% after first falling 30%, and a measure with no stop and no exit rule quietly assumes you were there to take it. That gap between "moved" and "you profited" is exactly where most scanner marketing lives.

The decision that dominates your result isn't the scanner's. Entry selection is a small part of a trade's outcome next to position size and exit discipline. A tool that improves the shortlist you start from cannot rescue an exit plan you don't have — which is the real reason "does this scanner work?" is a less useful question than it sounds, and "what job am I giving it?" is a better one.

Five questions to ask any crypto scanner

Including ours. This is the whole post compressed into something you can use on a landing page in about a minute:

1. Is there a result from data the settings were not chosen on? If the parameters and the proof come from the same period, there is no proof.
2. Are fees and slippage deducted? If the answer is unclear, assume not.
3. What did random entry do over the same window? No baseline, no claim.
4. Are there error bars, and do they account for coins moving together? A suspiciously tight interval on crypto data usually means each signal was counted as independent evidence.
5. Have they ever published a result that made them look bad?

The fifth question does most of the work. A tool that has never published an unflattering number has either never looked, or has looked and not told you. If the answer to it is no, the answers to the other four are marketing.

If you want the mechanics behind the score itself, what makes a good crypto setup breaks down the points and penalties, and screening 200+ pairs without checking charts manually covers the workflow a scanner is supposed to fit into.

← Back to all posts