PROOF · proof · Updated 2026-11-05 · Derek William Frazier

What Is Data-Mining Bias in a Backtest, and How Is It Different From Curve-Fitting?

Data-mining bias is the statistical inflation that appears when many different strategies or rule ideas are tested against the same block of history and only the best-performing one gets published. Some of those strategies will look like a genuine edge purely by chance — not because any one of them was tuned or cheated, but because testing enough different things against the same noisy data guarantees a few will come out looking good, the same way enough coin-flippers will produce a few people with an eight-flip winning streak.

That's the answer to the search query. The rest of this page covers the mechanism that makes testing many ideas dangerous even when every single one is tested honestly, how this is different from curve-fitting one strategy's own parameters, why walk-forward testing doesn't close this particular gap by itself, and what actually controls for it.

PLATE · ONE WINNER OUT OF THE BATCH28 honestly-tested ideas, identical history, zero tuning on any of them.every idea tried, same historyflat — no real edge1 of 28 drifted up — picked, publishedthe other 27 were never shown — same honesty, no real edgePicking the best of many is its own bias — even when nothing was cheated to get there.CYCLICAL MARKETS · THE CYCLITECNICAL METHODEducational. Not advice. No performance promise.

How does testing many strategies make one look good by luck alone?

Every backtest run against real market history is really a statistical test against noise, and noise occasionally lines up in a strategy's favor for reasons that have nothing to do with the strategy. Test one honestly-built rule against ten years of data and there's some chance it looks good purely by coincidence. Test a hundred different rules against that same ten years, and the chance that at least one of them looks good by pure coincidence climbs fast — and that one coincidental winner is exactly the one that gets pulled out, polished, and shown around. The other ninety-nine that looked ordinary or bad never get mentioned again.

Nobody needs to touch a single parameter for this to happen. Each of the hundred rules can be tested exactly once, start to finish, with zero tuning — and the bias still shows up, because the act of searching across many ideas and keeping the best one is itself a form of cheating the sample, even when no individual idea was cheated.

Rules testedA rough feel for how likely at least one looks "good" purely by chance
1Low — close to whatever the base rate of a lucky-looking result actually is
10Noticeably higher — a handful of chance winners becomes close to expected
100High enough that finding at least one standout is close to guaranteed
1,000+Near-certain — at that scale, not finding an impressive-looking winner would be the surprise

(Illustrative shape only — the real numbers depend on how the strategies are built and how correlated they are with each other, not a specific claim about any real search.)

How is this different from curve-fitting a single strategy's own parameters?

Curve-fitting — covered in full on what curve-fitting actually looks like in a backtest — is about one strategy getting its own knobs turned until it matches history unusually well. More parameters, more turning, more risk that the fit is matching noise specific to that one stretch of data rather than anything that generalizes.

Data-mining bias is a step upstream of that, and it doesn't need any knob-turning at all. It's the bias introduced by searching across many separate ideas — different entry logic, different instruments, different indicators entirely — and reporting only the one that happened to test best. A researcher can walk-forward test every single one of a hundred candidate strategies with perfect discipline, never touching a parameter after seeing the holdout result, and still end up promoting a strategy that has no real edge, simply because it was the lucky one out of a hundred honest tests. The two problems compound when they happen together — a strategy search that also lets the winner's own parameters get tuned afterward carries both biases stacked on top of each other.

Does a walk-forward test fix data-mining bias?

Only partially, and it's worth being precise about which part it covers. A walk-forward test — choosing a strategy's parameters on one stretch of history and testing that exact, unchanged configuration on a separate stretch it never saw — closes the parameter-overfitting leak inside a single strategy. It does nothing about the separate act of searching across many different strategies and keeping only the one that passed.

Here's the gap concretely: run a hundred different strategy ideas through an honest walk-forward test, each one truly never touching its own out-of-sample stretch during parameter selection. A handful of those hundred will still pass the walk-forward test by chance alone — a lucky strategy doesn't know it's lucky, so it clears the same honest holdout a genuinely good strategy would clear. Picking the one or two walk-forward "winners" out of a hundred attempts, without accounting for how many were tried, reintroduces the exact bias the holdout was supposed to prevent — just one layer further out.

What actually controls for data-mining bias?

Questions traders ask

Does a bigger sample size fix data-mining bias?

No. A larger sample narrows the noise around any one test, but it does nothing about the separate problem of running many different tests and reporting only the best one. A thousand-trade backtest searched across a thousand different rule variations still has the same data-mining exposure as a hundred-trade backtest searched the same way — the fix is accounting for how many things were tried, not how much data each one used.

Is data-mining bias the same thing as cherry-picking a track record?

Related, but they happen at different points. Cherry-picking usually describes a published record that quietly omits real segments after the fact — a losing stretch left out of a public summary. Data-mining bias happens earlier, during the search and discovery phase, before a single real trade is ever placed: it's about how many different ideas were tried before landing on the one being shown, not about what got left out of that one idea's own results afterward.

Can automated strategy-testing software make this worse?

Yes, and it usually does without anyone intending it to. The cheaper and faster it becomes to test another variation, the more variations tend to get tested — a tool that can run ten thousand rule combinations overnight makes the search-and-select bias far easier to fall into than hand-testing ten ideas ever could, unless the total count of everything tried is tracked right alongside the one result that looked best.

Is a strategy found this way automatically worthless?

No — a real edge can absolutely come out of a broad search; the search itself isn't the problem. What matters is what happens next: whether the winner gets re-tested on fresh data it never influenced, and whether the number of things it beat gets disclosed alongside its result. A strategy that holds up again on new data, after an honest search, is in a very different position than one that's simply assumed to be good because it was the best of the batch.

Where this fits the bigger picture: a backtest's real edge, once curve-fitting, data-mining bias, survivorship bias, and sample size have each been accounted for, is a much smaller and more honest number than the raw headline figure a wide search tends to produce. How this site's own graded calls hold up against that bar is on the methodology page.

Keep reading

I Graded Four Years of My Own Public Calls. Here's the Scoreboard.How a Trading Educator Should Score Their Own CallsHow to Verify a Trading Guru's CredentialsMy 2022 Posts Beat My 2026 Posts. Badly.
Get our AI tools and everything we send out New AI trading tools and our Claude Code builds, plus session invites. All free.
Get them free