How we test trading strategies

The word "backtest" gets thrown around loosely. Here's what our pipeline actually does — step by step — and why each step matters.

The core problem

A standard TradingView backtest answers the wrong question. It tells you whether a rule profited on the data it was designed for — but that's not the same as whether it will profit on data it hasn't seen yet. The gap between those two things is where most strategy edge disappears.

We run every strategy through five distinct checks. All five must pass for a DEPLOYABLE verdict.

Step 1 — Out-of-sample split

What: Hold out the most recent 30% of data

We never let the strategy see the held-out window during any calibration. The verdict is based exclusively on this unseen period. If the edge disappears on data it wasn't tuned on, it's not an edge — it's curve-fitting.

Step 2 — Cost adjustment

What: Realistic bid-ask + commission, then 3× stress

We apply a per-trade cost that reflects realistic retail execution (spread + commission). Then we run a second pass at 3× that cost. A strategy that collapses under cost pressure was never robust. The stress-test Sharpe must also be positive.

Step 3 — Probabilistic Sharpe Ratio (PSR)

What: Probability that the true Sharpe exceeds a benchmark of zero

The standard Sharpe ratio is noisy — a Sharpe of 0.5 from 2 years of daily data might be indistinguishable from luck. PSR accounts for the sample size, return distribution skewness, and excess kurtosis. We require PSR > 0.6 (meaning >60% probability the true Sharpe is positive). Below that, the strategy hasn't proven its edge against sampling noise.

Step 4 — Block-bootstrap significance test

What: Resample the return series 1,000 times; require p < 0.10

We resample blocks of the OOS returns (preserving autocorrelation) and ask: how often does a random resampling produce a Sharpe this high by chance? The bootstrap p-value must be below 0.10. This catches strategies that happened to align with a single lucky stretch in the OOS window.

Step 5 — Parameter perturbation + cross-asset breadth

What: Perturb parameters ±20%; test on 6+ other assets and timeframes

We shift each parameter by ±20% and check that the median Sharpe across perturbed variants remains positive. A strategy that only works at exactly one parameter setting is almost certainly overfit.

We also run the same logic on at least 6 other tickers and timeframes. A strategy with BROAD breadth means the edge generalises beyond the specific asset it was tested on. MIXED means partial evidence. Both are better than no breadth test at all.

Verdict definitions

What this doesn't tell you

DEPLOYABLE means the edge survived these tests at small size — not that it's guaranteed to profit, not that it works at any size, and not that market conditions won't change. It's a research tool, not financial advice. Strategies can degrade; that's why we recommend live forward-tracking with degradation alerts.

Want your strategy tested?

Submit any strategy — a name, a plain-English description, or a TradingView Pine Script — and we'll run it through the same pipeline and publish the results.

Verdict	Meaning
DEPLOYABLE	Passes all 5 checks. Positive OOS Sharpe, cost-resistant, PSR ≥ 0.6, bootstrap p < 0.10, parameter-stable, broad or mixed breadth. Recommended at small size (≤25% of per-strategy risk budget).
BORDERLINE	Positive OOS Sharpe, but one or more checks are marginal. Paper-trade first. May degrade in live conditions.
REJECT	Fails one or more checks. The apparent backtest edge does not survive rigorous OOS + cost + overfitting scrutiny.