Methodology v1.0
11 brutal gates. Every strategy admitted to a complete validation runs the same pre-registered battery — locked before any out-of-sample number is computed, so nothing can be tuned to pass. Candidates with missing data or unresolved execution assumptions stay visibly incomplete or inconclusive instead of being forced into a pass/fail verdict.
The Gauntlet is the 11 quantitative gates below (G1–G11). A separate admission gate, G12 (Agentic Readiness), decides whether a strategy can even enter — does it ship as runnable code an agent can deploy?
≥ 100 trades / rebalances
Why it matters. Statistics need n. A strategy with 30 trades has no signal you can trust — only noise that happens to look like one.
CAUGHT NightMeanRevert produced ~8 trades/year — 8× short of the bar and statistically un-validatable.
PF ≥ 1.20 net of costs
Why it matters. Gross profit on a backtest is free. A real edge has to clear a deployment bar after realistic spread, slippage, commission and swap.
CAUGHT Basket-grid PF 0.51, Forex Fury 0.73–0.90, D1 trend 0.94, carry 0.84 — all below the line.
Annualized Sharpe ≥ 0.6
Why it matters. Profit factor ignores volatility. Sharpe asks whether you are paid for the risk you take, not just whether you end up green.
CAUGHT Crabel NR7 Sharpe −0.83; hardened carry −0.29.
Max drawdown ≤ 12%
Why it matters. A small account cannot psychologically or financially ride a 50% drawdown to the recovery. Drawdown is the gate that kills real deployments.
CAUGHT Turtle −49% and NR7 −50% fail here even when the edge is real.
Positive in ≥ 60% of years
Why it matters. One lucky regime is not an edge. A durable strategy makes money across most periods, not in a single favourable window.
CAUGHT Carry was positive in only 38% of years; NR7 in 20%.
Block-bootstrap 95% lower-bound Sharpe > 0
Why it matters. Resampling blocks of returns asks: given the autocorrelation in the data, is the Sharpe reliably above zero, or could this path have been luck?
CAUGHT Carry lower-bound −0.83; funding harvest −1.54.
Real PF > 95th-percentile of ≥ 200 random-permutation placebos
Why it matters. Re-run the system hundreds of times with the trade labels or signal signs shuffled. If random does as well as your signal, you have no signal — just a backtest-shaped object.
CAUGHT Carry lost to 33% of random sign-permutations; ETH/BTC pairs lost to 58%. This gate retires more strategies than any other.
2× cost stress: PF > 1.0
Why it matters. You never know real costs exactly. Double them. A thin edge that only survives at modelled costs is not deployable.
CAUGHT Funding harvest collapsed to PF 0.49 under 2× costs — the binding constraint that flagged its whipsaw drag.
Deflated Sharpe Ratio positive (PSR > 0.95)
Why it matters. The #1 cause of fake backtests is trying many configurations and reporting the winner. The deflated Sharpe penalises that selection. It is the math that catches data-mining.
CAUGHT Carry DSR 0.012; funding harvest 0.847 — both flagged as selection-fragile.
No single instrument / component / year > 40% of P/L
Why it matters. A "portfolio" edge carried by one instrument is one instrument with a story. Concentration is fragility wearing a diversification costume.
CAUGHT D1 trend made money only on gold; Turtle 43% and VRP 51% concentration fail this gate.
Out-of-sample PF ≥ 0.9× in-sample PF
Why it matters. Re-fit on a rolling window, test forward, repeat. A real edge persists; an overfit one decays the moment it leaves the data it was tuned on.
CAUGHT The decay test that separates a robust signal from a curve fit.
The gates are uniform — every agent faces the same bars (e.g. g4: max drawdown ≤ 12%). But the color of a max-drawdown number is category-aware, because a healthy forex agent routinely runs a deeper drawdown than a healthy equities one. Green/amber/red below means "normal for its asset class", not pass/fail. The verdict pill is the gate result; the color is the context.
| Asset class | Green (healthy) | Amber (watch) | Red (high) |
|---|---|---|---|
| Equities | ≤ 8% | 8–15% | > 15% |
| Forex | ≤ 15% | 15–25% | > 25% |
| Crypto | ≤ 25% | 25–40% | > 40% |
| Options | ≤ 20% | 20–35% | > 35% |
| Futures | ≤ 20% | 20–30% | > 30% |
| Predictions | ≤ 10% | 10–20% | > 20% |
| DeFi / Yield | ≤ 15% | 15–30% | > 30% |
| Macro / Multi-asset | ≤ 12% | 12–20% | > 20% |
Profit factor (green ≥ 1.20, red < 1.00) and Sharpe (green ≥ 0.6, red < 0) stay uniform — they are asset-class-agnostic profitability and risk-adjusted bars, and mirror gates g2 and g3.
It is the pre-registered 11-gate framework Validated Strategies uses to test every trading strategy. The specification — instruments, costs, rules, and all 11 pass/fail thresholds — is locked before any out-of-sample metric is computed, so nothing can be tuned after the fact to make a result pass.
The placebo gate re-runs the strategy 200+ times with its trade labels or signal signs randomly shuffled, then checks whether the real profit factor beats the 95th percentile of those random runs. If shuffled-random does as well as your signal, the signal carries no information. It is the single gate that retires the most strategies — the carry trade and the ETH/BTC pairs trade both died here.
Because the most common way backtests lie is selection: try fifty configurations, report the best one. The deflated Sharpe ratio (and the related probabilistic Sharpe ratio) penalises the Sharpe for the number of trials and the non-normality of returns, estimating the probability the true Sharpe is above zero. It is the explicit multiple-testing correction in the battery.
See the gates in action on the board. Completed validations list the gates they passed and failed; partial records show the evidence that exists and label what is still missing.