Minerva's Luck and Robustness Measures

Edge (Luck vs. Skill) & Robustness (Signal vs. Noise)

Minerva provides optimizations of trading strategies that require large amounts (hundreds of thousands to millions) of computationally intensive backtests.

Traders’ goals are finding the strategies with the highest Annualized Returns or Sharpe Ratio, for instance.

However, in the statistics and data science of trading, it is well-known that performing many tests will always reveal an attractive result purely by chance. So, how should a researcher interpret large searches for optimized trading strategies?

Minerva solves this by asking two questions. The first: is the good strategy I found lucky, or does it represent a skill (an edge)?

We can think of luck versus edge like this: is my result exceedingly rare (such as flipping a coin heads 50 times in a row—likely not lucky but instead sign of a weighted, or skilled, coin), or does it fall within the expected variance of randomness?

If it does fall within the expected variance of randomness, it likely means that the high Return or Sharpe we found appears inflated by getting lucky, or in other words, we picked a strategy that merely beat the others out of sheer randomness.

To summarize: a high Sharpe found from a single backtest among many backtests that could be performed by varying the dates, the parameters, etc., may be the result of those dates and parameters being lucky.

The only way to know whether we are seeing edge (a skilled strategy) or luck is to run more backtests and compare all of their results together to all of the results that randomness would give us.

The second question that Minerva asks is this: does my strategy contain signal, or is it just noise. This is a subtle but importantly different question to the first one and addresses the problem of robustness, regardless of whether or not our high Sharpe is edge or luck.

Signal persists; noise means the strategy happened to have skill for a little while, but not for any good reason. Only because it randomly stumbled into skill briefly, like a broken clock that is correct twice a day.

So, we can think of signal versus noise and robustness as being akin to persistence. If a strategy contains true signal, and isn’t noise, it means that—so long as the overall global regime doesn’t change—the strategy may behave similarly in the future.

Think of this as the similarity of the Sharpe over time (at any magnitude of Sharpe). Whether or not our strategy has a high Sharpe or a low Sharpe, one should understand if the Sharpe measured is representative of the true Sharpe expected going forward.

To describe signal versus noise more precisely: if I find a strategy on past data and measure its performance in-sample, the question is whether performance is similar on out-of-sample data.

Minerva has developed multiple interesting ways of looking at robustness. One of them is to think about whether your strategy’s parameters sit on a plateau of Sharpes, or a narrow peak in the Sharpe. For instance, if my strategy has a high Sharpe when I use a 90-day moving average but much worse Sharpes with an 89- or 91- day MA, it certainly means my strategy is on a peak rather than a plateau. And, if that Sharpe is also high—which is what the previous luck vs. edge question asked about—then I have overfit.

How Minerva Does It

Minerva measures luck vs edge, signal vs. noise, and overall overfitting using a panel of known statistical tools plus its own proprietary measures.

Luck vs. Edge

Minerva answers the first question: is a high Sharpe Ratio that an optimization found just a lucky result? In other words, did the optimization merely pick the best parameters for the data it looked at and so overfit on the in-sample data?

For this, we measure the LuckFactor in optimizations and the LuckFactor+ in reports. Both are variations of the Deflated Sharpe Ratio (DSR). In a report, the MinervaScore also counts the Sharpe on the amount-held basis.

Signal vs. Noise

Minerva answers the second question: if I perturb this strategy by changing its parameters slightly--and if I test those changes on future (out-of-sample) data--does the strategy still produce a similar Sharpe?

For this, we measure the RobustGrade™, which includes plateau drop, the Superior Predictive Ability p-value, CPCV worst-12 and the minimum track record.

The Two Combined

Before trading a strategy, one should seek answers to both of those questions. Neither can predict the future; nothing can. But, the two measurements together can tell you whether you should expect to see behavior close to what you tested, or not, relative to other trading strategies.

For this, we combine all of the above in our proprietary measure, the MinervaScore™.

Beside it, the report shows the Probability of Overfitting (PBO), which contains different and useful information to the other measures outlined above. All are explained in this document, below.

To obtain a MinervaScore, Minerva takes a trading strategy, parameterizes it in its engine, and runs a few million backtests across multiple optimizations to produce a report. The tests in the report use considerable compute to comprehensively probe parameters around the region of skillful (non-lucky) candidate strategies. It is a powerful tool that takes your original strategy and fine-tunes it for both edge and robustness. The MinervaScore is this final measure, 0–99, of edge and robustness.

The report and the MinervaScore provide insight into the relative skill and persistence of a strategy. While many traders test strategies that seem to perform well shortly after their tests but then do not—-and then label this phenomenon “alpha decay”--at Minerva we believe that some alpha decay is merely a poorly understood strategy performing exactly as expected. The MinervaScore provides that expectation.

Definitions

Backtest
One historical test of a strategy, at one frozen choice of its settings, over a chosen stretch of dates: the trades it made, its value over time, and its results.
Optimization
A search for a strategy's best settings. Every parameter in a strategy is parsed out so that a series of backtests can be performed by sweeping through the values of all its parameters. Furthermore, Minerva splits past dates into a development period (and, further even more, performs Combinatorially Purged Cross-Validation, or CPCV, on this development period) and then a subsequent "future" holdout period. The optimization performs tens of thousands of backtests using a genetic search across many parameter settings during the development period, picks one and freezes it, then tests that one setting once on the holdout period. Optimizations deliver a LuckFactor™ and a RobustGrade™ in addition to Sharpe and other performance and robustness measures.
Report
Minerva's full strategy report runs several optimizations on one strategy, then many focused backtests around the best region they find, and names up to three best candidate strategies. Each candidate gets every measure and score in this document, the MinervaScore among them, and a thorough analysis based on the millions of backtests across many optimizations that the report performs.
No-edge history
We build this to test against pure luck. It is a rebuilt history in which every setting keeps its exposure to the benchmark and its day-to-day swings and loses its average gain beyond that exposure. The days are reshuffled in blocks of 21 bars--about a month's worth of trading days--the same way for every setting and the benchmark.
No-edge pick
The strategy's parameter settings that pure luck would pick. I.e., the setting a report's own selection rule picks in a no-edge history. A report makes 800 of them by rerunning its pick, and 8,000 when fewer than 10 of the first 800 match a candidate. An optimization's pick comes from its search and is never rerun, so an optimization has none.
CPCV
Combinatorial Purged Cross-Validation. The development period is cut into 10 slices of dates. For each of the 120 ways to set 3 slices aside, the search chooses on the other 7 (in-sample periods) and is scored on the 3 it never saw (out-of-sample periods). Dates just before each set-aside slice are dropped from the 7 (the purge), so no trade or signal still running reaches into the scored dates, and so are dates just after it (the embargo), so the days that follow cannot leak in. CPCV gives us a wealth of statistical data in addition to helping us understand how the strategy performed over a wide range of historical conditions, out-of-sample.
Development period
The dates a search uses to choose its setting: every date before the holdout period.
Sharpe Ratio
The headline of every backtest, optimization and report: the Sharpe of the strategy's own daily returns, after costs: their average divided by how much they swing (their volatility), annualized. Minerva widens the swing when good or bad days come in runs (the Newey-West correction), so streaky returns are not flattered. An optimization's Sharpe Ratio covers its holdout period.
Development Sharpe
The Sharpe that the Deflated Sharpe Ratio (DSR, see below) and LuckFactor use: the average return per bar on the amount-held basis over the development period, divided by the volatility of those returns, scaled to a year. It has no Newey-West correction: Bailey and López de Prado's DSR formula is written for this plain Sharpe and makes its own allowance for skew and fat tails.
Holdout period
The last stretch of dates, kept out of the optimization's search across the development period; as such, the holdout period represents a blind "future" to the strategy. The chosen setting is frozen first and then tested once on these dates, so the search's choosing cannot influence its holdout results.
Benchmark
What you could have held instead: the same stock, for a single-stock strategy; the list's members on each date held in equal amounts, rebalanced every bar, before costs, for every rotation (long-only, short-only or long-short) and every report.
Amount-held basis
The strategy's daily returns minus its amount held times its benchmark's daily returns. The amount held is how much of its benchmark the strategy holds on average: the slope of the strategy's daily returns against its benchmark's. Every luck and robustness measure and score is computed on it, for every kind of strategy: DSR, LuckFactor, plateau drop, CPCV worst 12, the Superior Predictive Ability p-value, the minimum track record, RobustGrade, the MinervaScore and PBO. No number is shown on two bases.

Luck vs. Edge

1 Sharpe

Luck vs. edge · backtests, optimizations & reports · higher is better

The Sharpe of the strategy's daily returns on the amount-held basis (see Amount-held basis, above): their average divided by how much they swing (their volatility), annualized. Two strategies can gain the same with very different Sharpes; the less volatile one scores higher. Minerva widens the swing when good or bad days come in runs (the Newey-West correction), so streaky returns are not flattered. It is not the strategy's Sharpe Ratio: the Sharpe Ratio is the same Newey-West calculation on the strategy's own returns, and is also shown in optimizations and reports.

Mathematically
SR = ā ⁄ √L · √252
L = c0 + 2·Σj=1..q (1 − j⁄(q+1))·cj

ā: average daily return on the amount-held basis · cj: covariance of those returns j days apart · q = ⌊T1/3⌋ · T: days

0.81.01.21.41.6A: +45% over 3 years, Sharpe 2.03A: Sharpe 2.03B: +45% over 3 years, Sharpe 0.64B: Sharpe 0.64year 0year 1year 2year 3Same gain, +45%. A swings less, so its Sharpe is higher.

2 Deflated Sharpe Ratio (DSR)

Luck vs. edge · optimizations & reports · 0 to 1, higher is better

Back to our example in the intro, if we see we flipped a coin that landed on heads 50 times in a row and so is likely weighted, DSR measures how many times a fair coin would produce the same 50-heads result, and compares that to your result. DSR keeps Bailey and López de Prado’s probability formula; only SR₀ changes, from assumed to measured.

SR₀ is the development Sharpe that choosing the best of the search's settings produces by itself, when no setting has an edge. It is measured, like the chosen setting's Sharpe, on the amount-held basis. The published formula gets it by treating every setting tried as an independent coin. Minerva's settings are not independent: settings one step apart trade mostly the same stocks on the same days. So Minerva measures it, the same way in an optimization and in a report. On each of the 120 CPCV splits, the search picks a setting on 7 slices, and the gap is that setting's Sharpe on those 7 slices minus its Sharpe on the 3 slices it never saw. With no edge, the Sharpe on the unseen slices is zero on average, so the gap is what the choosing produced. SR₀ is the average of the 120 gaps, scaled to the dates the search read: by √(7/10) in an optimization, because a split chooses on 7 slices and the search on all 10.

Panel A. One value per split: the gap of that split's winner. The average of the 120 gaps is 1.70, and 1.70 × √(7/10) is SR₀, 1.42.

Panel B, real. The chosen setting's development Sharpe, 2.00, with the estimation error that 8 years of daily returns leave. DSR is the probability that its true Sharpe is above SR₀: 0.95. With the published formula's assumed SR₀, 1.82, DSR would be 0.69.

Mathematically
DSR = Φ( (SR − SR0)·√(T−1) ⁄ √(1 − γ3·SR + ¼(κ+2)·SR2) )
SR0 = √(Dsplit ⁄ Dsearch) · 1120 Σs ( SRsin − SRsout )
assumed SR0, replaced = σ · [(1−γ)·Φ−1(1 − 1⁄N) + γ·Φ−1(1 − 1⁄(N·e))]

SR: the chosen setting's development Sharpe · SRsin, SRsout: the Sharpe of the winner of split s on the 7 in-sample slices and on the 3 out-of-sample slices · Dsplit ⁄ Dsearch: the dates a split's choice read over the dates the search read, 7⁄10 in an optimization · T: development bars · SR and SR0 per bar in the formula · γ3: skew · κ: excess kurtosis · Φ: the normal curve's area · N: settings tried · σ: the spread of the settings' Sharpes · γ = 0.5772 · every Sharpe on the amount-held basis

A · The 120 CPCV splits: the gap of each split winnerOne value per split: the winner's Sharpe in-sample minus out-of-sample01020gap 0.40 to 0.50: 1 splitsgap 0.60 to 0.70: 3 splitsgap 0.70 to 0.80: 1 splitsgap 0.80 to 0.90: 6 splitsgap 0.90 to 1.00: 2 splitsgap 1.00 to 1.10: 3 splitsgap 1.10 to 1.20: 4 splitsgap 1.20 to 1.30: 4 splitsgap 1.30 to 1.40: 9 splitsgap 1.40 to 1.50: 8 splitsgap 1.50 to 1.60: 9 splitsgap 1.60 to 1.70: 10 splitsgap 1.70 to 1.80: 7 splitsgap 1.80 to 1.90: 4 splitsgap 1.90 to 2.00: 8 splitsgap 2.00 to 2.10: 15 splitsgap 2.10 to 2.20: 5 splitsgap 2.20 to 2.30: 9 splitsgap 2.30 to 2.40: 4 splitsgap 2.40 to 2.50: 2 splitsgap 2.50 to 2.60: 2 splitsgap 2.60 to 2.70: 2 splitsgap 2.80 to 2.90: 1 splitsgap 2.90 to 3.00: 1 splitsaverage gap 1.70× √(7/10) = SR₀ 1.42number of splits00.511.522.533.5Sharpe in-sample minus Sharpe out-of-sample, on the amount-held basisB · The chosen setting's development Sharpe and the estimation error of that SharpeMeasured 2.00 on 8 years of daily returns. The curve is where the true Sharpe can be(normal, centred on 2.00, standard error 0.35).the area above SR₀: DSR 0.95SR₀ = 1.42, from ADSR = 0.95the shaded area:the probability that thetrue Sharpe is above SR₀0.511.522.53development Sharpe on the amount-held basis (annualized)Published formula, settings assumed independent: SR₀ = 1.82, DSR = 0.69.

3 LuckFactor

Luck vs. edge · optimizations & reports · lower is better

Minerva's LuckFactor is a measurement of how lucky a Sharpe is. It is DSR read from the other side: 100% − DSR, rounded up to a whole percent. That is the chance that the chosen setting's development Sharpe is no higher than SR₀, the Sharpe that choosing the best setting produces by itself. Lower is better. Example, from the DSR figure: DSR is 0.949, so 100% − 94.9% is 5.1%, shown as 6%. The LuckFactor is used in optimizations.

The LuckFactor+ is used in reports. A report reruns its pick on rebuilt histories with no edge, so the LuckFactor+ counts instead of using a formula: of every 100 no-edge picks, how many reach the candidate's DSR. Example: 17 of the 800 no-edge picks reach it, so (1 + 17) ÷ (1 + 800) is 2.2%, shown as 3%.

Mathematically
LuckFactor = 100% − DSR, rounded up to a whole percent

DSR: measure 2, on the amount-held basis

LuckFactor+ = ⌈ 100 × 1 + k1 + N ⌉ %
k = #{ h : DSRh ≥ DSRc }

h: one no-edge pick · c: the chosen setting · N: the reruns the candidate's MinervaScore uses, 800 or 8,000 · DSR: measure 2, with the search's one SR₀ for the chosen setting and for every no-edge pick

LuckFactor = 100% − DSRThe curve of DSR's panel B: where the chosen setting's true Sharpe can be.DSR is the area right of SR₀. LuckFactor is the area left of SR₀.the area right of SR₀: DSR 0.949the area left of SR₀: 100% − DSR, 5.1%SR₀ = 1.42LuckFactor = 6%the area left of SR₀:100% − 94.9% = 5.1%,rounded upDSR = 0.949the area right of SR₀0.511.522.53development Sharpe on the amount-held basis (annualized)Of every 100 no-edge picks, how many reach the chosen setting's DSR?reaches the chosen setting's DSRreaches the chosen setting's DSRreaches the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSRdoes not reach the chosen setting's DSR17 of 800 no-edge picksreach the chosen setting's DSR:1 + 171 + 800= 2.2%rounded up: LuckFactor+ 3%

Signal vs. Noise

4 Plateau drop

Signal vs. noise · in-sample · optimizations & reports · lower is better

In a Minerva report, many nearby settings of a strategy's parameters are backtested so that you can see whether the strategy is sensitive to minor perturbations. If it is, it will likely quickly collapse in real-world trading going forward. Said another way, does the optimization's resulting Sharpe depend on hitting one exact setting? Plateau drop is the delta between the Sharpe and the Sharpe of the settings nearby. It is the winning setting's Sharpe minus the lowest Sharpe among the settings one step away from the chosen setting (one parameter moved one step, the other parameters fixed). A strategy on a plateau barely drops; a strategy on a narrow peak drops a lot. If a 90-day average works but 89 and 91 days do not, plateau drop is large and the strategy is not robust. Plateau drop is measured in-sample: on the dates the setting was chosen on, which is the development period of an optimization or the whole window of a report.

Mathematically
plateau drop = SR(c) − min{ SR(c), SR(n) : n one step from c }

c: the chosen setting · n: the nearest tested value on either side of one setting, the others fixed · SR: the Sharpe on the amount-held basis, each setting with its own amount held · optimization: SR is the average over the CPCV splits · report: SR is over the whole window, and the report's scores use plateau drop ÷ SR(c)

0.16lookback 5, hold 10: Sharpe 0.160.30lookback 10, hold 10: Sharpe 0.300.35lookback 15, hold 10: Sharpe 0.350.33lookback 20, hold 10: Sharpe 0.330.33lookback 25, hold 10: Sharpe 0.330.12lookback 30, hold 10: Sharpe 0.120.16lookback 35, hold 10: Sharpe 0.160.44lookback 5, hold 20: Sharpe 0.440.47lookback 10, hold 20: Sharpe 0.470.83lookback 15, hold 20: Sharpe 0.831.18lookback 20, hold 20: Sharpe 1.180.97lookback 25, hold 20: Sharpe 0.970.56lookback 30, hold 20: Sharpe 0.560.13lookback 35, hold 20: Sharpe 0.130.39lookback 5, hold 30: Sharpe 0.391.02lookback 10, hold 30: Sharpe 1.021.43lookback 15, hold 30: Sharpe 1.431.91lookback 20, hold 30: Sharpe 1.911.36lookback 25, hold 30: Sharpe 1.360.74lookback 30, hold 30: Sharpe 0.740.13lookback 35, hold 30: Sharpe 0.130.43lookback 5, hold 40: Sharpe 0.430.93lookback 10, hold 40: Sharpe 0.932.01lookback 15, hold 40: Sharpe 2.012.54lookback 20, hold 40: Sharpe 2.541.94lookback 25, hold 40: Sharpe 1.940.75lookback 30, hold 40: Sharpe 0.750.38lookback 35, hold 40: Sharpe 0.380.38lookback 5, hold 50: Sharpe 0.380.93lookback 10, hold 50: Sharpe 0.931.41lookback 15, hold 50: Sharpe 1.411.90lookback 20, hold 50: Sharpe 1.901.48lookback 25, hold 50: Sharpe 1.480.81lookback 30, hold 50: Sharpe 0.810.54lookback 35, hold 50: Sharpe 0.540.14lookback 5, hold 60: Sharpe 0.140.54lookback 10, hold 60: Sharpe 0.541.04lookback 15, hold 60: Sharpe 1.041.02lookback 20, hold 60: Sharpe 1.020.90lookback 25, hold 60: Sharpe 0.900.56lookback 30, hold 60: Sharpe 0.560.27lookback 35, hold 60: Sharpe 0.27-0.02lookback 5, hold 70: Sharpe -0.020.27lookback 10, hold 70: Sharpe 0.270.58lookback 15, hold 70: Sharpe 0.580.24lookback 20, hold 70: Sharpe 0.240.51lookback 25, hold 70: Sharpe 0.510.27lookback 30, hold 70: Sharpe 0.270.07lookback 35, hold 70: Sharpe 0.07510152025303510203040506070setting 1 (lookback bars)setting 2 (hold bars)Best testedSharpe 2.54Lowest of its4 neighbours 1.90Plateau drop2.54 − 1.90 = 0.64As a share ofthe Sharpe0.64 ÷ 2.54 = 25%

5 Superior Predictive Ability p-value

Signal vs. noise · out-of-sample · optimizations & reports · lower is better

SPA asks this: during the "future" out-of-sample holdout period, did the strategy beat noise? In other words, how often would a strategy with no signal (i.e., a signal that carries no real information) do at least as well as this strategy did, by chance alone? Minerva takes the holdout's returns, removes the average of those returns, reshuffles the days in short blocks 9,999 times, and counts how often a reshuffle does at least as well as the real result. An SPA p-value (probability) of, say, 0.01 means a random strategy with no signal does as well as the real result 1 time in 100, implying that the strategy's holdout result is likely to be due to signal rather than noise. The SPA p-value does not use a Sharpe: the SPA p-value tests the average daily return on the amount-held basis.

Mathematically
x = r − amount held × b on the holdout · x̄ = mean(x)
p = 1 + #{ x̄* − x̄ ≥ x̄ }1 + 9,999

x̄*: the mean of x redrawn in circular blocks of ⌊n1/3⌋ bars · n: holdout bars · none under 30 bars

0200400600-11.5 to -11.0 bp a day: 2 reshuffles-11.0 to -10.5 bp a day: 1 reshuffles-10.5 to -10.0 bp a day: 1 reshuffles-10.0 to -9.5 bp a day: 10 reshuffles-9.5 to -9.0 bp a day: 11 reshuffles-9.0 to -8.5 bp a day: 27 reshuffles-8.5 to -8.0 bp a day: 28 reshuffles-8.0 to -7.5 bp a day: 27 reshuffles-7.5 to -7.0 bp a day: 38 reshuffles-7.0 to -6.5 bp a day: 76 reshuffles-6.5 to -6.0 bp a day: 88 reshuffles-6.0 to -5.5 bp a day: 145 reshuffles-5.5 to -5.0 bp a day: 176 reshuffles-5.0 to -4.5 bp a day: 226 reshuffles-4.5 to -4.0 bp a day: 262 reshuffles-4.0 to -3.5 bp a day: 295 reshuffles-3.5 to -3.0 bp a day: 366 reshuffles-3.0 to -2.5 bp a day: 442 reshuffles-2.5 to -2.0 bp a day: 467 reshuffles-2.0 to -1.5 bp a day: 536 reshuffles-1.5 to -1.0 bp a day: 521 reshuffles-1.0 to -0.5 bp a day: 631 reshuffles-0.5 to 0.0 bp a day: 625 reshuffles0.0 to 0.5 bp a day: 614 reshuffles0.5 to 1.0 bp a day: 652 reshuffles1.0 to 1.5 bp a day: 595 reshuffles1.5 to 2.0 bp a day: 579 reshuffles2.0 to 2.5 bp a day: 458 reshuffles2.5 to 3.0 bp a day: 397 reshuffles3.0 to 3.5 bp a day: 373 reshuffles3.5 to 4.0 bp a day: 307 reshuffles4.0 to 4.5 bp a day: 263 reshuffles4.5 to 5.0 bp a day: 207 reshuffles5.0 to 5.5 bp a day: 156 reshuffles5.5 to 6.0 bp a day: 117 reshuffles6.0 to 6.5 bp a day: 81 reshuffles6.5 to 7.0 bp a day: 61 reshuffles7.0 to 7.5 bp a day: 47 reshuffles7.5 to 8.0 bp a day: 24 reshuffles8.0 to 8.5 bp a day: 24 reshuffles8.5 to 9.0 bp a day: 14 reshuffles9.0 to 9.5 bp a day: 10 reshuffles9.5 to 10.0 bp a day: 7 reshuffles10.0 to 10.5 bp a day: 3 reshuffles10.5 to 11.0 bp a day: 2 reshuffles11.0 to 11.5 bp a day: 3 reshuffles11.5 to 12.0 bp a day: 4 reshufflesthe strategy: 7.0 bp a dayp = 0.014 (1 in 71)-10-505109,999 reshuffles with the signal removedaverage daily return on the amount-held basis (basis points)

6 CPCV worst-12

Signal vs. noise · in-sample · optimizations & reports · higher is better

During an optimization, the CPCV process slices the development period into 10 time ranges, and there are 120 ways to pick 3 of the 10 time ranges. The result of this CPCV process is 120 measurements of Sharpe across the entire tested period, each containing information about their respective slice of history. While these don't necessarily represent any future slices, they do imply a range of Sharpes that one may observe in the future. CPCV worst-12 takes the winning setting's Sharpe on each of those 120 combinations and averages the 12 lowest of the 120 Sharpes. A high value of the CPCV worst-12 means even the setting's worst stretches of dates showed good performance: a promising sign for robustness.

Mathematically
S(1) ≤ … ≤ S(120): the setting's Sharpe on the amount-held basis on each set of 3 of the 10 slices
CPCV worst 12 = S(1) + … + S(12)12
-0.500.511.521th lowest: Sharpe -0.192th lowest: Sharpe -0.123th lowest: Sharpe 0.034th lowest: Sharpe 0.085th lowest: Sharpe 0.106th lowest: Sharpe 0.207th lowest: Sharpe 0.238th lowest: Sharpe 0.239th lowest: Sharpe 0.2710th lowest: Sharpe 0.2911th lowest: Sharpe 0.3412th lowest: Sharpe 0.3513th lowest: Sharpe 0.3714th lowest: Sharpe 0.3815th lowest: Sharpe 0.3916th lowest: Sharpe 0.4317th lowest: Sharpe 0.4718th lowest: Sharpe 0.4819th lowest: Sharpe 0.4920th lowest: Sharpe 0.4921th lowest: Sharpe 0.5022th lowest: Sharpe 0.5423th lowest: Sharpe 0.5724th lowest: Sharpe 0.5925th lowest: Sharpe 0.6026th lowest: Sharpe 0.6127th lowest: Sharpe 0.6228th lowest: Sharpe 0.6429th lowest: Sharpe 0.6530th lowest: Sharpe 0.6731th lowest: Sharpe 0.7032th lowest: Sharpe 0.7033th lowest: Sharpe 0.7134th lowest: Sharpe 0.7435th lowest: Sharpe 0.7436th lowest: Sharpe 0.7637th lowest: Sharpe 0.7838th lowest: Sharpe 0.7939th lowest: Sharpe 0.7940th lowest: Sharpe 0.8241th lowest: Sharpe 0.8442th lowest: Sharpe 0.8643th lowest: Sharpe 0.8644th lowest: Sharpe 0.8645th lowest: Sharpe 0.8646th lowest: Sharpe 0.8747th lowest: Sharpe 0.8748th lowest: Sharpe 0.9049th lowest: Sharpe 0.9150th lowest: Sharpe 0.9151th lowest: Sharpe 0.9452th lowest: Sharpe 0.9653th lowest: Sharpe 0.9654th lowest: Sharpe 0.9955th lowest: Sharpe 0.9956th lowest: Sharpe 1.0057th lowest: Sharpe 1.0058th lowest: Sharpe 1.0259th lowest: Sharpe 1.0260th lowest: Sharpe 1.0461th lowest: Sharpe 1.0762th lowest: Sharpe 1.0863th lowest: Sharpe 1.0864th lowest: Sharpe 1.0865th lowest: Sharpe 1.1066th lowest: Sharpe 1.1067th lowest: Sharpe 1.1068th lowest: Sharpe 1.1169th lowest: Sharpe 1.1470th lowest: Sharpe 1.1571th lowest: Sharpe 1.1772th lowest: Sharpe 1.1873th lowest: Sharpe 1.1874th lowest: Sharpe 1.1975th lowest: Sharpe 1.2076th lowest: Sharpe 1.2177th lowest: Sharpe 1.2278th lowest: Sharpe 1.2279th lowest: Sharpe 1.2280th lowest: Sharpe 1.2781th lowest: Sharpe 1.2882th lowest: Sharpe 1.2983th lowest: Sharpe 1.2984th lowest: Sharpe 1.3085th lowest: Sharpe 1.3086th lowest: Sharpe 1.3187th lowest: Sharpe 1.3188th lowest: Sharpe 1.3289th lowest: Sharpe 1.3390th lowest: Sharpe 1.3491th lowest: Sharpe 1.3692th lowest: Sharpe 1.3893th lowest: Sharpe 1.3894th lowest: Sharpe 1.4095th lowest: Sharpe 1.4196th lowest: Sharpe 1.4297th lowest: Sharpe 1.4398th lowest: Sharpe 1.4599th lowest: Sharpe 1.46100th lowest: Sharpe 1.47101th lowest: Sharpe 1.48102th lowest: Sharpe 1.49103th lowest: Sharpe 1.51104th lowest: Sharpe 1.51105th lowest: Sharpe 1.51106th lowest: Sharpe 1.56107th lowest: Sharpe 1.56108th lowest: Sharpe 1.59109th lowest: Sharpe 1.59110th lowest: Sharpe 1.60111th lowest: Sharpe 1.60112th lowest: Sharpe 1.68113th lowest: Sharpe 1.69114th lowest: Sharpe 1.69115th lowest: Sharpe 1.72116th lowest: Sharpe 1.72117th lowest: Sharpe 1.73118th lowest: Sharpe 1.83119th lowest: Sharpe 1.84120th lowest: Sharpe 1.94worst 12 average: 0.151306090120the candidate's 120 Sharpes, lowest to highest

7 Minimum track record

Signal vs. noise · out-of-sample · optimizations & reports · lower is better

MTR measures, for a backtest, how long of a time period is needed before one can be 95% sure that the Sharpe is truly above zero. To understand this we need to invoke basic statistics: all measurements have noise (error), and the error of an average shrinks as 1/√T as the number of observations T grows. So a Sharpe measured over T bars carries noise of about 1/√T. In order to be 95% sure our Sharpe is greater than zero given that noise, the holdout period must be at least as long as the measurement needed for the noise to shrink well below the measured Sharpe. Minerva shows the minimum track record in bars and as the dates those bars cover from the holdout's first day, beside the holdout's own bars and dates. Example: a holdout Sharpe on the amount-held basis of 1.6, with some skew and fat tails, needs 283 bars, 2024-07-08 to 2025-08-21; the holdout has 488 bars, 2024-07-08 to 2026-06-16, so the holdout is long enough. Seen the other way, a strategy with no signal (no information, just noise) shows a Sharpe at least this high over 488 bars with a chance of 0.0152, below 0.05 because the holdout is longer than the minimum track record. A Sharpe at or below zero never qualifies.

Mathematically
MTR = 1 + (1 − γ3·SR + ¼(κ+2)·SR2) · (1.645SR)2 bars
pMTR = 1 − Φ( SR·√(T−1) ⁄ √(1 − γ3·SR + ¼(κ+2)·SR2) )

SR: the holdout's Sharpe per bar, on the amount-held basis · T: holdout bars · γ3: skew · κ: excess kurtosis · p ≤ 0.05 exactly when T ≥ MTR

0.1 y0.3 y1 y3 y10 y30 ythis holdout: 488 bars (1.9 years)holdout long enoughtoo shortSharpe 0.5: 2,767 bars, 11.0 yearsSharpe 0.5: 11.0 yearsSharpe 2.5: 121 bars, 0.5 yearsSharpe 2.5: 0.5 yearsSharpe 1.6: 283 bars, 1.1 yearsthe example, Sharpe 1.6:283 bars (1.1 years)0.511.522.53annual Sharpe of the holdoutRecord length needed to be 95% sure the Sharpe is above zero

8 RobustGrade™

Signal vs. noise · optimizations & reports · 0 to 99, higher is better

Minerva summarizes these four signal vs. noise metrics in one number. Two parts are measured in-sample: on the development period, the dates the search chose the winning parameters from. Two parts are measured out-of-sample: on the holdout, the "future" dates the optimization never saw. We combine both to see whether we see signal in both the development and the holdout period alike. The RobustGrade is simply the weakest of the four. Any one of the four failing is enough for future results to also fail. A long, significant holdout with a strong MTR says nothing about what happens one step away, etc. In other words: strategies are fragile, and most show alpha decay. If one sees a strategy that shows a high RobustGrade, it means all four measures are strong.

Each of the four components of the RobustGrade attempts to discern whether the strategy truly carries signal (information), or is just noise:

  • Plateau drop, in-sample. When weak: one parameter step away and the strategy does not perform.
  • CPCV worst 12, in-sample. When weak: on the weakest stretches of dates the strategy makes nothing, so the result depends on a few good periods (think mean vs. median)
  • Minimum track record, out-of-sample. When weak: the holdout is too short for the holdout's Sharpe to stand out from noise.
  • Superior Predictive Ability p-value, out-of-sample. When weak: on the holdout, the gain is no more than noise gives.
Mathematically
P(x, T) = Φ( x·√(T−1) ⁄ √(1 − γ3·x + ¼(κ+2)·x2) )
RobustGrade = ⌊100 × min( 1 − pMTR, 1 − pSPA, P(one step away), P(worst 12) )⌋

x: a Sharpe on the amount-held basis, per bar · T: the bars behind x: for the Sharpe one step away, the development bars in an optimization and the whole window's bars in a report; for CPCV worst 12, the bars of 3 of the 10 development slices · γ3, κ: skew and excess kurtosis of the chosen setting's development returns on the amount-held basis · pMTR: the minimum track record's p · pSPA: the Superior Predictive Ability p-value

Robust means holding on all four, so the lowest of the four is the gradeComponent and its valueChance it is not noise (1 − p)Plateau drop · in-sample0.64, so one step away the Sharpe is 1.90Plateau drop · in-sample: 100.0%>99.9%CPCV worst 12 · in-sample0.15CPCV worst 12 · in-sample: 59.3%59.3%the lowest: it sets the gradeMinimum track record · out-of-sampleneeds 283 bars; the holdout has 488Minimum track record · out-of-sample: 98.5%98.5%SPA p-value · out-of-samplep = 0.014SPA p-value · out-of-sample: 98.6%98.6%RobustGrade = ⌊100 × the weakest⌋ = ⌊100 × 0.593⌋ = 59

The Two Combined

9 The MinervaScore™ (composite p-value)

Edge and robustness · reports only · 0 to 99, higher is better

The MinervaScore combines evidence for an edge with robustness across time and parameter choices. The edge term, ℰ, averages the ranks of the Sharpe ratio on the amount-held basis and the DSR statistic, the dedicated comparison against luck. The robustness term, ℛ, combines the ranks of plateau drop, CPCV worst-12, the SPA p-value and minimum track record.

The conversion 𝒞 compares the candidate with the picks of the same search on histories rebuilt with every setting’s estimated gain beyond its benchmark exposure removed. It counts how often a no-edge pick matches the candidate on the combined statistic T, then expresses that frequency as an integer from 0 to 99. A score of 33 means a match probability of at most 1 in 10, 66 at most 1 in 100, and 99 at most 1 in 1,000. The MinervaScore is served in reports only.

Mathematically
M = 𝒞[0.2 ℰ + 0.8 ℛ] = 𝒞[T]
ℰ = c1 + c22
ℛ = 5c3 + c4 + c5 + c68
p = 1 + k1 + N
𝒞[T] = min(99, ⌊33 × log10(1 ⁄ p)⌋)

M: the MinervaScore · ℰ: edge evidence · ℛ: robustness · T = 0.2 ℰ + 0.8 ℛ · cm: on measure m, how many of the other no-edge picks are equal to or worse · 1 Sharpe on the amount-held basis, 2 DSR, 3 plateau drop as a share of measure 1, 4 CPCV worst-12, 5 Superior Predictive Ability p-value, 6 minimum track record · k: the no-edge picks whose T is at least the candidate's, both counted against the same other no-edge picks · N: 800 reruns, or 8,000 when fewer than 10 of the first 800 match · never rounded up

Expanding T gives weights of one tenth each for measures 1, 2, 4, 5 and 6, and one half for measure 3. The 20–80 split expresses the existing score’s weights on ranks before the conversion to score points.

1 · Simulated: the candidate against the no-edge picksEach no-edge pick's combined result on the six measures, 8,000 reruns01002003002 to 4: 6 no-edge picks4 to 6: 20 no-edge picks6 to 8: 25 no-edge picks8 to 10: 34 no-edge picks10 to 12: 51 no-edge picks12 to 14: 92 no-edge picks14 to 16: 76 no-edge picks16 to 18: 94 no-edge picks18 to 20: 116 no-edge picks20 to 22: 134 no-edge picks22 to 24: 144 no-edge picks24 to 26: 173 no-edge picks26 to 28: 180 no-edge picks28 to 30: 209 no-edge picks30 to 32: 220 no-edge picks32 to 34: 240 no-edge picks34 to 36: 252 no-edge picks36 to 38: 251 no-edge picks38 to 40: 233 no-edge picks40 to 42: 303 no-edge picks42 to 44: 280 no-edge picks44 to 46: 318 no-edge picks46 to 48: 269 no-edge picks48 to 50: 279 no-edge picks50 to 52: 281 no-edge picks52 to 54: 288 no-edge picks54 to 56: 269 no-edge picks56 to 58: 297 no-edge picks58 to 60: 275 no-edge picks60 to 62: 271 no-edge picks62 to 64: 261 no-edge picks64 to 66: 248 no-edge picks66 to 68: 234 no-edge picks68 to 70: 237 no-edge picks70 to 72: 184 no-edge picks72 to 74: 193 no-edge picks74 to 76: 186 no-edge picks76 to 78: 131 no-edge picks78 to 80: 134 no-edge picks80 to 82: 114 no-edge picks82 to 84: 95 no-edge picks84 to 86: 92 no-edge picks86 to 88: 79 no-edge picks88 to 90: 51 no-edge picks90 to 92: 29 no-edge picks92 to 94: 31 no-edge picks94 to 96: 18 no-edge picks96 to 98: 3 no-edge picksthe candidate52 of 8,000 no-edge picksmatch or beat the candidate:p = 53 ÷ 8,001, 1 in 1500255075100worse than every other no-edge pickbetter than every other2 · The scale: 33 points for every tenfoldUnder each mark: how often a no-edge pick matches the candidate0every time331 in 10431 in 20661 in 100991 in 1,000 or rarerMinervaScore 71MinervaScore 71In the first 800 reruns, 3 no-edge picks matched the candidate, fewer than 10, so the report ran on to 8,000.

The MinervaScore, step by step

A · The report's searchSharpe of each of the 25 settings1.2Sharpe 1.250.6Sharpe 0.610.7Sharpe 0.750.8Sharpe 0.800.9Sharpe 0.870.8Sharpe 0.790.8Sharpe 0.791.0Sharpe 0.991.0Sharpe 1.000.9Sharpe 0.861.7Sharpe 1.661.3Sharpe 1.261.1Sharpe 1.081.0Sharpe 1.041.0Sharpe 1.040.9Sharpe 0.921.2Sharpe 1.161.2Sharpe 1.170.9Sharpe 0.901.0Sharpe 1.041.0Sharpe 0.970.7Sharpe 0.721.0Sharpe 1.000.8Sharpe 0.771.1Sharpe 1.081234512345setting 1setting 2Outlined: the candidate.Dashed square: the settings that can be chosen.B · One no-edge historyEvery edge removed, days reshuffled in blocks−0.3Sharpe -0.27−0.3Sharpe -0.31−0.1Sharpe -0.090.0Sharpe 0.03−0.2Sharpe -0.15−0.3Sharpe -0.33−0.3Sharpe -0.250.1Sharpe 0.150.0Sharpe 0.020.1Sharpe 0.06−0.2Sharpe -0.190.0Sharpe 0.020.1Sharpe 0.120.1Sharpe 0.07−0.1Sharpe -0.060.3Sharpe 0.320.3Sharpe 0.300.1Sharpe 0.070.0Sharpe 0.02−0.1Sharpe -0.080.0Sharpe 0.000.2Sharpe 0.170.0Sharpe 0.010.0Sharpe -0.01−0.1Sharpe -0.131234512345setting 1setting 2Outlined: the no-edge pick of this history,chosen by the same rule.C · The candidate and the 800 no-edge picksEach dot: the pick of one no-edge historyplateau drop, % of the Sharpe050100150+the candidate: Sharpe 1.26, plateau drop 38% of the Sharpethe candidate-0.500.511.5Sharpe on the amount-held basisShaded: at least as good on both. 2 of the 800 are there.150+: 150% or more, or no Sharpe above zero to lose.
A made-up report. A: the Sharpe of each setting on the real history. B: the same settings in one no-edge history (steps 1 and 2). C: the candidate against the picks of 800 no-edge histories on two of the six measures (steps 3 to 7); 2 of the 800 are at least as good on both.
  1. Rebuild the history 800 times. Every setting keeps its daily swings and its exposure to the benchmark, and loses its average gain beyond that exposure. Only luck and noise remain. The days are reshuffled in blocks of 21 bars, about a month, the same way for every setting and the benchmark.
  2. Rerun the report's selection rule in each rebuilt history. The setting the rule picks is that history's no-edge pick.
  3. Measure the candidate and every no-edge pick on the same six measures, all on the amount-held basis. The amount held is measured again in each rebuilt history.
  4. Turn each measure into a percentile: the share of no-edge picks equal to or worse than the candidate.
  5. Use one yardstick. To compare the candidate with a no-edge pick, set that pick aside and rank both against the other no-edge picks.
  6. Combine edge and robustness. The edge term ℰ averages the ranks of Sharpe and DSR (measures 1 and 2). The robustness term ℛ gives weight 5/8 to plateau drop (measure 3) and 1/8 each to CPCV worst-12, the SPA p-value and minimum track record (measures 4–6). Combine the two terms as T = 0.2 ℰ + 0.8 ℛ.
  7. A no-edge pick matches the candidate when the pick's combined result is at least the candidate's. Ties count against the candidate.
  8. p = 1 + the no-edge picks that match the candidate1 + the reruns. A history in which the rule picks no setting counts as no match.
  9. Second stage. When fewer than 10 of the 800 match, the reruns continue to 8,000, and everything above is counted over all 8,000.
  10. MinervaScore = min(99, ⌊33 × log10(1 ⁄ p)⌋), from 0 to 99, never rounded up.
MinervaScoreA no-edge pick matches the candidateReruns that match, at most
33at most 1 time in 1079 of 800
43at most 1 time in 2038 of 800
66at most 1 time in 10079 of 8,000
85at most 1 time in 37620 of 8,000
991 time in 1,000 or rarer7 of 8,000
MeasureComputed for the candidate and each no-edge pick, on the amount-held basisBetter
Sharpe on the amount-held basisThe Sharpe of the daily returns over the report's whole window (a no-edge pick: over its rebuilt history)higher
DSRThe DSR statistic of the development returns, with the search's one SR₀ for the candidate and every no-edge pickhigher
Plateau dropThe Sharpe minus the lowest Sharpe among the setting and the settings one step away, as a share of the Sharpelower
CPCV worst-12The average of the 12 lowest Sharpes over the 120 ways to take 3 of the 10 development sliceshigher
Superior Predictive Ability p-valueThe final part's average daily return, tested against zero, 9,999 resampleslower
Minimum track recordThe bars the final part needslower

Candidate 2 is chosen for steadiness: the setting whose rolling one-year Sharpe Ratio is below zero for the least time. Candidate 2 is compared only with the no-edge picks for Candidate 2. Steadiness gets half the weight: ½ × the count on time below zero (rolling Sharpe on the amount-held basis) + ½ × the combined result of step 6.

10 Probability of Overfitting (PBO)

Shown beside the MinervaScore · reports only · lower is better

PBO asks whether choosing the best setting mattered: do the settings that rank best in-sample also rank best out-of-sample? If the ranking does not carry over, the optimization's choice was no better than picking a setting at random. This is a different question from robustness. RobustGrade tests the chosen setting in-sample and out-of-sample; PBO tests only whether the ranking of the settings carries from in-sample to out-of-sample. Because PBO reads only the ranking, PBO cannot see how good the settings are: when every setting works about equally well, the ranking is random and PBO is high; when one narrow peak wins every time, PBO is low. So PBO counts in no score and is shown beside the MinervaScore.

To compute it, Minerva takes the 10 CPCV slices of the development period and puts 5 in-sample and 5 out-of-sample, in all 252 ways. In each split, the setting with the best average Sharpe in-sample is ranked against every other setting out-of-sample. PBO is the share of the 252 splits in which that in-sample best lands below the middle setting: 0.22 in parts 1 and 2 of the figure. Near 0, the ranking repeats. Near 0.5, the ranking is random. Near 1, the ranking reverses: each setting gained on particular dates and lost on the others.

Part 3 of the figure shows four kinds of search: no setting has an edge (PBO 0.40), one setting has an edge (0.01), every setting has the same edge (0.45), and each setting gains on different dates (0.99). Parts 4 and 5 set PBO beside plateau drop: on a flat curve plateau drop is small (0.16) and PBO is 0.41; on a narrow peak plateau drop is large (0.88) and PBO is 0.00; on a broad hill plateau drop is small (0.17) and PBO is 0.00. Read PBO with RobustGrade: a high RobustGrade with PBO near 0.5 means the result holds and the exact setting matters little; a high RobustGrade with PBO near 0 means the chosen setting is lastingly better than the others.

Mathematically
PBO = splits where the in-sample best ranks below the middle setting out-of-sample252

in-sample, out-of-sample: 5 of the 10 development slices each, in all 252 ways · a setting's score: the average of the setting's Sharpes on those 5 slices · settings: every distinct setting the report tried · each on the amount-held basis, with its own amount held

1 · Two of the 252 splits of the 10 slices: 5 in-sample, 5 out-of-sampleBlue: in-sample, where the best setting is chosen. Grey: out-of-sample, where that setting is checked.in-sample (blue):20 settings, rankedout-of-sample (grey):the same 20, rankedbestworstmiddle2ndThe in-sample best is 2nd of 20 out-of-sample:above the middle. Not counted.in-sample (blue):20 settings, rankedout-of-sample (grey):the same 20, rankedbestworstmiddle19thThe in-sample best is 19th of 20 out-of-sample:below the middle. Counted.2 · All 252 splits: where the in-sample best ranked out-of-sample020401st of 20 out-of-sample: 56 splits2nd of 20 out-of-sample: 42 splits3rd of 20 out-of-sample: 25 splits4th of 20 out-of-sample: 16 splits5th of 20 out-of-sample: 17 splits6th of 20 out-of-sample: 11 splits7th of 20 out-of-sample: 9 splits8th of 20 out-of-sample: 8 splits9th of 20 out-of-sample: 7 splits10th of 20 out-of-sample: 5 splits11th of 20 out-of-sample: 5 splits12th of 20 out-of-sample: 2 splits13th of 20 out-of-sample: 6 splits14th of 20 out-of-sample: 4 splits15th of 20 out-of-sample: 10 splits16th of 20 out-of-sample: 5 splits17th of 20 out-of-sample: 9 splits18th of 20 out-of-sample: 8 splits19th of 20 out-of-sample: 7 splitsmiddlebelow the middle in 56 of 252 splits:PBO = 56 ÷ 252 = 0.221st (best)5th10th15th20th (worst)out-of-sample rank of the in-sample best, among the 20 settings3 · The same count for four kinds of searchEach chart: where the in-sample best ranked out-of-sample in that search's 252 splits.Under each chart: PBO, and what is true of the strategy.No setting has an edgebestworstPBO 0.40nothing worksOne setting has an edgebestworstPBO 0.01works at one setting onlyEvery setting has the same edgebestworstPBO 0.45works at every settingEach setting gains on different datesbestworstPBO 0.99in-sample best, out-of-sample worst4 · The Sharpe of each setting along one parameterMade-up broad hill. The same curve is read three ways.Sharpe on the amount-held basis00.511.52setting 1: Sharpe 0.34setting 2: Sharpe 0.61setting 3: Sharpe 0.83setting 4: Sharpe 1.02setting 5: Sharpe 1.10setting 6: Sharpe 1.18setting 8: Sharpe 1.28setting 9: Sharpe 1.25setting 10: Sharpe 1.27setting 11: Sharpe 0.99setting 12: Sharpe 0.98setting 13: Sharpe 0.75setting 14: Sharpe 0.43setting 15: Sharpe 0.21plateau drop 0.17Sharpe one step away: 1.18above zero: the plateaupart of RobustGradethe chosen setting: Sharpe 1.35the chosen setting: 1.35what PBO reads:the differencesbetween all thesettings, top tobottom, andwhether theyrepeat out-of-samplePBO 0.00151015settings along one parameter, lowest value to highest5 · The same three readings on three shapes of curveFlat curve0plateau drop 0.16one step away 1.43PBO 0.41Broad hill0plateau drop 0.17one step away 1.18PBO 0.00Narrow peak0plateau drop 0.88one step away 0.51PBO 0.00