Minerva provides optimizations of trading strategies that require large amounts (hundreds of thousands to millions) of computationally intensive backtests.
Traders’ goals are finding the strategies with the highest Annualized Returns or Sharpe Ratio, for instance.
However, in the statistics and data science of trading, it is well-known that performing many tests will always reveal an attractive result purely by chance. So, how should a researcher interpret large searches for optimized trading strategies?
Minerva solves this by asking two questions. The first: is the good strategy I found lucky, or does it represent a skill (an edge)?
We can think of luck versus edge like this: is my result exceedingly rare (such as flipping a coin heads 50 times in a row—likely not lucky but instead sign of a weighted, or skilled, coin), or does it fall within the expected variance of randomness?
If it does fall within the expected variance of randomness, it likely means that the high Return or Sharpe we found appears inflated by getting lucky, or in other words, we picked a strategy that merely beat the others out of sheer randomness.
To summarize: a high Sharpe found from a single backtest among many backtests that could be performed by varying the dates, the parameters, etc., may be the result of those dates and parameters being lucky.
The only way to know whether we are seeing edge (a skilled strategy) or luck is to run more backtests and compare all of their results together to all of the results that randomness would give us.
The second question that Minerva asks is this: does my strategy contain signal, or is it just noise. This is a subtle but importantly different question to the first one and addresses the problem of robustness, regardless of whether or not our high Sharpe is edge or luck.
Signal persists; noise means the strategy happened to have skill for a little while, but not for any good reason. Only because it randomly stumbled into skill briefly, like a broken clock that is correct twice a day.
So, we can think of signal versus noise and robustness as being akin to persistence. If a strategy contains true signal, and isn’t noise, it means that—so long as the overall global regime doesn’t change—the strategy may behave similarly in the future.
Think of this as the similarity of the Sharpe over time (at any magnitude of Sharpe). Whether or not our strategy has a high Sharpe or a low Sharpe, one should understand if the Sharpe measured is representative of the true Sharpe expected going forward.
To describe signal versus noise more precisely: if I find a strategy on past data and measure its performance in-sample, the question is whether performance is similar on out-of-sample data.
Minerva has developed multiple interesting ways of looking at robustness. One of them is to think about whether your strategy’s parameters sit on a plateau of Sharpes, or a narrow peak in the Sharpe. For instance, if my strategy has a high Sharpe when I use a 90-day moving average but much worse Sharpes with an 89- or 91- day MA, it certainly means my strategy is on a peak rather than a plateau. And, if that Sharpe is also high—which is what the previous luck vs. edge question asked about—then I have overfit.
Minerva measures luck vs edge, signal vs. noise, and overall overfitting using a panel of known statistical tools plus its own proprietary measures.
Minerva answers the first question: is a high Sharpe Ratio that an optimization found just a lucky result? In other words, did the optimization merely pick the best parameters for the data it looked at and so overfit on the in-sample data?
For this, we measure the LuckFactor in optimizations and the LuckFactor+ in reports. Both are variations of the Deflated Sharpe Ratio (DSR). In a report, the MinervaScore also counts the Sharpe on the amount-held basis.
Minerva answers the second question: if I perturb this strategy by changing its parameters slightly--and if I test those changes on future (out-of-sample) data--does the strategy still produce a similar Sharpe?
For this, we measure the RobustGrade™, which includes plateau drop, the Superior Predictive Ability p-value, CPCV worst-12 and the minimum track record.
Before trading a strategy, one should seek answers to both of those questions. Neither can predict the future; nothing can. But, the two measurements together can tell you whether you should expect to see behavior close to what you tested, or not, relative to other trading strategies.
For this, we combine all of the above in our proprietary measure, the MinervaScore™.
Beside it, the report shows the Probability of Overfitting (PBO), which contains different and useful information to the other measures outlined above. All are explained in this document, below.
To obtain a MinervaScore, Minerva takes a trading strategy, parameterizes it in its engine, and runs a few million backtests across multiple optimizations to produce a report. The tests in the report use considerable compute to comprehensively probe parameters around the region of skillful (non-lucky) candidate strategies. It is a powerful tool that takes your original strategy and fine-tunes it for both edge and robustness. The MinervaScore is this final measure, 0–99, of edge and robustness.
The report and the MinervaScore provide insight into the relative skill and persistence of a strategy. While many traders test strategies that seem to perform well shortly after their tests but then do not—-and then label this phenomenon “alpha decay”--at Minerva we believe that some alpha decay is merely a poorly understood strategy performing exactly as expected. The MinervaScore provides that expectation.
Luck vs. edge · backtests, optimizations & reports · higher is better
The Sharpe of the strategy's daily returns on the amount-held basis (see Amount-held basis, above): their average divided by how much they swing (their volatility), annualized. Two strategies can gain the same with very different Sharpes; the less volatile one scores higher. Minerva widens the swing when good or bad days come in runs (the Newey-West correction), so streaky returns are not flattered. It is not the strategy's Sharpe Ratio: the Sharpe Ratio is the same Newey-West calculation on the strategy's own returns, and is also shown in optimizations and reports.
ā: average daily return on the amount-held basis · cj: covariance of those returns j days apart · q = ⌊T1/3⌋ · T: days
Luck vs. edge · optimizations & reports · 0 to 1, higher is better
Back to our example in the intro, if we see we flipped a coin that landed on heads 50 times in a row and so is likely weighted, DSR measures how many times a fair coin would produce the same 50-heads result, and compares that to your result. DSR keeps Bailey and López de Prado’s probability formula; only SR₀ changes, from assumed to measured.
SR₀ is the development Sharpe that choosing the best of the search's settings produces by itself, when no setting has an edge. It is measured, like the chosen setting's Sharpe, on the amount-held basis. The published formula gets it by treating every setting tried as an independent coin. Minerva's settings are not independent: settings one step apart trade mostly the same stocks on the same days. So Minerva measures it, the same way in an optimization and in a report. On each of the 120 CPCV splits, the search picks a setting on 7 slices, and the gap is that setting's Sharpe on those 7 slices minus its Sharpe on the 3 slices it never saw. With no edge, the Sharpe on the unseen slices is zero on average, so the gap is what the choosing produced. SR₀ is the average of the 120 gaps, scaled to the dates the search read: by √(7/10) in an optimization, because a split chooses on 7 slices and the search on all 10.
Panel A. One value per split: the gap of that split's winner. The average of the 120 gaps is 1.70, and 1.70 × √(7/10) is SR₀, 1.42.
Panel B, real. The chosen setting's development Sharpe, 2.00, with the estimation error that 8 years of daily returns leave. DSR is the probability that its true Sharpe is above SR₀: 0.95. With the published formula's assumed SR₀, 1.82, DSR would be 0.69.
SR: the chosen setting's development Sharpe · SRsin, SRsout: the Sharpe of the winner of split s on the 7 in-sample slices and on the 3 out-of-sample slices · Dsplit ⁄ Dsearch: the dates a split's choice read over the dates the search read, 7⁄10 in an optimization · T: development bars · SR and SR0 per bar in the formula · γ3: skew · κ: excess kurtosis · Φ: the normal curve's area · N: settings tried · σ: the spread of the settings' Sharpes · γ = 0.5772 · every Sharpe on the amount-held basis
Luck vs. edge · optimizations & reports · lower is better
Minerva's LuckFactor is a measurement of how lucky a Sharpe is. It is DSR read from the other side: 100% − DSR, rounded up to a whole percent. That is the chance that the chosen setting's development Sharpe is no higher than SR₀, the Sharpe that choosing the best setting produces by itself. Lower is better. Example, from the DSR figure: DSR is 0.949, so 100% − 94.9% is 5.1%, shown as 6%. The LuckFactor is used in optimizations.
The LuckFactor+ is used in reports. A report reruns its pick on rebuilt histories with no edge, so the LuckFactor+ counts instead of using a formula: of every 100 no-edge picks, how many reach the candidate's DSR. Example: 17 of the 800 no-edge picks reach it, so (1 + 17) ÷ (1 + 800) is 2.2%, shown as 3%.
DSR: measure 2, on the amount-held basis
h: one no-edge pick · c: the chosen setting · N: the reruns the candidate's MinervaScore uses, 800 or 8,000 · DSR: measure 2, with the search's one SR₀ for the chosen setting and for every no-edge pick
Signal vs. noise · in-sample · optimizations & reports · lower is better
In a Minerva report, many nearby settings of a strategy's parameters are backtested so that you can see whether the strategy is sensitive to minor perturbations. If it is, it will likely quickly collapse in real-world trading going forward. Said another way, does the optimization's resulting Sharpe depend on hitting one exact setting? Plateau drop is the delta between the Sharpe and the Sharpe of the settings nearby. It is the winning setting's Sharpe minus the lowest Sharpe among the settings one step away from the chosen setting (one parameter moved one step, the other parameters fixed). A strategy on a plateau barely drops; a strategy on a narrow peak drops a lot. If a 90-day average works but 89 and 91 days do not, plateau drop is large and the strategy is not robust. Plateau drop is measured in-sample: on the dates the setting was chosen on, which is the development period of an optimization or the whole window of a report.
c: the chosen setting · n: the nearest tested value on either side of one setting, the others fixed · SR: the Sharpe on the amount-held basis, each setting with its own amount held · optimization: SR is the average over the CPCV splits · report: SR is over the whole window, and the report's scores use plateau drop ÷ SR(c)
Signal vs. noise · out-of-sample · optimizations & reports · lower is better
SPA asks this: during the "future" out-of-sample holdout period, did the strategy beat noise? In other words, how often would a strategy with no signal (i.e., a signal that carries no real information) do at least as well as this strategy did, by chance alone? Minerva takes the holdout's returns, removes the average of those returns, reshuffles the days in short blocks 9,999 times, and counts how often a reshuffle does at least as well as the real result. An SPA p-value (probability) of, say, 0.01 means a random strategy with no signal does as well as the real result 1 time in 100, implying that the strategy's holdout result is likely to be due to signal rather than noise. The SPA p-value does not use a Sharpe: the SPA p-value tests the average daily return on the amount-held basis.
x̄*: the mean of x redrawn in circular blocks of ⌊n1/3⌋ bars · n: holdout bars · none under 30 bars
Signal vs. noise · in-sample · optimizations & reports · higher is better
During an optimization, the CPCV process slices the development period into 10 time ranges, and there are 120 ways to pick 3 of the 10 time ranges. The result of this CPCV process is 120 measurements of Sharpe across the entire tested period, each containing information about their respective slice of history. While these don't necessarily represent any future slices, they do imply a range of Sharpes that one may observe in the future. CPCV worst-12 takes the winning setting's Sharpe on each of those 120 combinations and averages the 12 lowest of the 120 Sharpes. A high value of the CPCV worst-12 means even the setting's worst stretches of dates showed good performance: a promising sign for robustness.
Signal vs. noise · out-of-sample · optimizations & reports · lower is better
MTR measures, for a backtest, how long of a time period is needed before one can be 95% sure that the Sharpe is truly above zero. To understand this we need to invoke basic statistics: all measurements have noise (error), and the error of an average shrinks as 1/√T as the number of observations T grows. So a Sharpe measured over T bars carries noise of about 1/√T. In order to be 95% sure our Sharpe is greater than zero given that noise, the holdout period must be at least as long as the measurement needed for the noise to shrink well below the measured Sharpe. Minerva shows the minimum track record in bars and as the dates those bars cover from the holdout's first day, beside the holdout's own bars and dates. Example: a holdout Sharpe on the amount-held basis of 1.6, with some skew and fat tails, needs 283 bars, 2024-07-08 to 2025-08-21; the holdout has 488 bars, 2024-07-08 to 2026-06-16, so the holdout is long enough. Seen the other way, a strategy with no signal (no information, just noise) shows a Sharpe at least this high over 488 bars with a chance of 0.0152, below 0.05 because the holdout is longer than the minimum track record. A Sharpe at or below zero never qualifies.
SR: the holdout's Sharpe per bar, on the amount-held basis · T: holdout bars · γ3: skew · κ: excess kurtosis · p ≤ 0.05 exactly when T ≥ MTR
Signal vs. noise · optimizations & reports · 0 to 99, higher is better
Minerva summarizes these four signal vs. noise metrics in one number. Two parts are measured in-sample: on the development period, the dates the search chose the winning parameters from. Two parts are measured out-of-sample: on the holdout, the "future" dates the optimization never saw. We combine both to see whether we see signal in both the development and the holdout period alike. The RobustGrade is simply the weakest of the four. Any one of the four failing is enough for future results to also fail. A long, significant holdout with a strong MTR says nothing about what happens one step away, etc. In other words: strategies are fragile, and most show alpha decay. If one sees a strategy that shows a high RobustGrade, it means all four measures are strong.
Each of the four components of the RobustGrade attempts to discern whether the strategy truly carries signal (information), or is just noise:
x: a Sharpe on the amount-held basis, per bar · T: the bars behind x: for the Sharpe one step away, the development bars in an optimization and the whole window's bars in a report; for CPCV worst 12, the bars of 3 of the 10 development slices · γ3, κ: skew and excess kurtosis of the chosen setting's development returns on the amount-held basis · pMTR: the minimum track record's p · pSPA: the Superior Predictive Ability p-value
Edge and robustness · reports only · 0 to 99, higher is better
The MinervaScore combines evidence for an edge with robustness across time and parameter choices. The edge term, ℰ, averages the ranks of the Sharpe ratio on the amount-held basis and the DSR statistic, the dedicated comparison against luck. The robustness term, ℛ, combines the ranks of plateau drop, CPCV worst-12, the SPA p-value and minimum track record.
The conversion 𝒞 compares the candidate with the picks of the same search on histories rebuilt with every setting’s estimated gain beyond its benchmark exposure removed. It counts how often a no-edge pick matches the candidate on the combined statistic T, then expresses that frequency as an integer from 0 to 99. A score of 33 means a match probability of at most 1 in 10, 66 at most 1 in 100, and 99 at most 1 in 1,000. The MinervaScore is served in reports only.
M: the MinervaScore · ℰ: edge evidence · ℛ: robustness · T = 0.2 ℰ + 0.8 ℛ · cm: on measure m, how many of the other no-edge picks are equal to or worse · 1 Sharpe on the amount-held basis, 2 DSR, 3 plateau drop as a share of measure 1, 4 CPCV worst-12, 5 Superior Predictive Ability p-value, 6 minimum track record · k: the no-edge picks whose T is at least the candidate's, both counted against the same other no-edge picks · N: 800 reruns, or 8,000 when fewer than 10 of the first 800 match · never rounded up
Expanding T gives weights of one tenth each for measures 1, 2, 4, 5 and 6, and one half for measure 3. The 20–80 split expresses the existing score’s weights on ranks before the conversion to score points.
| MinervaScore | A no-edge pick matches the candidate | Reruns that match, at most |
|---|---|---|
| 33 | at most 1 time in 10 | 79 of 800 |
| 43 | at most 1 time in 20 | 38 of 800 |
| 66 | at most 1 time in 100 | 79 of 8,000 |
| 85 | at most 1 time in 376 | 20 of 8,000 |
| 99 | 1 time in 1,000 or rarer | 7 of 8,000 |
| Measure | Computed for the candidate and each no-edge pick, on the amount-held basis | Better |
|---|---|---|
| Sharpe on the amount-held basis | The Sharpe of the daily returns over the report's whole window (a no-edge pick: over its rebuilt history) | higher |
| DSR | The DSR statistic of the development returns, with the search's one SR₀ for the candidate and every no-edge pick | higher |
| Plateau drop | The Sharpe minus the lowest Sharpe among the setting and the settings one step away, as a share of the Sharpe | lower |
| CPCV worst-12 | The average of the 12 lowest Sharpes over the 120 ways to take 3 of the 10 development slices | higher |
| Superior Predictive Ability p-value | The final part's average daily return, tested against zero, 9,999 resamples | lower |
| Minimum track record | The bars the final part needs | lower |
Candidate 2 is chosen for steadiness: the setting whose rolling one-year Sharpe Ratio is below zero for the least time. Candidate 2 is compared only with the no-edge picks for Candidate 2. Steadiness gets half the weight: ½ × the count on time below zero (rolling Sharpe on the amount-held basis) + ½ × the combined result of step 6.
Shown beside the MinervaScore · reports only · lower is better
PBO asks whether choosing the best setting mattered: do the settings that rank best in-sample also rank best out-of-sample? If the ranking does not carry over, the optimization's choice was no better than picking a setting at random. This is a different question from robustness. RobustGrade tests the chosen setting in-sample and out-of-sample; PBO tests only whether the ranking of the settings carries from in-sample to out-of-sample. Because PBO reads only the ranking, PBO cannot see how good the settings are: when every setting works about equally well, the ranking is random and PBO is high; when one narrow peak wins every time, PBO is low. So PBO counts in no score and is shown beside the MinervaScore.
To compute it, Minerva takes the 10 CPCV slices of the development period and puts 5 in-sample and 5 out-of-sample, in all 252 ways. In each split, the setting with the best average Sharpe in-sample is ranked against every other setting out-of-sample. PBO is the share of the 252 splits in which that in-sample best lands below the middle setting: 0.22 in parts 1 and 2 of the figure. Near 0, the ranking repeats. Near 0.5, the ranking is random. Near 1, the ranking reverses: each setting gained on particular dates and lost on the others.
Part 3 of the figure shows four kinds of search: no setting has an edge (PBO 0.40), one setting has an edge (0.01), every setting has the same edge (0.45), and each setting gains on different dates (0.99). Parts 4 and 5 set PBO beside plateau drop: on a flat curve plateau drop is small (0.16) and PBO is 0.41; on a narrow peak plateau drop is large (0.88) and PBO is 0.00; on a broad hill plateau drop is small (0.17) and PBO is 0.00. Read PBO with RobustGrade: a high RobustGrade with PBO near 0.5 means the result holds and the exact setting matters little; a high RobustGrade with PBO near 0 means the chosen setting is lastingly better than the others.
in-sample, out-of-sample: 5 of the 10 development slices each, in all 252 ways · a setting's score: the average of the setting's Sharpes on those 5 slices · settings: every distinct setting the report tried · each on the amount-held basis, with its own amount held