0

I sometimes read articles where author X presents a new trading strategy, and then shows backtest results to "prove" that the trading strategy was more profitable than some chosen benchmark. (Let's assume that X has no intention to deceive; X did not cherry-pick the data.)

  • X's premise: If my trading strategy is worthwhile, the backtest will show profitable results.
  • X's observation: The backtest shows profitable results.
  • X's conclusion: My trading strategy is worthwhile.

Isn't X committing the logical fallacy of affirming the consequent? For example:

  • Premise: If an animal is a healthy cat, it will have four legs.
  • Observation: That horse has four legs.
  • Conclusion: That horse is a healthy cat.

From my understanding as shown above, backtesting cannot be used to "prove" the validity of a trading strategy. So what is the use of backtests? Is backtesting only useful for disproving trading strategies (using modus tollens)? What did I get wrong?

2
  • More like "If my trading strategy is worthwhile, it will work in a large variety of different situations", "The backtest show profitable results in many different situations", "My trading strategy is probably worthwhile".
    – chepner
    May 13, 2021 at 12:36
  • @chepner: But the problem is that the backtest only shows what happened in ONE situation (or combination of situations), which is what actually happened in the past.
    – jamesqf
    May 13, 2021 at 16:49

5 Answers 5

2

Backtesting has a problem with overfitting the data. Two ways to combat this are out-of-sample testing and forward testing.

1
  • 1
    There's a difference between back testing and optimization. The latter is curve fitting. May 8, 2021 at 11:32
2

The purpose of backtesting is to provide additional information about the robustness of a strategy across different market conditions.

Note that back testing does not mean optimization. Optimization is curve fitting where in hindsight you determine the parameters that would have provided the best result.

Ideally, one would seek to identify a strategy that worked for some period of time and then layer upon that something that indicated when that strategy would not work.

The ultimate test of a strategy is real time. Only then can you claim true viability.

2
  • "The ultimate test of a strategy is real time" — But how can you be sure that the performance is not due to very good luck or very bad luck?
    – Flux
    May 8, 2021 at 16:06
  • 1
    What can you be sure of in the market, whether your positions are directional or non directional? If there's risk, it can be realized, despite your best efforts. Bear in mind that there's more to a strategy than just putting on the position and watching what happens. There's risk management as well as nuances to the strategy as you react to price change. A strategy tends to evolve in real time as well. The bottom line is that if you believe that you have an edge, you have to plunk some money down to find out if it indeed is an edge. May 8, 2021 at 18:29
1

It's a fact, that nobody can know the future of a chart (without insider information). This also means you can not "prove" a trading strategy other than executing it successfully with real assets.

Despite that, people always try to guess the future based on the past. That's why back testing often is a taken as a (invalid) proof and norm to judge about a trading strategy.

Another reason for back testing is, that there is no other indicator for the success of a trading strategy.

Even if someone successfully executes a trading strategy for a certain period, when you enter the same strategy afterwards, the success story of the trader is just worth as much as back testing, since you can not know the future.

5
  • Not all trading strategies are based on guessing what the future will be. May 8, 2021 at 18:31
  • @BobBaerker do you have examples, where you still use back-testing?
    – Tarion
    May 15, 2021 at 13:07
  • @BobBaerker "Not all trading strategies are based on guessing what the future will be" — Could you give some examples?
    – Flux
    May 16, 2021 at 7:31
  • No, I do not have any examples where I still use backtesting. I have been in the market for 40 years and have also traded for the past 20 years so at this point, I know what I want to do as well as what I can do. I'm no longer looking for new strategies. May 16, 2021 at 11:40
  • Example? Trading volatility. Short term option implied volatility before earnings announcements often expands dramatically and far more than subsequent expirations. IV is going to contract post EA. No prediction needed to make that happen. May 16, 2021 at 11:41
1

Only a partial answer, but note that what you describe as "affirming the consequent" is the same as a classical scientific experiment but without a control condition. (For instance, if this fertilizer works well, and I add fertilizer to the plant, it will grow big; it grew big; therefore my fertilizer works well.) If a person really engages in the logic you describe, then yes, that is fallacious.

However, in at least some cases the people touting a strategy are comparing the backtest against an alternative "control" condition, such as the performance of an S&P 500 index fund over the same period of time. Then it is no longer just affirming the consequent, because it is varying the experimental conditions (buy the S&P 500 vs follow my cool strategy) and observing the effect on the result. Of course there can still be debate about whether the baseline was an appropriate choice, whether the backtesting itself is a good way to evaluate the strategy, etc., but from a logical perspective a claim with a baseline is on firmer ground.

-3

This is a logical fallacy. Consider this:

  • premise 1 (what you said): If my trading strategy is worthwhile, the backtest will show profitable results.
  • premise 2 (what I add): If my trading strategy is worthwhile, the backtest will show profits that exceed the total commissions (that can be quite high)
  • observation 1 (what you observed): The backtest shows profitable results.
  • observation 2 (what I observed): The backtest shows profitable results, but the profits are lower than the commissions of the trades
  • Erroneous conclusion: My trading strategy is worthwhile.
  • Correct conclusion: My trading strategy is not worthwhile.

For a trading strategy to make any sense at all, several properties need to be true:

  • The market inefficiency must never be fully taken advantage of. So if someone else has also observed the trading strategy to make sense, it soon becomes public knowledge and the market inefficiency is fully taken advantage of. If you are a large investor, the mere use of that trading strategy probably fully takes advantage of all trading inefficiencies so you using that trading closes the window of opportunity for profit. Keep every strategy secret!
  • The trading strategy must work at all times. It is easy to come up with 10 000 trading strategies. Backtesting may show 1000 those work with historical data. If you randomly choose some trading strategy that worked with historical data, most likely it doesn't work with future data. If you have some a priori reason to assume some trading strategy should work (and keep the reasons secret), and if backtesting shows every 10-year period shows profit during the last 100 years, then perhaps maybe the trading strategy could work in the future as well. Remember to statistically prove that the results cannot be random at a significance level of 1% or so. And remember if you have 100 strategies you attempt to statistically prove at a significance level of 1%, on the average you get 1 fluke -- 1 trading strategy that was proved statistically but is complete garbage.
  • But (and there's always a big but): typically market inefficiencies are very minor, so to make any profit from trading, you must have low commissions. Usually low commissions are restricted to largest investors. If you are a large investor, it could be the case that being a large investor you affect the pricing of the market so much that mere usage of that trading strategy could close the window of opportunity by itself and therefore the trading strategy no longer works. If you are a small investor, the trading strategy could otherwise work in theory but you probably pay so high commissions that it doesn't make sense.

I don't believe it makes sense for an average investor to trade. The best risk-adjusted return can be obtained not by maximizing the turnover of your portfolio, but by minimizing your costs.

1
  • Commission costs are not the most pressing issue related to whether an individual investor should actively trade [biggest concern would likely be risk through diversification]. In particular, there are many commission-free services to amateur investors. As well, this doesn't really get at the core of the question itself. May 14, 2021 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.