Reading Assignment: Common Backtesting Mistakes

Any Strategy that could work, could be applied to even newest companies or assets, but if you want to design a long term strategy that could be able to behave exactly has is desire, it is suggested to test your strategy on different time ranges. IF your strategy is capable to work on very long range of times like 9 to 11 years (or 7 like ETH) you could think that is a solid strategy that can work for the long run.

If you have any more questions, please let us know so we can help you! :slight_smile:

Carlos Z.

And what is “long term”? Is there any convention of that definition? Because for example, if I take the daily candles, I would have 365 samples per year, but If I take the hourly candles, I would have 24*365 candles (in the case of BTC or any other cryptocurrency), and If keep going on and on I would have even more. So, what is a long-term strategy? Is it defined by a number of candles or is it defined by a time frame?

Because I could take a minute candle and I could define long-term? as 1 month for example? or even with HFT … I guess you could even define a day as a “long-term” strategy. But then, it all comes down on where you’re getting your information from, i mean if they can give you each minute candle, or 30sec candle or 1 sec candle and so on.

1 Like
  1. What is so dangerous about over-optimization?
  • The danger lies in that you could unintenionally create a common mistake condition known as curve fitting, wherein your trading strategy does extremely well with the past data you used to back test it on but then performs very poorly or not as well as expected when used outside this past data set.
  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • The longer the testing period the more we will be able to ensure that curve fitting is not unintentionally applied. A data period of 9-11 years of back testing is recommended to allow for a large amount of market conditions to be available.
  1. Why should you avoid asymmetric trading signals?
  • Using asymmetric trading signals insert increases in the strategy’s degrees of freedom and makes it excessively more prone to curve fitting.
1 Like

It can be defined by a time frame, candles just represent data of a specific time frame.

For me (my personal opinion), time frames are divided in 3 groups:

  • short-term: less than 6 months.
  • mid-term: from 6 months to 2 years.
  • long-term: more than 2 years.

Now what i understood from the course is: a good strategy is capable to behave exactly has expected in different time frames (hours, days, months or years), if you had programmed a strategy that goes well in hourly time frame, but goes unprofitable in a monthly time frame, means that your strategy could not work well in longer time frames, might need more backtesting for different time frames.

I mean, if you want a short term strategy (a bot trading hourly or daily) thats ok, just be sure its works fine for that time frame, then you could improve that strategy for different time frames.

If you have any more questions, please let us know so we can help you! :slight_smile:

Carlos Z.

1 Like

1 You don´t want to fit your algorithm to previous data, because the future will be different!
2 9-11 Years of data
3 Adding different criteria for longs and shorts automatically increase the strategy´s degrees of freedom and makes it more prone to be curve-fitted.

1 Like
  1. curve fitting.
  2. Optimizations should be carried out for long periods of time, ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.
  3. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.
1 Like

1.Its dangerous because it leads to curve fitting. Curve fitting is where you begin to form your strategy around the chart and indicators.

  1. 9-11 years of data should be tested.

  2. They are often driven buy past data and a poor indicators for future data therefore they lead to curve fitting.

1 Like
  1. What is so dangerous about over-optimization?
  • When we have created a strategy, we test it on some past data set and optimize for greater profits. However, if we adjust the strategy to this specific data set too much, it may decrease its profitability in other situations.
  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • It should be 9 to 11 years. In crypto space, though, we have to make use of what we have.
  1. Why should you avoid asymmetric trading signals?
  • Because their profitability will be greatly affected by the macroeconomic environment. Technical analysis as such doesn’t go into that.
  • In crypto, though, I think strategies should be created differently for bull and bear markets as we do have clear cycles. For now, a long position generally should be a safer bet than a short position.
1 Like
  1. Over optimization can lead to overfitting your model to the sample data. That means that the model will be effective for the particular set of data but not in general.
  2. Nine to eleven year should be the data in general, and one should consider to test the model in different periods too, for example you can test the model for one year data and after tweaking it testing the model for a different year.
  3. With asymmetric trade signals you are in danger of overfitting, because currencies different of interest rate changes can change your model efficiency.
1 Like

What is so dangerous about over-optimization?

When done incorrectly, optimization leads to curve-fitted systems which are “fit” to test profitably in the past but fail to profit in the same way in the future. Because the future may look nothing like the past in a particular market – the “fitting” of parameters onto the past “curve” of data may cause big problems on the future data curve, causing the system to be out of phase and potentially causing investors losses.

Imagine a system that buys or sells newly created tokens on a breakout (after an ICO event) abd was back tested on above or below the market high or low for the past X number of days. When testing the system on the past data, the testing may show $3,000 in profits when using a 10 day high/low, $60,000 in profits when using a 20 day high/low, and $100,000 when using a 30 day high/low.

How long should a testing period be if you are serious about building a profitable trading strategy?

5 years+

Smaller periods of time are not statistically relevant according to long term changes in market volatility.

Tip: For Bitcoin I would go since the beginning, 2009. The problem with altcoins is that some of them do not have a long trading history, and therefore cannot be traded with a bot. An alternative solution would be to run Pearson Correlation tests, against other coins with similar properties e.g ETH has 93% correlation with BTC. Generally speaking short lived token cannot be traded.

For more information see link

Why should you avoid asymmetric trading signals?

Because it does not take into consideration macro economic variables of the asset. The highs and lows of BTC in 2020 compared to 2017 are dedicated by different events e.g. Covid-19 was responsible for the March drop, which did not exist on 2017.

For more information see link

1 Like
  1. Curve fitting, you adjust the algorithm to much to past events.
  2. Best is to test it over a longer period - 1-2 year old data
  3. The context of the events was probably different when you use such a strategy e.g. macro economic situation, interest rates.
1 Like

Excellent answers (this one and others from you in different topics), It’s easy to understand. Please keep them like that :muscle:

Carlos Z.

  1. What is so dangerous about over-optimization?

The danger is to over-fit the model to the past data. That could be extremely dangerous because the trader could think that the model is extremely good when in fact it just was extremely good during the data set in which it was developed, and not in future datasets. That is why simplicity is a feature that ensures the lack of a model over-fitting.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

I would say that it depends on the strategy itself. 9 to 11 years is a good amount of data in order to create a model that could work indefinitely, as it would have data from different market conditions.

  1. Why should you avoid asymmetric trading signals?

Because it makes it specially prone to curve-fitted models.

1 Like

1.Reasons why over optimization can be dangerous.

  • Optimization period length
  • Reliability of the simulation
  • Ignoring the results surroundings
  • Find grid optimization
  • Reoptimizing after optimizing
  1. The testing period for a great trading strategy should be approximately 9 to 11 years.
  2. Asymmetric trading signals should be avoided, because they will work better in one direction than the other. Because we can’t make assumptions of the future moves, it is better to have a strategy work equally well in both ways. Asymmetric signals can also indicate that we have fitted our strategy too much to suit our simulation.
1 Like
  1. What is so dangerous about over-optimization? Curve fitting.
    The less complexity and less parameters available within a given strategy the less probable it is that it will ever be curve fitted as systems that don’t have complex criteria tend to be unable to “fit” to the data if a true inefficiency is not present.

  2. How long should a testing period be if you are serious about building a profitable trading strategy? 9-11 years of data.

  3. Why should you avoid asymmetric trading signals?
    Leads to more complexity and curve-fitting.

1 Like
  1. What is so dangerous about over-optimization?
    Over optimization leads to curve fitting. basing your strategy on the exact same “old” patterns. The patterns in the future will never be exactly the same. History doesn’t repeat but it rimes.
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    The testing period sould be 9 to 11 years !!!
  3. Why should you avoid asymmetric trading signals?
    It would probably lead to over optimization…
1 Like

1- Our goal is to produce systems that achieve good performance in the past with the highest possible guarantee that that performance will be repeated in the future. If we adapt the code too much to a specific set of data that is very unlikely to repeat exactly we will be having great results on past data but bad ones on future.

2- Long periods of time, around 10 years.

3- Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

1 Like
  1. Curve fitting your system to past data and therefore your system is unable to behave similarly on a different data set.

  2. Simulations on periods should be are as long as possible. The longer the trading period, the more statistically significant the data set is and the less likely it is to allow the curve fitting of your system. Long time periods introduce a wide variety of market conditions which make curve fitting very difficult. However the number of trades is also very important. Having more trades for a given long trading period is better (against curve-fitting) since it implies a larger number of market conditions in which the EA was able to trade and succeed.
    Optimizations should be carried out for long periods of time, ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.
    Strictly speaking, optimizations are not bound to be meaningful fit they are done within periods of less than 5 years given that smaller periods of time are not statistically relevant according to long term changes in market volatility. Using a smaller period will most likely “fit” your strategy to very specific market conditions and will make it unable to perform correctly as the market changes.

  3. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

1 Like
  1. Over-optimization is dangerous because it can easily result in a curve-fitted system.

  2. The testing period for building a profitable trading strategy should be 9-11 years on historical data and 1-2 years on out-sample testing.

  3. Asymmetric trading signals should be avoided in system development because they do not take in to consideration macro economic variable like interest rates and global pandemic! Having separate criteria for short/long positions make the system excessively prone to curve-fitting.

1 Like
  1. What is so dangerous about over-optimization?
    You could start curve fitting your strategies to past datasets.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    9 to 11 years

  3. Why should you avoid asymmetric trading signals?
    because the results cannot be guaranteed that they will produce similar results in the future do to the factors of interest rate differentials or such similar macro economic variables that inevitably change through economic cycles.

1 Like