Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#422

My analysis from the reading

Over fitting, also know as over optimization of your model, can lead to a disastrous trading experience. Due to the use of past data to predict future outcomes, we are not 100% certain what will happen i.e. another war, another pandemic. The period that is recommended in the reading is from 9 to 11 years of data to create the model and a period of 1 to 2 years to test/train your model (ensuring we have all the possible market conditions i.e. Black Swan events). Another thing to take to consider when creating a trading strategy, are the trade signals (always use the same MA for shorts and longs, i.e. using the 20 days MA on shorts, they recommend using 20 days MA on Longs too), as this symmetric strategy WILL reduce the degrees of freedom from your model, making it more accurate and by consequence more profitable.

Any constructive comment is welcome, thanks!!!
1 Like
#423
  1. What is so dangerous about over-optimization?
    it will allow to look for any possible change required in the code for any deviation of the original idea: curve-fitting
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    between 9-11 years; that reduce the probability of curve-fitting and explore multiple scenarios possbiles through years of data
  3. Why should you avoid asymmetric trading signals?
it will be excessively prone to curve-fitting; 
 where 
   adjust code for longs and shorts;
hence making the code longer and double  of optimization is required
1 Like
#424
  1. curve fitting
    2.10 years
    3.increases freedom which is prone to curve fitting
1 Like
#425

  1. Over-optimization can lead to generating strategies good results in the past yet have poor results going forward.

  2. 9-11 years

  3. Adding asymmetry increase degrees of freedom and excessively prone to curve-fitted solutions.

1 Like
#426

  1. What is so dangerous about over-optimization?

    curve fitting

  2. How long should a testing period be if you are serious about building a profitable trading strategy?

    9-11 years

  3. Why should you avoid asymmetric trading signals?

    Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

1 Like
#427
  1. The model might not be good for predicting future events if it is curve fitted to much to past events.
  2. 9-11 years if possible
  3. It makes the model more prone to curve fitting.
1 Like
#428
  1. What is so dangerous about over-optimization?
    • Over optimization can lead to funneling your algorithm to a specific data set or curve fitting. When the algorithm is so over optimized that is performs perfectly in a specific data set it won’t have the flexibility to adapt once it gets executed in real time with a data set that can have new parameters not encountered in the tested data set leading to a failure in proper execution of your algorithm.
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    • Ideally you should be using long periods of testing data between 9-11 years of historical data if available.
  3. Why should you avoid asymmetric trading signals?
    • Because it tends to be prone to curve fitted solutions
1 Like
#429

  1. its dangerous because having too many things going on at once instead of having a system thats simple, and easily understandable.

  2. testing period should be around 9-11 years worth of previous data used in the algo.

  3. because they are too focused on macro data which goes against the grain of a simpler trading strategy essentially leading to curve fitting.

1 Like
#430

  1. Because then we make curve fitting. The problem is we are working with data form the past. Therefore it can not fit for the future, when we over optimize it.

  2. 10 years

  3. Because then we take the “Freedom of the Algorithm/Strategy” and a curve fitting is so more likely.

1 Like
#431

1. What is so dangerous about over-optimization?
We are restricting our ability to make a profitable strategy for future datasets
2. How long should a testing period be if you are serious about building a profitable trading strategy?
If possible 10 years. The more data the better.
3. Why should you avoid asymmetric trading signals?
It overcomplicates the strategy unnecessarily.

2 Likes
#432
  1. Over optimization is dangerous because past performance is no guarantee of future results.
  2. WE should different short periods for testing
  3. Adding different criteria for long and shorts automatically increases the strategy.
#433

  1. Over-optimization leads to a false sense of accuracy because it assumes the past data will apply to the future. But history doesn’t repeat, it only rhymes.

  2. Approximately 10 years (plus or minus one year)

  3. Asymmetric trading signals make the strategy excessively prone to curve fitting.

#434

1. What is so dangerous about over-optimization?
Overall, over-optimization is dangerous because it can result in trading strategies that are highly specific to the past data and do not generalize well to new market conditions, leading to poor performance and potentially significant losses

2. How long should a testing period be if you are serious about building a profitable trading strategy?
A general guideline is to test your strategy over a period of at least 5 to 10 years of historical data and 1 to 2 year blocks.

3. Why should you avoid asymmetric trading signals?
Asymmetric trading signals, where separate criteria are used for entering and exiting long and short trades, should be avoided in order to reduce the risk of overfitting a strategy to historical data. While it may be true that up and down trends have developed differently in the past, this may not continue in the future as market conditions change

#435
  1. What is so dangerous about over-optimization?

It is dangerous because you can almost perfectly fit your trading system to a particular past historic market dataset that you are working with, which may have some specific market conditions that may not be present in the future thus it will not be able to adapt to new market conditions making the system unprofitable.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

The testing period must not be less than 5 years, preferably you should work with a testing period of 9 to 11 years and you should also use a one year or two of historic data outside the optimization dataset you are using in order to test your trading system in other market conditions outside the period it was designed and optimized for.

  1. Why should you avoid asymmetric trading signals?

You should avoid asymmetric trading signals (like having different criteria for opening long and short positions) as it makes your trading strategy prone to be curve-fitted.

#436

1. What is so dangerous about over-optimization?

Over-optimization leads to curve-fitted systems that are “fit” to test profitably in the past but fail to profit in the same way in the future.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

Optimizations are not relevant if they are done within periods of less than 5 years because smaller periods of time are not statistically relevant according to long term changes in market volatility. Optimizations should be carried out for long periods of time, ideally 10 years of data should be used in order to ensure that a large number of market conditions are taken into consideration. If a simple strategy yields profitable results across a ten-year period then the probability the system has been curve fitting is greatly reduced because the system has limited degrees of freedom to artificially “fit” all those different market conditions.

3. Why should you avoid asymmetric trading signals?

Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#437

  1. What is so dangerous about over-optimization?
    curve-fitted strategies

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    ideally 9-11 years of data should be used.

3, Why should you avoid asymmetric trading signals?
separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#438
  1. What is so dangerous about over-optimization?

Over-optimization, or curve fitting, risks tailoring the strategy too closely to past data, reducing its ability to perform well in the future since past market events may not reoccur.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

A testing period should span 9-10 years to ensure the strategy performs across a range of market conditions.

  1. Why should you avoid asymmetric trading signals?

Asymmetric signals increase curve-fitting risks by adding complexity. They assume past conditions will repeat, which is uncertain, leading to less robust strategies.