Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#101
  1. What is so dangerous about over-optimization?

That our trading algorithm will “copy/paste” past data or the lack of data will be fitted artificially on our trade strategy ; when doing a backtest the results might not be accurate nor trustworthy.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

Between 9 months and 12 months.

  1. Why should you avoid asymmetric trading signals?

To increase freedom to the trading strategy which in return will be prone to curve-fitted solutions.

(paraphrasing from the article : Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.)

#102

  1. When you keep tweaking an algorithm to fit a specific fixed time period data set you enter the realm of curve fitting, in which the algorithm becomes too complex for good probability trades and though performing well on the bias data set will likely fail to perform well in other time periods of same data set. By being too fixed on a single, specific moment in time, you lose the ability to see trade opportunities “before they happen”.

  2. For a serious test of a profitable trading strategy it is suggested to work with approximately 10 years of data for any given instrument-exchange set.

  3. Asymmetric events (transients) are poor subjects for trading signals as they are not likely to re-occur in the exact same way every time. You might get struck by lightning once…but again? Conversly, symmetric events (cycles) are better subjects for trading signals as they follow re-occuring patterns. This is what good traders do, find repeating patterns that provide profitable entry and exit points within their risk tolerance.

#103

What is so dangerous about over-optimization?
It transforms the strategy in a curve-fitting of past data that will never be the same as future data.

How long should a testing period be if you are serious about building a profitable trading strategy?
9-11 years

Why should you avoid asymmetric trading signals?
It make it prone to curve-fitting.

#104

  1. What is so dangerous about over-optimization?
    You begin to have curve-fitting - fit the model to essentially confirmation bias.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    10 year period provides far more than needed because the model would depend on your timeframes.

  3. Why should you avoid asymmetric trading signals?
    Curve fitting again with bias of trend.

#105

  1. Over-optimization is dangerous when it turns into curve-fitting. We don’t want to generate trading strategies with absolutely astonishing results that will not be achievable going forward. Since our goal is to produce systems that achieve good performance in the past with the highest possible guarantee that that performance will be repeated in the future it becomes vital to take steps in order to ensure that optimization does not deliver curve-fitted strategies. When done incorrectly, optimization leads to curve-fitted systems which are “fit” to test profitably in the past but fail to profit in the same way in the future.

  2. If I am serious about building a profitable trading strategy the testing period should be between 9 -11 years optimally and a minimum of 5 years.

  3. I should avoid asymmetric trading signals because adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#106

  1. Over-optimization leads to curve fitting. Curve fitting is where you start to form your strategy around the chart and indicators.

  2. 9-11 years of data should be used to test the strategy to ensure large amount of market conditions become available.

  3. Asymetric signals or adding different criteria increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#107
  1. too much tweaking )curve fitting) by working with historic past data might not give a useful indicator for future - even the basic idea of one´s strategy might get lost in the process
  2. longterm, for example 6-12 months or more (not only a few minutes…), but also use different historic data from different time-frame to check if one’s program-outline leads to the desired result
  3. sorry, I am not really sure what is meant by “asymmetric signals”…perhaps some unprecedented events which are out of the norm and not representative - or, what is it?
#108
  1. over-optimization leads to curve fitting which can make it suitable for past events but not flexible in handling future ones
  2. At least one year testing on a data set 9 to 11 years old is best.
  3. Asymmetric trading signals leads to more complicated situations which quickly leads to curve fitting.
#109

  1. the biggest problem with over optimization is curve fitting.

  2. 9 to 11 years

  3. Because asymmetric trading signals are more for short term trading?

#110
  1. What is so dangerous about over-optimization?

Since our goal is to produce systems that achieve good performance, over-optimization poses the threat of having no predictive power, rendering it useless when programming a strategy.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

9-11 years is a good testing period to build a profitable trading strategy, with 2 years of out-sample testing.

  1. Why should you avoid asymmetric trading signals?

Asymmetric trading signals present a higher probability of making a mistake, and a higher likelihood to over-optimize.

#111

Reading Assignment: Common Backtesting Mistakes

  1. What is so dangerous about over-optimization?

By doing curve fitting with a small set of data, or to try to match a set of data to get the fantastic profit on a specific pattern then hope that this pattern will be repeated. As later on the almost exact pattern can do the opposite.

2.How long should a testing period be if you are serious about building a profitable trading strategy?

By developing simple symmetric strategies with limited degrees of freedom and reliable simulations over long periods of time with one or two years of out-sample testing the possibility to find a curve fitted solution will be extremely unlikely.

3.Why should you avoid asymmetric trading signals?

When you have one strategy to buy and another to sell, the risk is increasing for mistake as it’s more complicated, and a grater risk for over optimization as there are more parameters to adjust and tweak.

#112

one can perfect the backtesting results so much so that the results are amazing but do not work in the future this is called curve fitting.

they should be carried out for 9 to 11 years as this usually give the best idea of the market.

having different rules for going long and short limits the freedom of the program and can cause curve fitting.

#113

  1. It makes it more likely to curvefit

  2. 9-11 years of data ideally. this gives a large spread of market condition to be tested on

  3. By adding to the complexity of the system. it makes it more prone to curve fitting

#114

  • What is so dangerous about over-optimization?
    A: Curve fitting.

  • How long should a testing period be if you are serious about building a profitable trading strategy?
    A: Ideally 9-11 years.

  • Why should you avoid asymmetric trading signals?
    A: Because asymetric criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#115 
1. What is so dangerous about over-optimization?

Over-optimization leads to curve fitting and potential losses in the future when we apply the strategy.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

Preferably over long periods of time, usually 9-11 years is a good range, with one or two years of out-sample testing.

3. Why should you avoid asymmetric trading signals?

Asymmetric trading singnals increases the strategy’s degrees of freedom and complexity making it more prone to curve-fitted solutions.

#116

Over optimisation will create a trading system with good results based on a given data set but unable to behave similarly on a different data set.

Testing should be based on periods of around 10 years

Using asymmetric trading signals will lead to curve fitting by increasing strategy’s degree of freedom required to accommodate these type of signals.

#117

1: It is dangerous because optimizing the curve for past time may not correlate accurately for the future. Having a high degree of success with past history after optimization will most likely not give as good result in the future.

2: 10 to 11 years of back data is a good idea

3: Because you are using separate criteria for short and long trades it makes it more prone to curve fitting

#118
  1. Overfitting your algorithm could lead to you losing money. You have to try to keep your bias out of your implementation, and look for conditions that will generalize to future data. Look for patterns and repetition rather than edge cases to define your rules and conditions.
  2. This is kind of difficult to answer, but you should cross validate your data, and maybe even run it on different sets to see how it stacks up against unpredictable data. Test your results and compare against control data (hodl data), or certain thresholds to determine if it is worth moving forward.
  3. Asymmetric signals are not generalizable, and could cause unexpected behavior in your algorithm. Look for patterns instead.
#119

1. What is so dangerous about over-optimization?

The better and tighter the optimization, the more curve-fitted the system will become, this is a reason why neural networks, which are excellent at optimization, tend to fail in successful trading systems as they always curve-fit their data excessively.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

Optimizations should be carried out for long periods of time, ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available. If a simple strategy yields profitable results across a ten year period then the probability of curve fitting is greatly reduced as the system has limited degrees of freedom to artificially ‘fit’ all those different market conditions.

3. Why should you avoid asymmetric trading signals?

Market uptrends tend to be more gradual and downtrends tend to be sharper and steeper and become cascading declines and the daily range in prices tends to be higher during downtrends than uptrends.
People prefer avoiding losses to acquiring equivalent gains. Some studies suggest that losses are twice as powerful, psychologically, as gains. This bias skews our assessments of probability.

#120

Answer:

  1. Over-optimization can be dangerous resulting in curve-fitting which replicates the historical movement instead of future trend. Also, over-optimization lacks the required level of statistical significance in the back test result preventing a more flexible and less complex system to adopt any different market condition.

  2. Ideally, it shall be 9-11 years as this timeline of horizon should covers all kind of market cycles conditions depicting crisis and bull-run during that period.

  3. It is because no one trading signals mimics exactly to one another due to macro economic variables and interest rate differential. By keeping it symmetrical allows more freedom and thus avoiding curve fittings.