Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#81

1.) Over-optimization can eventually lead to curve-fitting. This is when one adjusts the trading parameters to fit the given data.

2.) A serious testing period should be across 9-11 years.

3.) You should avoid asymmetric trading signals because it increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#82
  1. It can produce curve fitting, over optimization will simulate events that hardly can be replicate in the future.
    2.Between 9-11 years, ideally you want to have an long period of data.
  2. It can lead to curve fitting, separate criteria include more variables and much uncertainty
#83

  1. What is so dangerous about over-optimization?
    Will be perfect working on your history, unfortunatley results out of the history will never plan the future exactly

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    9 -11 years though take 6 months to iterate and test your strategy

  3. Why should you avoid asymmetric trading signals?
    This lead to mistakes in the long run

#84

A:“optimization”, a process by which the results with many different parameter sets are compared and the best ones amongst those are chosen. Optimization is a natural part of system development since changes in certain things – like indicator periods, stop loss and take profit values – can dramatically affect the performance of a trading strategy. However one of the main problems of performing optimizations is the dreaded word : curve fitting

A: ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available

A: it is true that under past data up and down trends might have developed differently in currencies this cannot be guaranteed to continue in the future as these differences rely on interest rate differentials or such similar macro economic variables that inevitably change through economic cycles. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solution

#85
1. What is so dangerous about over-optimization?

Over optimization leads to the strategy working only on the specific data set it was trained on. This leads to poor performance on new data.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

10 years so that the strategy has exposure to a wide variety of market conditions as possible.

3. Why should you avoid asymmetric trading signals?

Asymmetric conditions make the strategy more prone to curve fitting. Even if an asymmetric strategy appears to work well in the past these same conditions may not hold in future data. A symmetric approach is also simpler and simpler is better.

1 Like
#86
  1. History never repeats itself. So if the code to optimised for a specific time frame in the past, is to rigid it is likely to fail in other situations. It’s like panda bears. Only eating the very top of some type of bamboo trees just don’t make them very adaptable to environmental changes.
  2. 9-11 years.
  3. I guess in doing so there this is already building in a sort of curve fitting. It’s already a biased optimisation
#87
  1. That the over-optimization works with known data but not wel with future data…
  2. You can use several sets of data like 1 year, 6 months and a month. You should also use data set when the market is bullish or bearish. And improve your script.
  3. You will stuck with a lot of bad bags… try to trade symmetric and gain steady gains.
#88
  1. Over-optimization make your strategy to be over adapted to past events whit great results but maybe, with terrible results in the future.
  2. the longer, the better, more then 5 years ideally, time frame greater than 30 min.
  3. More parameter, more probability of making a mistake, a strategy must be as simple as possible with more freedom for the future market condition.
#89

  1. What is so dangerous about over-optimization?
    The main danger is that it overly optimized strategy would be working perfectly on past market fluctuation, but would very likely bring unpredictable results. It is opposite to the aim of strategy, which is catching a tendency and working on updating and optimizing the strategy in future.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    We should take different short periods for testing as well, like 6 months or a year. However ideally, it is to check 9 to 11 years worth of market history.

  3. Why should you avoid asymmetric trading signals?

Because it adds more complexity to the strategy. It must be simple and ‘‘elegant’’, which will allow the strategy to adjust better.

#90
  • What is so dangerous about over-optimization?
    Congestion
  • How long should a testing period be if you are serious about building a profitable trading strategy?
    Several years period
  • Why should you avoid asymmetric trading signals?
    Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solution
#91

1 - What is so dangerous about over-optimization?

Avoid curve fitting should be a very important part of any system developer’s efforts.
Avoid simulating systems that trade on time frames lower than 30 minutes, or systems with very small take profit and stop loss targets (below 10 times the spread), as the results will not be viable and a lot of curve fitting to past data will most likely take place under optimization.

2 - How long should a testing period be if you are serious about building a profitable trading strategy?

Optimizations should be carried out for long periods of time, ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

3 - Why should you avoid asymmetric trading signals?

Asymmetric trading signals adds more complexity, which further refines the strategy towards a specific dataset, thus the curve-fitting effect.

#92

What is so dangerous about over-optimization?

  • your strategy will ONLY be optimized for the given dataset

How long should a testing period be if you are serious about building a profitable trading strategy?

  • different historical multiple Year (>5year) data would be needed to build a robust system

Why should you avoid asymmetric trading signals?

  • keep it simple stupid to avoid seperate criteria for longs and shorts, this could avoid curve fitting to occure.
#93
  1. You don’t want to fit your algorithm to previous data, because the future will be different
  2. 9 - 11 years, apparently. (the article also said not to perform strategies on data under 30minutes. I wonder if that’s true even when trading defi?)
  3. Having different criteria for long and short signals, will increase the programs vulnerability to variability and thus increase the motivation to curve fit, i think.
#94

  1. Over-optimisation can lead to curve fitting which may reflect high performance using past data, however it can be less useful moving forward into less predictable environments.

  2. Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

  3. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#95

  1. What is so dangerous about over-optimization?
    curve-fitting

  2. How long should a testing period be if you are serious about building a profitable trading strategy?

you will have to look back at a timeframe of about 9-11 years.

  1. Why should you avoid asymmetric trading signals?
    as it can lead to curve-fitting
#96

Isn’t it suppose to be a symmetrical system not asymmetric?

#97
  1. Curve fitting: generate trading strategies with absolutely astonishing results that will not be achievable going forward.
  2. 9-11 years
  3. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.
#98
  1. What is so dangerous about over-optimization?

With over-optimization we might end up with so called curve-fitted strategy which is adjusted to the past performances and it is so specific that is unlikely that its performance will be repeated in the future

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

Ideally backtesting should be implemented for around 10 years of data in order to ensure that a large amount of market conditions are included in the strategy.

  1. Why should you avoid asymmetric trading signals?

Adding different criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it more prone to be curve-fitted.

#99
  1. You might be doing curve fitting and adapt it for specific past events. This will affect the ability to use yous strategy on future market.
  2. 5 years, though >10 would be ideal.
  3. Could get you back into curve fitting.
#100

1 It’s dangerous (and basically useless) because optimizing an algorithm that beautifully works with past datas doesn’t absolutely ensure a good performance with data in the future. It’s better to code an algorithm that can predict events in a more general way.

2 It’s better a minimum of 5 years. 10 years is even better and the preferred choice.

3 Because dividing criteria for longs and shorts will make it excessively prone to curve-fitted solutions.

1 Like