Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#21
  1. What is so dangerous about over-optimization?

The risk of 'curve_fitting"

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

in the initial article it recommended 9-10 years. then in the 1st article it links to It suggests long as possible in the second (5 mistakes) it recommended >5 years.

longer data sets are more accurate at the correct resolution so I’m going with > 9. This will also fight against “curve-fitting”

  1. Why should you avoid asymmetric trading signals

asymmetry allows for to much ‘wiggle room’ and adds to much complexity. the more variables the more prone it will be to “curve-fitting”

1 Like
#22

1
so called curve fitting is dangerous: too complex, only past-related data tested, asymmetric, doing unreliable simulations, testing period length not appropriate…

2
longer time frames than 30mins and testing of 9-11 years of data used - according to the blogpost. Well the question is if this is applicable in the new world of crypto.

3
prone to curve-fitting;
interest rate differentials or such similar macro economic variables that inevitably change through economic cycles. (Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.)

1 Like
#23
  1. What is so dangerous about over-optimization?

It can lead to over fitting or manipulation of the data that is not compliant with reality. Looks good in theory but doesn`t work in practise. Should avoid systems that trade on time frames lower than 30 minutes or systems with very small take profit and stop loss targets (below 10 times the spread). Should code simple systems that are symmetric both long and short, long testing periods and test the system on Out of sample data to best avoid data mining or fitting.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

Should be around 10 years of testing data (9-11 years).

  1. Why should you avoid asymmetric trading signals?

Because the results can be skewed and curve fitting as it doesn`t account for macro developments in the economy. What worked before might not work in the future.

1 Like
#24

How long should a testing period be if you are serious about building a profitable trading strategy?

My question is that for a crypto currency, such as chainlink, that just came out and hasn’t been around for even 1 year, does this mean that since i cant go back 10 years, cant do effective testing, thus wont be able to build a profitable strategy? Or do i do testing relative to the length of the crypto currency time frame. in general how should i approach cryptocurrencies that have relatively no long history

1 Like
#25
  1. What is so dangerous about over-optimization?
    May not be suitable for future markets or future data sets
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    9 to 11 years
  3. Why should you avoid asymmetric trading signals?
    It increases the degrees of freedom, therefore prone to curve fitting
1 Like
#26

  1. The dangers of over-optimization or curve-fitting are not immediately apparent, as it will show very positive results in your simulation. This can, however, mean that the strategy is too complex and too adjusted to the simulation and isn’t very likely to work in the market ahead.

  2. A testing period should be one to two years long, but you should also include other sets of 1 to 2 years with different market conditions. In traditional markets we should test with at least 10-year time frames.

  3. Asymmetric trading signals should be avoided, because they will work better in one direction than the other. Because we can’t make assumptions of the future moves, it is better to have a strategy work equally well in both ways. Asymmetric signals can also indicate that we have fitted our strategy too much to suit our simulation.

1 Like
#27

  1. What is so dangerous about over-optimization?
    The dangerous thing about over-optimization is curve fitting, trying to fit too much, the less complexity, less parameters the less likely the code will be curve fitting.

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    If you are serious about building a profitable trading strategy, you would need about 10 years of data to test market conditions to make sure it is reliable.

  3. Why should you avoid asymmetric trading signals?
    you should avoid asymmetric trading signals because they lead to curve fitting.

1 Like
#28

What is so dangerous about over-optimization?
Curve fitting
How long should a testing period be if you are serious about building a profitable trading strategy?
At least a year or longer
Why should you avoid asymmetric trading signals?
Makes it prone to curve-fitted solutions.

#29

The main problem to happen is curve-fitting, or adequate your system to past data and therefore your system is unable to behave similarly on a different data set.
2.
9 to 11 years, so the system goes through different market cycles. After the system is tested run an outside period of 2 years for backtesting and avoiding curve-fitting.
3.
Having asymmetric parameters for entering a long and short position makes it excessively prone to curve-fitted solutions.

#30

1. What is so dangerous about over-optimization?
Main problem would be "curve fitting.
2. How long should a testing period be if you are serious about building a profitable trading strategy?
The longer the testing period, the better. Lets say a few years. Hey, we are talking about
Crypto.
3. Why should you avoid asymmetric trading signals?
The complexity makes your model excessively prone to curve-fitted solutions.

#31

What is so dangerous about over-optimization?
Optimization is a normal part of back testing. Over optimization can introduce unwanted results usually in the form of curve fitting which can skew the results.

How long should a testing period be if you are serious about building a profitable trading strategy?
Normally a testing period of around ten years is ideal. obviously this isn’t possible with bitcoin unless you use the entire history.

Why should you avoid asymmetric trading signals?
The trading signals need to be similar in order to avoid biases which wouldn’t currently apply. Using one set of conditions to enter them a separate set to exit is likely to give unreliable results in testing and performance going forward.

#32

  1. Over optimization can lead to “curve-fitting”, which is the unwanted tuning of the strategy to fit specific past data.

  2. According to the article, approx 10yrs on higher time frames for accuracy.

  3. Asymmetric signals provide more complexity which tends to lean towards tuning the strategy for specific datasets… curve-fitting.

#33

1. What is so dangerous about over-optimization?
Over-optimizing can cause your model to appear highly profitable when in reality it is merely catered to a specific chart and not actually identifying valid trends/market signals. This will result in an un-fit model that, when used in the future, does not actually function properly, which may result in significant losses.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
According to the author, 9-11 years of data. However, I find this surprising as market trends/macro conditions can vary significantly year-over-year. Perhaps for investing in large stagnant companies (like railroad companies, for example.) this would apply, but for rapidly changing industries like AI, blockchain/crypto, etc. I can’t see this being applicable.

3. Why should you avoid asymmetric trading signals?
This would be creating bias within the model as it will factor in macro factors for only one trading direction

#34

1. What is so dangerous about over-optimization?

Over-optimization leeds to curve-fitting, which should be avoidet at any part of any system developer’s efforts, as we don’t want to generate trading strategies with absolutely astonishing results that will not be achievable going forward. If we optimize to much the system to fit and be perfect for one period of time it doesn’t mean it will fit again in another period.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

As i understand it every test you make must cover over one year period. After you optimize your programm in this one year you test it again for one year in another period of time for one year again and this you do it 10 times so test period of one year in long term data of ten years. For example make the strategy for 2019 but look if it fits also for 2018 and 17’ and 16’ until you get to 2009.

3. Why should you avoid asymmetric trading signals?

Asymetric trading can be helpful for experts or day traders but for programming trading it can confuse and over-optimize to a concrete period of time so it would not fit in future trades.

#35

  1. What is so dangerous about over-optimization?
    the danger is that curve fitting might come out of it

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    backtesting must be done intensively using large periods of time (one year is a good time frame)

  3. Why should you avoid asymmetric trading signals?
    because it adds complexity that may lead to curve-fitting

#36

  1. One of the dangers of over-optimization is curve -fitting your strategy.

  2. Your testing period should be as long as possible 9-11 years.

  3. Asymmetric trading signals lead to complexity and complexity leads to no good.

#37

1, It can lead to curve fitting which is where the updates to your programming adapt more and more to the historical data and may not necessarily be of use when applied to the data going forward.
2, back testing should be ideally carried out over a 9-11 year time frame. If a simple strategy yields profitable results there is less chance of curve fitting.
3, Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitting solutions.

#38
  1. It’s dangerous because if we over optimize something, it won’t work in some other way. So we should avoid curve fitting and do only neccessary optimizations.
  2. 9-11 years
  3. Asymmetric signals mean more complexity so then we get back to curve-fitting.
#39
  1. You can fall in the trap of curve fitting, which means that you can perfectly explain the development of a curve in the past but which would be meaningless for the future.
  2. The ideal case would be 10 years. At least it should be a meaningful time.
  3. Because again you can expose yourself to curve fitting when you apply different parameters for going long and short (buying and selling at both ends).
#40
  1. cause it may lead to curve-fitting
  2. circa 10 years or more is the best
  3. cause it adds complexity which leads to curve-fitting