Reading Assignment: Common Backtesting Mistakes

[Julia]

1. Over-optimalization produces a trading algorithm that produces amazing results for the period it is curve fitted for but will not produce the same results in the future as it is fitted according to past market movements without any guarantee or probability that it will hold for future markets as well.

2. If you are serious about building a profitable trading strategy the testing period should be at least 9-11 years

3. Asymmetric trading signals have different rules for up trends and down trends. In some markets up and down trends behave differently but as the macroeconomic environment changes so does the up and down trends and in the future they may be symmetric and not asymmetric.

[Dale]

1. What is so dangerous about over-optimization? Over-optimization is dangerous because it alters the date of the past that will not match the probability of the future trading.
2. How long should a testing period be if you are serious about building a profitable trading strategy? Compiling about 10 years of data.
3. Why should you avoid asymmetric trading signals? Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

[Ouzo69]

1. What is so dangerous about over-optimization?
Over-optimization is prone to curve-fitting which will result in an algorithm that is perfect for the history data you backtested, but will fail in the future. In order to prevent curve-fitting your strategy should not be too complex because a high degree of freedom in the different parameters of your strategy is very prone to curve-fitting.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
A testing period should be as long as possible since long time periods introduce a wide variety of market conditions which make curve fitting very difficult. However you should not backtest your strategy on the same historical dataset over and over again, because this may lead to a curve-fitting algorithm, which only has a high performance on this particular historical dataset. It is good practice in system development to keep about one year or two of your historical data “outside” the optimization set. In this way your algorithm cannot curve-fit these data it has never been backtested on during the optimization phase. You can use this dataset to test the robustness of your trading algorithm. Another good practice in the optimization process is not to ignore your best results surroundings. If a slight change of the best value for a parameter will not produce similar but much worse results, this parameter has probably generated a curve-fitted and therefore useless trading algorithm.

3. Why should you avoid asymmetric trading signals?
Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions. However in the future market volatility and conditions may fluctuate. If your economic analysis can change its exit and entry values in a dynamic way, you can add another layer of protection against curve-fitting.

[bitcoinjim]

1. Unreliable systems can get you into trouble by curve-fitting and you won’t be able to get the best use of you resource or data.
2. 1-2 years but more accurate if you can find 10 years of data.
3. If you add separate criteria then system will not be accurate.

[Leo]

1. Because the strategy can fall into the over-optimization problem of curve-fitting: showing great performance results for the past that will not be happen in the future.

2. On traditional markets, 9 to 11 years, with a minimum of 5 years to avoid curve-fitting.

3. Adds complexity and more parameters to “tweak”.
   As the author says “although it is truth that under past data up and down trends might have developed differently in currencies this cannot be guaranteed to continue in the future as these differences rely on interest rate differentials or such similar macro economic variables that inevitably change through economic cycles”.
   Therefore, using separate criteria for entering and exiting short and long trades automatically increases the strategy’s degrees of freedom and the possibility of having a curve-fitted solutions.

[biffen96]

What is so dangerous about over-optimization?
It can lead to curve-fitting. And that makes the trading strategy suitable for past data but not for future data.

How long should a testing period be if you are serious about building a profitable trading strategy?
Ideally 9-11 years of data.

Why should you avoid asymmetric trading signals?
Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitting solutions.

[Kav]

What is so dangerous about over-optimization?

Overoptimization can be dangerous in its self-deception.
It can create a false sense of EA profitability that encourages
the GREED within us to try our “Utopian” EA on a real account –and
thereby subject our account to the flame of what in fact is a truly
terrible system that can burn it the ground.

How long should a testing period be if you are serious about building a profitable trading strategy?

normally 6 months, but honestly, if the past results do not reflects in future results, testing can be not very usefully

3.Why should you avoid asymmetric trading signals?
because of curve-fitting risk being bigger

[Andres]

Too many rules or conditions, too few trade occurrences per rule, erratic results with small adjustments in input parameters…
2.
Ideally 9-11 years of data should be used in order to ensure large amount of market information is available to be more accurate.
3.
Asymmetric information can lead to moral hazard or adverse selection, both result in market failures.

[rigamortizzz]

1. What is so dangerous about over-optimization?
   We don’t want to generate trading strategies with absolutely astonishing results that will
   not be achievable going forward.

2. How long should a testing period be if you are serious about building a profitable trading
   strategy?
   9-11 years of data should be used for the process in order to ensure that a large amount of
   market conditions become available.

3. Why should you avoid asymmetric trading signals?
   Adding separate criteria for longs and shorts automatically increases the strategy’s
   degrees of freedom and makes it excessively prone to curve-fitted solutions.

[Zee2020]

1. What is so dangerous about over-optimization?

Over-optimizing can lead to curve-fitting. This is bad, that means your strategy has been tailored to produce results based on a previous outcome or market trend. A curve-fitted strategy will lack the ability to produce profitable results in the current ever-changing market.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

Ideally your testing period should be 9 to 11 years. Less than that time frame, especially 5 years or less could put your strategy at risk for curve-fitting.

3. Why should you avoid asymmetric trading signals?

Simply put it greatly increases the chances of curve-fitting. It is best to work with the same set of criteria as creating another set will cause asymmetric trading signals.

[Hsu_Li]

1. What is so dangerous about over-optimization?
   Over-optimization leads to curve-fitting, which is the downfall of any method of optimization.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
   10 years.

3. Why should you avoid asymmetric trading signals?
   Because using them means that one is curve-fitting.

[FuriousD]

1. Over-optimization results in algorithms that are too complex and fit very well to the given data but will not perform as well with future data.

2. More than 5 years and preferable 9-11 years.

3. Using different parameters for short and long trades results in excessive complexity and curve-fitting that reduces profitability with future data.

[bitbooz]

1.The most common mistake when doing optimization is -without a doubt- the length of the testing period used to optimize It is very important to note that in order for optimizations to be valid, simulations need to be valid. alsot to consider the surrounding since they give you an idea of the possible changes of profitability you will get if the market changes enough so that your “optimal” settings are no longer that good. In general, the coarser the optimization the less risk there is to curve fit a strategy since the fitting is done in a “lose way” and results that may over estimate profits and underestimate future draw downs are also avoided to a good extent.

2. 9 to 11 years

3.It is to avoid excessively correlated and exhaustive optimization of trading systems. Trading systems should be optimized one variable at a time or using two cross-related variables at a maximum without going into excessive detail.

[KryptoDr]

1. Every curve/ data set can be described in a way of a mathematical function. That means you can optimize the function in the way it can predict with a very high accuracy all events which happened in the past data. But this over optimized solution may not capable of predicting any data in the future. You have to built a model which can predict or detect some events in general and not in every detail.

2. To avoid curve fitting (over optimization) you should at least use past data for a period of at least five years. But it is recommended to use a period of 9-11 years.

3. To avoid high degrees of freedom in your model (high degrees of freedom increases the probability of over optimization) you should use symmetric trading signals (that means use the same criteria for long and short for example, don´t separate them). Asymmetric signals increases the degrees of freedom of your model.

[donald77]

1. What is so dangerous about over-optimization?
   The model might be so tuned to the dataset that it works well for that data but will not perform well for the future data that has not yet been introduced.
2. How long should a testing period be if you are serious about building a profitable trading strategy?
   It should be for the longest possible time period.
3. Why should you avoid asymmetric trading signals?
   Asymmetric trade signals lead to over fitting the model to the data.

[andreas_nygren]

1. What is so dangerous about over-optimization?
   We run the risk of Curve fitting , in other words fitting the setup to close to past data and makes it less probable for future optimization.

2. How long should a testing period be if you are serious about building a profitable trading strategy?

9-11 years and the use of out sample testing is also recommended.

3. Why should you avoid asymmetric trading signals?

The less complexity and less parameters available within a given strategy the less probable it is that it will ever be curve fitted as systems that don’t have complex criteria tend to be unable to “fit” to the data if a true inefficiency is not present.

[Federico]

1. What is so dangerous about over-optimization?
   Over optimization implies a lot of curve fitting, which means potential failing trading systems.

2. How long should a testing period be if you are serious about building a profitable trading strategy?
   From 9 to 11 years.

3. Why should you avoid asymmetric trading signals?
   Because on long time frames some variables might change, compromising the trading system.

[diogozas]

1- Curve fitting - Adapting our model to a specific time frame. Our model should be tested in different time frames and in all of them produce good results.

2- Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.

3- Because they are more prone to curve-fitted solutions

[Howmoney.work]

1. Over optimizing is dangerous because past performance is no guarantee of future results.
2. A testing period should be as long as possible, like 10 years or so.
3. Asymmetric signals should be avoided because adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

[DreamtGalactic]

1. Over-optimization is dangerous because correlation does not equal causation. Over adjusting your optimization strategy (otherwise known as “curve fitting”)

2. According to the article 9-11 years of market data is optimal for being able to fit your trading strategy around the possibilities of the market.

3. Asymmetric trading signals should be avoided because it increases the strategies degrees of freedom, which is something that we should avoid doing for fear of leaving us prone to curve fitting.