Reading Assignment: Common Backtesting Mistakes

TA & Algo Trading Course
#61

1 over optimization would cause curve fitting
2 best around 10 years
3 because they lead to curve fitting

#62

#1 - What is so dangerous about over-optimization?
The danger of over-optimization is, that your tradingsystem will more likely become “curve fitted”, meaning it is optimized for a specific data set of the past. The performance in the future therefore is more than questionable.

#2 - How long should a testing period be if you are serious about building a profitable trading strategy?
About 10 years

#3 - Why should you avoid asymmetric trading signals?
It is not certain that some kind of balance of the past is going to continue in the future

#63
  • What is so dangerous about over-optimization?
    In over optimizing you run the risk of refining the parameters too much (curve fitting) making them less likely to fit with future market movements which naturally occur over a period of time
  • How long should a testing period be if you are serious about building a profitable trading strategy?
    A plan should be built from a window of say 6 months and back tested ideally over a minimum period of 5 years up to 10+ years if possible
  • Why should you avoid asymmetric trading signals?
    Using asymmetric trading signals can lead to curve fitting in the respect that you can find yourself adapting the signals for a long or short trade which although relevant to that specific time period, may well not follow the same or similar pattern in future markets
#64
  1. We don’t want to generate trading strategies with absolutely astonishing results on historical data that will not be achievable going forward. This would happen bey overly curve fitting them.
  2. Optimizations should be carried out for long periods of time, ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.
  3. Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.
#65

  1. Making a mistake could result in curve-fitted strategies which do not act well in future market scenarios.

  2. Ideally 9 to 11 years of data so that various market conditions are taken into consideration.

  3. Separate criteria would increase the strategy’s scope and therefore making it likely to fail due to excessive unknown variables.

#66

  1. The danger in over optimization is that it leads to curved fitting. This is unadvised tuning of the trading strategy to fix only one type of data set.

  2. The safe time frame to optimize a data set is 9-11 years according to the article.

  3. The asymmetric signal set up will lead to curve fitting beaqcuse the signals will be from a specific past data set and wont be easily able to adjust to the different type of market conditions out of its condition set.

#67
  1. Over-optimisation is the act of tweaking your strategy to fit historical data to perfection, which can result in the strategy not being fit for future markets.
  2. A testing period should be > 10 years of historical data, with ideally 1-2 years of forward testing to see how profitable your strategy plays out in a future market.
  3. Asymmetric trading signals is a form of curve-fitting as you are adapting rules to fit upward and downward trends separately.
#68

1: It can be turn into algorithm that run perfect on test set but have poor results on real time market.
2: Best 9-11 years
3: signals can be too specific so it can make the algo curve fitting

#69

1.What is so dangerous about over-optimization? Answer: One should guard against over-optimization which can lead to curve-fitting, because creating seemingly great strategies that will not perform in the future will lead to poor strategies that can lead to lost funds.

2. How long should a testing period be if you are serious about building a profitable trading strategy? Answer: Ideally, 9 to 11 years of data should be used to establish a wide variety of possible market conditions. Time frames lower than 30 minutes and small take profits below 10 x’s the spread will not be reliable, curve fitting to past data likely to happen under optimization, results will be meaningless, and exploitation of backtesting calculations from surrounding known values and broker dependency will render a prehistoric part.

3. Why should you avoid asymmetric trading signals? Answer: While some past price trends might have developed asymmetrically, there is no guarantee of future performance due to economic variable and cycles. Using different criteria for longs and shorts increases the strategy’s degrees of freedom and makes it extremely prone to curve-fitted solutions.

#70

  1. What is so dangerous about over-optimization?
    -curve fitting

  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    -Simulating systems that trade on time frames lower than 30 minutes should be avoided,
    -ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available

  3. Why should you avoid asymmetric trading signals?
    -Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#71
  1. What is so dangerous about over-optimization?

Over-optimization can lead to “curve fitting” which will fit your strategy to historical data, rendering it susceptible to failure in the future.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

A testing period should be at least 9-11 years, with a time frame over 30 minutes.

  1. Why should you avoid asymmetric trading signals?

Asymmetric trading signals can be shown to work for the past and not in the future. There are too many macro economic variables to consider.

#72
  1. What is so dangerous about over-optimization?
  • Because there is the risk that tweaks are not a real optimization but an attempt to curve fitting the strategy.
  1. How long should a testing period be if you are serious about building a profitable trading strategy?
  • Ideally 9-11 years of data should be used for the process in order to ensure that a large amount of market conditions become available.
  1. Why should you avoid asymmetric trading signals?
  • Because adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.
#73
  1. What is so dangerous about over-optimization?

You run the risk of “fitting the curve.” This means you may build a strategy or program to reproduce similar results when back testing with historical data, but it wont necessarily be effective for the future. The idea is to keep the the strategy loose enough that it can still produce positive results even with evolving marketing conditions.

  1. How long should a testing period be if you are serious about building a profitable trading strategy?

9-11 years is ideal. 5+ minimum.

  1. Why should you avoid asymmetric trading signals?

You are more likely to run into the issue of curve fitting again.

#74

  1. Over-optimization is dangerous because it transforms the strategy in a curve-fitting of past data that will be never identic to the future data.

  2. 9-11 years of past data is idealic.

  3. Asymmetric signals provide more complexity meaning the trading with such algorithm includes to much uncertainties.

#75
  1. What is so dangerous about over-optimization? Overtuning your strategies so it wont work with different varibles. Keep it simple, keep it stupid. Robust
  2. How long should a testing period be if you are serious about building a profitable trading strategy?
    More history can be used the better. Article suggested 9-11years of data to analyze.
  3. Why should you avoid asymmetric trading signals?
    It may work in past but its questionable in the future. Asymmetric trading signals can correlate in the past. Setting different varibles on longs and shorts limits strategies freedom.
#76

Over-optimization can lead to a formula that works with a specific set of data only. It is too specific and does not work when a different set of data is used. With an over-optimized algorithm the main strategy is lost although with a specific set of data it may still show positive results.

The test period should be sufficient to take into account most possible scenarios. According to the article, in forex markets, a ten year period is reasonable. However a ten year sample might not be feasible in crypto markets at present. In addition to the main sample an out-sample will also be needed.

Asymmetric signals cater for more complex adjustments which in turn open the door to over-optimization. It is therefore recommended that asymmetric signalling is avoided.

#77

1 it is dangerous to adapt the system to a certain period of data… adapt to much

2 9 to 11 years of data

3 Adding separate criteria for longs and shorts automatically increases the strategy’s degrees of freedom and makes it excessively prone to curve-fitted solutions.

#78

  1. The risk of over-optimizing is that the model becomes curve-fitting. This in turn will be dangerous in a sense where it seldom works. It is important to recon that histroy does not repeat it self, it only rhymes. With that said - history will never be identical to the future events/charts.

  2. Most optimal would be 10 years +. In BTC it is not really possible to go that far back so go back as far as possible.

  3. This increases the degrees of freedom for the trading signals. More likely that curve fitting will occur.

#79

  1. If you are relating to the lectures example if you optimize to previous data in the past and go in to the future with that it could be very dangerous and you aren’t really testing youre strategy by doing this. You don’t want to fit it to previous data. You want to perfect you’re strategy

  2. I’m not sure long enough for you to back test until it’s working and you’re happy with it.

  3. It can lead to curve fitting

#80 
1. What is so dangerous about over-optimization?
	a. curve fitting
	b. it maybe over adjusted or tailored to past events (back testing) and may not be suitable for the future markets or new future data sets.
	
2. How long should a testing period be if you are serious about building a profitable trading strategy?
	a. 9-11 years
	
3. Why should you avoid asymmetric trading signals?
	a. it makes it excessively prone to curve fitting