Upon completion of the first steps of the process to build the trading system, the developer must validate the model with a defined confidence level. This evaluation should provide specific metrics to assess the model’s capability to generate profits and the risk of using it.
In this educational article, we will discuss the testing process of a trading system, including the evaluation and optimization process.
The testing process is a critical stage in the trading system development; in this phase, the developer must validate the system’s behavior in a simulated market context with real data. This process leads the developer toward deciding how much data will be needed to verify the model.
The data size should be significant enough to provide results with a confidence level, such as in statistical terms, a 95% confidence. Considering that volatility and market dynamics changed in the last 40 years, the consideration of 40 years of data could not be adequate to evaluate the model. In this regard, 5 to 15 years of market data could be enough for the right estimation of the system’s performance.
In the first step, the system developer should realize a back-test evaluating the system’s construction logic and the performance with historical data. The results must be studied with an objective quantitative method as the statistical inference. This process’s results could drive the system’s developer to discover some optimization model based on the market’s synchronicity.
After this optimization, the next step in the testing process is in terms of Jaekle and Tomasini, the walk-forward analysis. A walk-forward analysis is a series of multiple and successive out-of-sample test over different chunks of the data series. The data used in the walk-forward tests should be unused portions of the historical data.
Once verified the trading system capability to generate profits with an acceptable risk level, the system could be tested using real-time data and paper money to evaluate its performance in front of new market conditions. In this context, the time to run the system should be flexible and dependent on the developer’s experience.
A Question of Samples
A sample is just a portion of the whole phenomenon under study; in a trading system, the event under analysis corresponds to the results from a trading entry series. In this context, the final result corresponds to the profit/loss level generated on each trade. In other words, a sample with one trading signal could not represent the trading system’s capability to generate profits. Therefore, the trading signals generated by the system should be significatively bigger to evaluate its performance, and in consequence, the sample used should be representative of the whole series.
The result from the sample evaluation will be an average of the potential returns of the trading system. Considering the variability of the results, the trading system developer will have to analyze the degree of variation of the returns with respect to the average return. In statistical words, the developer will have to study the standard deviation of the trading system.
An example of this analysis could be the return average of the trading system is $100 per trade with a standard deviation of $15 per trade, this means that the trading system may show returns between $85 to $115 per trade 68 percent of the time.
Is it Necessary to Optimize the System?
The optimization process is a way to adjust some variables oriented not only to maximize the profits but also to reduce the risk taken on each trade and improve the trailing stop methodology. It could also have to target the reduction of false trade signals, for example, to avoid the market entries when the price action realizes a false breakout. However, according to Jaekle and Tomasini, there exist the possibility of incurring an over-optimization, which could reduce the system’s performance.
The testing process is a step intended to evaluate the trading system’s ability to generate profits and identify the variability level of its results with a confidence level. In this way, the system developer must analyze the system’s performance, using both historical and real-time market data. At the same time, the period required to study the system’s performance in real-time will depend on the developer’s experience.
The requirement of optimization will depend on the variable to need to be improved. This process could carry the trading system to reduce its efficiency in its capability to generate trading signals.
Finally, in our next educational article, we will expand the optimization process and some metrics to evaluate the trading system performance.
– Jaekle, U., Tomasini, E.; Trading Systems: A New Approach to System Development and Portfolio Optimisation; Harriman House Ltd.; 1st Edition (2009).