Testing – Forex Academy

Introduction

The development of a trading strategy requires steps to evaluate its reliability during its execution in real markets. To achieve this, the developer must develop a testing process to determine its robustness and viability as a previous step before its optimization.
In this educational article, we’ll introduce the steps of a trading strategy’s testing process.

Getting Started with Testing the Trading Strategy

Once the developer completed the programming of a candidate trading strategy, it’s time to confirm if the strategy works as the conceptual model assumes. In this regard, the strategist should follow the following steps:

Verify the preliminary profitability of the trading strategy.
Assessing the robustness delivered by the strategy.

The robustness concept relates to the ability to continue generating substantial profits despite adverse market conditions, such as trend changes or extremely volatile conditions. An alternative method for verifying the strategy’s robustness is by assessing if it continues being profitable under a wide basket of markets.

Another critical part of the testing process is to verify the trading rules. As the strategy’s complexity increases, rules also increase in complexity. In this context, the developer must validate that the execution of the buy and sell signals happens at the levels triggered by the strategy’s rules. The verification of entry and exit signals will allow the developer to identify any programming error.

Analyzing Profitability

Once the programming stage is verified, the developer should estimate the trading strategy’s profitability considering a reasonably lengthy historical price series.

As a guide, a short-term strategy may be tested using two years of price data, a mid-term would need up to four years, and a long-term up to eight years of historical data. Nevertheless, the window size may vary depending on the strategy type or the market conditions.

With this information, the strategist will have a panoramic overview of the trading strategy’s profitability and risk. In this regard, if the strategy’s performance is deemed acceptable, the developer could advance to the next stage of the testing process.

Nevertheless, if the strategy has a poor performance, the developer should judge if it could be redesigned or discard it.

Finally, for a proper evaluation, the strategy must be run using a standard one unit trading size. This way, the profits would result in multiples of the risk.

The Multimarket and Multiperiod Test

The multimarket and multiperiod test corresponds to the last stage of the testing strategy’s performance. During this historical simulation, the developer must study a set of parameters of the strategy considering a small basket of diversified markets over a broad range of historical periods. In other words, the developer must develop a historical simulation taking a group of different assets using different timeframes.

The developer must select a small portfolio of non-correlated assets; in other words, markets that don’t relate to each other. An example of a non-correlated portfolio is a mix of a commodity, a bond, and a stock index.

Concerning the length of the test period, Robert Pardo suggests that to obtain solid results, the developer should use ten years of historical data for each market. However, it could start from five years of historical prices. Pardo’s figures are related to long-term stock trades using daily timeframes. Intra-day trading systems, as already stated, would require smaller data ranges.

The results obtained from the historical simulation will provide an objective overview of the trading strategy’s profit and risk. Depending on the results obtained, the developer may terminate if the strategy is robust and produces reasonable returns or if it performs poorly and should be rejected. Likewise, the strategist may observe that the strategy presents mixed results, so it should not be rejected entirely.

Conclusions

During the testing process of a trading strategy, the developer must evaluate a broad range of aspects that ensure the correct work during its evaluation.

For example, the developer must verify if the opening and closing trades’ programming rules execute as the strategy requires. After this step, the robustness degree will drive the strategist to conclude if it is viable in real markets, needs improvements, or should be rejected.

Finally, on a multimarket and multiperiod test, the developer must evaluate the strategy’s performance in a small non-correlated portfolio using different timeframes. Once this historical simulation is made, the strategist would be able to confirm if the strategy is robust and viable or reject it before continuing with the optimization stage.

Suggested Readings

Jaekle, U., Tomasini, E.; Trading Systems: A New Approach to System Development and Portfolio Optimisation; Harriman House Ltd.; 1st Edition (2009).
Pardo, R.; The Evaluation and Optimization of Trading Strategies; John Wiley & Sons; 2nd Edition (2008).

Introduction

Upon completion of the first steps of the process to build the trading system, the developer must validate the model with a defined confidence level. This evaluation should provide specific metrics to assess the model’s capability to generate profits and the risk of using it.
In this educational article, we will discuss the testing process of a trading system, including the evaluation and optimization process.

Testing Process

The testing process is a critical stage in the trading system development; in this phase, the developer must validate the system’s behavior in a simulated market context with real data. This process leads the developer toward deciding how much data will be needed to verify the model.

The data size should be significant enough to provide results with a confidence level, such as in statistical terms, a 95% confidence. Considering that volatility and market dynamics changed in the last 40 years, the consideration of 40 years of data could not be adequate to evaluate the model. In this regard, 5 to 15 years of market data could be enough for the right estimation of the system’s performance.

In the first step, the system developer should realize a back-test evaluating the system’s construction logic and the performance with historical data. The results must be studied with an objective quantitative method as the statistical inference. This process’s results could drive the system’s developer to discover some optimization model based on the market’s synchronicity.

After this optimization, the next step in the testing process is in terms of Jaekle and Tomasini, the walk-forward analysis. A walk-forward analysis is a series of multiple and successive out-of-sample test over different chunks of the data series. The data used in the walk-forward tests should be unused portions of the historical data.

Once verified the trading system capability to generate profits with an acceptable risk level, the system could be tested using real-time data and paper money to evaluate its performance in front of new market conditions. In this context, the time to run the system should be flexible and dependent on the developer’s experience.

A Question of Samples

A sample is just a portion of the whole phenomenon under study; in a trading system, the event under analysis corresponds to the results from a trading entry series. In this context, the final result corresponds to the profit/loss level generated on each trade. In other words, a sample with one trading signal could not represent the trading system’s capability to generate profits. Therefore, the trading signals generated by the system should be significatively bigger to evaluate its performance, and in consequence, the sample used should be representative of the whole series.

The result from the sample evaluation will be an average of the potential returns of the trading system. Considering the variability of the results, the trading system developer will have to analyze the degree of variation of the returns with respect to the average return. In statistical words, the developer will have to study the standard deviation of the trading system.

An example of this analysis could be the return average of the trading system is $100 per trade with a standard deviation of $15 per trade, this means that the trading system may show returns between $85 to $115 per trade 68 percent of the time.

Is it Necessary to Optimize the System?

The optimization process is a way to adjust some variables oriented not only to maximize the profits but also to reduce the risk taken on each trade and improve the trailing stop methodology. It could also have to target the reduction of false trade signals, for example, to avoid the market entries when the price action realizes a false breakout. However, according to Jaekle and Tomasini, there exist the possibility of incurring an over-optimization, which could reduce the system’s performance.

Conclusions

The testing process is a step intended to evaluate the trading system’s ability to generate profits and identify the variability level of its results with a confidence level. In this way, the system developer must analyze the system’s performance, using both historical and real-time market data. At the same time, the period required to study the system’s performance in real-time will depend on the developer’s experience.

The requirement of optimization will depend on the variable to need to be improved. This process could carry the trading system to reduce its efficiency in its capability to generate trading signals.

Finally, in our next educational article, we will expand the optimization process and some metrics to evaluate the trading system performance.