Introduction
Before proceeding to discuss forward-testing and Montecarlo permutations, we need to know the basics of the methodology and statistical foundations of evaluation and hypothesis testing. To achieve that, I’ll use as a reference the excellent book Evidence-Based Technical Analysis – Applying the Scientific Method and Statistical Inference to Trading Signals, by David Aronson.
According to Dr. Aronson, traditional Technical Analysis is nowadays where medicine stood before it evolved from faith-based art into a system of knowledge based on science. His book’s central theme is that TA must grow from anecdotal and unproven evidence into rigorous observational science since the scientific method is the only rational way to extract useful information from market data.
Definitions: propositions, claims, belief and knowledge
Declarative statements
The fundamental block of knowledge is the declarative statement, also known as a proposition or claim. A declarative statement is distinguished from others in that it exhibits a true or false value. The statement “Now it’s raining” or “the earth is round” are declarative, because we can assess if they are true or false. Statements such as “what is that?” or “buy me a sandwich!” do not hold any true or false value, so they are not declarative statements.
Regarding TA, an example of a proposition might be MA crossovers have an edge, and the goal of our work, when testing these rules, is to determine which of such statements justify our belief.
Beliefs and cognitive content
Therefore, what’s the meaning, of ” I believe I can buy K for $10 “? It means I expect to be able to buy K for $10 if I go to the market. But the command “buy me K!” or “I’m not happy with that price!” doesn’t have that property.
To conclude, we recognize any statement as a candidate for a belief if it holds something that we could expect or experience. Such a class of assertions is said to have cognitive content, something that can be known.
Sometimes, although a declarative statement seems to hold cognitive content, it does not. These pseudo-declarative statements are meaningless claims or empty propositions.
Although empty claims are not valid candidates for belief, this is not reason enough to stop people from believing them. Astrology pages are still prevalent in newspapers, and there are channels dedicated to astrological predictions, and pseudo-curators claim they can cure cancer by imposing their hands.
A way to detect if a statement has cognitive content is the discernible-difference test: Propositions with cognitive content make claims that are true or false, if it holds cognitive content then we can discern a difference between those two states, meaning that its truth-state is distinguishable from its false one. Testing a claim based on the discernible difference is central to the scientific method.
What is knowledge?
The best definition comes from David Aronson: Knowledge can be defined as a justified true belief. Hence, for a declarative statement to qualify as knowledge, not only must it be a candidate for belief, because it has cognitive content, but it must meet two other conditions as well. First, it must be true (or probably true). Second, the statement must be accepted with justification. A belief is justified when it is based on sound inferences from solid evidence.
Some statements seem to be true, but they aren’t. An example is ancient people believing the Sun was orbiting the earth. It seems true, but we know it’s false. These people weren’t in possession of knowledge. Let’s suppose there was a person at that time that believed the appearance of the Sun by the east every morning was due to the rotation of the Earth. Even if his belief was true he had no evidence to support that theory, so we cannot say this person had knowledge. We can call this kind of belief false knowledge because it is erroneous or there is no evidence to support it.
Finally, even when we have evidence to infer that something is knowledge, that’s not enough guarantee that we really know. Uncertainty is inherent in the scientific method, but knowledge improves with time when using the scientific method.
Technical Analysis and erroneous knowledge
There are two kinds of technical analysis (TA). Subjective TA and objective TA.
Subjective TA is a not-well-defined analytical method because it cannot be expressed as a set of precise rules. Therefore, it requires the interpretation of the analyst. As a consequence, it is impossible to confirm or deny its efficacy.
Objective TA is well-defined and repeatable. That allows to implement it as a computer algorithm and back-test it using historical data.
It’s evident that a subjective TA method cannot be called knowledge since no one can reproduce it and there is no way that, even the same person, was capable of producing the same results on the same dataset. These subjective TA methods are problematic in that they present the illusion of true cognitive content, but, really, they are meaningless claims.
Objective TA can deliver erroneous beliefs, but they come from a different path. The fact that it has been profitable on a back-test is not enough to guarantee its validity. Success on a back-test is necessary but not sufficient. Past performance could be the result of overfitting or luck.
The Scientific Method: A method to get more knowledge
The scientific method is the most valuable knowledge the West has given to the world, according to C. Van Doren because it’s a set of procedures to acquire new knowledge. The rigorous rules of the scientific method protect us from the weaknesses of our minds. Informal observation and inference from inadequate or insufficient data are likely to fail with complex or noisy data.
Traditional TA is one of the branches of our practice that has not been applied using scientific methods. There is no surprise that many TA practitioners are against using scientific methods and say that objective TA does not capture the subtleties of all parameters involved; that only a human brain can do that. That happened in medicine as well, and when alchemy developed into chemistry.
The scientific knowledge is objective
Science aims for the highest objectivity by restricting itself solely to demonstrable facts about the world, although we understand that wholly objective knowledge is never feasible. That eliminates subjective opinions that are inherently personal.
Therefore, scientific knowledge must be open to verification by others, and it must be public to promote the maximum agreement between independent observers.
Scientific knowledge is quantitative
Observations must be translated into numbers to be analyzed rigorously. Quantification allows the application of robust statistical methods. Quantification is the best way to ensure objectivity and maximize its potential to be tested.
The purpose of a scientific theory is to explain and predict
One of the goals of a scientific theory is to discover rules that predict new facts, and an explanation for past observations. Explanatory theories go one step beyond predictive theories in that they tell us why A follows B instead of just saying that it does.
The most important type of scientific law is function. It describes a set of observations in the form of an equation, such as:
Y = f(Xt)
Once we find a functional description, we can predict future events of Y by introducing known values to the function f(). Functions can be found by two methods: Deduced by analytical theories or estimated from historical data by fitting a function, regression analysis for example. TA falls into this second category.
Logic and Science
Science relies on logic and empirical evidence to arrive at conclusions, as opposed to the informal reasoning that tends to find support in authority and tradition.
The fundamental principle of formal logic is the rule of consistency. It is backed by two laws: The law of excluded middle and the law of noncontradiction.
“The law of the excluded middle requires that a thing must either possess or lack a given attribute. There is no middle alternative. Or said differently, the middle ground is excluded.” {1}
“Closely related to the law of the excluded middle is the law of noncontradiction. It tells us that a thing cannot both be and not be at the same time.”{2}
Propositions and arguments
A Proposition is a declarative statement that may be true or false
An Argument is a set of propositions, one of which is the conclusion, derived from the previous propositions, called premises.
A logical inference has two forms: deduction and induction.
Deductive logic and plausible reasoning
“A deductive argument is one whose premises are claimed to provide conclusive, irrefutable evidence for the truth of its conclusion” (Aronson).
Categorical syllogisms
A usual form of deductive argument is the categorical syllogism, credited to Aristotle (4th century B.C.), formed by two strong syllogisms called premises and one syllogism called conclusion. Example:
Premise 1: All mammals have warm blood
Premise 2: A dog is a mammal
Conclusion: A dog has warm blood
The general form of a categorical syllogism is:
Premise 1: All members of A are members of B
Premise 2: C is a member of A
Conclusion: C is a member of B
Deductive logic is appealing because of the certainty of the conclusion. But this only happens if the premises that are the basis of the conclusion are true and expressed in a valid form. Truth and false are properties of the propositions. Validity defines the correctness of the logical inference linking premises with the conclusion. We can demonstrate its validity using diagrams called Euler circles.
Conditional syllogisms
Another form of deductive argument, and of crucial importance to scientific reasoning, is the conditional syllogism, which is the basis for the discovery of new knowledge. It’s also composed of three propositions, two premises, the first one being a conditional proposition and a conclusion.
A conditional proposition is a composite statement that mixes two propositions using the words if and then. The general form is:
if(antecedent clause), then (consequent clause),
for example
if it is a mammal, then it has warm blood
If this TA rule is predictive, then its back-tested return will be positive
The second premise is a premise that affirms or denies the truth of either the antecedent or the consequent clause if the first proposition. For example, it could state:
It is a mammal
It is not a mammal
It has warm blood
It does not have warm blood
The conclusion of the conditional syllogism affirms or denies the truth of the remaining clause. As an example, let’s see the complete syllogism:
if it is a mammal, then it has warm blood
It is a mammal (validates the truth of the antecedent)
Therefore, it has warm blood (establishes the truth of the consequent)
Valid forms of conditional syllogisms
Affirming the antecedent:
Premise 1: If A is true, then B is true
Premise 2: A is true
Valid Conclusion: Therefore, B is true
Denying the consequent:
Premise 1: If A is true, then B is true
Premise 2: B is not true
Valid Conclusion: Therefore, A is not true
This is the form that uses science to prove that a hypothesis is false. If we can prove that a hypothesis is false, we can indirectly prove that some other is true. This is the way to acquire new knowledge that we wish to establish as true. For example, that a TA signal is more predictive than a random entry.
Invalid forms of the conditional syllogism
People using informal logic tend to commit two errors: Affirming the consequent and denying the antecedent. An example of affirming the consequent is:
If it is a mammal, it has warm blood
It has warm blood
Therefore it is a mammal
The fact that the animal has warm blood it doesn’t mean it’s a mammal. This fallacy is common in TA. Let’s see an example:
If this TA strategy has predictive power, then it should be profitable in a back-test
the back-test is profitable
Invalid conclusion: the TA strategy has predictive power.
The other form of invalid conditional syllogism is denying the antecedent
If it is a mammal, it has warm blood
It’s not a mammal
Therefore, it does not have warm blood
Even in the absence of complete information, a conditional syllogism’s conclusion may enhance our knowledge about A or B:
If A is true, then B is true,
B is true
Therefore, A becomes more plausible
The evidence does not prove A to be true, but verification of some of its consequences gives us more confidence that A is true.
A weak form of reasoning using the same strong premise is:
If A is true, then B is true,
A is false
Therefore, B becomes less plausible
B is not proven false, but one of the reasons for it being true has been discarded.
Finally, a plausible reasoning is even weaker:
If A is true, then B becomes more plausible
B is true
therefore, A becomes more plausible.
In his book Probability theory, the logic of science, E.T. Jaynes presents a practical case:
“Suppose some dark night a policeman walks down a street, apparently deserted. Suddenly he hears a burglar alarm, looks across the street, and sees a jewelry store with a broken window. Then a gentleman wearing a mask comes crawling out through the broken window, carrying a bag which turns out to be full of expensive jewelry. The policeman doesn’t hesitate at all in deciding that this gentleman is dishonest.”
What’s the policeman’s reasoning process to deduct the man with a mask was a burglar? There might be a totally innocent explanation for this situation: The gentleman with the mask was the owner of the jewelry, coming from a masquerade party and while walking near his store a truck accidentally throws a big stone that broke the jewelry’s window. In the end, he was protecting his merchandise.
So why the policeman’s actions seem right? It is so because the probability of this explanation to be true is quite low. If the policeman had been experiencing this kind of situation often because there were a lot of trucks that usually throw stones at jewelry’s windows while its owner comes from masquerade parties the policeman would soon stop worrying when observing people with masks getting out with a bag full of jewels.
Therefore, our reasoning depends very much on prior information.
Inductive Logic
Induction tries to extract knowledge about the world by going beyond the knowledge contained in the premises. The new knowledge comes at the price of it being uncertain. Wikipedia philosophical definition is quite good: “the premises of an inductive logical argument indicate some degree of support (inductive probability) for the conclusion but do not entail it; that is, they suggest truth but do not ensure it. “
Inductive logic goes in the opposite direction of deductive logic. Deductive starts from general facts and concludes about a particular fact. Inductive starts from a sample of particular examples to reach a conclusion
As an example of a deductive argument:
All known life on earth depends on liquid water to exist
Therefore, it is highly probable that all biological life in the universe depends on liquid water to exist
Generalizations need not be universal. We may say as well:
X percent of A’s are B’s
Or
X Percent of B’s hold attribute A
Or
B’s hold attribute A with probability X
Unlike deductive logic, inductive logic allows for the conclusion to be false, even when the premises are true. Instead of being true or false, inductive arguments are strong or weak, with describes how likely the conclusion is right.
Inductive logic is also known as hypothesis construction because it allows us to make conclusions based on current knowledge and further predictions.
One common form of induction is based in enumeration. It starts from a premise that enumerates the evidence contained in a set of observations and a conclusion predicting the properties of observations outside the known set.
Premise: This TA rule gave 500 buy signals over a 5-year period using hourly charts, and in 300 of them the market moved markedly higher over the next 20 bars.
Conclusion: In future appearances of this TA signal there is a 60% chance that the market moves higher along the first 20 hours.
The strength or weakness of the conclusion is enhanced or weakened by the quantity and the quality of the evidence shown by the premise.
Supposing we have only ten samples with a 60% success rate the evidence on the goodness of the signal is quite poor. In that case, further evidence may differ greatly from the 60% probability in one way or another.
The quality of the evidence is also important. Some methods are better than others. The gold standard to gather evidence is the controlled experiment with all its parameters held constant, except the one subject to test. TA does not permit that kind of test, but some methods are better than others, and we should be especially careful to avoid systematic errors such as data mining bias and confirmation bias.
Common biases
The availability bias causes people to depend primarily upon information that’s easily accessible to them. As an example, when asked people to signal the importance of a list of facts, they rank based on the recent news by newspapers or by their personal beliefs or experiences, disregarding less obvious evidence.
The confirmation bias is the natural inclination to seek confirmatory rather than denial evidence about their beliefs. For example, people tend to seek corroborative evidence that some TA signal is predictive, rather than disprove it.
The predictable world bias describes the tendency to perceive order where there is just randomness. Gamblers and traders find patterns where there is none, and they believe they can predict outcomes from past data.
The law of small numbers bias tends to assign a predicting value to what essentially is random noise caused by a short streak of good or bad luck. For example, when a trader has five straight wins thinks his system is excellent attributing to the system what might have been just a very lucky winning streak.
The data mining bias is the result of extensive searching for patterns on a historical database using excessive parameters or continually improving the parameter mix until we find a good result.
Sample selection bias happens when available data is not representative of the whole possible scenarios. For example, we test a TA strategy for the last five years of historical data, but it happens that during this time that market was trending up, so the strategy is not representative in downtrends.
Critique of induction
David Hume, in his work Treatise on Human Nature, defined the problem with induction: how to distinguish true knowledge from inferior forms of wisdom such as opinions.
Before Hume, there was a consensus that the difference was related to the quality of the method employed. Hume said that the belief that A causes B or is correlated to B, just because A preceded B, was a habit of the mind. He said that no amount of observed evidence was satisfactory, and there was no rule to tell us then we have evidence enough.
Supporters of induction claimed that generalizations coming from induction were correct in a probabilistic way. However, critics said that that justification was flawed. The probability that A predicts B is equal to the number of times that A is followed by B, divided by the total number of samples, but, because an infinite number of cases will happen in the future, the result is zero no matter what the number of past observations.
It took more than two hundred years to understand the paradox between Hume’s critique and the accumulation of scientific discoveries. William Whewell was the first to understand the role of induction in the formulation of hypothesis. He said that scientific discovery starts with an inductive guess, but then it’s followed by deduction. After a hypothesis has been induced, predictions are deduced in the form:
If the hypothesis is true, then specific future observed events would occur.
The hypothesis is antecedent clause, and the prediction is the consequence clause in a deductive syllogism:
If A predicts B then future A events will be followed by B events.
When there is a cause-effect between A and B the following proposition is used:
If A causes B then if A is removed B should not happen.
Therefore, if B does not follow future observations of A, then the hypothesis is proven false by the valid deductive “falsification” of the consequent (B).
If A, then B
Not B
Valid conclusion: therefore, Not A
But if future appearances of B follow observations of A, the hypothesis IS NOT proven true, because we have to remember that affirming the consequent is not valid deductive form:
If A, then B
B
Not Valid: therefore X
Karl Popper and Falsification
Karl Popper extended the Whewell’s insight and redefined the logic of scientific discovery. Popper’s central allegation was that scientific studies were unable to confirm a hypothesis. Rather, scientific efforts were limited to identify which hypothesis was false.
Popper’s method of falsification goes against common sense, which favors confirmatory evidence. He argued that the absence of required evidence is sufficient to establish that a hypothesis is false, but the appearance of the expected evidence is not enough to determine its truth.
Provisional and cumulative knowledge
One implication of Popper’s method of falsification is that scientific knowledge is provisional. Every currently accepted theory may be replaced in the future by a more correct one. The net result is a body of knowledge on continuous improvement, building upon prior successful theories, and discarding wrong ideas.
Restriction to testable statements
Another consequence of Popper’s method is that science must limit itself to a testable hypothesis: propositions that generate predictions on events not yet uncovered (past or future).
Distinction between Science and pseudo-science
A major consequence of Karl Popper’s method is that it solved a fundamental problem in the philosophy o science: Differentiating between science and non-science: Science is limited to those propositions that make predictions that can be refuted using empirical evidence.
The information content of a scientific hypothesis
A hypothesis is informative if it makes testable predictions, with the possibility that they can be found false. Therefore, the information content of a hypothesis is linked to its falsifiability.
A high information-content hypothesis makes precise and high quantity predictions. Then, it offers ample opportunities to be falsified. A low information-content hypothesis makes fewer and less accurate predictions. Therefore it is more difficult to falsify.
The hypothetico-deductive model
The hypothetico-deductive model also called the H-D method is a proposed procedure for the construction of a scientific theory. Dutch physicist Christiaan Huygens (1629-95) introduced the original version.
The five stages
- Observation: A possible pattern or relationship is observed in a set of prior data
- Hypothesis: By insight and prior knowledge, an inductive generalization is made that the pattern is not due to random causes, but one that should be found in similar sets. In this case, the only, assertion is that the pattern is real.
- Prediction: A prediction is made from the hypothesis and enclosed in a conditional proposition. The antecedent clause being the hypothesis and the consequent clause the prediction.
- Verification: New observations are obtained and compared with the specified predictions. In some sciences, this is acquired by a controlled experiment. In others, it is an observational research.
- Conclusion: An inference about the validity of the hypothesis is made. This stage involves statistical inference methods such as confidence intervals and hypothesis tests.
In the next part, we will examine the basics of statistical analysis and hypothesis testing. Both needed to continue our path to do proper trading system testing and validation.
References:
Evidence-based Technical Analysis, David Aronson
Probability theory, the logic of Science, E.T. Jaynes
Wikipedia searches on inductive and deductive logic
Encyclopaedia Britannica: https://www.britannica.com/science/hypothetico-deductive-method
©Forex.Academy
Among its functions we can highlight:
In the money multiplier process, the key is in a very important proportion: the ratio between the reserves of the banks (they correspond to their deposits in the central bank) and the deposits of the public in the banks plus other liabilities.
Interest rate
Inflation shows the upward or downward movements of thousands of prices, which implies the difficulty in identifying the continuous and sustained changes in the set.
