Hitchhiker’s Guide to Investment Research, Part 4: Trust but falsify

Last week, I wrote a trilogy of posts on how to approach investment research. I called it the Hitchhiker’s Guide to Investment Research and staying true to the theme, this is part 4 in this trilogy. It is my criticism of the tendency in both finance and economics to dismiss falsifications of theories and rather stick with confirming evidence.

In the hard sciences, one of the key principles is Popper’s Falsification Principle. It states that a theory must be testable and presumed to be true until evidence to the contrary appears. The classic example is the theory that “all swans are white”, which for centuries was considered true until seafarers encountered black swans in Australia. Note that it suffices to observe only one black swan to falsify the theory and make us look for a new, expanded, or adapted theory, such as “all European swans are white”.

This may sound like common sense, but compare this to the practice in finance and economics to come up with a theory (say, the CAPM) to make predictions about the world and then, when empirical evidence violates the predictions of the theory, either claim that the evidence is insufficient or didn’t properly test the theory. In the face of hundreds of violations of the CAPM, some people still insist that it is a useful theory in practice.

One of the quotes, I have heard too often to let go anymore is “that extraordinary claims require extraordinary evidence”, something that was popularised by Carl Sagan but can be traced back as far as Thomas Jefferson. Let me tell you that this is not true, not true at all.

The thing about science is that it doesn’t look for confirmation. Confirmations are boring because they tell you something you already know. Science looks for contradictions and falsifications. And no matter how small they are, falsifications trigger new research into ‘why this and that theory is contradicted by the real world’. Einstein’s Special Theory of Relativity replaced Newton’s Mechanics based on the results of the Michelson-Morley experiment. It was a single experiment that contradicted Newton’s Mechanics but could be explained by theories developed by Mach and Einstein. So, the world switched to special relativity. A range of experiments with subatomic particles showed that Newton’s mechanics don’t hold there as well. So physicists developed quantum mechanics. If finance were a true science, the majority of academics would dedicate their life to formulating and testing better theories of asset pricing. And while some academics try to adjust the CAPM with small bells and whistles, nobody is really looking into a fundamental re-assessment of our understanding of asset pricing. It’s as if we have all these falsifications in place, but nobody really cares.

Science is hard because it goes against human nature. We aren’t naturally inclined to look for evidence that contradicts our views. The Wason selection task is the classical experiment to show this. It goes something like this:

Assume you are an administrator in a university. Your job is to check student report cards, each of which has a grade from A to F on one side and a number code on the flip side. The rule you have to check is that every student who has a ‘D’ has to have an odd number on the flip side. Below you see four cards. Which one of these cards do you have to turn over to make sure this rule is true?

1. Card showing grade ‘D’

2. Card showing grade ‘A’

3. Card showing number ‘3’

4. Card showing number ‘8’

Less than 10% of participants in this task get the answer right, namely, to turn over the first and the last card. Instead, about three out of four people turn over the first card and the third card (i.e. the one showing the number ‘3’).

Why is turning over the third card with the number ‘3’ not helpful? Because you are looking for confirmation. If you turn over the card with the number ‘3’ and it shows grade ‘D’ you haven’t learned anything new. If it shows a different grade (e.g. ‘A’) you haven’t violated the rule. The rule only says that if someone has a grade ‘D’, the reverse side must show an odd number. It didn’t say that someone with a grade ‘A’ cannot also have an odd number on the flip side.

But if you turn over the fourth card with the number ‘8’ and you discover that it shows grade ‘D’, you have falsified the rule and shown that the rule does not hold in practice. And that means that you can abandon the rule and look for a better one.

If all of that is too abstract for you, let’s think about it this way:

Assume you are the owner of a bar. Your job is to make sure you can keep running your business and you don’t violate drinking laws. The rule you have to check is that everyone who drinks alcohol is at least 18 years old. You notice four young people at a table. Which one of these do you have to check to ensure, your bar doesn’t violate drinking rules?

1. Person who drinks beer

2. Person who drinks Cola

3. Person who is 18 years old

4. Person who is 15 years old

In this example, most people immediately understand that they have to check the 15-year-old person because if that person drinks alcohol, the bar violates the law.

But when people come to the finance and economics gurus of this world and tell them they found a 15-yar old drinking beer, all they say is “Well, that is just one underage drinker, but look at all the 18-year-olds having a good time at our place”.

The problem is that if you want to better understand how the world works, you need to emphasise instances where the world does NOT do what you expect it to do. If your investments are not working, use them to better understand why they are not working, and don’t be afraid to throw established wisdom out the window. Even a small contradiction weighs more than dozens of examples of confirmations of a theory. None of these confirmations increase the weight of the evidence. That is the foundation of the scientific process. To think that more and more examples of your theory increase the weight of it in practice is plain wrong.

After the financial crisis, it took me several years to accept that the expansion of the central bank balance sheets did not create inflation, even though monetary theory said it should. I also did not make the link between zero interest rates and the underperformance of value vs. growth stocks. Sticking to theories that no longer worked cost me a lot of performance. But I eventually learned from my mistakes and adopted different approaches to investing that were better adapted to the reality of markets.

Ironically, with the current inflation surge, it could be that we are again at a watershed moment in markets where pre-existing theories of how markets work become falsified and contradicted. So far, I am not fully convinced, and I think the jury is still out if inflation will become entrenched or not (to name just one example). But I have promised myself I won’t stick around defending an outdated theory for as long as I did ten years ago. If the facts change, I change my mind. What do you do?