The replication crisis
|Joachim Klement||Jun 3, 2019|
We live in an age when politicians think they can get away with falsehoods that are blatantly obvious. Thank goodness, researchers in economics, medicine, and the natural sciences are more careful about their work and don’t spread falsehoods. Or are they?
Fake research is indeed very rare in the sciences. According to a meta-study of surveys amongst academic researchers, about 2% of researchers have likely falsified research at least once. That is a relatively low number, though it is a shame it is even that high. The much bigger crisis in both the natural sciences and in economics and finance is the replication crisis. A worryingly large number of scientific studies published in academic, peer-reviewed journals cannot independently be replicated.Colin Camerer and his colleagues tried to replicate 18 economic studies published in highly respected economics journals between 2011 and 2014. They found that only 11 out of the 18 studies could be replicated and that the results were on average about one third smaller in size than the original published study. These results fit the picture we get from psychology and other sciences: about 40% of all published studies cannot be replicated afterwards and the reported effect size is typically twice as high as what can be replicated afterwards.
If academic studies are so often the result of false positives, how bad do you think the situation is in practitioner research from investment banks and independent research houses? The research produced by many research houses is often not thoroughly vetted or based on spurious correlation. Recently, I came across a chart from a prominent investment bank that tried to show the impact a growth slowdown in China would have on other economies. The chart showed that the Swedish economy would be amongst the hardest hit economies in the world. That does not make sense, since the economic ties between Sweden and China are relatively small compared to Asian countries or Germany for that matter. It was instantly clear to the thinking observer that this chart could not be true and probably measured some spurious correlation. Yet, the chart made it through the entire publication process of the firm and on the clients’ desks.
My way of dealing with this problem is to try to replicate a finding by another research firm whenever it looks surprising, unintuitive or otherwise noteworthy. If I can’t replicate it in a short time, it is a good indication that the results are due to data mining and overly specific data specifications. And that is when I know not to rely on the results.
And then there are the fortuitous cases when I can replicate the results and this replication creates additional insights and raises new questions. Take for instance a recent study by Oxford Economics that showed that the probability of a recession follows a slightly U-shaped curve. The analysts at Oxford Economics looked at GDP growth in 14 developed countries since 1950. In my replication I looked at GDP growth in the same 14 countries since the end of World War II and used the common definition of a recession as two negative quarters in a row. The chart below shows a result that is similar to the original chart of Oxford Economics but I took the liberty to add the individual empirical observations to the chart.
The results make intuitive sense. Economic expansions are very fragile shortly after a recession. Even small shocks to the economy can end a nascent recovery and push a country back into recession. Once an economic expansion has gained traction, however, it becomes harder to end.
In an economic recovery, the recovery itself sows the seeds for the next recession and crisis. Imbalances increase in an economic expansion and eventually become so large that a small shock is enough to derail the system and trigger a recession. This shock can be endogenous like the central bank raising interest rates just a little bit too much or too fast, or it can be exogenous like a financial crisis in another country. But no matter the cause, it means that economic recoveries eventually die of old age, which explains why it is increasingly likely for a recession to start if the economic recovery has lasted longer than 8 years or so. Historically, the lowest probability of a recession is observed for an economic recovery that is about 4 to 7 years old.
But the insights from replicating the Oxford Economics study also led to another question. Historically, 90% of the economic recoveries ended 12 years or less after a recession. However, the remaining 10% of recoveries that made it past this 12-year mark lasted much, much longer than that. There is not a single economic recovery that ended between year 12 and year 16 after a recession and except for two cases, every cycle that made it past the 12-year mark also made it to the 17-year mark. It is, as if a cycle that lasts longer than 12 years turns into a “double cycle” that then goes on for at least another 5 years (or roughly a typical economic expansion). What causes an economic expansion to extend for so long and why do these cycles seem to become more “robust” once they pass the 12-year mark is a mystery to me – but an interesting question to investigate.
Cycle length and probability of a recession