Superforecasting: The Art and Science of Prediction

This chapter explores the mystery of why, in 1956, Archie Cochrane, a pioneer of 20th-century statistical testing in the field of medicine, was so quick to accept a specialist surgeon’s unfounded pronouncement that Cochrane had terminal cancer. In the end, a pathologist who examined the biopsy found that Cochrane did not have cancer at all. The authors consider that the error was made both because Cochrane failed to doubt the specialist and because the specialist failed to doubt his own judgment. They argue that all humans fall into such cognitive errors when we have “been too quick to make up our minds and too slow to change them” (25).

Such cognitive errors dominated medicine for centuries as doctors drew upon guesswork and their experiences of what had previously worked. While ignorance and overconfidence were defining features of medical treatments, patients were often better off waiting out their illnesses than seeking professional advice. The 20th-century innovation of randomized controlled trials, which involved careful measurement and favored using large samples, paved the path for greater accuracy. The physicist Richard Feynman coined the idea that doubt, as opposed to overconfidence, would be what advanced science; scientists must be willing to challenge the hypotheses that seemed so obvious to them. This idea impacted medicine too.

Ironically, Cochrane, the very man who fell for his surgeon’s misdiagnosis, was a key proponent of randomized controlled trials in the 1950s, as he complained that too many healthcare decisions were made without adequate scientific validation. He observed that physicians were stuck in the illusion of their own expertise and were lulled into complacence about their longstanding methodologies. However, when it came to his own treatment, he took his surgeon’s claims for granted.

The authors’ hunch is that Cochrane fell victim to “System 1 thinking” before he could access the more deliberative mode of “System 2 thinking,” concepts originating with psychologist Daniel Kahneman: While System 2 is focused, conscious, reflective thought—the kind that might be used in complex problem-solving—System 1 is the automatic process that delivers hasty, though often inaccurate, conclusions. System 1 thinking is a product of human evolution, as Paleolithic hunter-gatherers had no time to assess whether a shadow on the grass might be a lion and had to make a snap judgment, often based on previous experience.

System 1 thinking also emerges when people are uncomfortable with doubt and uncertainty; however, the quick leap from uncertainty to a satisfying conclusion often produces error. This was the case during the 2011 bombing in Oslo, Norway, when early speculators automatically inculpated Islamic terrorist groups that had previously attacked other European cities. In fact, the Oslo bombing was the brainchild of Anders Breivik, an anti-Islamist and opponent of multiculturalism. The thinking errors that produced the inaccurate conclusion were a result of what psychologists term “confirmation bias,” in which speculators failed either to seek out or to consider information that might contradict their initial explanation; humans naturally seek out evidence that confirms their preexisting beliefs, hence the term confirmation bias. While “this is a poor way to build an accurate mental model of a complicated world, […] it’s a superb way to satisfy the brain’s desire for order because it yields tidy explanations with no loose ends” (39).

Meditating further on Cochrane’s mistake regarding his diagnosis, the authors conclude that he employed the ineffective psychological strategy of “bait and switch,” whereby he substituted the harder question of whether he had cancer with the easier question of whether the specialist would be the sort of person who would know whether he had cancer (39). Cochrane’s mistake shows that even experts are susceptible to cognitive errors.

The authors maintain that intuition can be right or wrong in equal measure. Indeed, what seems like intuition on the part of experts is often simply pattern recognition formed over time. Like the 20th-century pioneers of randomized medical trials, all humans need to honor the fact that we know far less than we think we do about the future.