Michael Lewis does not trust his gut. He doesn’t trust yours, either. He’s spent too long writing about irrational behaviour and collective delusions to have much faith in opinions expressed with professional certainty. Given the choice between an expert and an algorithm, he’ll ask the data every time.
So when he gets a new hip, as the pain caused by his deteriorating cartilage is telling him to, he won’t be looking for a surgeon with a calm, confident manner and decades of experience. He’ll read the stats, meet the medics, and consciously bet against unconscious bias. “If there’s a tall, handsome, self-assured doctor who wants to replace my hip, he’s not going to do it. I want to find somebody who really doesn’t look the part,” he says.
Doesn’t he look the part himself? A patrician son New Orleans, educated at Princeton and the London School of Economics and tempered at Salomon Brothers bank (an experience he described in his first book, Liar’s Poker) Lewis is the most famous non-fiction writer in the United States of America. At fifty-six, gammy hip notwithstanding, he is tall and handsome, in a Michael Douglas sort of way, and as self-assured as ever.
“You know what I really don’t look the part of? An author,” he says. “I look the part of a carefree Southern lawyer. You’re supposed to be troubled and unhappy and difficult and all the rest when you’re a writer. And I really don’t fit that stereotype… I’m actually a superficial, happy person, and I happen to do this.”
Lewis’s heroes tend to be outsiders, marginalised and ridiculed in their attempts to reform a system that they perceive, correctly, to be full of shit. In Moneyball, he told the story of a team of misfits assembled by Oakland A’s manager Billy Beane and his assistant Paul DePodesta, who analysed advanced statistics to find elite baseball players among the cast-offs.
In the Big Short, he described how a handful of fund managers and traders made fortunes betting against the financial industry’s insane, self-serving conviction that the housing boom would never end and that securities backed with junk mortgages were as safe as government bonds.
Both books are packed with arcane knowledge and impenetrable jargon – Collateralised Debt Obligations, Credit Default Swaps and On-Base Plus Slugging Percentage – but also hugely entertaining, and accessible enough to catch Hollywood’s eye.
Lewis’s new book, The Undoing Project, is the final installment in this trilogy about how people fool themselves, although he doesn’t present it as such. It begins with two behavioural economists responding to Moneyball, writing, in effect: “Lewis doesn’t know it, but he’s discussing the findings of Daniel Kahneman and Amos Tversky.”
The two Israeli psychologists did more than anyone to reveal that we are irrational, self-deceiving creatures, liable to make poor decisions when we are convinced that we’re right, see patterns where none exist, ignore facts that don’t fit our theories and jump to conclusions based on tiny sample sizes. Why are so many people convinced that their arthritic pain is worse on wet days, when science proves this is not so? Kahneman and Tversky have the answer.
Cognitive psychology papers can be dry, and rather daunting, but this being Lewis, there’s plenty of drama to keep the pages turning. Kahneman spent much of his childhood hiding from Nazis in Vichy France, and both men fought in the series of wars that established Israel’s modern-day borders. They were also in love with each other, platonically. The letters they wrote to each other when they eventually fell out are open heart surgery.
Kahneman lives in Berkeley, not far from Lewis. They were put in touch by a mutual friend. “We would go on these long walks in the hills, and he would talk about Amos the way you’d talk about an old lover. To this day he’s obsessed with him,” Lewis says. Tversky died in 1996. Six years later, Kahneman was awarded the Nobel Prize in economics for work they had done together.
“They were clearly in up to their necks in this emotional drama, and Danny was very forthcoming about it a long time ago,” Lewis says. “I remember going out to lunch with him and he said ‘I’m in a confessional mood, and I’m just going to tell you what nobody knows. Everybody thinks we just drifted apart. It wasn’t like that at all; it was horrible.’”
Lewis claims that at first, he had “no ambition” to write a book about the partnership. Kahneman had a book of his own to finish, and then had to be convinced that if he didn’t share his story now, someone else would tell it after his death. Lewis travelled to Israel to meet Tversky’s family and dig through his effects three times before sending the proposal to his publisher.
Tversky was supremely confident, aggressive in his opinions. A colleague once devised an intelligence test: “The sooner you figure out that Amos is smarter than you, the smarter you are.” Kahneman’s default mode, by contrast, is self-doubt. “He makes you aware of how much more doubt you should have in yourself,” Lewis says. “You leave Danny Kahneman feeling like confidence is idiocy.”
In a collaboration lasting fifteen years, Tversky and Kahneman often sat beside each other at a single typewriter, laughing at human stupidity (including their own) and hammering out papers that challenged some of the most deeply-held assumptions of psychology and economics. “We were sharing a mind” is how Kahneman puts it.
They came up with the “availability heuristic” to explain that when assessing odds, people are influenced by things they can easily recall: we drive more carefully after passing the scene of a car crash, and worry more about the possibility of divorce if our friends are separating.
They showed how anticipation of regret affects decision-making, proved we are more likely to take risks to avoid a loss than to increase a gain, and devised countless elegant tests that illustrate how easily the human mind tricks itself. To demonstrate the “anchoring heuristic” they told subjects to spin a wheel of fortune numbered from 1 to 100, then asked them to estimate what percentage of the countries in the United Nations are African. People who rolled higher numbers tended to make higher guesses.
Kahneman and Tversky’s influence on our ‘big data’ era has been profound. It can be seen in the primacy of evidence-based medicine over clinical judgement, for instance, and the rise of index funds at the expense of expert stock-pickers. “The world has moved as if it understands Danny and Amos’s writing,” says Lewis.
When Moneyball came out, a fair few sportswriters penned articles saying that Lewis didn’t understand baseball, and that the A’s owed their success to three great starting pitchers, but the dominance of the analytical approach that the team pioneered has since become indisputable.
“I remember thinking that the negative reaction was a status dispute,” Lewis says. “The people were commentators or former athletes, very upset at the thought that these people who just had computers and numbers had a place at their table and maybe knew stuff that they didn’t, in the same way that political reporters have consistently been offended by Nate Silver.”
Silver is the most-respected political polling guru in the USA. Of all the analysts trying to make sense of the polling data in the run-up to the recent presidential election, he was the most bearish about Hillary Clinton’s chances. He didn’t predict Donald Trump’s victory, but he said there was a one in three chance of it happening.
What about you, I ask Lewis, did you see it coming? “Ha!” He lets fly a party-popper guffaw. “No way! My takeaway is Danny and Amos’s: that the world is a far more uncertain place than our mind is prepared to accept.
“The stock market, elections, a lot of these things, it’s basically unknowable and it’s a mug’s game trying to predict it. The problem is people don’t like that, so after the fact they make it seem as if it was knowable, because they can’t live with the idea that we have that level of uncertainty in our lives. But we do!”
In 1973, Kahneman called for a “transformation of cultural attitudes to uncertainty and to risk”. His lecture began by noting how troubling it is to consider “an organism equipped with an affective and hormonal system not much different from that of the jungle rat being given the ability to destroy every living thing by pushing a few buttons.” For all the progress made in limiting our exposure to human error, that remains unchanged.