This book isn’t just about prediction, or even the limits of knowledge. It is about the ascent of man. According to Nate Silver, the American electoral analyst, the digital age and its explosion of knowledge constitute a great turning point in human history. Never before have we had so much evidence on which to base our predictions of the future. Yes, there have been setbacks along the way, but we should feel optimistic about the direction of travel.
Yet there is a paradox about the evolution of prediction. Innovations in technology and information make it possible to analyse data far more quickly and extensively. The availability of so much raw data, however, means that much of the analysis is woeful. There are many more bullets; but also many more inept marksmen. T. S. Eliot’s question has never been more apt: ‘Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?’
The Signal and the Noise acknowledges this, but Silver remains an optimist, hoping to build on the tradition of Enlightenment self-confidence. A similar case for optimism was memorably expressed 15 years ago by Peter L. Bernstein, whose Against the Gods argued that modern finance constituted the front line of the great march of knowledge over uncertainty. The revolutionary idea that defines the boundary between modern time and the past is the mastery of risk, argued Bernstein. ‘Until human beings discovered a way across that boundary, the future was the murky domain of oracles and soothsayers who held a monopoly over knowledge of future events.’ Unlike Bernstein, Silver has to temper his optimism to account for a massive financial crisis — which the combined predictions of regulators, ratings agencies and experts failed to spot.
This book ranges confidently across sport, politics, weather and gambling. Perhaps the less you know about the subject, the more there is to enjoy. I was riveted by the chapter on chess. Until recently, the best chess champions were able to defeat even the most powerful chess computer. That changed in 1997 when ‘Deep Blue’ beat a tetchy Gary Kasparov. Ironically, Kasparov was not defeated by a masterpiece of computer code. He got spooked by a bug, which he misinterpreted as higher intelligence — and ended up chasing shadows, identifying depths of strategic insight that weren’t always present.
Silver emerges as a modest, engaging narrator and an admirably clear thinker. But he might have explored more deeply why we find it so hard to disentangle two separate threads when thinking about the future. The first is prediction: what will happen? The second is strategy: how can I capitalise on what happens? Silver’s book assumes, not untypically, that the latter relies on the former — that a good strategy must be predictive.
But there is an alternative view. Rather than hitching ourselves to the game of prediction, what if we develop strategies that reduce our exposure to uncertainty by being more sceptical about the future — and hence more flexible and adaptive? The best strategist I’ve met — he was professional sportsman before becoming an educationalist — disliked the term strategy and was impatient with prediction. Instead of wasting time trying to predict, he focused on being open to change — hence benefitting from the light-footedness of not being attached to a predictive view of the world. His strategy, in effect, was anti-strategy. By decoupling the arts of prediction and strategy, he embodied what Nassim Taleb, author of The Black Swan, calls ‘antifragility’ (the subject of Taleb’s new book).
This approach is often misrepresented — usually by people who are scared of uncertainty or have an ‘evidence-driven’ spreadsheet to deliver to their boss — as pessimistic and unpractical. In fact, antifragility is sceptical but not remotely fatalistic.
There is a further problem with the thirst for statistically-driven prediction. It can encourage the mistake of turning all disciplines into pseudo-science. Journalism is a case in point. There is nothing innately ‘rigorous’ about over-liberally sprinkling dubious findings from soft social sciences on top of every argument. Some insights can stand on their own two feet.
Perhaps that is another way of saying there are many different kinds of truth, that science does not have a monopoly on describing reality. Which more truthfully captures human nature: Anna Karenina or the latest dodgy experiment that claims to have discovered the ‘neurological causes’of love (‘his pre-frontal cortex was firing off so much oxytycin that he jumped off his horse and offered his hand in marriage’)?
The ability to make data-derived predictions belongs to a much broader intellectual challenge: distinguishing which subjects can be addressed scientifically and which cannot. As a practitioner, Silver scored hits by making predictions about baseball and US elections. As he points out, both disciplines lend themselves to statistical analysis. But the approaches that work for baseball and elections are not universal. Summoned to speak at a conference about terrorism, Silver can only agree with the critic who replies, ‘That’s all very well, Nate. But how in the hell is this applicable to terrorism?’
So does the genuine advancement of certain forms of prediction outweigh the harm done by overconfident predictions in spheres that do not lend themselves to statistical analysis? Silver says it does. Even those who are less convinced will enjoy his balanced, intelligent and erudite advocacy.