“It is much easier after the event to sort the relevant from the irrelevant signals. After the event, of course, a signal is always crystal clear; we can now see what disaster it was signaling, since the disaster has occurred. But before the event it is obscure and pregnant with conflicting meanings. It comes to the observer embedded in an atmosphere of ‘noise’ i.e.; in the company of all sorts of information that is useless and irrelevant for predicting the particular disaster.”

-Roberta Wohlstetter Pearl Harbor: Warning and Decision (1962)

Forgive me for sounding redundant, but The Signal And The Noise is about the challenge in seeing the signal in the noise. The truth among the lies. The wheat from the chaff. The essential from the extraneous. The book is a document of human ingenuity and folly.

“This need of finding patterns, humans have more than any other animal,” I was told by Thomas Poggio, an MIT neuroscientist who studies how our brains process information.”Recognizing objects in difficult situations means generalizing. A newborn baby can recognize the basic pattern of a face. It has been learned by evolution, not by the individual.”

The problem, Poggio says, is that these evolutionary instincts sometimes lead us to see patterns that aren’t there. “People have been doing that all the time,” Poggio said. “Finding patterns in random noise.”

Nate Silver is a statistical journalist who makes predictions in sports and politics for media outlets like ESPN and The New York Times. Here, he examines the successes and failures of many industries and pastimes that are on the forefront of prediction making; including meteorology, chess, poker, epidemiology, political elections, sports, and the stock market.

The tone of the book is cautiously optimistic because the author is weary of the notion that more data makes for better predictions. Instead, he contends that more data is mostly more noise for the truth to be buried in. Things might be worse than ever, considering that humanity is generating around 2.5 quintillion bytes of data each day, exponentially more than ever before.

The human brain is quite remarkable; it can store perhaps three terabytes of information. And yet that is only about one-millionth of the information that IBM says is now produced in the world each day. So we have to be terribly selective about the information we choose to remember.

Meanwhile, if the quantity of information is increasing by 2.5 quintillion bytes per day, the amount of useful information almost certainly isn’t. Most of it is just noise, and the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine – but a relatively constant amount of objective truth.

Information is no longer a scarce commodity; we have more of it than we know what to do with. But relatively little of it is useful. We perceive it selectively, subjectively, and without much self-regard for the distortions that it causes. We think we want information when we really want knowledge.

He sees the intellectual history of prediction science as a pendulum swinging back and forth between Laplace’s Demon and Heisenberg’s Uncertainty Principal. In other words, between incautious and hedonistic optimism, and the humbling realization that the future is unwritten and so cannot be foreseen.

Our views on how predictable the world is have waxed and waned over the years. One simple measure of it is the number of times the words ‘predictable’ and ‘unpredictable’ are used in academic journals. At the dawn of the twentieth century, the two words were used almost exactly as often as one another. The Great Depression and the Second World War capitulated ‘unpredictable’ into the dominant position. As the world healed from these crises, ‘predictable’ came back into fashion, its usage peaking in the 1970s. ‘Unpredictable’ has been on the rise again in recent years.

Further reading:

Excerpt from the book on FiveThirtyEight.com
Bob Lefsetz on Nate Silver
What the professionals had to say—Wall Street Journal book review
Buy from Amazon
The Signal and the Noise: Why So Many Predictions Fail — but Some Don’t

Interesting anecdotes and new vocabulary

Laplace’s Demon — A hypothetical omniscient beast that was 18th century mathematician/astronomer Pierre-Simon Laplace’s rationalization for a predictable universe that could be comprehensible to humanity if only they had enough information.
Heisenberg’s Uncertainty Principal — “How can you predict where something is going to go when you don’t know where it is in the first place? You can’t.”
Schoolhouse Blizzard 1888 —a blizzard hit that unexpectedly in the American Great Plains on January 12, 1888 on a relatively warm day. The temperature dropped almost 30 degrees in a few hours and a blinding blizzard caught people unaware. Hundreds of children, leaving school as the blizzard hit, died of hypothermia on their way home.
Lisbon Earthquake 1775 — “No type of catastrophe is more jarring to our sense of order than an earthquake. They quite literally shake our foundations. Whereas hurricanes descend upon us from the heavens and have sometimes been associated with metaphors for God’s providence, earthquakes come from deep underneath the surface and are more often taken to be signs of His wrath, indifference, or nonexistence. (The Lisbon Earthquake of 1755 was a major spark for the development of secular philosophy.) pg. 145”
Brownian noise — noise produced by the motion of particles, like a waterfall. “If you listen to true white noise, which is produced by random bursts of sounds over a uniform distribution of frequencies, it is sibilant and somewhat abrasive. The type of noise associated with complex systems, called Brownian noise, is more soothing and sounds almost like rushing water. pg.173”
Livingston Survey — an organized effort to predict economic variables like GDP.
1976 Swine Flu Outbreak — A political disaster reminiscent of the current Ebola scare. The H1N1 strand responsible for the Spanish Flu of 1918-20 and 50 million deaths kills one Army lieutenant at Fort Dix. A series of dire predictions soon followed. Gerald Ford;s secretary of health, F. David Mathews, predicted that one million Americans would die. So Ford took the resolute step of asking Congress to authorize some 200 million does of vaccine, and ordered a mass vaccination program, the first of its kind since Polio. Overwhelming majorities in both houses approved his plans at a cost of $180 million. The next flu season came and no cases of H1N1 were reported outside of Fort Dix the previous year. The Ford Administration doubled down, releasing a series of ominous public service announcements that served more to terrify the population of the government and the vaccine itself than of the flu.
anosognosia — When a possibility is unfamiliar to us, we don’t even think about it.
sibilant — (of a speech sound) sounded with a hissing.

Highlighted passages:

Books had existed prior to Gutenberg, but they were not widely written and they were not widely read. Instead, they were luxury items for the nobility, produced one copy at a time by scribes. The going rate for reproducing a single manuscript was about one florin (a gold coin worth about $200 in today’s dollars) per five pages, so a book like the one you’re reading now would cost around $20,000. It would probably also come with a litany of transcription errors, since it would be a copy of a copy of a copy, the mistakes having multiplied and mutated through each generation.

Shakespeare’s plays often turn on the idea of fate, as much drama does. What makes them so tragic is the gap between what his characters might like to accomplish and what fate provides to them. The idea of controlling one’s fate seemed to have become part of the human consciousness by Shakespeare’s time – but not yet the competencies to achieve that end.

The story the data tells us is often the one we’d like to hear, and we usually make sure that it has a happy ending.

The importance of the Industrial Revolution is hard to overstate. Throughout essentially all of human history, economic growth had proceeded at a rate of perhaps 0.1 percent per year, enough to allow for a very gradual increase in population, but not any growth in per capita living standards. And then, suddenly, there was progress when there had been none. Economic growth began to zoom upward much faster than the growth rate of the population, as it has continued to do through to the present day, the occasional global financial meltdown notwithstanding.

We face danger whenever information growth outpaces our understanding of how to process it. The last forty years of human history imply that it can still take a long time to translate information into useful knowledge, and that it we are not careful, we may take a step back in the meantime.

The 1970s were the high point for ‘vast amounts of theory applied to extremely small amounts of data,’ as Paul Krugman put it to me. We had begun to use computers to produce models of the world, but it took us some time to recognize how crude and assumption laden they were, and that the precision that computers were capable of was no substitute for predictive accuracy. In fields ranging from economics to epidemiology, this was an era in which bold predictions were made, and equally often failed. In 1971, for instance, it was claimed that we would be able to predict earthquakes within a decade, a problem that we are no closer to solving forty years later.

If there is one thing that defines Americans – one thing that makes us exceptional – it is our belief in Cassius’s idea that we are in control of our own fates. Our country was founded at the dawn of the Industrial Revolution by religious rebels who had seen that the free flow of ideas had helped to spread not just their religious beliefs, but also those of science and commerce. Most of our strengths and weaknesses as a nation – our arrogance and our impatience – stem from our unshakable belief in the idea that we choose our own course.

But the new millennium got off to a terrible start for Americans. We had not seen the September 11 attacks coming. The problem was not want of information. As had been the case in the Pearl Harbor attacks six decades earlier, all the signals were there. But we had not put them together. Lacking a proper theory for how terrorists might behave, we were blind to the data and the attacks were an ‘unknown unknown’ to us.

“This need of finding patterns, humans have more than any other animal,” I was told by Thomas Poggio, an MIT neuroscientist who studies how our brains process information.”Recognizing objects in difficult situations means generalizing. A newborn baby can recognize the basic pattern of a face. It has been learned by evolution, not by the individual.”

The problem, Poggio says, is that these evolutionary instincts sometimes lead us to see patterns that aren’t there. “People have been doing that all the time,” Poggio said. “Finding patterns in random noise.”

The human brain is quite remarkable; it can store perhaps three terabytes of information. And yet that is only about one-millionth of the information that IBM says is now produced in the world each day. So we have to be terribly selective about the information we choose to remember.

Meanwhile, if the quantity of information is increasing by 2.5 quintillion bytes per day, the amount of useful information almost certainly isn’t. Most of it is just noise, and the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine – but a relatively constant amount of objective truth.

Information is no longer a scarce commodity; we have more of it than we know what to do with. But relatively little of it is useful. We perceive it selectively, subjectively, and without much self-regard for the distortions that it causes. We think we want information when we really want knowledge.

The signal is the truth. The noise is what distracts us from the truth.

One of the pervasive risks that we face in the information age is that even if the amount of knowledge in the world is increasing, the gap between what we know and what we think we know may be widening. This syndrome is often associated with very precise-seeming predictions that are not at all accurate.

It’s easy to adopt a goal of ‘exploit market inefficiencies.’ But that doesn’t really give you a plan for how to find them and then determine whether they represent fresh dawns or false leads. It’s hard to have an idea that nobody else has thought of. It’s even harder to have a good idea – and when you do, it will soon be duplicated.

That is why this book shies away from promoting quick-fix solutions that imply you can just go about your business in a slightly different way and outpredict the competition. Good innovators typically think very big and they think very small. New ideas are sometimes found in the most granular details of a problem where few others bother to look. And they are sometimes found when you are doing your most abstract and philosophical thinking, considering why the world is the way it is and whether there might be an alternative to the dominant paradigm. Rarely can they be found in the temperate latitudes between these two spaces, where we spend 99 percent of our lives. The categorizations and approximations we make in the normal course of our lives are usually good enough to get by, but sometimes we let information that might give us a competitive advantage slip through the cracks.

The key is to develop tools and habits so that you are more often looking for ideas and information in the right place – and in honing the skills required to harness them into W’s and L’s once you’ve found them.

Bak’s favorite example was that of a sand pile on a beach. If you drop another grain of sand onto the pile (what could be simpler than a grain of sand?), it can actually do one of three things. Depending on the size and shape of the pile, it might stay more or less where it lands, or it might cascade gently down the small hill toward the bottom of the pile. Or it might cascade gently down the small hill toward the bottom of the pile. Or it might do something else: if the pile is too steep, it could destabilize the entire system and trigger a sand avalanche. Complex systems seem to have this property, with large periods of apparent stasis marked by sudden and catastrophic failures. These processes may not literally be random, but they are so irreducibly complex (right down to the last grain of sand) that it just won’t be possible to predict them beyond a certain level.

If you’re looking for an economic forecast, the best place to turn is the average or aggregate prediction rather than that of any one economist. My research into the Survey of Professional Forecasters suggests that these aggregate forecasts are about percent more accurate than the typical individual’s forecast at predicting GDP, 10 percent better at predicting unemployment, and 30 percent better at predicting inflation. This property – group forecasts beat individual ones – has been found to be true in almost every field in which it has been studied.

As the statistician George E.P. Box wrote, “All models are wrong, but some models are useful.” What he meant by that is that all models are simplifications of the universe, as they must necessarily be. Everything else is leaving out some sort of detail. How pertinent that detail might be will depend on exactly what problem we’re trying to solve and on how precise an answer we require.

Successful gamblers – and successful forecasters of any kind – do not think of the future in terms of no-lose bets, unimpeachable theories, and infinitely precise measurements. These are the illusions of the sucker, the sirens of his overconfidence. Successful gamblers, instead, think of the future as speckles of probability, flickering upward and downward like a stock market ticker to every jolt of new information. When their estimates of these probabilities diverge by a sufficient margin from the odds on offer, they may place a bet.

The need for prediction arises not necessarily because the world itself is uncertain, but because our understanding it fully is beyond our capacity.

However, when a field is highly competitive, it is only through this painstaking effort around the margin that you can make any money. There is a ‘water level’ established by the competition and your profit will be like the tip of an iceberg: a small sliver of competitive advantage floating just above the surface, but concealing a vast bulwark of effort that went in to support it.

In theory, the value of a stock is a prediction of a company’s future earnings and dividends. Although earnings may be hard to predict, you can look at what a company made in the recent past (Shiller’s formula uses the past ten years of earnings) and compare it with the value of the stock. This calculation – known as the P/E or price-to-earnings ratio – has gravitated toward a value of about 15 over the long run, meaning that the market price per share is generally about fifteen times larger than a company’s annual profits.

There is fairly unambiguous evidence, instead, that insiders make above-average returns. One disturbing example is that members of Congress, who often gain access to inside information about a company while they are lobbied and who also have some ability to influence the fate of companies through legislation, return a profit on their investments that beats market averages by 5 to 10 percent per year, a remarkable rate that would make even Bernie Madoff blush.

In today’s stock market, most trades are made with someone else’s money. The 1990s and 2000s are sometimes thought of as the age of the day trader. But holdings by institutional investors like mutual funds, hedge funds, and pensions have increased at a much faster rate. When Fama drafted his thesis in the 1960s, only about 15 percent of stocks were held by institutions rather than individuals. by 2007, the percentage had risen to 68 percent.

These statistics represent a potential complication for efficient-market hypothesis: when it’s not your own money on the line but someone else’s, your incentives may change.

Some theorists have proposed that we should think of the stock market as constituting two processes in one. There is the signal track, the stock market of the 1950s that we read about in textbooks. This is the market that prevails in the long run, with investors making relatively few trades, and prices well tied down to fundamentals. It helps investors to plan for their retirement and helps companies capitalize themselves.

Then there’s the fast track, the noise track, which is full of momentum trading, positive feedbacks, skewed incentives and herding behavior. Usually it is just a rock-paper-scissors game that does no real good to the broader economy – but also perhaps no real harm. It’s just a bunch of sweaty traders passing money around.

Over the very long run, the stock market essentially always moves upward. But this tells you almost nothing about how it will behave in the next day, week, or year.

“There is a tendency in our planning to confuse the unfamiliar with the improbable. The contingency we have not considered seriously looks strange; what looks strange is thought improbable; what is improbable need not be considered seriously.” -Thomas Schelling”

Groups and individuals have all sorts of aspirations that they make no effort to act on because they doubt their ability to achieve them.

Whatever range of abilities we have acquired, there will always be tasks sitting right at the edge of them. If we judge ourselves by what is hardest for us, we may take for granted those things that we do easily and routinely.

Information becomes knowledge only when it’s placed in context. Without it, we have no way to differentiate the signal from the noise, and our search for the truth might be swamped by false positives.

What isn’t acceptable under Bayes’s theorem is to pretend that you don’t have any prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many. To state your beliefs up front – to say ‘Here’s where I’m coming from’ -is a way to operate in good faith and to recognize that you perceive reality through a subjective filter.”

Staring at the ocean and waiting for a flash of insight is how ideas are generated in the movies. In the real world, they rarely come when you are standing in place. Nor do ‘big; ideas necessarily start that way. It’s more often with small, incremental, and sometimes accidental steps that we make progress.

Our views on how predictable the world is have waxed and waned over the years. One simple measure of it is the number of times the words ‘predictable’ and ‘unpredictable’ are used in academic journals. At the dawn of the twentieth century, the two words were used almost exactly as often as one another. The Great Depression and the Second World War capitulated ‘unpredictable’ into the dominant position. As the world healed from these crises, ‘predictable’ came back into fashion, its usage peaking in the 1970s. ‘Unpredictable’ has been on the rise again in recent years.

These perceptions about predictability are more affected by the fashions of the sciences and the shortness of our memories – has anything really bad happened recently? – than by any real change in our forecasting skills. How good we think we are at prediction and how good we really are may even be inversely correlated. The 1950s, when the world was still shaken by war and was seen as fairly unpredictable, was a time of more economic and scientific productivity than the 1970s, the decade when we thought we could predict everything, but couldn’t.