All this week, CoxBLUE will be blogging live from Hubspot’s Inbound 13 Conference in Boston.
Relative to every other University of Chicago-educated economics geek who spends his days hunched over a laptop crunching numbers, Nate Silver is a rock star. He’s taken his love of statistics and applied it to every field he’s passionate about, from poker to sports to politics, and he’s conquered all of them.
After finishing college in 2000, Silver worked for three years as a consultant with professional services firm KPMG in Chicago. He soon became burned out there, so he began playing poker online full-time; from there, he moved on to forecasting baseball. He became a cult celebrity in 2008, when his PECOTA model correctly predicted that the expansion Tampa Bay Rays, laughingstock of the American League for a decade, would leap from last place to first in one season. The Rays advanced to the World Series that October, and Silver’s name was on the map.
Next, he took on the political world. Using his economics background, some common sense and a whole lot of polling data, he devised a model to predict the outcomes of presidential and congressional elections. Silver’s “538 method” saw moderate success in 2008, with the election of Barack Obama, and again in 2010, when the Republicans seized control of Congress. Then, in 2012, he achieved national renown by correctly calling all 50 states on the electoral map as Obama defeated challenger Mitt Romney. In the week leading up to Election Day, Silver’s popularity peaked so high that Google Analytics briefly showed searches for his name surpassing those of Vice President Joe Biden.
Silver downplays his momentary fame, self-deprecatingly pointing out that Justin Bieber’s Google presence still absolutely dwarfs that of himself and Biden combined. Addressing a packed house at the 2013 Inbound Marketing Conference in Boston, Silver also pointed out that he’s living in the perfect era for the world to accept his newfangled approach to analytical thinking.
“There certainly there has been paradigm shift in the availability of digital information,” Silver told the crowd. “The World Wide Web was only invented in 1990. I’m just barely old enough to remember when I had go out in the morning and get the newspaper to see how the Detroit Tigers had done every day. But not anymore. Every bit of information you want available at your fingertips. It’s available instantly.”
When baseball met Moneyball
When Silver first began tackling baseball statistics in the middle of the last decade, he was hardly the godfather of that movement. He was drawing from the earlier findings of writer Bill James, who invented what’s now known as “sabermetrics,” and piggybacking off of work already performed by several forward-thinking teams, most notably the Oakland Athletics and Boston Red Sox.
It was only natural that baseball would be the first sport to fall in love with statistical analysis, Silver noted.
“Baseball has the best data set of anything ever, really,” he said. “Every play from every game over the last 150 years has been recorded. There are people who will spend their entire weekend going back through box scores from 1921 to see if Wally Moon was given a double instead of a triple.”
With the proliferation of knowledge of the game that’s out there today, it’s relatively easy to perform research. The tricky part, though, was finding out how specifically to apply it. The “Moneyball” era, a term coined by author Michael Lewis in 2003, was the beginning of baseball’s attempts to bring the burgeoning “big data” craze into the game. Silver was instantly drawn to it.
“What [Oakland general manager] Billy Beane and the early adopters like the Red Sox did was find out how to go a long way with relatively simple stuff,” Silver said. “On-base percentage is worth a lot more than batting average. College players are more reliable draft picks than high school players. It’s very, very simple stuff, but applying that could go a long way amid a culture that wasn’t very friendly to analytics.”
The game has changed now, and eggheads like Silver are employed by every major league organization. Once the stats revolution had fully conquered baseball, Silver moved on. After the front office, he next set his sights on the Oval Office.
Playing politics
Countless people attempt to predict the outcome of the presidential election every four years. Bloggers, newspaper columnists and TV talking heads everywhere are making their picks, and by and large, their results tend to be unimpressive. Everyone knows that New York will be a blue state every four years while Texas goes red, but interpreting minor fluctuations in swing states is a precise science, and few can master it.
But master it, Silver did – after going 49-for-50 in 2008 (he guessed incorrectly that John McCain would carry Indiana), he was perfect in 2012. Pundits everywhere marveled at his wizardry, but Silver insisted that his method was relatively simple. There were three steps, he said – you average the polls, you count to 270 and you account for margin of error.
“Really, it’s not a ‘big data’ problem,” he said. “It’s more of a ‘medium data’ problem. When you only have a couple of thousand polls that you’re looking at, you’re not working with data on quite the scale that large multinational corporations are.”
The difference between Silver’s method and everyone else’s was that he removed all subjective thought from the equation, basing his projections on only the facts. Some might make their picks with a bias, either to the left or the right, but Silver didn’t waste time with partisan politics. Nearly all of his counterparts let their opinions skew the data.
“When people begin to have a lot more information than they’re used to having, they become very biased in the facts they choose to interpret,” Silver noted.
He cited the Drudge Report as an example. Rather than compile all the polls and present them fairly, blogger Matt Drudge instead went about cherry-picking the ones that were favorable to Romney and painting his own picture of the data. That only serves to deceive the reader, Silver said.
“You should know better,” he cautioned. “But if you don’t, and you read this site on election morning, you’re going to be sorry on election night.”
Blocking out the noise
In his new book, “The Signal and the Noise,” Silver explores in depth the art of interpreting statistics. He notes that a lot of people are like Matt Drudge, selectively picking out one point or another without zooming out to see the full picture. That, he explained, is where you get into trouble.
“You’re looking for relationships here, really,” Silver said. “People think of statistics, and they think it means a factoid. Like, you know, ‘Dustin Pedroia hit .308 last year.’ That’s not really a statistic the way I use statistics. That’s just a data point. What you really want is actionable information and knowledge. Statistics is a science that ought to aid us with our decision-making. You’re thinking about connections.”
Silver explained two more examples to conference attendees in Boston. One was the legendary 1997 chess showdown between world champion Garry Kasparov and IBM supercomputer Deep Blue. In the first round of their six-game match, the computer made a highly unorthodox move that puzzled the human champion – as it turns out, the play was the result of a bug, as Deep Blue had simply gotten stuck, given up and made a random play. Kasparov attributed the move to “superior intelligence,” and he began to show anxiety and play an overly defensive style, afraid of the computer.
Example No. 2 was an ordinary cab ride that Silver took in his current home city of New York. His cab driver had a GPS, and the device told him that taking a shortcut through Central Park would help him get across town faster because there was zero traffic on the street. Of course, that was too good to be true – it turns out the road was closed, and Silver’s cab driver had to turn around and find a detour to get around the park.
Both of the above cases are examples of a human overreacting to one fact, failing to think critically about the larger perspective. In the future, Silver says, big data scientists should seek to overcome this temptation.
“We should talk about what makes data rich,” he said. “I prefer rich data over big data.”
What about business?
Silver has already mastered gambling, sports and politics. Naturally, the next question is how his insights can apply to the business world, where everyone is looking to capitalize on the “big data” craze.
“If you pick up Businessweek, or the Harvard Business Review, or the Wall Street Journal, you can look at the ads, and almost every other ad is for ‘big data,'” Silver said. “There’s a lot of hopes there. And I do think it will accomplish some things for some people – but some people see it as a kind of magic elixir that will solve all of their problems.”
Data will never solve anything. In order for businesses to truly achieve success, they need to take their statistical models and infuse them with critical thinking. Furthermore, they need to make three major improvements to the stockpiles of data they already have – they must improve the quality, quantity and variety of information at their disposal.
We’re only now beginning to see the economic effects of the information revolution. Silver noted that throughout hundreds of years of innovation in technology, there was little impact on the world’s economy. The printing press came around in the 1440s, revolutionizing the concept of mass production, and the steam engine arrived in 1775, dramatically changing transportation. But it wasn’t until 1990, when the World Wide Web came around, that per-capita GDP spiked worldwide.
The world is still struggling to figure out what to make of that. The Internet and the data movement have been around for a couple of decades now, but relatively speaking, they’re still in their infancy. If people want to discover new ways to apply big data, they must be willing to experiment and make mistakes.
Silver closed his speech by quoting the Danish scientist and author Piet Hein, who said the road to wisdom was “plain and simple to express.”
“Err and err and err again,” he said, “but less and less and less.”
Thanks to Silver and other innovators like him, the world is moving in that direction.
