October 29th, 2012
Some people seem to be sharpening the knives to go after Nate Silver if Obama loses. I’d like to register an early dissent.
There are valid criticisms of black box statistical modeling. On the one hand, we’re asked to view the results credibly without knowing what the special sauce used to bake them is. You could mount that criticism of Silver as much as you could of any other modeler. Or pollster, for that matter. So at the end of the day you have to either make your peace with the black boxes, or write them off as value-less.
I happen to find some value in them. They aren’t predictive–but I’d argue they’re not really meant to be. They’re simply informative–just more data points from which we cobble together our understanding of a system (an election) which is so multi-variate that, as Scott Fitzgerald once wrote about Hollywood, is so complex that no more than a handful of men can keep the entire equation in their heads.
What’s more, Silver’s a very agile writer. Like Michael Lewis he has a gift for explaining complicated numerical concepts. (I would not agree with the charge that Silver often makes simple mathematical concepts sound grandiose and complex.) And finally, Silver hedges. Always and everywhere. Some people might take this to be weasely on his part, but it strikes me as just the opposite: It’s humility. Silver is in the numbers business, but he understands that the numbers don’t tell us everything. So you’ll never hear him say, “X has happened so Y must happen.” Just the opposite, actually. Silver understands the limits of his own models. He acknowledges those limits nearly every time he writes. I think this ought to be applauded.
If Romney wins should that discredit Silver’s models? Only so far as anybody ever used them as oracular constructs instead of analytical tools.
One final word: People seem to think that it would reflect badly on Silver if Romney were to win while Silver’s model shows only a 25 percent chance of victory. But isn’t 25 percent kind of a lot? If I told you there was a 1-in-4 chance of you getting hit by a bus tomorrow, would you think that 25 percent seemed like a big number or a little number? Or, to put it another way, a .250 hitter gets on base once a game, so you’d never look at him in any given at bat and think there was no chance he’d get a hit.
Ultimately I’d suggest that the real test for Nate Silver is the same as it is for any analyst, on any subject. Not “did he predict an outcome correctly” (or “did he predict the outcome I prefer”) but “does his work add value to our understanding of the subject.”
Speaking only for myself, the answer to that question is an unqualified yes.
Update: On Twitter, Jim Henley (@UOJim) pointed to a meditation he wrote on probability. It’s written from the perspective of a D&D nerd with cancer and it’s very much worth reading on its own, apart from its tangential bearing on our larger discussion here.
Also, Galley Friend A.W. offers the following dissent:
I like Silver quite a lot. But I have one major quibble: His numerical specificity occludes the enormous subjectivity inherent in his weighing and discounting of various polls. (See http://fivethirtyeight.blogs.nytimes.com/methodology/)
He’s selling a false certainty. It’s the Washington equivalent of Wall Street’s now-infamous “value at risk” (VaR) models at the center of recent Wall Street meltdowns. (E.g., http://www.futuresmag.com/2010/12/01/var-the-number-that-killed-us)
In end, my problem with Silver’s presentation is the same as Naked Capitalism’s indictment of VaR:
“But VaR is a particularly troubling example, more so because it is sufficiently, dangerously simple minded enough that regulators and managers a step or two removed from markets have become overly attached to its deceptive simplicity.” (http://www.nakedcapitalism.com/2012/05/jp-morgan-loss-bomb-confirms-that-its-time-to-kill-var.html)
Obviously, in scrutinizing and combining polls, such relative judgments are unavoidable. But Silver ought to be more transparent and up-front in presenting how those subjective judgments affect his bottom-line numbers.
-
Not buying it. With his insisting that Obama is 75-80 percent in, one of only two scenarios can be true after the election. Either Silver is the acknowledged guru of polling data whose opinion ought to be taken more seriously than anyone else’s. Or he’s revealed as a partisan hack hiding behind pretty language about selective data, and is sent back to amuse the Kos kids.
-
I don’t get the Tigers-Giants analogy. That certain sportswriters believed one team or another was superior based on watching them all season is definitionally different than predicting the outcome of an election that polls people who will actually determine its outcome. If you had polled players from both teams on which would win the Series, you’d presumably end up with a 25-25 tie.
Silver’s interpretation of the data gives Obama a 3 in 4 chance of winning, which seems out of whack with everything else I read. If Romney gets over 300 electorals, Silver is, imo, fatally discredited. If Obama wins by even a single electoral, Silver has to be taken seriously forever more.
-
If I get 4 pieces of paper and write Obama on 3 and Romney on 1 and put them into a hat, you would agree that if I pick out a paper at random there’s a 75% chance it will say Obama on it, right? What if the paper I picked out said Romney? Would you say there was never a 75% chance to begin with that an Obama paper would be chosen?
I also don’t quite understand your point about polling baseball players. Polls use representative (hopefully at least) samples of the population. If you polled every single voter in the US then you’d probably have a pretty good idea of the outcome. As it stands now, though, like Jonathan mentioned, anyone who uses polls for their so-called predictive powers clearly doesn’t understand the actual value that polls have.
-
If Romney wins, it doesn’t mean that Nate was wrong, nor does it mean that his model can’t be tested. The model builds in correlations and trends. And he can, and no doubt will, look at “conditional probabilities”. That is, right now he claims that there is a 25+% chance of a Romney win. As many have pointed out, that’s far, far from zero. (Would you play Russian Roulette with two bullets in the gun???)
But Nate can take his model, and under the condition that Romney wins, describe what that win should look like – in terms of percentage of vote, electoral college, etc., etc. A post hoc analysis of a Romney win should show that *how* he won fell within the “likely scenarios” of the 538 model for a Romney win.
And if they don’t, e.g., Romney wins 355 electoral votes (I’ve seen conservative web sites claiming this will happen), I’m pretty sure you’ll see Nate Silver say that he missed something or the polls missed something.
Likewise, if Obama wins with 350+ electoral votes, you won’t hear Nate Silver claiming that was vindication of his 75% prediction. Quite the opposite, I’m sure. He’ll point to that as the model or the polls having failed. That’s not a likely scenario even under the “condition” that Obama wins.
And that’s where the value of Nate’s work comes in. It describes not only the odds of Obama or Romney winning, but gives a pretty good mathematical description of where the “movable factors” are. Granted, a lot are obvious – Romney really needs to flip Ohio, and then some; North Carolina & Florida are among Obama’s least likely and important swing states. But it also puts numbers to other scenarios. Electoral vote tie, which I’ve seen countless articles on (a whole hour devoted to it on NPR’s “On Point”) is a 1/200 chance. Electoral/popular split is maybe 1/20. A recount being important is 1/10. All these numbers make a lot of mathematical sense.
There will be lots of ways of telling how on target Silver was, even if Romney wins.
-
[…] Jonathan V. Last: People seem to think that it would reflect badly on Silver if Romney were to win while Silver’s model shows only a 25 percent chance of victory. But isn’t 25 percent kind of a lot? If I told you there was a 1-in-4 chance of you getting hit by a bus tomorrow, would you think that 25 percent seemed like a big number or a little number? Or, to put it another way, a .250 hitter gets on base once a game, so you’d never look at him in any given at bat and think there was no chance he’d get a hit. Tweet FILED UNDER: Campaign 2012, Quick Picks, Steven Taylor, US Politics […]
-
No, he’s a remarkably untalented writer. There’s no single glaring example, so just grab any one of this year’s leaden, ponderous “now-cast” updates at random. An example of a careful but lucid technical writer might be that old Mass family court justice David H. Kopelman from 18 years ago–now that is how you reduce a complicated subject into 50-some intelligible pages. Whereas Silver clearly gets paid by the pound.
-
Though he’s a better writer than this “Why I Am Pro-Life” feeb
-
[…] as The Weekly Standard’s Jonathan Last pointed out, Silver’s estimate doesn’t mean that there is no chance Romney will win: “a .250 hitter [in […]
-
Silver shouldn’t use a decimal point: e.g., Obama has a 76.8% chance of winning. It makes him look silly. He probably should use racetrack-style odds to convey the uncertainty better: e.g., Romney is at 3 to 1 against, improved from 7 to 2 against last week.
-
As a non-gambler, I don’t find racing odds to be terribly intuitive, especially when comparing fractions with different denominators. I end up having to do math with fractions in my head for a moment. It’s not hard, granted, but when he says, “Obama went from a 80% chance of winning to 72% chance,” I instantly understand it.
And no, I’m not terribly upset by the lack of decimal. After all, 72% is already 0.72. I don’t think it matters much if you push that out another significant digit to 0.724 or whatever.
-
Yep, Silver claiming candidate A/B will get “273.00106 electoral votes under this model” is a shopworn trick, affecting a spurious air of precision to lend authority to his interpretive reports; Mr. A.W.’s invocation of Black-Scholes and other quant crutches is apt. Likewise w/ Silver’s excess trust in verbosity (which is plainly shared among his online groupies)
-
I’d second A.W.’s point above. Whatever value Silver’s work has as an informative/analytical tool is undercut a good bit by the subjectivity in his weighting of various polls. This piece by Josh Jordan at NRO does a good job of getting into the details:
http://www.nationalreview.com/articles/331192/nate-silver-s-flawed-model-josh-jordanI wish Silver had stuck w/baseball.
-
The thing about Silver’s analysis and Moneyball is that it doesn’t work.
The A’s won no Championships. None.
-
ETA: Since 1998.
-
I tend to like Silver, even if I question some of what he does. Some of the math and weighting he does I’d wager makes no sense. But there’s a real, REAL divergence between state and national polling. After the election, a lot of people are going to be wrong no matter what happens.
If Romney wins the Presidency, what will be interesting is to see how Silver adapts future models. People tend to forget…. the man did ONE presidential election. He was a highly intelligent and entertaining writer, but it was still just one election. In his congressional models in 2010, he lowballed the GOP until it became blatantly obvious they were going to win in a romp. That might just mean there’s a problem with his model. We need a bigger sample size and more data. So if he were to lose this one, it wouldn’t mean he’s a hack, unless he managed to get almost every state wrong. And quite simple, even a novice is going to get 40 out of 50 states right. If silver is still doing this 10-20 years from now, we can begin to see if he really is the pontiff of polling, or just another opinion maker who is as accurate as a coin flip.
-
Well it’s probably JL’s last Beltway-mumbo-jumbo post before then (I do look forward to more lightsabers & iPads) so I’m gonna call it now: Obama croaks over the line in a ’48-style squeaker, with negative coattails; the Electoral College won’t matter this year, as usual. This was how I felt after Romney got the nom and I never changed my mind, even after “Pennsylvania was in play” (bwahaha)
R campaign will compliment itself on winning demographically whitened Iowa, though. Akin wins, but so does Warren, by a hair of a sliver of a fraction of a cat’s whisker. GOP still pick up Montana, N. Dakota, Nebraska, and “surprisingly” Wisconsin; much self-congratulation despite blowing Ohio & Fla. (though Virginia really was a reach). The idea of the Dems gaining in the House is too ludicrous to countenance–they didn’t even improve in California, thanks to the new nonpartisan primary–so I commend you to this sharp overview of their bungling Illinois machine:
http://www.thedailybeast.com/articles/2012/10/13/democratic-dreams-of-big-wins-in-illinois-fading.htmlMy prognostication: based on watching lots & lots of TV, scientifically of course.
-
* Crowd source map:
http://www.270towin.com/americas-electoral-map/
* Mine–contest ended today so I’m not worried of being copied, natch
http://www.270towin.com/2012_election_predictions.php?mapid=bbjm -
OK, I am now on board with the “that putz Akin” mob, and Iowa, you’re dead to me
-
[…] V. Last makes related […]
-
[…] So I now understand why some conservatives have it in for Nate Silver. It’s because of stuff like this, where Paul Krugman uses Silver’s work to say that […]
Nylund October 31, 2012 at 11:37 am
It’s perfectly possible that Obama does indeed have a 75% chance of winning, but will actually lose, not invalidating anything Silver has done. But, you’re right in the sense that whether or not Obama wins will be interpreted as a major coup for Silver, or proof that he’s a hack. But that doesn’t say anything about Silver’s actual work. That’s more a statement on those who judge him.
If Romney wins, all these people will say that since an event happened that Nate Silver said might happen, that proves he’s an idiot. Yes, it makes no sense. The problem is, too many people are thinking that since Nate says Obama has a high probability of winning that he’s stating, with certainty, that Obama will win, when he’s doing anything but!
Look at the predictions for the World Series prior to game 1. Nearly every “expert” picked the Tigers to win. And unlike Nate, they weren’t stating probabilities. They were picking WITH CERTAINTY, all or nothing, yes or no, picks. Of course, the Tigers lost. Does that mean all those experts should be fired and ignore forever? I doubt a single one will lose any “expert” status despite picking the loser. And here we have Nate, not picking a winner or lose, making a statement that allows either outcome to be possible, and people are acting as if he should be banished to Siberia if the event he says has a lower probability of happening end up being the event that happens.