Daniel Filan
A discussion on the usefulness on 538's forecasts
Recently, an article was published decrying the usefulness of 538’s forecasts of political events and Nate Silver’s opinions. I thought that this was largely misguided, and so got in an argument on Facebook about it. The argument is preserved for posterity, because I basically agree with what I said.
My first response to the article:
He bases his claim to have succeeded off his having given Trump a somewhat higher probability of a win than some other people.
Make that a significantly higher probability of a win than anyone else who was forecasting based off poll data (rather than yard signs/halloween costumes/feelings). I’m pretty sure the closest contender was the upshot, who gave Trump half the chance of winning than Silver did. That’s a pretty significant difference.
Silver makes sure to hedge every statement carefully so that he can never actually be wrong. And when things don’t go his way, he lectures the public on their ignorance of statistics. After all, probability isn’t certainty, he didn’t say it would definitely happen.
Sure, but things usually go his way. You can check this by looking at all the races that he predicted this year and in previous years - he ends up looking relatively good.
But recognize what it means: even when Silver isn’t wrong, because he’s hedged everything carefully, he’s still not offering any information of value.
Of course he’s offering information of value. If you think that Donald Trump has a 25% chance of being president, you’re going to be significantly more interested in preparing for that eventuality than if you think he has a 0.5% chance of becoming president, and significantly less than if you think that he has a 75% chance of becoming president.
But for anyone interested in the actual human lives affected by political questions, Silver’s analyses are of almost no help. They can tell us today that Silver thinks Trump has a 5% chance of winning. But then we might wake up tomorrow and find that Silver now thinks Trump has a 30% chance of winning.
If you think that Trump has a 5% chance of winning, than more likely than not you should think that his chances will decrease over time, not increase. Maybe they eventually shoot up to 100%, but there’s only a 5% chance of that - that’s just what the 5% number means.
And the important question for anyone trying to affect the world, as opposed to just watching the events in it unfold, is how those chances can be made to change.
If you want to affect the world, you need to know how much you can affect it, and part of that involves knowing what the chances of certain outcomes are.
The problem is that poll data analysts are completely fucking useless in a crisis. They don’t understand anything that’s going on around them, and they’re powerless to predict what’s about to happen next.
This is just not true. Probabilistic forecasts are useful for predicting what’s about to happen next, as demonstrated by their track record in 2008, 2012, and 2016, because that’s literally what they’re about.
The response of someone who posted the article:
But recognize what it means: even when Silver isn’t wrong, because he’s hedged everything carefully, he’s still not offering any information of value.
Of course he’s offering information of value. If you think that Donald Trump has a 25% chance of being president, you’re going to be significantly more interested in preparing for that eventuality than if you think he has a 0.5% chance of becoming president, and significantly less than if you think that he has a 75% chance of becoming president.
This doesn’t seem to engage with what Robinson’s criticism is. Robinson isn’t saying it wouldn’t be important to know that Trump has a 25% chance of becoming president. He’s saying that probability of that eventuality is not the thing reported by the number put on the 538 website. What is reported on the website is Silver’s guess about what that actual probability is, and a guess based on a methodology that most consumers of media do not understand and one that seems incredibly sensitive to…something (why was the number today 15% less than the number yesterday?)
If it’s difficult to tell what the relationship is supposed to be between the number Silver puts up and the number we’re actually interested in, then we have a problem that isn’t statistical.
My response to that comment:
He’s saying that probability of that eventuality is not the thing reported by the number put on the 538 website. What is reported on the website is Silver’s guess about what that actual probability is, and a guess based on a methodology that most consumers of media do not understand and one that seems incredibly sensitive to…something (why was the number today 15% less than the number yesterday?)
Firstly, I didn’t get the sense that that’s what Robinson was complaining about – what quotes made you think that this was the concern?
Secondly, I think that there’s good reason to think that the 538 forecast is pretty close to the probability that you would assign if you knew everything there was to know - you can give scores to probabilistic forecasts that reward them for being more certain rather than less and at the same time ensure that events given 90% probability happen 90% of the time. I think that these scores put the 538 forecasts in a good light (see for instance BuzzFeed’s analysis). I’d be interested to hear reasons why the probabilities are bad other than referring to a few specific instances where they were wrong.
Thirdly, you say that the 538 forecasts are “a guess based on a methodology that most consumers of media do not understand and one that seems incredibly sensitive to…something (why was the number today 15% less than the number yesterday?)”. Admittedly, by the nature of probabilistic forecasts, they have to be a guess. I’m sure that most media consumers don’t understand them, but they could if they read 538’s in-depth explanation. Regarding the claim that they’re incredibly sensitive, I don’t really buy this. I can find exactly one day where the polls-only model jumps by 15%, right after the DNC when polling was really good, producing a bounce that that particular model didn’t adjust for. The polls-plus model, which does know about the conventions, didn’t show such a bounce. Why do you think that there’s so much of a problem that the forecast is basically meaningless?
OP’s response:
To the first thing - I’m describing what I would guess statistical forecasting would seem like to someone who isn’t in the relevant know. I take Robinson to be gesturing at the interpretation problem here: “Similarly, Silver will make predictions that have multiple components, so that if one part fails, the overall prediction will seem to have come true, even if its coming true had no relation to the reasons Silver originally offered.”
And here: “The myth of Nate Silver’s continued usefulness is based on a careful moving of goalposts”
Though in both spots he blames Silver, what seems to me to be at work is an underlying unclarity about what is being communicated.
Second, it may be the case that the 538 forecast is the probability that I would assign if I had all the data. But I’m certainly not claiming that predictions are useless, and in Robinson’s careful moments he doesn’t either.
Third - 15 percent was an arbitrary and hyperbolic number, I’m actually pretty surprised that that ever happened. My point is, again, on the side of a lay person just refreshing their screen and seeing a different number, and trying to figure out for themselves what has changed about the universe such that Trump has a better chance of winning today than he did yesterday. My guess would be that many visitors to his website would be at a loss to explain that sort of thing, which is useful to think about in terms of reporting stats. And again, I’ll emphasize that the uselessness of statistical forecasts is Robinson’s position, not mine - I don’t have any problem with statistical forecasts or any particular bone to pick with Silver.
Me again:
“Similarly, Silver will make predictions that have multiple components, so that if one part fails, the overall prediction will seem to have come true, even if its coming true had no relation to the reasons Silver originally offered.” And here: “The myth of Nate Silver’s continued usefulness is based on a careful moving of goalposts”
The first quote is in the context of Silver randomly saying stuff, which is probably legit. The second one is referring to the forecast, which as I’ve pointed out is better than he is acting like it is, see e.g. the Buzzfeed analysis.
But I’m certainly not claiming that predictions are useless, and in Robinson’s careful moments he doesn’t either.
I’m sure he has some moments where he doesn’t say that predictions are useless, but he also says “They can tell us today that Silver thinks Trump has a 5% chance of winning. But then we might wake up tomorrow and find that Silver now thinks Trump has a 30% chance of winning. And the important question for anyone trying to affect the world, as opposed to just watching the events in it unfold, is how those chances can be made to change”, and the only way I can reasonably interpret that is “probabilities are unimportant because they can change”.
My point is, again, on the side of a lay person just refreshing their screen and seeing a different number, and trying to figure out for themselves what has changed about the universe such that Trump has a better chance of winning today than he did yesterday.
Firstly, I just don’t buy that this is what Robinson is talking about (is someone here friends with him so that he can be tagged?). Secondly, if this was your actual concern, the forecasts had an ‘updates’ tab which included polls and how they moved the numbers. 538 also regularly had pieces and podcasts explaining why the numbers changed (link to the most recent one).
OP:
The first quote is in the context of Silver randomly saying stuff, which is probably legit. The second one is referring to the forecast, which as I’ve pointed out is better than he is acting like it is, see e.g. the Buzzfeed analysis.
Even if it is better than he is acting like it is when properly interpreted, it doesn’t follow that it is better than he is acting like it is on common, actual interpretations.
I’m sure he has some moments where he doesn’t say that predictions are useless, but he also says “They can tell us today that Silver thinks Trump has a 5% chance of winning. But then we might wake up tomorrow and find that Silver now thinks Trump has a 30% chance of winning. And the important question for anyone trying to affect the world, as opposed to just watching the events in it unfold, is how those chances can be made to change”, and the only way I can reasonably interpret that is “probabilities are unimportant because they can change”.
That strikes me as an entirely unfair interpretation. The point isn’t well made but it’s incoherent stick him with the claim that he doesn’t think probabilities are important if his warrant is that he thinks its important to change probabilities. Maybe he hasn’t earned a lot of rope but we should do better than that.
Firstly, I just don’t buy that this is what Robinson is talking about (is someone here friends with him so that he can be tagged?). Secondly, if this was your actual concern, the forecasts had an ‘updates’ tab which included polls and how they moved the numbers. 538 also regularly had pieces and podcasts explaining why the numbers changed (link to the most recent one).
I started this discussion thread out by saying that this article wasn’t fair to Silver, and these are the things I had in mind. So this I concede straightaway, with the caveat that points to my general interest in articles like this: that consumers are culpably negligent in consuming information in the way that they do does not mean that producers of information are off the hook. If that culpable negligence is predictable then we might ask questions about further steps producers should take, and this article shows some stuff to take stock of. I don’t take it that Robinson is a particularly unsophisticated reader (I concede off the bat that having an axe to grind can make someone otherwise competent functionally equivalent to a bad reader, but in this case I don’t think that is the whole story)
Me:
Even if it is better than he is acting like it is when properly interpreted, it doesn’t follow that it is better than he is acting like it is on common, actual interpretations.
Sure, but it seems like 538 have taken great pains to help people interpret it better, and if you think “well it’s just hard to communicate probabilistic forecasts and 538 could have done it better”, then that’s fine but seems separate to the original article.
That strikes me as an entirely unfair interpretation. The point isn’t well made but it’s incoherent stick him with the claim that he doesn’t think probabilities are important if his warrant is that he thinks its important to change probabilities.
It does seem like he thinks that probabilistic forecasts are unimportant, given the above quote and the end of the article: “That doesn’t mean there’s anything wrong with Nate Silver, just that nobody should ever pay any attention to him. Nate Silver will probably always be the best poll data analyst. The problem is that poll data analysts are completely fucking useless in a crisis. They don’t understand anything that’s going on around them, and they’re powerless to predict what’s about to happen next… [Silver] tells you entirely about the world as it looks to him right now, rather than the world as it could suddenly be tomorrow.” The most straightforward readings of this I can make are either “The 538 forecast measures current sentiment but is bad at predicting the state of the race” (which I think is just factually false) or “Probabilistic forecasts are unimportant because they could change given effort”. I just can’t understand what else he could possibly mean.
I started this discussion thread out by saying that this article wasn’t fair to Silver, and these are the things I had in mind. So this I concede straightaway, with the caveat that points to my general interest in articles like this: that consumers are culpably negligent in consuming information in the way that they do does not mean that producers of information are off the hook. If that culpable negligence is predictable then we might ask questions about further steps producers should take, and this article shows some stuff to take stock of.
I think that this is pretty interesting, but almost disjoint to what I understood the article and the section you quoted to be about. Discussion about what the article means aside, I sort of agree, and think that 538 could have done better (e.g. by letting you sample maps from their forecasts), but at the same time think that they did do relatively well, especially to readers who read their articles about the forecast.
OP:
Sure, but it seems like 538 have taken great pains to help people interpret it better, and if you think “well it’s just hard to communicate probabilistic forecasts and 538 could have done it better”, then that’s fine but seems separate to the original article.
That’s not what I’m thinking. I’m thinking something more along the lines of “well what is it that you’re communicating when you report statistics in the sort of media context that we have?”
It does seem like he thinks that probabilistic forecasts are unimportant, given the above quote and the end of the article: “That doesn’t mean there’s anything wrong with Nate Silver, just that nobody should ever pay any attention to him. Nate Silver will probably always be the best poll data analyst. The problem is that poll data analysts are completely fucking useless in a crisis. They don’t understand anything that’s going on around them, and they’re powerless to predict what’s about to happen next… [Silver] tells you entirely about the world as it looks to him right now, rather than the world as it could suddenly be tomorrow.” The most straightforward readings of this I can make are either “The 538 forecast measures current sentiment but is bad at predicting the state of the race” (which I think is just factually false) or “Probabilistic forecasts are unimportant because they could change given effort”. I just can’t understand what else he could possibly mean.
From the looks of it, he’s arguing against the kind of fatalism people can develop when confronted with the sort of epistemic authority that statistics are often used to claim. Don’t let Nate tell you that battleground state is a lock for the RNC, says Robinson: go out and canvass anyway, because no matter what the polls tell Nate today, tomorrow is another day. Maybe you’re scratching your head and wondering why Nate is supposed to disagree with something like that - and that’s not without justice, as the thought that what I just said pits one against forecasting is at best confused - but if you are just scratching your head then you, as I said in the beginning, probably aren’t engaging with the perspective that Robinson seems to be inside of and speaking to.
I started this discussion thread out by saying that this article wasn’t fair to Silver, and these are the things I had in mind. So this I concede straightaway, with the caveat that points to my general interest in articles like this: that consumers are culpably negligent in consuming information in the way that they do does not mean that producers of information are off the hook. If that culpable negligence is predictable then we might ask questions about further steps producers should take, and this article shows some stuff to take stock of.
I think that this is pretty interesting, but almost disjoint to what I understood the article and the section you quoted to be about. Discussion about what the article means aside, I sort of agree, and think that 538 could have done better (e.g. by letting you sample maps from their forecasts), but at the same time think that they did do relatively well, especially to readers who read their articles about the forecast.
I don’t think so. If this use of statistics speaks so poorly to a class of otherwise engaged readers (on the guess that Robinson isn’t alone here) then I wonder what statistics for world-changers could look like. We have some thoughts here about how it would have to succeed in communicating about itself.
At this point I got tired of responding.