Posted by I love reading Nate Silver’s Five Thirty Eight blog (now part of NY Times). Reading it for the political insight is great, but I actually read it for his methodologies. The way he leverages the trends of polls to create his own forecast is fascinating (and more accurate than the individual polls themselves). I also like how he adds his level of certainty for each prediction – it adds another level of insight that talking heads on TV neglect to consider.
I wondered, how did Silver do this election cycle? Pretty well.
Silver’s original map (Senate)
Actual Results
Silver was 92% correct on Senate races (missed AK, NV, & CO) and 95% on Governor races (missed IL & FL). But his record is even more impressive than this…
In 56 of 73 races for Senate/Governor, Silver assigned a 90% probability or higher that he was correct. He got all of them right. This included some tricky races including the Feingold upset, the crowded Senate race in Florida, and the close race in PA.
But even on the races that he had less confidence in, Silver delivered. He was 71% in races he assigned an 80% probability to, and 70% in toss ups (50-80%). Clearly, Silver understands the weaknesses in his model and compensates for it with lower certainty.
Further analysis
To really stress test Silver’s prognostications, I looked at how well he fared on margin of victory. I compared his margin of victory to actual and found that he was within 2 points over 50% of the time for the Senate (30% of the time for Governor). The accuracy is mind-blowing.
Where can Silver improve?
I looked for bias is Silver’s predictions. First, I looked at predictions by party. In both the Senate and for Governor, independents got a higher percentage of the vote than Silver predicted. There wasn’t a consistent bias between Democrat and Republican though.
I also wondered if Silver was better or worse in predicting margin of victory in close races versus blowouts. I compared his “certain” predictions to the ones where he had less certainty to see if there was any difference in the margin of victory error.
Silver did better predicting close races than blowouts. I thought it would be the other way around, since close races typically have a large swing/undecided vote that can go either way, but in landslides, I would have guessed that there is a smaller swing-vote contingent. Perhaps it is driven by more polling in close races, which allows Silver to refine his model more than blowouts. Either way, it’s impressive.