Polls, Predictions, and Puke

In the interest of full disclosure and advance explanation of likely snarkiness on my part, I should mention that in four days I begin taking Mid-Terms in three graduate courses. The significance of this fact is that I have had much less time to work on poll analysis than I would like, and so this article will necessarily be more general than usual in thought.

It’s October in an election year, which means that while all the kiddies are picking out imaginative costumes to wear, the pollsters are going all out to find the most imaginative spin for their press releases, because this is their big chance to be – drum roll please – “newsmakers”.

If you find that notion, that polls should be considered news of themselves, more than slightly irritating, join the club. Polls are news in the same way that former Representative Foley would be a good babysitter for your boys, or Nancy Pelosi has a valid comprehension of U.S. National Security. I have spent what little free time I had during the past couple weeks reading predictions for the 2004, 2002, and 2000 elections, and frankly if I were a polling group I would be more than a little bit humble about what I claimed. Granted, most polls played CYA in the late months, making calls both ways so that they could come back later, bury the wrong guesses, shine a light on the lucky guesses and call themselves “experts”. I will claim bluntly that I am as much or more an expert as John Zogby or Larry Sabato on predicting elections, but that’s not claiming much – anyone could hope to do better than those clowns, yet they continue to get paid for shoddy work and biased claims. There was not even one truly reliable predictor of the last three elections, and the reader should be aware of this whenever considering a poll report.

The first reason most reports cannot and do not predict the outcome of an election to any degree of reliability, is the nature of the organization which releases the report. Polls, for example, are at best no more than a snapshot of the mood of the public. And polls are not all that equal, bias finding its way into most of them to varying degrees, sometimes so blatantly as to create comedy rather than veracity. Should the reader find him or herself interested in examining a poll, I will repeat here the basic caveats of poll analysis:

- All valid polls which release results to the public will also release internal data
- All valid polls will use a consistent methodology from one occasion to the next, and will report that method in their summary
- All valid polls will weight according to Census norms and known demographic political balances
- All valid polls will make archive data available for comparison to current polls


To be blunt, very few polls meet this standard. Gallup, Pew, The New York Times, NBC News/Wall Street Journal, Newsweek and Survey USA are the only organizations which perform to these standards (in the case of the NYT, Newsweek and NBC/WSJ, these companies sponsor polls and demand consistent standards as I described) on a regular basis. This does not mean that I agree with their stated conclusions, but rather that their transparency means that suspected bias can be backed out and the numbers examined in an unweighted or rebalanced state if so desired.

It is also in the nature of election predictors to fall into two common blunders. If a group you have never heard of suddenly claims spectacular results from its predictions, it may well not be a coincidence. Many groups which “predict” elections do nothing of the sort – rather than risk seeing their prediction blow up in their face, as Sabato and Zogby had happen in 2004, they post a number of mildly worded opinions, ranging to cover all reasonable possibilities but nothing to call unusual attention to any one position, then later on pretend that their wisdom and mathematical genius foretold the results. The other common practice is the last-minute pick; Tradesports and similar shills make a fair piece of change by promoting the “accuracy” of their positions immediately before election day – they never mention the wild variance in their mid-season numbers. To put this in perspective, the election campaign effectively begins when the general candidates are known, say by late spring. This gives us, roughly, 180 days of campaign during which time someone may make a prediction. To make a call two days before the election is like predicting the winner of a football game with half a minute to go in the fourth quarter; it’s gutless and has no value as a predictor. I will harass John Zogby for being so partisan in the 2004 election, but at least he had the guts to call his prediction in the summer. He was completely wrong and destroyed his credibility as a pollster, but at least he called his shot plainly (though it should be noted that Zogby began back-tracking from his prediction a few weeks later).

Next: Poll Integrity