A Reminder About Polls

I have noticed that a lot of the media noise right now is about polls, Job Approval polls, estimates of leads in opinion polls, generic party preference polls, take your choice. I did a search under the News section of Google for “polls” and got 17,600 hits. Under “opinion polls elections” I got 1,490 hits. This is under ‘News’, and that’s no mistake – the media is regularly treating opinion polls as news events in themselves. Even the blogs do this, so every so often it is important to stop and go back to the basics, to understand the information in a proper and organized fashion.

To be blunt, there are good polls and there are bad polls, and I do not mean that a poll is good if it says what I want to hear, or vice versa. In fact, I may disagree with the stated conclusions of a poll, yet consider it a good one. This is because, as I judge them, a “good” poll is one which basically tells you how it got to the results it posts, especially noting the demographics of the respondent pool, the ‘ingredients’ which make the poll what it is. The really bad polls are the ones which either hide the internal data, which use an inconstant methodology, select demographic weighting significantly at odds with known standards, or which present a proprietary poll for a client but then attempt to present it to the public as an objective effort without financial influence from an outside party. It can be tricky, but after you learn which polls are consistent and reliable, you can get a better feel for what is going on.

Rule number one – don’t be fooled by headlines. More than a few agencies, especially national wire services, have a peculiar habit of taking one item from a poll and ignoring significant information. A case in point would be the recent claims of poor support for President Bush, without observing A – that the President, even at his lowest point, enjoyed significantly better support and approval than Congress, or either or the two major political parties, and B – that the President’s numbers have slowly but steadily been rising for several months. If you are going to consider the results of a poll, take the time to read everything, especially how the questions are phrased and in what order.

Rule number two – Look for a poll’s internal data, including demographics of the respondent pool and weighting. Some polls will claim that they need to protect their methodology from copying, but that’s simply dishonest. ANY poll which releases information to the public, owes support for its claims. No exceptions, no excuses. This is critical, because the balance of things like urban/suburban/rural, men to women, or race proportions has everything to do with the results. If I did a poll mostly on the East Coast, but claimed it was a national poll, that would hardly be a true statement, and if I overcounted minorities the results would unquestionably be skewed. If a poll will not give specifics and support its numbers, they are not being honest, no matter whether you like what it says or not.

Rule number three – reading a poll can be interesting, but never take any single poll by itself for your source of opinion. Always consider the results from at least three polls in any two-week period of time on any question, or you risk letting an anomaly throw you off.

And finally, rule number four - even if a poll is valid and accurate, and done with a conscientious application of accepted standards and practices, it is only a snapshot of the moment. Never count too much into any poll.