The Sordid Business of Opinion Polls

As I was riding the Park n Ride to work one morning last week, the bus passed a fortune-teller’s store, which had gone out of business. The place had been doing poorly for a very long time. As I pondered the painted word ‘psychic’ on the side of the building, I wondered why the medium did not know this was going to happen. For all of that, there are many fortune-tellers who make good money, telling folks stories which sound good, which is what the customers really want. Of course, there are many people who won’t be taken in by horoscopes and fortune-tellers; they know better than that. Yet, many of those fine people read the poll reports in the papers and watch the polls in television, so they will know who’s going to win the election. They believe the claims of people they do not know about what will happen in the future, just because they like the story.

I have followed opinion polls for some years now, and I have great respect for a poll which is properly done, and properly reported. Unfortunately, there are a lot of polls which get published without providing readers or viewer with the proper context, and worse there are a number of major polls which hide vital parts of their internal data. The media, it must be remembered, is not and never has been in the business of reporting facts. Instead, they want to sell a story, in order to claim lots of attention which will increase readership or ratings. And in this election, the media has overwhelmingly been biased in favor of Obama.

Before I continue, I need to go back and say again how a poll should be properly done. There are two major authoritative poll accreditation bodies;

The National Council on Public Polling (NCPP), and
the American Association for Public Opinion Research(AAPOR). Any valid poll should include a disclosure that it complies with NCPP or AAPOR standards of methodology, but most do not do so.

The NCPP has published a guide for journalists to use when discussing the report from an opinion poll. Some of the questions they say a journalist should ask include

Who did the poll?
Who paid for the poll and why was it done?
How many people were interviewed for the poll? (This, by the way, is why the state polls are less statistically valid than the nation polls. They usually involve a much smaller poll of respondents than the national polls use)
How were the people chosen to take part in the poll?
What area were those people from?
What was the response rate to contact attempts? (almost no one answers that one)
How does the poll describe the results?
What questions were asked, and how were they worded?
What order was used for questions? (this is an old, old trick – if a controversial issue is asked just before a certain candidate is named, often this creates a false connection between the candidate and the controversy in the mind of respondents)
What events are likely to have impacted the poll results?

The plain fact is, that almost no media outlet or polling group reveals the answers to all of these kinds of questions. And that damages their credibility as valid reflections of the public’s opinion.

The AAPOR warns that polls almost always fall into one of seven categories of sponsorship:

1. Academic institutions
2. Federal, State and Local Governments
3. Media Organizations
4. Non-profit groups or Foundations
5. Special interest groups
6. Businesses and Corporations
7. Political campaigns, consultants, and candidates

They also warn about many of the same problems in poll reporting as the NCPP, and the AAPOR goes further to warn that many of these sponsors may be biased, even when they claim to be objective. The first question, in any poll result, is 'who benefits’?

This brings me to Rasmussen. In an earlier article, I rather harshly said that Rasmussen was in it for the money. I have always asked the questions recommended by the NPCC/AAPOR, and in doing so I have noticed how some major polls refuse to answer important questions. I learned a long time ago that if you had enough information, you could work out the mechanics of most major conclusions. I found Gallup, AP-Ipsos, and CBS News very open about their methodology and the internal data of their polls. But I have always been disappointed in the secretive and sometimes disingenuous behavior of Zogby, NBC, ABC, and Rasmussen. As it happens, Scott Rasmussen has enjoyed a lot of success at his business, but if you want internals it will cost you, and if you want to reverse-engineer his polls to test for bias, he simply won’t allow you a way to have all the information needed. That, again being blunt, strikes me as both mercenary and essentially dishonest. And therefore, no matter what his polls say I will always see a little red flag next to his published claims. My rule of thumb is very simple – it does not matter whether the poll says what I’d like to hear, it matters whether all the supporting data is available.

Another thing that is troubling, is the desire for instant gratification. For example, Sarah Palin clearly beat Joe Biden in the debate they had last week (although ardent democrats are just as sure Biden triumphed), and so there are people on both sides looking at the current polls for signs of their guys big win. But there are two reasons why you will not see that, at least right now. First, even the invalid daily tracking polls take their results from three days of polling, which means the first poll with full post-debate reaction will not be released until Monday. And I have to warn you that the polls will not reflect an immediate valid effect – if they show a sudden jump for either side it is far more likely to mean that the polls have – once again – changed the party affiliation weights, a practice as dishonest as putting one foot on the floor while you weigh yourself. The reason is because while people do change their minds, historically it has always been a few people at a time and in small steps. One proof of that is that as popular as Presidents Reagan and Clinton were, neither saw a major shift in party identification during his Administration. People supported the man, not necessarily the party, and in this election it would foolish indeed to imagine that people who supported Biden were also becoming more pro-democrat than before, or that those who supported Palin were becoming more pro-republican than before. After all, the biggest statistical group of undecideds consider themselves to be independents, and while they will come over to one candidate or the other, that does not lock them in with the whole party. Party identification should be a static measure for the duration of a campaign, and manipulating party affiliation is simply cheating the results to create a story out of imagination.