Poll Integrity

First Tuesday of November, 2004. Months of wrangling, haranguing, pleading and begging all came down to the bottom line. And as always happens in modern elections, the polling groups had their teams out to ask voters how they voted. Knowing how close the 2000 Presidential Election had turned out, everyone was naturally eager to see how the voters responded, and since the polls would not close until evening, the media was hungry for the exit polls.

And what a story they told. Early (and unverified) reports about exit polls not only showed a Kerry lead, they showed large leads in some of the battleground states. Kerry advocates began to openly claim they were headed for a landslide Kerry victory.

The actual vote was significantly different. The paranoid and the bitter voiced baseless claims of Conspiracy, while even more mature minds asked how such a discrepancy could happen between the exit polling and the actual results. This was not due to any cataclysmic collapse of the polling process, nor any sinister conspiracy on anyone’s part. The exit polls have always been unreliable in their raw form, but this time the rush to a story caused invalid conclusions to be drawn from the data.

The Mystery Pollster did a good run-down right after the election on the problems with exit polls. Suffice to say, exit polls are essentially the same as any other opinion poll, but since they are rushed and inconstant on numerous points, to my mind they have a much higher volatility rate – the chance they will be badly wrong – than other polls. In the case of the 2004 exit polls, early clues were present in the demographics; the early polls claimed 59% of the respondents were women, and a heavy under-30 participation was noted. This conflicted with both historical electorate demographics and observed participation in early voting and film footage – what happened was that the polls used a lot of inexperienced field personnel, who found it more appealing to poll women than men, and young people more receptive than the old. The pressure to get quick results corrupted the sample, in short.

The damage from the poor work on the exit polls has led to all sorts of collateral damage, including fueling some incredibly stupid conspiracy theories, muddying up the issue of improving election transparency, and lowering the public trust in polling institutions. Since 2004, an increasing number of people have begun to claim that they deliberately mislead pollsters in the hopes of skewing the results. While this practice cannot be confirmed in fact and is unlikely to reach a threshold of actually invalidating overall trend analysis in opinion polling, it is a salient concern that such an attitude is on the rise. This brings me, at last, to my topic of Poll Integrity.

I have discussed specific polls before, and plan to do so again. For now, it is important to consider the source when a poll is released to the public. I would be repeating myself to state the most important aspects of a valid poll, but then again, the points are important enough to deserve repeating:

- All valid polls which release results to the public will also release internal data
- All valid polls will use a consistent methodology from one occasion to the next, and will report that method in their summary
- All valid polls will weight according to Census norms and known demographic political balances
- All valid polls will make archive data available for comparison to current polls

The astute reader may have realized that in my mind, ‘valid’ and ‘integrity’ are synonymous for polls. “Integrity”, after all, simply means ‘complete’ or ‘whole’. I have been privileged to exchange phone calls and e-mails with some people in a number of polling organizations, and need to say that in the main, even those polls with which I disagree may well be professional, honest, and careful both in their analyses and their reports. That said, there are clear differences in the classes of performance, and some polls simply do not demonstrate the sort of integrity to set themselves apart from the rest.

In this final month before the mid-term elections, there will be a flood of polling reports from both well-known and some almost-unknown polling groups. In the main, polling groups may be private companies which accept contracts to perform polling for clients (Gallup is the best known of these), they may be a partisan group which clearly prefers one side over the other but whose research is released to the public (Lake Snell Perry or Zogby, for example), they may be a university which conducts ongoing research and builds on empirical samples (Quinnipiac and Pew are well-known here), or they may be associated with a news organization, usually as a sponsor (the Princeton Survey Research, for example). There are many other sorts of polling groups, but the key is to begin by noting who is paying for the survey. Most polls will be direct about who sponsored the effort. If a poll hides this information, this is a big red flag.

Demographics are a critical component to polls. All polls weight their results, because no significant sample correctly matches the public as a whole. Demographics include gender ratio as well as race, urban/suburban/exurban/rural breakdown, as well as key professional and political norms. That last piece, political identification, is the most contentious and the most common factor to be manipulated or slanted, because many polling groups insist on their own preference for political identity among voters; despite the clear trend of Republican gains in the past decade, implying strong identification growth of the GOP, many polling groups – especially those sponsored by media outlets – refuse to correct obsolete balances held over from the Clinton era. Keep that fact in mind.

The next element in poll integrity is methodology. This is both a very stable, yet controversial area. Random Digit Dialing, or RDD, is the dominant method of contacting respondents, generally at their home by telephone in the evening and on weekends. The controversy comes from the fact that fewer and fewer and fewer Americans are likely to be at home, use a landline telephone for their primary contact, or be inclined to participate in unsolicited contact from strangers. Also, both sample sizes and the randomness of the respondent poll has come into question – a poll should automatically suspect which does not show at least 1,024 registered voters as a respondent pool, or which cannot confirm that they have a method in place to prevent repeat contacts with respondents – increasingly, polls have been asking respondents if they can contact them again, a practice which can easily lead to unethical manipulation of the poll for future polls.

And any poll which only queries “adults” is garbage.

Also, if a polling group mixes methodology, as Zogby did by adding phone poll responses to online poll responses, treat that poll as you would a child predator – it cannot be trusted, ever.

In conclusion, many polls can be useful, but not all polls are equal – indeed, some should be rejected from the start. It is a critical factor to know what sort of poll is producing the press release.