Poll Bias 2008: What the Numbers Say

Back in 2004, I determined a general measure of bias in the Presidential election by polls. I did the same thing for this year, which may be useful in examining the post-Convention stretch. I began by noting the 2-way poll results from Real Clear Politics from January 1 2008 through August 25.

I made a spreadsheet with the name of the group or agency which conducted the poll, the starting and ending dates of the polling, and the results for Obama and McCain. From this information, I drew up another spreadsheet which produced the daily average for the polls. I then compared each poll against the average for the days polled, to see where each poll stood against the average, noting whether Obama or McCain was getting more or less support than the average in a poll. I then was able to apply that information to determine the average bias, the aggregate variance, and essentially the stability of each poll.

I want to stop at this point, and make an observation about the fairness of these polls. As polls make statements which favor Obama or McCain, it is becoming common to see this poll or that dismissed as being unreliable or unbelievable. This is generally unwise, because in examining the polls presented, I was struck by how well, overall, most polls were conducted. While I have concerns about the demographics base used, and the weighting assumptions inherent in modern polling, the conclusions of each poll were consistent with other polls at the same time. Since twenty-six national agencies or groups have conducted national polls used for my research, such consistency indicates a high level of professional standards. However, within that acknowledgment, there were some clear trends and characteristics, which I think are worth mentioning.

As I said, there were twenty-six separate groups or agencies which conducted polls. Of these, six conducted only one or two polls this year (Battleground, McLaughlin, Franklin & Marshall, Ipsos,POS/GQR, and NPR). Accordingly, I have not considered these groups for the purposes of comparative bias.
That leaves another twenty groups, however, which have conducted a measurable pattern in their polls. For this report, I will note the five polls which most favor Obama, the five which are least favorable to Obama, the five which are most favorable to McCain, the five which are least favorable to McCain, and the five which are, as an aggregate, the least likely to be off the average measure for that date. I would like to emphasize that bias is used here not as an attack on the poll’s integrity or veracity, but simply marks a tendency likely to continue in future polls between now and the election. It remains to be seen which of these polls will prove the most accurate in the final measure.
Here are the top five in each of the five categories, then:

Obama Favorable
1. ABC News/Washington Post (+3.33 pts)
2. CNN (+3.00 pts)
3. Quinnipiac (+1.97 pts)
4. Pew Research (+1.54 pts)
5. USA Today/Gallup (+0.92 pts)

Obama Unfavorable
1. Investor Business Daily/TP (-2.80 pts)
2. FOX News (-2.30 pts)
3. Cook/RT Strategy (-2.28 pts)
4. Hotline/FD (-1.62 pts)
5. LA Times/Bloomberg (1.26 pts)


McCain Favorable
1. Democracy Corps (+3.27 pts)
2. USA Today/Gallup (+2.83 pts)
3. Rasmussen (+1.89 pts)
4. CNN (+1.51 pts)
5. ABC News/Washington Post (+1.40 pts)

McCain Unfavorable
1. Investor Business Daily/TP (-4.55 pts)
2. CBS News (-2.80 pts)
3. Hotline/FD (-1.88)
4. Quinnipiac (-1.57 pts)
5. Time (-1.40 pts)


Aggregate Stability
1. NBC News/Wall Street Journal (1.81 avg total variance)
2. AP-Ipsos (2.06 avg total variance)
3. Gallup (2.17 avg total variance)
4. Pew Research (2.40 avg total variance)
5. CBS News/NY Times (2.53 avg total variance)


I will let the reader draw his/her own conclusions, after making three suggestions:

1. Notice which groups are favorable or unfavorable to both Obama and McCain

2. Aggregate Stability is a near-unknown quality, but represents a reason to have great confidence in a poll. If a poll does not miss the average by very much, it represents a reliable place for quick vitals.

3. Most of the variance shown is within the standard deviation for the population base used in the polling.