Odds at Ends – The Pew and Battleground Polls, with a Gallup Chaser

I’ve laid out a pretty harsh accusation against the polls this year, by claiming that all the major polls are far from accurate. The cause of this, in essence, has been that the polls made some key assumptions about turnout, the independents, and the undecided voters. Assumptions which they never tested, and now are finding cannot be trusted. Poll results vary wildly from one another, and not just at different times. The variance for current polls listed at Real Clear Politics for this morning ranges from the Pew poll which advertises a 14-point lead for Obama, to the Battleground poll which says the lead is only 2 points. The variance is too great (and there are polls relatively close to both ends, demonstrating proof of statistical invalidity for the published confidence level) for even the casual observer to accept as a reasonable. There are four polls which show a 10 point lead or greater for Obama, and another five which show a 6 point lead or less. It is mathematically impossible for so many polls to be valid, yet disagree to such a degree with valid methodology. I said this when McCain was ahead, again when Obama climbed in front, and I am repeating it yet again. The starting point to discussing the polls this year, is understanding that the methodology in common use is flawed, and is producing results which cannot be depended upon.

A quick word here about validity. Opinion polling relies on statistical math, which depends on certain key tests. When a group of respondents exceeds a certain size, a pattern of responses emerges which is generally symmetrical, with few outliers. A p-test can be done to confirm that the results are consistent with the requisite conditions. This produces what is known as a confidence level, which in common words means the likelihood that the process, if repeated using samples from the same data pool and using the same method, will result in the same conclusion. This is called reproducibility, and it is the most signifcant test for human behavior testing. The confidence level basically sets out how often the same results should be expected to repeat. The most common published level of confidence in opinion polling is 95%, which predicts that the same method used at the same time will produce results within the published margin of error no less than 19 out of 20 times. RCP is listing twelve major polls with variances from each other which cannot be covered by the MOE, which proves the model is invalid by its own definition. One poll as an outlier could be explained, but the range is too great to explain the variance between the rest.

People have specifically asked me about the Pew and Battleground polls, since their twelve-point variance is the largest. I will have to say that in both cases, the error has been the same – disregarding historical norms in favor of introducing subjectively chosen demographics, things like over-sampling urban areas, younger voters, and democrats which creates a false image relative to the voting demographic. But to get a sense of the numbers, I would like to examine the Pew, Battleground, and Gallup polls in the context of their direct movement, and in reweighting their party affiliation to historic norms.

First, here are the recent results from Pew:

Sept 17 – 46% Obama, 44% McCain, 10% undecided
Sept 30 – 49% Obama, 42% McCain, 9% undecided
Oct 13 – 50% Obama, 40% McCain, 10% undecided
Oct 20 – 52% Obama, 38% McCain, 10% undecided

Pew is showing what is effectively a zero-sum game, with Obama gaining directly at McCain’s expense, with around 10 percent remaining unsure each time.

Now, the Battleground trend:

Sept 25 – 45% Obama, 47% McCain, 8% undecided
Oct 3 – 45% Obama, 41% McCain, 14% undecided
Oct 9 – 48% Obama, 38% McCain, 14% undecided
Oct 16 – 47% Obama, 40% McCain, 13% undecided
Oct 22 – 49% Obama, 47% McCain, 4% undecided

Battleground shows support movement more independent between the two candidates, and the most recent undecided numbers are much lower than what we saw before.


OK, with that in mind, let’s check the Gallup numbers for the same range of dates:

Gallup Daily Tracking
Sept 14: Obama 47%, McCain 45%, 8% undecided
Sept 21: Obama 48%, McCain 44%, 8% undecided
Sept 28: Obama 50%. McCain 42%, 8% undecided
Oct 5: Obama 50%, McCain 42%, 8% undecided
Oct 12: Obama 51%, McCain 41%, 8% undecided
Oct 19: Obama 52%, McCain 41%, 7% undecided

This model appears to be similar to Pew’s, zero-sum balancing with a constant undcided portion.

Gallup ‘expanded voter’
Oct 8: Obama 52%, McCain 43%, 5% undecided
Oct 15: Obama 51%, McCain 45%, 4% undecided
Oct 21: Obama 52%, McCain 42%, 6% undecided

This model allows a smaller undecided portion, suggesting that undecideds are pressed for a clear decision. Also, Gallup has admitted that this model has no precedent, and uses over-samples of urban and youth voters, in the presumption that they will sharply increase participation this year.


Gallup ‘traditional’
Oct 8: Obama 50%, McCain 45%, 5% undecided
Oct 15: Obama 49%, McCain 47%, 4% undecided
Oct 21: Obama 51%, McCain 44%, 5% undecided

This model removes the urban and youth voter overweights, but otherwise is the same as the ‘expanded voter’ model. This is because Gallup abandoned the true historical model, and so can only attempt to recreate it to some degree by using data sets it knows have been corrupted by invalid methodology.

Now, let’s see what happens to these results when the internal data is reweighted to historical party affiliation norms:

Pew:
Sept 17: 46-44 Obama becomes 45-46 McCain
Sept 30: 49-42 Obama becomes 48-44 Obama
Oct 13: 50-40 Obama becomes 48-43 Obama
Oct 20: 52-38 Obama becomes 50-42 Obama

Bear in mind that this accepts Pew’s polling methodology, which may have over-sampled other demographics besides just democrats. For example, in the Oct 20 poll Pew undersamples seniors and oversamples the 50-64 age group, oversamples high school only education by a large amount, and fails to note regional breakdowns or the urban/suburban/rural split. These are critical points which Pew fails to address, and which hshould make the reader wary.


Battleground:
Sept 25: 45-47 McCain becomes 44-47 McCain
Oct 3: 45-41 Obama becomes 46-44 Obama
Oct 9: 48-38 Obama becomes 48-41 Obama
Oct 16: 47-40 Obama becomes 49-42 Obama
Oct 22: 49-47 Obama (no internal data to reweight yet)

Battleground does not change much, but this reinforces that something odd has happened in the recent poll, most likely among independents.


Next to check is the reweight of Gallup’s polling:

Gallup Daily Tracking (d=daily, e=expanded, t=traditional)
Sept 14: 47-45 Obama (d) becomes 43-44 McCain
Sept 21: 48-44 Obama (d) becomes 43-42 Obama
Sept 28: 50-42 Obama (d) becomes 43-42 Obama
Oct 5: 50-42 Obama (d) becomes 44-41 Obama
Oct 12: 51-41 (d), 53-43 (e), 51-44 (t) Obama becomes 47-39 Obama
Oct 19: 52-41 (d), 52-42 (e), 51-44 (t) Obama becomes 45-42 Obama

Note that the reweights of these polls using historical norms are much more consistent with each other.

The next thing I suggest is looking at comparable metrics. First, base party support:

Support for Obama by Democrats
Pew: 87% 92% 91% 91%
Battleground: 82% 81% 83% 86%
Gallup: 86% 86% 86% 87% 88%

Support for McCain by Republicans
Pew: 90% 86% 91% 89%
Battleground: 85% 82% 80% 81%
Gallup: 84% 84% 82% 82% 84%

Note that Pew reports the highest support within each party. Note also that McCain’s support by republicans is reported to be dropping.

Next, independent support:

Independents

Pew: (for Obama) 38% 38% 45% 51%
Pew: (for McCain) 45% 46% 37% 33%

Battleground: (for Obama) 32% 38% 41% 43%
Battleground: (for McCain) 40% 36% 34% 33%

Gallup: (for Obama) 22% 22% 23% 33% 27%
Gallup: (for McCain) 31% 31% 32% 25% 34%

Note that Pew reports a commanding lead for Obama among independents, while Gallup shows McCain in consistent advantage.


Undecideds

Pew: 08% 07% 08% 07%
Battleground: 15% 15% 15% 14%
Gallup: 16% 16% 15% 14% 14%

Battleground and Gallup agree that there is a lot of the population still waiting to be won over. The election is certainly well within any reasonable boundaries of doubt.


Conclusion

It’s difficult to work with limited internal data, especially when the polling group has altered more than one category of demographic. But it is interesting to note that when the reported data is reweighted to a consistent historical norm, that even the varied results from three different polling groups start to trend in the same way. Obama supporters can take comfort from the indications that his lead stands up to inspection, albeit not as large, while McCain supporters can take heart in the evidence that the election is not at all decided, that the level of turnout, the choice of the independents, and which way the undecideds break (including the choice to stay home) is a vital part of the decision still to be resolved.


Pew’s data comes from here

Battleground data comes from here

Gallup’s polling reports come from here

Gallup’s internal affiliation numbers are reported here