Cooking Polls – State Poll Edition

My articles on polling have received a great deal of discussion. Most of it has been emotional in focus rather than factual, and I am amused by those blogs which reference the articles but ignore the context and substance in order to attack something I never even said. There have been a few reasonable questions, however, and one of them addresses the state polls. The national polls are all over the place, but what about the state polls? Don't they show Obama leading in most states, and don't the state polls basically agree? Those questions are good ones, and so state polling is the focus of today's article.

The first thing that jumps out at you if you read the state polls, is that there are a lot more polling groups doing polls at the state level, than at the national level. Also, most of the polling groups which do national polls, do not also do state polling, probably because it is expensive and difficult to try to cover all of the states on a consistent and timely basis. I have written before that national polls often focus on urban centers, which means that many of the states would require a functionally different methodology to work than what is used nationally. State polling tends to be smaller in respondent pool size, smaller in budget, and less frequent. Some polling groups only do one poll for the whole campaign, and it's common for even major groups to do a poll only once a month.

So anyway, I'm looking at the state polls and I notice that there's there's quite a range of opinion there, just like the national polls. California, for example, is all about Obama, but it ranges from +24 down to +16, which is statistically significant. No, it hardly means Cali is in play, but that degree of variance in a deep blue state indicates on the state level much of what I have been noting on the national level. Moving ahead alphabetically, Colorado looks pretty stable, but on the other hand RCP only shows a single poll done there all month. Something to think about, that. Even the Obama people admit Florida is hard to call, polls there taken in October show anything from Obama +8 to McCain +5. Indiana is just as weird, running from Obama +10 to McCain +7 in October polls. Iowa is like Colorado, an important state but with only two polls done there this month. Minnesota is strange as well, with a range of 18 points between reports from the eleven polls done there in October. Looking at Missouri, there have been nine polls done and they range from Obama +8 to McCain +3. Even in safe states there's some hinkiness, as New Jersey has a range of 15 points between polls taken in October. Like California, it's not in doubt but the volatile range of results is sending a signal about the polls' validity, just like the national polls. I don't think I need to go through all of the states to show what I'm saying here, go check out RCP and drill down to specific polls on specific states. The state polls are showing the same volatility that I noted in the national polls, and there's likely a common reason for it.

A reader mentioned Survey USA earlier this week, and I'd like to use them as an example of what I mean. First off, I like Survey USA for making internal data available; it really helps me take apart their process to see what they were thinking. And I found an interesting trend, something which is consistent with the national polls and which explains both the volatility and the invalidity of the current model.

2006 was a bad year for republicans, a year when republicans stayed home and democrats used the opportunity to win a number of close races and take over control of the House and Senate. In a number of states, therefore, it's not surprising that democratic party supporters gained a few points (usually 1 to 3 points) relative to 2004 in voter participation. So I went back and looked at voters by party affiliation, and compared those balances to this year's weighting by Survey USA. In thirty-six states, the party affiliation weights for democrats used by SUSA was five points or more higher than in 2006, a high-water mark for democrats. In twenty states, the party afiiliation weights for democrats used by SUSA was ten points or more higher than in 2006, and in eight states, the party affiliation weights used for democrats by SUSA was thirteen points or more higher than in 2006. Significant battleground states affected by this bias are as follows:

Pennsylvania: D+5 in 2006, SUSA using D+19, 15 point variance
Indiana: R+14 in 2006, SUSA using R+1, 13 point variance
Nevada: R+7 in 2006, SUSA using D+6, 13 point variance
Colorado: R+3 in 2006, SUSA using D+9, 12 point variance
Iowa: R+2 in 2006, SUSA using D+10, 12 point variance
Virginia: R+3 in 2006, SUSA using D+9, 12 point variance
Ohio: D+3 in 2006, SUSA using D+13, 10 point variance
Missouri: R+1 in 2006, SUSA using D+7, 8 point variance
North Carolina: R+1 in 2006, SUSA using D+5, 6 point variance

I've looked at the publicly available records on historical election participation, 2008 new voter registrations, and the Census information on these states, but I can find no valid reason for such large and arbitrary changes in political affiliation weightings. I would therefore submit that the models being used for many of the state polls have design flaws, which threaten the credibility of their published results.