State Poll Accuracy

I have spent a great deal of time analyzing faulty procedures in the national political polls for the presidential race this year. I have noted proofs of invalidity and the significance of party affiliation weighting in the published claims of such polls. But a number of people have observed that Barack Obama is also leading in a number of state polls, and they quite reasonably ask if those polls are valid or not. The chief difficulties in the state polls, are the facts that there are so many polling groups performing state polling (Real Clear Politics has reports from 49 different polling groups on state polls for the presidential race), there are different demographic norms for each state, and whatever is skewing the national polls is likely to be influencing reported state results as well. But while the past does not guarantee the future, I did find some interesting points in the 2004 state polls.

Not every polling group which is publishing polls this year also polled and published in 2004. And many of the polling groups are limited to only a few states, or just one. The Field Poll, for example, is only concerned with California. So I limited my analysis to nine major polling groups which conducted state polls in a large number of states in 2004. I was looking for polls which released polls just prior to the election, and which also released polls during the first two weeks of October of 2004. The results of that examination are presented here. I hope you will find them illuminating.

Survey USA

Survey USA conducted polls in 30 states just before the 2004 election. Of those 30 polls, in 8 state polls the published support for Bush was off by 3 points or more, as many as 6 points wrong. In 5 states Bush’s support was over-estimated, in 23 states his support was under-estimated. In 5 state polls the published support for Kerry was off by 3 points or more, as many as 10 points wrong. In 10 states Kerry’s support was over-estimated, in 19 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 14 times, off by 5 points or more in 6 states.

Survey USA conducted polls in 30 states during the first two weeks of October 2004. of those 30 polls, in 14 states support for Bush was off by 3 points or more, as many as 6 points wrong. In 3 states Bush’s support was over-estimated, in 23 states his support was under-estimated, In 11 state polls the published support for Kerry was off by 3 points or more, as many as 11 points wrong. In 6 states Kerry’s support was over-estimated, in 19 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 14 times, off by 5 points or more in 7 states.


American Research Group (ARG)

ARG conducted polls in 7 states just before the 2004 election. Of those 7 polls, in 3 state polls the published support for Bush was off by 3 points or more, as many as 4 points wrong. In 0 states Bush’s support was over-estimated, in 6 states his support was under-estimated. In 1 state poll the published support for Kerry was off by 3 points or more, as many as 3 points wrong. In 1 state Kerry’s support was over-estimated, in 5 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 2 times, off by 5 points or more in 1 state.

ARG conducted polls in 49 states during the first two weeks of October 2004. of those 49 polls, in 40 states support for Bush was off by 3 points or more, as many as 11 points wrong. In 2 states Bush’s support was over-estimated, in 45 states his support was under-estimated, In 23 state polls the published support for Kerry was off by 3 points or more, as many as 9 points wrong. In 8 states Kerry’s support was over-estimated, in 32 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 28 times, off by 5 points or more in 24 states.


Rasmussen

Rasmussen conducted polls in 11 states just before the 2004 election. Of those 11 polls, in 3 state polls the published support for Bush was off by 3 points or more, as many as 5 points wrong. In 0 states Bush’s support was over-estimated, in all 11 states his support was under-estimated. In 3 state polls the published support for Kerry was off by 3 points or more, as many as 5 points wrong. In 1 state Kerry’s support was over-estimated, in 8 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 3 times, off by 5 points or more in 2 states.

Rasmussen conducted polls in 31 states during the first two weeks of October 2004. of those 31 polls, in 19 states support for Bush was off by 3 points or more, as many as 10 points wrong. In 2 states Bush’s support was over-estimated, in 29 states his support was under-estimated, In 13 state polls the published support for Kerry was off by 3 points or more, as many as 10 points wrong. In 5 states Kerry’s support was over-estimated, in 23 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 12 times, off by 5 points or more in 9 states.


Mason-Dixon

Mason-Dixon conducted polls in 21 states just before the 2004 election. Of those 21 polls, in 11 state polls the published support for Bush was off by 3 points or more, as many as 7 points wrong. In 0 states Bush’s support was over-estimated, in all 21 states his support was under-estimated. In 14 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 1 state Kerry’s support was over-estimated, in 20 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 6 times, off by 5 points or more in 2 states.

Mason-Dixon conducted polls in 19 states during the first two weeks of October 2004. of those 19 polls, in 11 states support for Bush was off by 3 points or more, as many as 7 points wrong. In 1 state Bush’s support was over-estimated, in 18 states his support was under-estimated, In 15 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 2 states Kerry’s support was over-estimated, in 17 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 8 times, off by 5 points or more in 4 states.


Zogby

Zogby conducted polls in 13 states just before the 2004 election. Of those 13 polls, in 9 state polls the published support for Bush was off by 3 points or more, as many as 6 points wrong. In 0 states Bush’s support was over-estimated, in all 13 states his support was under-estimated. In 3 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 6 states Kerry’s support was over-estimated, in 4 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 10 times, off by 5 points or more in 4 states.

Zogby conducted polls in 15 states during the first two weeks of October 2004. of those 15 polls, in 12 states support for Bush was off by 3 points or more, as many as 8 points wrong. In 1 state Bush’s support was over-estimated, in 14 states his support was under-estimated, In 5 state polls the published support for Kerry was off by 3 points or more, as many as 11 points wrong. In 5 states Kerry’s support was over-estimated, in 7 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 10 times, off by 5 points or more in 8 states.


CNN

CNN conducted polls in 6 states just before the 2004 election. Of those 6 polls, in 4 state polls the published support for Bush was off by 3 points or more, as many as 5 points wrong. In 2 states Bush’s support was over-estimated, in 4 states his support was under-estimated. In 4 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 3 states Kerry’s support was over-estimated, in 3 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 5 times, off by 5 points or more in 5 states.

CNN conducted polls in 11 states during the first two weeks of October 2004. of those 11 polls, in 4 states support for Bush was off by 3 points or more, as many as 4 points wrong. In 2 states Bush’s support was over-estimated, in 7 states his support was under-estimated, In 6 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 2 states Kerry’s support was over-estimated, in 9 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 9 times, off by 5 points or more in 3 states.


Research 2000

Research 2000 conducted polls in 9 states just before the 2004 election. Of those 9 polls, in 5 state polls the published support for Bush was off by 3 points or more, as many as 5 points wrong. In 0 states Bush’s support was over-estimated, in all 9 states his support was under-estimated. In 0 state polls the published support for Kerry was off by 3 points or more. In 2 states Kerry’s support was over-estimated, in 7 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 4 times, off by 5 points or more in 2 states.

Research 2000 conducted polls in 12 states during the first two weeks of October 2004. of those 12 polls, in 11 states support for Bush was off by 3 points or more, as many as 7 points wrong. In 1 state Bush’s support was over-estimated, in 11 states his support was under-estimated, In 5 state polls the published support for Kerry was off by 3 points or more, as many as 6 points wrong. In 2 states Kerry’s support was over-estimated, in 8 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 6 times, off by 5 points or more in 5 states.


Quinnipiac

Quinnipiac conducted polls in 3 states just before the 2004 election. Of those 3 polls, in 1 state polls the published support for Bush was off by 3 points or more, as many as 3 points wrong. In 0 states Bush’s support was over-estimated, in all 3 states his support was under-estimated. In all 3 state polls the published support for Kerry was off by 3 points or more, as many as 5 points wrong. In 0 states Kerry’s support was over-estimated, in all 3 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 1 time.

Quinnipiac conducted polls in 5 states during the first two weeks of October 2004. of those 5 polls, in 0 states support for Bush was off by 3 points or more. In 1 state Bush’s support was over-estimated, in 2 states his support was under-estimated, In 3 state polls the published support for Kerry was off by 3 points or more, as many as 12 points wrong. In 0 states Kerry’s support was over-estimated, in all 5 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 3 times, off by 5 points or more in 1 state.


Fox News

Fox News conducted polls in 4 states just before the 2004 election. Of those 4 polls, in 1 state poll the published support for Bush was off by 3 points or more, as many as 8 points wrong. In 0 states Bush’s support was over-estimated, in all 4 states his support was under-estimated. In 2 state polls the published support for Kerry was off by 3 points or more, as many as 5 points wrong. In 1 state Kerry’s support was over-estimated, in 3 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 3 times, off by 5 points or more in 1 state.

Fox News conducted polls in 3 states during the first two weeks of October 2004. of those 3 polls, in 1 state support for Bush was off by 3 points or more, as many as 4 points wrong. In 0 states Bush’s support was over-estimated, in all 3 states his support was under-estimated, In all 3 state polls the published support for Kerry was off by 3 points or more, as many as 5 points wrong. In 0 states Kerry’s support was over-estimated, in all 3 states his support was under-estimated. The margin between the candidates was off from actual election results by 3 points or more 1 time.


SUMMARY

Even this brief examination reveals that the state opinion polls were often wrong, and often to a materially significant degree. In 2004, both Bush and Kerry's support were commonly under-estimated, but in several groups' work, all of Bush's support was underestimated and Kerry's support was much more likely to be over-estimated. The margin was also often wrong, and to a significant degree. State polls, therefore, are not proven to be a valid predictor of election results.


[ the dominant source for this examination was the archive of 2004 state polling results, provided by Real Clear Politics]