State Poll Accuracy 2008

The election of 2008 having been concluded for the main part, examination of the numbers is becoming possible. Some of the national polls were quite a bit off, but others seem to have been correct, at least in their final submission. My own opinions on the national polls have been stated and debated and mocked by some, so there is no purpose now to revisiting them. The last venue of interest for now with regard to polls is the state polls. It should be noted that except for Rasmussen, most of the groups doing state polling did not also do national polls. There are two ways to look at the accuracy of state polls; comparing specific polls to the results, and comparing various indicators to the election results. The one may be used to grade the success of various polling agencies, and I shall post on that method in the near future, but the other gives a look at the overall efficacy of state polling, especially that popular practice of aggregating results. With that in mind, here are the results for the salient indicators. I am noting the historical average since 1948, the results from 2004, the Real Clear Politics average of polls, the reweighted average using 2006 demographics, and the actual election results.

Compared to the historical average since 1948, Barack Obama collected less support in 7 states, the same in 1, and more in 42. John McCain collected less support in 26 states, the same in 3, and more in 21 states. On average Obama outperformed the historical average by 5.30 points, while McCain outperformed the historical average by 0.36 points. The reason both candidates can outperform the historical average is the significant participation of third-party candidates in historical elections. The historical average does not appear to be useful as a vector for future behavior. Shorter terms will be examined, but if one appears to match the results from 2008, that term would have to be tested against earlier elections to see if the trend was applicable outside the moment.

Compared to the 2004 election, Barack Obama collected less support than John Kerry in 3 states, the same in 4, and more in 43. John McCain collected less support than George W. Bush in 43 states, the same in 3, and more in 4 states. On average Obama outperformed Kerry’s 2004 results by 4.76 points, while McCain underperformed Bush’s 2004 results by 4.58 points.

Compared the the RCP average for each state, Barack Obama collected less support in 4 states, more in 43, and exactly what was called in 3 states. John McCain collected less support than the RCP average in 5 states, more in 40, and exactly what was called in 5 states. It should be noted that the RCP averages had a certain amount of undecided weight, which is one reason why 83 out of 100 calls were less than the amount received in the actual election. On average Obama outperformed predicted support by RCP by 3.02 points, while McCain outperformed RCP support by 2.80 points.

Compared to the reweighted average for party affiliation, Barack Obama received less support in 2 states, more in 48, and the reweights were exactly right in no states. John McCain collected less support than the reweighted average in 20 states, more in 23, and the reweights were exactlty right in 7 states. On average Obama outperformed the reweighted averages by 5.82 points, while McCain’s average was identical to the reweighted averages. The significance of this datum, especially with 7 exact calls using historical reweights, is that for the republican candidate the reweighting corrects the polls’ undercount of support, but at least in this election, the democratic party candidate’s support was better measured by the unadjusted polls.

More information is needed for a better analysis, but the preliminary indication from this review, is that rather than a ‘Bradley effect’ being in play, the polls at the state level may have understated Obama’s support.