States and Shadow

Earlier this season, I wrote about the statistical effect of what I call “shadow”, the combination of a poll’s margin of error and the undecideds. In today’s article, I apply this again to the state polls and address the errors of aggregation and over-simplification.

The national polls are showing a tightening race in several polls, notably Gallup, Battleground, and the AP-Gfk poll. Of course, other polls are claiming a large lead for Obama, notably Pew and Newsweek. The state polls are also showing some movement, although its not as rapid for a number of reasons, not the least being that state polling is not done as regularly as national polling. Obama supporters have greatly enjoyed the RCP aggregate numbers for Obama, which they would, since the RCP aggregates at both the national and state level indicate Obama is winning easily over McCain. The problem, of course, comes when you start to look closely at the support for that belief.

If the methodology is sound, national and state polls should track in similar fashion. This does not mean that every state poll will reflect national support to the same degree, but if a national poll is done properly, it will include proportionate responses from every region of the country, ideally from every state, and so the national numbers will reflect the sum of the state supports. So, the tightening of the national race has to mean – assuming the polls are valid – that McCain is gaining support in some large states or in enough small to medium states to be reflected in national numbers. But as I said, major polling is done less often at the state level; most state-level polling is done less than once a month by polling agencies. Survey USA, for example, who has done more state polls than any other agency, has not done a state poll in the last two weeks in 35 states, and has not done a state poll in the last 10 days in 41 states. That’s important to keep in mind.

I need to address the problem of aggregation in polling now. Aggregates are popular because they are easy to read, and seem to be helpful in telling how much someone is ahead. After all, you don’t want to be fooled by paying attention to an outlier, and there is a sense that if most of the polls say the same thing, that’s most likely what’s really going on. The problem with that, is the assumption that all of the polls in an aggregate are valid, that all can be accepted with equal confidence. But that would be erroneous. First of all, not every polling group is really professional at what they do. Remember the disastrous early exit polling in 2004? In that case, a lot of brand new pollers were hired and hustled out without proper training, orientation, or supervision. Does anyone really think that was the only occasion where that happened? The fact is, a lot of polling errors get made without the public ever hearing about it, for a number of reasons, not the least being that if their results are what is expected, the error is not obvious. Also, even professional polling groups may look for different characteristics, such as polling adults, registered voters, people who have voted in recent elections for their ‘likely voter’ category, people who simply claim they are ‘likely’ to vote, and so on. Take a look at some of these state polls, and you will also find that it can be difficult to see how they arrived at their numbers; many simply do not provide access to the raw data or their internal demographics. As a result, a significant portion of the state polls are likely to be flawed in a functional manner, and aggregating such polls tends to magnify such errors, not eliminate them.

The next problem is over-simplification. This shows up most often in the way that polls are reported. Whether you like the results from a poll or not, it’s very important to understand that polls are sometimes just plain wrong, and even if a poll is valid, it’s only valid to the extent that it demonstrates a trend against its earlier report using consistent questions and methods, and polls have never predicted the surprise results, because they are modeled in a way which reflects the public’s assumptions far more often than the actual condition. Polls are opinion polls, after all, not predictors of future events. Polls only “predict” the results of an election to the degree that the voters behave in line with the poll’s assumptions.

So, with that said, I am addressing the state polls with respect to the statistical phenomenon of shadow. ‘Shadow’ is the total amount of uncertainty in a poll, the combination of the undecideds plus two times the published margin of error. For example, let’s say candidate A is leading candidate B in a poll, 51-44 with a published margin of error of 4%. Game over, it seems. But that 4% MOE means that either candidate could be as much as 4 points stronger or weaker, meaning its candidate A at 47 to 55, and candidate B at 40 to 48. Also, there are 5% undecideds in the poll, so while B looks to be out of it, it’s mathematically possible for the actual condition to be A 47, B 53. It could also end up being A 60, B 40, with the same level of probability as the other extreme. And of course, this does not consider the possibility of some voters changing their minds. I do not think that happens as wildly as the polling groups seem to claim, but it is a valid factor. Considering that, we can now examine the state polling condition.

I took the RCP aggregates (I know, I know, but I do not have the time or space to examine each and every state poll for validity, I don’t need to have anyone whining about ‘cherry picking’ polls, and I can make my point even by using the aggregate reports) and applied the percentages claimed to the 2004 voting results as a two-party vote split. If we count all of the states according to who leads according to the RCP aggregates, Barack Obama would take 50.2% of the popular vote to 43.0% for John McCain, and 364 electoral votes to 174. However, even using those aggregates, the numbers change considerably if we consider the effect of shadow. Applying the shadow rule (undecided plus double MOE), it becomes 200-118, Obama still in good shape but with 220 electoral votes still to be decided.

Before ending this article, I also looked at the trends and outliers in the polling I have seen, especially given certain key internals. I will not call it definitive, but in my opinion if the demographic weighting is corrected the popular vote becomes Obama 46.9%, McCain 46.6%, but with McCain taking the electoral vote 278-260. When the shadow effect is applied, the electoral numbers change to 147-71 McCain, with 320 to be decided. The message is clear then, that the race remains to be decided.