CAUTION: Watching Out For Error In the Ruffini Straw Poll

||||||||||||||

I have said several times that I found the Presidential Candidate poll being run by Patrick Ruffini to be invalid. There are a number of reasons for this, and now that Mr. Ruffini has begun to announce “results” , it’s important to go into the matter in order to avoid mistakes.

People like polls. It’s the combination of hinting at the future, and giving people a say in the call. The media picked up on these points a long time ago, and that’s paid a lot of paychecks for Polling Groups. I have done autopsies on a number of polls since last year’s campaign, showing elements of concern about methodology, weighting, and the ubiquitous bias gremlin. It’s a serious issue for even the most professional pollster. I like Patrick Ruffini and find his thinking generally sound, but he’s allowed his enthusiasm for his own poll to overlook some mistakes, and I must warn the reader that those mistakes can invalidate a lot of the conclusions reached.

The first mistake is the biggest one. Ruffini calls his poll for Giuliani, but Rudy, as much as I like him, only pulls 30.0%. That’s not a win, folks. John Kerry cleared more than 30% last year, and Al Gore cleared over 30% of the Electoral Vote. While certain Aluminati may claim otherwise, neither “Magic Hat” John nor Mr. Internet has signed any bills into law. Giuliani took the most votes in an informal, non-binding, non-scientific poll, with no ramifications beyond providing points for discussion.

The poll itself should not be confused in the slightest with any sort of election or primary contest. The people who took part in the poll, for instance, did not have to register, nor show up at a polling place, and there were fewer security measures taken to prevent repeat votes than in downtown Chicago. Also, the slate of candidates in this poll were selected by a committee of one - Patrick Ruffini. None of the conditions unique to the early primary states was applied to establish a more realistic venue. In that light, Mr. Ruffini’s attempts to project how the slate would do in a real primary seems specious to me. For instance, the early primaries in Iowa and New Hampshire have a much larger slate of candidates than the later ones, but they do not generally serve well as indicators of the eventual winner. As for the later polls, candidates drop out for financial and political reasons, and the winner of these primaries generally has a much more focused message, and voters have a clearer choice among fewer (and distinct) options.

Patrick called his poll “statistically valid", simply because he got a lot of participation. Sorry Patrick, but that just isn’t so. For one thing, as I observed before, you did not consider the likelihood of multiple votes, or the poll’s inabiliy to catch cheaters. As an example, I could have voted from my home computer, or any of several computers at work, and you’d have no way of knowing it, much less correcting for it. And I am hardly unique in that capability.

But more to the point, Ruffini’s poll was not weighted. In polling history, one of the most famous blown calls was the 1936 Presidential Election, in which the Literary Digest, a prominent national magazine, called the race for Alf Landon in a blowout. With more than two million people taking part in the LD poll, it seemed like a strong indicator. In the actual election, however, FDR rolled along with ease. The mistakes made by the editors at LD came from failing to understand the demographic balance and address it. And Ruffini makes the same mistake. His poll made no attempt to compensate, for example, for gender or race. Also, there was no weighting for geographic location, a crucial error given the fact that people in different parts of the country naturally find different candidates and different qualities which appeal to them. Taken as a scientific poll, this doesn’t meet even minimal standards.

I also take issue with a trick played by Ruffini in the poll. Ruffini chose not to include several qualified candidates in the poll as main choices, but threw them up as “fantasy” candidates. This is significantly invalid as a polling practice, as it simultaneously denies those individuals an equal place on the ballot, but also draws a connection between the “serious” candidates and the “fantasy” candidates on no more than the whim of Mr. Ruffini. It shows a basic lack of understanding of poll method, and basely insults the candidates demoted to second-class status. Count them on the SAME ballot, Mr. Ruffini, and let the voters decide.

At best, this poll was meaningless, because of the myriad errors in its development. At worst, it was a cheap shot at qualified candidates. That’s too bad, because the poll could have been run to some good effect. If, for example, Ruffini had paid less attention to meaningless statistics, but asked his voters to detail what made their candidate qualified, he could have helped draw out what voters are looking for in the next President. By asking them to specify what worried them about candidates they didn’t choose, Ruffini could have helped these candidates address those concerns early on. It’s just too bad that Ruffini chose a poll which has no effective meaning for the campaigns ahead, and that a false image was created this early. After all, at one time when people didn’t look too deep, a lot of people thought Howard Dean looked Presidential.