The Associated Press, never known in recent years for objective reporting, has released a poll claiming that nearly two out of three registered voters polled plan to put the Democrats in charge of Congress this fall. This is a rather bold claim, especially given the events of the spring and early summer so far, so I decided to take a look at the poll to see what is going on here. What I found is a lesson as much in assumption as in analysis.
First, the assumptions to correct: A lot of people trust large institutions in the media. As we know from Rathergate, a large corporation may well have unannounced agendas and a permanent bias towards a certain viewpoint. But also, it should be understood that even a biased institution can be useful in determining the public mood. I generally appreciate the CBS/New York Times polls, for example, even though I dispute their conclusions, because they practice a consistent methodology and present internal data for independent review, that is, we bloggers can take apart the poll to decide for ourselves whether it says what they claim. The AP-Ipsos poll is like that. The AP-Ipsos poll is taken roughly once a month, and while selected questions change from one poll to another, certain basic questions are always asked, and the methodology is consistent, which allows me to test their claims using their own data.
I would like to begin, therefore, but noting the stated results and compare it to prior polls by the same group for trends. The poll I am discussing here is Project #81-5139-99, released July 13 2006, for interview dates July 10-12, for one thousand adults, of which 789 reported they were registered to vote. In other words, a slightly less than desired number of registered voters (a thousand is pretty much the starting point for confidence), but in this midterm season not unusual for polling, and the poll was taken over three days, Monday through Wednesday, which is an important detail when we discuss the demographics later. Keep it in mind. But for now, let’s look at the key questions; President Bush’s Job Approval numbers, Job Approval numbers for Congress, the highlighted question of whom voters want to run Congress, and how sure those voters are in that decision.
First, the President’s Job Approval. One thing which I found interesting, was that while the AP touted Bush’s JA as hanging around 36%, they emphasized that it was not moving up very much. Actually, looking at the last year, the AP never showed much movement for Bush. In the past year, the “Strongly Approved” number, which AP marks at 18%, has never been below 15% or above 23%. Adding the “Somewhat Approve” numbers, Bush is at 27%, in a range during the last year between 27% and 33%. Adding in the “Lean Towards Approval” numbers, which brings us to the present 36% from the AP, we see a range in the past year of 32% to 42%. The reason I make a point of this range, is that we see the numbers in a better context. The AP poll has generally held Bush’s JA below that reported by other polls, and so the best measure is not to believe that President Bush is really at 36% approval, but to recognize that his present numbers are not significantly different from the past year, a year which has seen the appointment of two Supreme Court Justices, a solid year of GDP growth, home ownership and job growth, and signal successes in Iraq, including two elections, a constitution and the death or capture of key terrorist leaders. It is not reasonable to not see a bounce from these events. What this means should not be presumed simply from this question, but held to make a total picture.
A key comparison is the Approval for Congress. For example, where President Bush manages 18% “Strong Approval”, the AP reports only a 5% “Strong Approval” for Congress, regardless of party. Like the President’s numbers, the Approval numbers for Congress over the past year have not changed significantly, showing only mild movements in either direction. Like the President, the movement over the past year has been subtle and always within a certain range, which indicates a stable opinion more than a radical change.
With those two points in mind, we move now to the question which the AP touted in its release; the question about who will hold control of Congress this fall. From the headline, it’s no surprise that the AP announced the Democrats are in much better shape for the election than the Republicans. But let’s look closer at that question, and the components of the response.
Question 1a of the ‘Congressional Vote Study’ reads: “If the election for the U.S. House of Representatives were held today, would you vote for the Democratic or Republican candidate in your congressional district?” Well, that sounds reasonable enough, but as Betsy Newmark observed, there is a clear distinction between the results of a generic question by party, and what happens when real names are added.
Basically, this means either that the generic question fails to measure the impact of the incumbent, or else that the specific races are considered in a much different perspective than a general opinion of the party. Historically, this is no shock; many incumbents go through controversy, even scandal, yet are re-elected over and over again. Also, I would remind the reader that we must keep the demographics of the respondent pool in mind, so this question must be held until we have enough information to establish the veracity of the reported results. In this poll, the AP reported a 51-40 advantage for the Democrats, or an eleven-point margin. What else was interesting, before we move to the internals, is the response to CVS 1b, which asked how strongly the respondent felt about his support. You have to take apart how they phrased it, because the AP took the whole response, Democrat or Republican, then stated strength of support as a percentage of the whole responding a preference for a party, rather than in the original percentage of overall respondents. So, I backed out the numbers and reapplied them as percentages of the overall respondents, which produced this breakdown:
21% Sure to Vote Democrat
15% Sure to Vote Republican
15% Say Democrat But Could Change Mind
14% Probably will Vote Democrat
12% Say Republican But Could Change Mind
11% Probably will Vote Republican
4% Undecided
3% Chose Democrat or Republican but when asked how strong, said Not Sure
3% Support candidate besides Republican or Democrat
2% Support no candidate or will not vote
When seen in this breakdown, it becomes apparent that the responses in each section are much closer than the overall report from the AP, and should be considered by the smaller response pool to be well within the margin of error. The Democrats have an apparent lead, but it is smaller and much less certain. And we have not yet gotten to the respondent demographics.
At first glance, the AP-Ipsos poll appears to only slightly biased towards the Democrats. On page 9, the ‘Party Identification’ claims 28% Republican to 33% Democrat, with 27% Independent and 12% “None of these”. First off, let’s catch that semantics trick. For purposes of a poll about which major party should control Congress, “Independent” is the same as “None of these”, meaning an overweight of non-party identifiers, at 39%. A look at the last two elections for a base reference shows a close parity between the three groups of Republican/Democrat/Other, so that the Republican representation is clearly under-represented and the no-party-identifier is clearly over-represented. This is similar to what is done at the CBS News/NY Times poll and at biased college polls like Quinnipiac; the Census was six years ago, yet the polls refuse to adjust their demographics to match known facts. The excuse most commonly made is that political identification is volatile or an unknown, but again the Census has traditionally been used for this measure for many years; rejecting an inconvenient trend is simply unreasonable. We already know from the results of the last several elections that the increase in Republican support is real, yet many of the major polls resist this fact. So it is important to keep this bias in mind when evaluating results.
But there’s more. A breakdown in the ‘Party Identification’ results by AP shows that heaviest number consider themselves “Moderate”, word which can mean different things in different places. For instance, a “moderate” Republican could mean someone who supports the President in the Iraq War, but not on Abortion, or vice versa. The distinction about what, specifically, is liked or disliked about a party can make all the difference in an actual election.
Next, we come back to the fact that this poll was not taken over the weekend, when the most regular people are available, but during the week when most regular people are at work. That, right there, calls into question the respondent pool. What’s more, when you look at the Demographics section beginning on page 10, you see that the heaviest section by age was from the 18-34 age group, which leans liberal and is the least likely age demographic to actually vote. The poll also noted that 15% of respondents admitted they were unemployed, more than three times the actual unemployment rate for the nation as a whole, again a signal anomaly. Only 35% of the respondents have completed a college degree, which again calls into question the balance of the pool in reflecting the nation’s voting public. And 62% of the respondents do not have children, which is a critical difference from the nation as a whole.
I found it interesting, that the AP claimed a large percentage of Republicans might vote for the Democrat, and so I took a close look at the Regional breakdown. Again, you have to pay close attention – the poll broke down into Northeast, Midwest, South, and West, but failed to note, for example, whether ‘South’ included the whole range of states or just places like Atlanta and Miami, which are distinctly different from, say, Biloxi and Tulsa, or whether ‘West’ meant the whole West, or just focused on California and the Pacific Coast. The vagueness of the terms is suspicious, especially when one notes that only 19% of the respondents were from “rural” locations. So the AP stayed within the lines, but drawing very careful lines indeed.
So, in summary, what to make of the claim that the House of Representatives is “likely” to fall into the hands of the Democrats? I don’t buy it. Consider what we know:
1. The AP poll is based on demographics known to be at significant variance with the 2000 U.S. Census and the last two national election demographics.
2. The poll created a separate category of party descriptor to hide the overcount of no-party identifiers
3. The demographics of this poll weighed heavily on the responses of classes of the population which are historically unlikely to vote
4. 21% of the responses regarding the election were included from people who admit they are not even registered to vote, yet the poll did not list the results from only the registered voters as a comparison
5. The numbers for approval do not appear to respond significantly to even major events of political interest, in either direction, which raises the question of how interested in politics the respondent poll really is.
6. The poll only touches generic party preferences, which in the past have often proved to be unreliable as a predictor of specific races
I leave to the reader, to consider the balance of reporting, and any embedded bias, from the political reporting by the Associated Press in recent years, but recommend its addition to weighing the reported results of their poll. I commend the AP’s consistency and transparency in reporting the results of this poll, but cannot agree with the claims made in the release.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment