Almost any political debate has an element of “dueling citations”, where each party will come up with a brace of “recognized experts” – each with their own analysis of the matter at hand. And those experts will generally spout the results of some survey, or collection of surveys, in an effort to give their argument some gravitas.

However, on matters of public policy, surveys of the general public will frequently diverge widely in their results. And particularly partisan experts can then pick and choose those surveys which tend to reinforce their preconceived opinions. This is nowhere more true than on matters concerning Iraq.

It then becomes necessary for the debater to go to the rigor of critiquing the surveys themselves. Well, if the expert even cites what survey he/she is relying upon for the information (frequently not the case in brief op-ed pieces), the debater is lucky if the information source even makes details of their surveys available to those other than paid subscribers to their service. Then there’s the matter of actually dissembling the raw data and techniques – a bit of real drudgery, even for those of us with the skills to do it.

Well, for the past year or so, Mark Blumenthal, an opinion poll wonk, has been putting out a blog, Mystery Pollster, which gets to the bottom of these things for us. I, for one, couldn’t be more happy about it. In a post from a few days ago, he goes after the recent RT Strategies poll of public opinion relative to the Iraq War. This is quite extended, for a blog post, as Mark goes into excruciating scope and detail. But he writes at a level accessible to the average lay political blog reader.

I’ll just excerpt a particularly “meaty” piece here:

When pollsters move beyond general ratings to more specific questions about policy – as we do in almost every public political poll – we move to shakier ground.  Here Americans often lack preexisting attitudes, yet most will work to answer our questions, often forming opinions on the spot based on the text of the question.  When that happens, responses can be very erratic and contradictory across polls.  Very small variations in wording, the number of answer choices offered or the order of the questions can result in big and often surprising differences in the results. 

With that in mind, consider the three RT Strategies questions: 

Thinking about the war in Iraq, when Democratic Senators criticize the President’s policy on the war in Iraq, do you believe it HELPS the morale of our troops in Iraq or HURTS the morale of our troops in Iraq? (IF HELPS/HURTS, ASK:) And do you believe it (HURTS/HELPS) morale A LOT or just SOME

44% hurts a lot
26% hurts some
6% helps some
7% helps a lot
17% not sure

When Democrats criticize the President’s policy in Iraq, do you believe they are (ROTATE) Criticizing the President’s policy because they believe their criticisms will help the United States’ efforts in Iraq, OR, Criticizing the President’s policy to gain a partisan political advantage?

31% believe will help
51% to gain advantage
6% some of both (volunteered)
6% neither (volunteered)
7% not sure

And thinking about the future of our policies in Iraq, do you believe the U.S. military should…. (ROTATE FIRST TWO, ALWAYS ASK "Set a fixed timetable" last) Withdraw our troops immediately, regardless of the impact OR Withdraw our troops as the Iraqi government and military meet specific goals and objectives OR Set a fixed publicly available timetable for withdrawal.

16% withdraw immediately
49% withdraw when goals met
30% set fixed timetable
3% none (volunteered)
2% not sure

A few reactions:  First, all three of these questions fall into that second category of issues about which many Americans appear to lack preexisting attitudes.  Non-attitudes are most evident on the morale question (something that Armando at DailyKos picked up on).   The telltale clue is that 17% were completely unable to answer the question, a sure sign that many more came up with an answer on the spot.   The fact that nearly a third chose one of the softer "some" categories (26% hurts morale "some," 6% helps "some") is consistent with that argument.  Also consider the respondent who believes such criticism neither hurts nor helps troop morale, but does not realize that "neither" is an o[o[p]tion.  Odds are good they will end up in the "hurts a little" category. 

On the partisan advantage question, nearly one in five respondents (19%) could not choose between the two offered answer categories.  Finally, for reasons that I’ll discuss below, I’d argue that the large number of respondents in the middle category of the future policy question (49%) suggests that it was an attractive choice for those respondents who were simply not sure how to answer.   

Now MP is not averse to survey questions that offer new information and push respondents a bit to see where they might stand in debates they have not followed closely.  And in this case, the results of the RT "morale" and "criticism" questions are more or less consistent with the similar questions asked elsewhere.   For example, a Fox News poll in early November found that 58% of Americans agree that those "who describe U.S. military action in Iraq as a mistake" are "hurting U.S. troops."  Only 16% believed they were "helping."  The rest had mixed opinions (9%), believed the criticism had no effect (9%) or could not answer the question (8%). 

It is also worth noting that Americans tend to dismiss much of the debate in Washington as attempts to gain "partisan advantage," so the results of the RT question are not particularly surprising.  For example, back in September (9/8-11), Gallup asked about politics in the context of Hurricane Katrina:

"Do you think Democrats who criticize the way the Bush Administration has handled the hurricane response mainly want to find out what went wrong, or mainly want to use the issue for political advantage?"
36 find out, 60% use for advantage, 4% unsure

Seven years ago, ABC News and the Washington Post asked a similar question about the impeachment of President Clinton with nearly identical overall results:

"Do you think the House voted to impeach Clinton on the basis of the facts of the case, or on the basis of partisan politics?"
36% facts of the case, 61% Partisan politics, 3% no opinion

Questions that push respondents to consider questions for which they do not have pre-existing opinions do have a role in opinion research (one that should not be labeled as a fraudulent "push poll" — but that’s another subject for another day).  However, in those instances pollsters need to take care to provide respondents with new information in a way that does not bias subsequent questions.  For that reason, I am a bit surprised that RT Strategies asked two questions that mirrored the Bush administration talking points just before asking respondents their preference about prospective Iraq policy.  Would the responses to the third question have been different if they followed a question about say, whether Bush "intentionally misled the American people about the presence of weapons?"  We will never know, but it certainly seems possible that they would. 

To be fair, Gallup asked a very similar question a few weeks ago (11/11-13) with similar results:

"Here are four different plans the U.S. could follow in dealing with the war in Iraq. Which ONE do you prefer? Withdraw all troops from Iraq immediately. Withdraw all troops by November 2006 — that is, in 12 months’ time. Withdraw troops, but take as many years to do this as are needed to turn control over to the Iraqis. OR, Send more troops to Iraq." 
19% withdraw now, 33% withdraw within 12 months, 38% take as long as needed, 7% send more troops, 3% unsure. 

Note that Gallup showed 19% ready to withdraw immediately; RT Strategies show 16%.  Gallup shows 52% supporting withdrawal either immediately or within 12 months, RT shows 46% support withdrawal either immediately or on a fixed timetable. 

Having said this, I want to caution readers against taking these these prospective policy questions  at face value.  I also tend to agree with those who argue that the questions on the RT poll are, in essense, the wrong questions, that other measures give a better sense of true, pre-existing opinions on the Iraq War.  This is not necessarily a criticism of Riehle and Tarrance, merely a caution that focusing on these three questions alone can give a misleading impression.  For example, review the questions asked since Labor Day as posted by the Polling Report and you will find some highly consistent results: 

• Approval of Bush’s handling of the war in Iraq varied between 32% and 36%, with disapproval between 62% and 65%, as measured by six different pollsters.

• Differently worded questions about the worthiness of the war (asked by Gallup, CBS, ABC/Washington Post and NBC/Wall Street Journal) found between 31% and 40% that found the war worth the cost and between 52% and 60% that said it was not. 

• Differently worded questions about whether the decision to go to war was right or wrong (asked by Gallup, CBS and the Pew Research Center), found 42% to 45% who say the US made the right decision in going to war, between 50% and 54% who say we made the wrong decision or should have stayed out. 

However, look at the range of questions about prospective policy and the results are all over map.  Here is a sampling (full details at the Polling Report):

CNN/USA Today/Gallup  (11/30):  "If you had to choose, which do you think is the better approach for deciding when the U.S. should withdraw its troops from Iraq: to withdraw U.S. troops only when certain goals are met, or to withdraw U.S. troops by a specific date and stick to that time-table, regardless of conditions in Iraq at that time?"
59% when goals are met, 35% by a specific date, 6% unsure

FOX News/Opinion Dynamics (11/ 29-30):  "Do you think there should be a publicly announced timetable for withdrawing U.S. troops from Iraq?" 47% should, 41% should not, 12% unsure

Harris (11/8-13):  "Do you favor keeping a large number of U.S. troops in Iraq until there is a stable government there OR bringing most of our troops home in the next year?"
35% wait for stable government, 63% bring home next year, 3% unsure

FOX News/Opinion Dynamics (11/8-9): "What do you want U.S. troops in Iraq to do? Do you want them to leave Iraq and come home now or do you want them to stay in Iraq and finish the job?"
36% leave now, 55% finish the job, 9% unsure

NBC News/Wall Street Journal (11/4-7):  "Do you think that the United States should maintain its current troop level in Iraq to help secure peace and stability, or should the United States reduce its number of troops now that Iraq has adopted a constitution?"
36% maintain level, 57% reduce number, 4% both depends, 4% unsure

ABC News/Washington Post (10/30-11/2):  "Do you think the United States should keep its military forces in Iraq until civil order is restored there, even if that means continued U.S. military casualties; or do you think the United States should withdraw its military forces from Iraq in order to avoid further U.S. military casualties, even if that means civil order is not restored there?"
52% keep forces in, 44% withdraw forces, 4% unsure

CBS News (10/30-11-1): "Should the United States troops stay in Iraq as long as it takes to make sure Iraq is a stable democracy, even if it takes a long time, or should U.S. troops leave Iraq as soon as possible, even if Iraq is not completely stable?" 43% stay as long as it takes, 50% leave ASAP, 7% unsure

Pew Research Center (10/6-10): "Do you think the U.S. should keep military troops in Iraq until the situation has stabilized, or do you think the U.S. should bring its troops home as soon as possible?" 47% keep troops, 48% bring home ASAP, 5% unsure

So there we have it:  A consistent majority of at least 60% of Americans now disapproves of President Bush’s performance on the Iraq war and believes it not worth the cost.  A smaller majority now says that the war was a mistake.  The consistency of the results suggests these are real attitudes, not opinions formed on the spot in the response to the language of the question.