Misinterpreting survey questions

Social scientists and survey methodologists have been long aware of the complexity of the apparent simple process of asking and answering questions. They are also aware of the many pitfalls of this process and look for a number of common and widely recognized mistakes in question and questionnaire construction that await the unwary.

Who among us, for example, has misunderstood a question being asked of them? What’s the most common source of such misunderstanding? It may very likely be the difference in meaning that you and your questioner attribute to one or more terms in the question itself. There’s the classic example in Stanley Payne’s 1951 book, The Art of Asking Questions, which describes a survey that sought to measure public opinion toward the regulation of corporate profits. The survey asked people across the country about their opinions on this topic. The researchers discovered during their analysis, however, that Southern black women reported an unexpected distaste for the regulation of profits. To better understand this result, interviewers were asked to return to the field and ask respondents the same question, but then to follow with the question, “And why is that?” They quickly discovered that many respondents thought that they were being asked about their attitudes toward regulating “prophets,” not “profits.” Being the good Southern Baptists that many of these women were, they quickly and emphatically replied that it was no business of government to mess with prophets like Abraham, Esther, Isaac, or Sarah.

Survey questions can be questionable

The CBS/New York Times released the results of a RDD phone survey of 1,042 adult Americans on 9/24/09. Among the items heavily reported in the "liberal media" was the "fact" that nearly two-thirds of the public support a "public option" in the health care reforms being currently considered by Congress. But look at the actual wording of the question:

"q57 Would you favor or oppose the government offering everyone a government administered health insurance plan -- something like the Medicare coverage that people 65 and older get -- that would compete with private health insurance plans?"


Isn't it very likely that the specific mention of Medicare would have boosted the percentage of people in support of such a plan? Despite its many problems, Medicare is well regarded among the American people. It's a good brand. Including it in the question undoubtedly inflated the support of the survey's results.


Little noted were the results of another question in the same survey: "q38 Do you think you understand the health care reforms under consideration in Congress, or are they confusing to you?" 59 percent said they were confused. Should we put a lot of faith in the support of the "public option"?

Statistics as a foreign language

Pick up any textbook in statistics, and you will soon realize why statistics seem so foreign. It is. The language for many is not only new but mysterious. This is so for several reasons. The language of statistics is:

¨ paradoxically precise yet probabilistic;
¨ slightly askew from everyday usage and downright misleading in some instances;
¨ replete with instances in which the same word takes on substantially different meanings, even in a statistical context; and
¨ replete with double negatives (e.g., rejecting the null hypothesis is one of my favorites).

All these characteristics get in the way of understanding and communicating statistics, but they help make a decent wage for the statisticians who invent the jargon, use it, and criticize others’ misuse of it.

It's easier to lie without statistics

Mark Twain attributes this quote to the British statesman Disraeli: “There are three kinds of lies: lies, damned lies, and statistics.” An erudite American statistician, Fred Mosteller, quipped in response: “It’s easy to lie with statistics, but it’s easier to lie without them.” Considerable research in cognitive psychology and decision theory has demonstrated repeatedly that our guts, hearts, and heads play tricks on us that good data and statistics can help protect us from.