Misinterpreting survey questions

Social scientists and survey methodologists have been long aware of the complexity of the apparent simple process of asking and answering questions. They are also aware of the many pitfalls of this process and look for a number of common and widely recognized mistakes in question and questionnaire construction that await the unwary.

Who among us, for example, has misunderstood a question being asked of them? What’s the most common source of such misunderstanding? It may very likely be the difference in meaning that you and your questioner attribute to one or more terms in the question itself. There’s the classic example in Stanley Payne’s 1951 book, The Art of Asking Questions, which describes a survey that sought to measure public opinion toward the regulation of corporate profits. The survey asked people across the country about their opinions on this topic. The researchers discovered during their analysis, however, that Southern black women reported an unexpected distaste for the regulation of profits. To better understand this result, interviewers were asked to return to the field and ask respondents the same question, but then to follow with the question, “And why is that?” They quickly discovered that many respondents thought that they were being asked about their attitudes toward regulating “prophets,” not “profits.” Being the good Southern Baptists that many of these women were, they quickly and emphatically replied that it was no business of government to mess with prophets like Abraham, Esther, Isaac, or Sarah.

Survey questions can be questionable

The CBS/New York Times released the results of a RDD phone survey of 1,042 adult Americans on 9/24/09. Among the items heavily reported in the "liberal media" was the "fact" that nearly two-thirds of the public support a "public option" in the health care reforms being currently considered by Congress. But look at the actual wording of the question:

"q57 Would you favor or oppose the government offering everyone a government administered health insurance plan -- something like the Medicare coverage that people 65 and older get -- that would compete with private health insurance plans?"


Isn't it very likely that the specific mention of Medicare would have boosted the percentage of people in support of such a plan? Despite its many problems, Medicare is well regarded among the American people. It's a good brand. Including it in the question undoubtedly inflated the support of the survey's results.


Little noted were the results of another question in the same survey: "q38 Do you think you understand the health care reforms under consideration in Congress, or are they confusing to you?" 59 percent said they were confused. Should we put a lot of faith in the support of the "public option"?