Thursday, October 22, 2009

Misinterpreting survey questions

Social scientists and survey methodologists have been long aware of the complexity of the apparent simple process of asking and answering questions. They are also aware of the many pitfalls of this process and look for a number of common and widely recognized mistakes in question and questionnaire construction that await the unwary.

Who among us, for example, has misunderstood a question being asked of them? What’s the most common source of such misunderstanding? It may very likely be the difference in meaning that you and your questioner attribute to one or more terms in the question itself. There’s the classic example in Stanley Payne’s 1951 book, The Art of Asking Questions, which describes a survey that sought to measure public opinion toward the regulation of corporate profits. The survey asked people across the country about their opinions on this topic. The researchers discovered during their analysis, however, that Southern black women reported an unexpected distaste for the regulation of profits. To better understand this result, interviewers were asked to return to the field and ask respondents the same question, but then to follow with the question, “And why is that?” They quickly discovered that many respondents thought that they were being asked about their attitudes toward regulating “prophets,” not “profits.” Being the good Southern Baptists that many of these women were, they quickly and emphatically replied that it was no business of government to mess with prophets like Abraham, Esther, Isaac, or Sarah.

No comments:

Post a Comment