Where's the Evidence that Respondents Understand Your Survey Questions?

Abstract

Survey researchers try to align respondent question interpretations with their own by applying theoretical rules, such as writing questions simply, concretely, and without biasing words or double-barreled inquiries, or via post hoc plausibility checks. Few, however, conduct ex ante empirical evaluations of their application of these rules. The widely recognized best empirical approach is "cognitive debriefing," that follows a survey question with a detailed conversation with the respondent. Unfortunately, this time-consuming procedure is rarely used, especially for novel surveys on online platforms. This paradoxically leaves the veracity of the discipline's most used empirical methodology depending largely on theoretical assumptions. For cognitive debriefing to live up to its intended potential, we first formalize the essential interpretation-related assumptions underlying all survey questions. We then automate cognitive debriefing to satisfy these assumptions using a specially-tuned chatbot, apply it to larger numbers than heretofore possible, analyze the collection of transcripts, and summarize the interpretations it reveals. By iteratively rewriting questions and rerunning this procedure, researchers can help ensure respondents understand questions as they do. We also apply our procedure to some of the most common survey questions and demonstrate large disconnects between researcher intentions and respondent understandings. We make available easy-to-use open source software that implements all our suggestions.