In 2005, the Institute of Medicine called for the National Institutes of Health to “implement periodic comprehensive, representative, national surveys to assess the changes in the prevalence, patterns, perceptions, and costs of [complementary] therapy use…”. Therefore, it is not surprising that national surveys play a prominent role in NCCAM’s latest strategic plan, which calls for an increased understanding of “real world” patterns and outcomes of the use of complementary health approaches. The National Health Interview Survey, for example, provides important information on complementary approach use, cost, and spending in the United States. NCCAM needs reliable data about attitudes toward and use of complementary approaches to help ensure that our research priorities and public information materials address what is most important to the general public, providers, and policymakers.
Well-designed questionnaires are critical to this important research effort. There are many qualitative and survey methods that researchers use to refine surveys before they are ready for real-world studies. I’ve worked in this area for more than 20 years and have seen how qualitative evaluation can be used to identify and fix problems in questionnaire items before they are administered to the entire study population.
The development process for many surveys begins several years before the start of data collection. Often as an initial step, survey developers will convene expert panels to help determine important content areas. Focus groups, where participants share their thoughts and experiences, are another useful tool to hone in on the most salient aspects of a particular area of interest. In addition to these, a popular technique in questionnaire development is the use of “cognitive interviews” to learn what respondents are thinking while answering questions. A trained interviewer will read the draft questions to the respondent and then use probing techniques to find out how the respondent is interpreting the questions. Cognitive interviews are useful for identifying problems with question wording, sequencing, or formatting. Rather than making changes to the questionnaire based on one or two interviews, researchers will search for common themes across interviews using qualitative analysis techniques.
Once researchers have refined the draft questionnaire, they will often perform a “field test” to determine how the questionnaire performs under study conditions. By duplicating the planned survey operations, interviewers and others involved with the study are able to learn about potential problems in delivering the questionnaire and responder issues.
All of these methods are useful to find and resolve problems with survey questions before they are incorporated into the actual study. Researchers should make every effort to select a diverse group of participants for these activities, especially with regard to age, gender, race, ethnicity, and other socio-demographic, health, and literacy characteristics.
During my many years working in questionnaire development, I have learned several tips for designing effective survey questions:
- Keep questions specific rather than general.
- Split or break down questions that cover multiple concepts into single items.
- Make sure reference periods match the respondent’s ability to recall.
- Define unfamiliar or ambiguous terms.
- Keep literacy level at the 8th grade or below for general population surveys.
- Frame the topic and focus attention with introductory and transitional statements.
- Keep in mind that question ordering and placement will affect responses.
- Design response categories so they are consistent with the question and non-overlapping.
- Avoid asking multiple topics in a single question that only allows for one answer (double-barreled).