National Center for Complementary and Integrative Health (NCCIH)

NIH…Turning Discovery Into Health

Follow NCCIH Subscribe to our email update Subscribe to the NCCIH RSS feed Follow NCCIH on FacebookRead our disclaimer about external links Follow NCCIH on TwitterRead our disclaimer about external links

N C C A M Research Blog

Survey Development in Complementary Health Research

October 16, 2012
Barbara Stussman

Barbara Stussman is a survey statistician at NCCAM.  She has been involved in planning and implementing the National Health Interview Survey, an annual study conducted by the Centers for Disease Control and Prevention's National Center for Health Statistics.

In 2005, the Institute of Medicine called for the National Institutes of Health to “implement periodic comprehensive, representative, national surveys to assess the changes in the prevalence, patterns, perceptions, and costs of [complementary] therapy use…”. Therefore, it is not surprising that national surveys play a prominent role in NCCAM’s latest strategic plan, which calls for an increased understanding of “real world” patterns and outcomes of the use of complementary health approaches. The National Health Interview Survey, for example, provides important information on complementary approach use, cost, and spending in the United States.  NCCAM needs reliable data about attitudes toward and use of complementary approaches to help ensure that our research priorities and public information materials address what is most important to the general public, providers, and policymakers.

Well-designed questionnaires are critical to this important research effort. There are many qualitative and survey methods that researchers use to refine surveys before they are ready for real-world studies. I’ve worked in this area for more than 20 years and have seen how qualitative evaluation can be used to identify and fix problems in questionnaire items before they are administered to the entire study population.

The development process for many surveys begins several years before the start of data collection. Often as an initial step, survey developers will convene expert panels to help determine important content areas. Focus groups, where participants share their thoughts and experiences, are another useful tool to hone in on the most salient aspects of a particular area of interest. In addition to these, a popular technique in questionnaire development is the use of “cognitive interviews” to learn what respondents are thinking while answering questions. A trained interviewer will read the draft questions to the respondent and then use probing techniques to find out how the respondent is interpreting the questions. Cognitive interviews are useful for identifying problems with question wording, sequencing, or formatting. Rather than making changes to the questionnaire based on one or two interviews, researchers will search for common themes across interviews using qualitative analysis techniques.

Once researchers have refined the draft questionnaire, they will often perform a “field test” to determine how the questionnaire performs under study conditions. By duplicating the planned survey operations, interviewers and others involved with the study are able to learn about potential problems in delivering the questionnaire and responder issues.

All of these methods are useful to find and resolve problems with survey questions before they are incorporated into the actual study. Researchers should make every effort to select a diverse group of participants for these activities, especially with regard to age, gender, race, ethnicity, and other socio-demographic, health, and literacy characteristics.

During my many years working in questionnaire development, I have learned several tips for designing effective survey questions:

  1. Keep questions specific rather than general.
  2. Split or break down questions that cover multiple concepts into single items.
  3. Make sure reference periods match the respondent’s ability to recall.
  4. Define unfamiliar or ambiguous terms.
  5. Keep literacy level at the 8th grade or below for general population surveys.
  6. Frame the topic and focus attention with introductory and transitional statements.
  7. Keep in mind that question ordering and placement will affect responses.
  8. Design response categories so they are consistent with the question and non-overlapping.
  9. Avoid asking multiple topics in a single question that only allows for one answer (double-barreled).

Barbara Stussman and Richard Nahin, Ph.D., M.P.H., members of NCCAM’s survey research program, are available to talk with interested investigators.


Comments are now closed for this post.

For your “well-designed” NCCAM questionnaire, you forgot to mention these key elements: 1.) avoid defining CAM with any meaningful specificity, 2.) pad the surveys with practices that have been standard healthcare for decades (e.g. relaxation, massage), 3.) pad the surveys even with religious practices (e.g. “prayer for health”), and 4.) use results to gain Congressional funding for more surveys and more studies of the wildly implausible.

The 2007 National Health Interview Survey included questions on 36 types of complementary and alternative medicine therapies commonly used in the United States—10 types of provider-based therapies, such as acupuncture and chiropractic, and 26 other therapies that do not require a provider, such as herbal supplements and meditation. The results are based on data from more than 23,300 interviews with American adults and more than 9,400 interviews with adults on behalf of a child in their household. The CDC report is available here: “Prayer for health” is not included in this analysis.

Hi Barbara, agree with you on the pointers for survey questions -  and of course the need to have periodic surveys. Sites like SoGoSurvey have good survey samples for beginners.