National Center for Complementary and Integrative Health (NCCIH)

NIH…Turning Discovery Into Health

Follow NCCIH Subscribe to our email update Subscribe to the NCCIH RSS feed Follow NCCIH on FacebookRead our disclaimer about external links Follow NCCIH on TwitterRead our disclaimer about external links

N C C A M Research Blog

Observational Studies and Secondary Data Analyses To Assess Outcomes in Complementary and Integrative Health Care

June 25, 2012
Portrait of Dr. Nahin
Richard Nahin, Ph.D., M.P.H.

Senior Advisor for Scientific Coordination and Outreach
National Center for Complementary and Alternative Medicine

View Dr. Nahin’s biographical sketch

During the many years I have been with NCCAM, I have seen an exponential increase in both the quantity and quality of research investigating the efficacy and biological basis of many types of complementary health practices. For many reasons there has been less research on the real-world effectiveness of these therapies. However, promising analytic methods from other fields and emerging technologies such as electronic medical records can be used to take advantage of actual patient experiences so we can learn more about outcomes and effectiveness in real-world settings.

In response, NCCAM’s most recent strategic plan calls for research using the tools and methods of the disciplines of observational, survey, epidemiology, outcomes, health services, and effectiveness research. While well-designed pragmatic clinical trials are the most rigorous way to study effectiveness, observational studies and secondary analyses of existing datasets can also be of great value.

Although observational studies cannot provide definitive evidence of safety, efficacy, or effectiveness, they can: 1) provide information on “real world” use and practice; 2) detect signals about the benefits and risks of complementary therapies use in the general population; 3) help formulate hypotheses to be tested in subsequent experiments; 4) provide part of the community-level data needed to design more informative pragmatic clinical trials; and 5) inform clinical practice.

The main difficulty with causal inference in such observational studies has to do with the fact that participants or their providers choose which therapies the participants receive. Invariably, this “choice” means that participants choosing one therapy may not have the same characteristics as participants choosing another therapy; one or more of these differences may be the true cause of any observed effects rather than the use of one therapy or another. This is, of course, very different than a well-designed randomized clinical trial of sufficient size, where the same process that randomly assigns participants to one treatment or another also helps to balance the characteristics of individuals in each group.

A number of study designs and analytical techniques have been used to, at least partially, control for treatment self-selection. Of these, the most well-known and popular methods have used regression modeling to control for differences in participant characteristics between groups. Not as well known by those studying complementary health practices are techniques employing propensity scores or instrumental variables to match samples. Use of these and other various methods were presented in a symposium at the recent International Research Congress on Integrative Medicine and Health.

To help clinical researchers, including observational researchers, NCCAM has posted a Clinical Research Toolbox on its Web site. The Toolbox has information on, and examples of, many of the materials needed to conduct a clinical trial. Importantly, the Toolbox also includes a list of many NIH-supported longitudinal studies (95KB PDF) that have collected information on one or more complementary health practices. The datasets for these studies are publicly available for interested researchers. We encourage submission of grant applications to support rigorous, hypothesis-driven studies using these and other longitudinal datasets.

9/13/12 Update –  NCCAM will participate in the following funding opportunity announcement:  Secondary Analyses of Comparative Effectiveness, Health Outcomes and Costs in Persons with Multiple Chronic Conditions (R21).

* Note: PDF files require a viewer such as the free Adobe Reader.


Comments are now closed for this post.

Pragmatic trials are useful for comparing two treatments whose efficacy has already been established. They are popular among advocates of alternative medicine because they allow them to bypass rigorous trials to establish efficacy and go straight to a scenario where the placebo effects of treatment can shine. If a treatment only “works” as a placebo, it may look better in a pragmatic trial for various reasons. Prescribing placebos is unethical. We must inform patients when a treatment offers only non-specific treatment effects rather than misleading them by telling them it has been “proven to work” by a pragmatic trial

I’m afraid that observational data sometimes give misleading results and do real harm to public health. It’s been explained very well in the New York Times -see

I’m rather surprised to see a branch of NIH publishing ideas that are easily seen to be wrong by a journalist writing in a newspaper.

There is perceptibly a lot to know about this. I consider you made various nice points in features also.