Observational Studies and Secondary Data Analyses To Assess Outcomes in Complementary and Integrative Health Care
During the many years I have been with NCCAM, I have seen an exponential increase in both the quantity and quality of research investigating the efficacy and biological basis of many types of complementary health practices. For many reasons there has been less research on the real-world effectiveness of these therapies. However, promising analytic methods from other fields and emerging technologies such as electronic medical records can be used to take advantage of actual patient experiences so we can learn more about outcomes and effectiveness in real-world settings.
In response, NCCAM’s most recent strategic plan calls for research using the tools and methods of the disciplines of observational, survey, epidemiology, outcomes, health services, and effectiveness research. While well-designed pragmatic clinical trials are the most rigorous way to study effectiveness, observational studies and secondary analyses of existing datasets can also be of great value.
Although observational studies cannot provide definitive evidence of safety, efficacy, or effectiveness, they can: 1) provide information on “real world” use and practice; 2) detect signals about the benefits and risks of complementary therapies use in the general population; 3) help formulate hypotheses to be tested in subsequent experiments; 4) provide part of the community-level data needed to design more informative pragmatic clinical trials; and 5) inform clinical practice.
The main difficulty with causal inference in such observational studies has to do with the fact that participants or their providers choose which therapies the participants receive. Invariably, this “choice” means that participants choosing one therapy may not have the same characteristics as participants choosing another therapy; one or more of these differences may be the true cause of any observed effects rather than the use of one therapy or another. This is, of course, very different than a well-designed randomized clinical trial of sufficient size, where the same process that randomly assigns participants to one treatment or another also helps to balance the characteristics of individuals in each group.
A number of study designs and analytical techniques have been used to, at least partially, control for treatment self-selection. Of these, the most well-known and popular methods have used regression modeling to control for differences in participant characteristics between groups. Not as well known by those studying complementary health practices are techniques employing propensity scores or instrumental variables to match samples. Use of these and other various methods were presented in a symposium at the recent International Research Congress on Integrative Medicine and Health.
To help clinical researchers, including observational researchers, NCCAM has posted a Clinical Research Toolbox on its Web site. The Toolbox has information on, and examples of, many of the materials needed to conduct a clinical trial. Importantly, the Toolbox also includes a list of many NIH-supported longitudinal studies (95KB PDF) that have collected information on one or more complementary health practices. The datasets for these studies are publicly available for interested researchers. We encourage submission of grant applications to support rigorous, hypothesis-driven studies using these and other longitudinal datasets.
9/13/12 Update – NCCAM will participate in the following funding opportunity announcement: Secondary Analyses of Comparative Effectiveness, Health Outcomes and Costs in Persons with Multiple Chronic Conditions (R21).