Lessons from comparative effectiveness research methods development projects funded under the Recovery Act

Analytical Methods for a Learning Health System: 1. Framing the Research Question

Learning health systems use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning. Even without randomization, observational studies can play a central role as the nation’s health care system embraces comparative effectiveness research and patient-centered outcomes research. However, neither the breadth, timeliness, volume of the available information, nor sophisticated analytics, allow analysts to confidently infer causal relationships from observational data. However, depending on the research question, careful study design and appropriate analytical methods can improve the utility of EHD.

The introduction to a series of four papers, this review begins with a discussion of the kind of research questions that EHD can help address, noting how different evidence and assumptions are needed for each. We argue that when the question involves describing the current (and likely future) state of affairs, causal inference is not relevant, so randomized clinical trials (RCTs) are not necessary. When the question is whether an intervention improves outcomes of interest, causal inference is critical, but appropriately designed and analyzed observational studies can yield valid results that better balance internal and external validity than typical RCTs. When the question is one of translation and spread of innovations, a different set of questions comes into play: How and why does the intervention work? How can a model be amended or adapted to work in new settings? In these “delivery system science” settings, causal inference is not the main issue, so a range of quantitative, qualitative, and mixed research designs are needed.

We then describe why RCTs are regarded as the gold standard for assessing cause and effect, how alternative approaches relying on observational data can be used to the same end, and how observational studies of EHD can be effective complements to RCTs. We also describe how RCTs can be a model for designing rigorous observational studies, building an evidence base through iterative studies that build upon each other (i.e., confirmation across multiple investigations).

Assessing the long-term impact of public investments in comparative effectiveness research: conceptual framework and lessons learned

The American Recovery and Reinvestment Act (ARRA) of 2009 directed US$1.1 billion to the US Department of Health and Human Services for support of comparative effectiveness research (CER). As part of this investment, US Department of Health and Human Services commissioned a midstream evaluation of the ARRA CER portfolio. One goal of the evaluation was to identify issues to consider for a future evaluation of the long-term impact of this portfolio and other CER investments. In planning the ARRA CER evaluation, we developed and revised a conceptual framework and related policy research questions that may be useful to future efforts to assess the impact of CER or patient-centered outcomes research investments. In addition, we explored methodological challenges related to designing an evaluation to assess investments in CER that may be informative to any future plans to evaluate the long-term impact of ARRA CER as well subsequent investments made from the Patient-Centered Outcomes Research Trust Fund.