The evidence for evidence-based medicine

A Pragmatic Historical Assessment Tool: A New Systematic Framework for the Collation and Evaluation of Documented Empirical Effectiveness and Safety of Traditional Plant Medicines in the European Materia Medica

INTRODUCTION

Traditional plant medicines (TPMs) are plant-derived therapeutic products prepared and applied according to longstanding medical customs. Around the world they are widely used in primary and preventative health care. The World Health Organization (WHO) calls in its Traditional Medicine Strategy 2014-2023 for Member States to provide a regulatory framework so that the formal contribution of traditional therapeutics can be advanced in national systems of health care. Evidence of effectiveness and safety is paramount for the regulatory integration of TPMs; however, a presumed lack of such "evidence" is one obstacle for full integration. The consequential health policy question is how to systematically evaluate therapeutic claims relating to herbal remedies when the extant evidence is predominantly based on historical and contemporary clinical usage, i.e., is empiricist in nature. This paper introduces a new method along with several illustrative examples.

METHOD

Our research design employs a longitudinal, comparative textual analysis of standard textbooks of the professional European medical literature from the early modern period (1588/1664) onwards to today. It then triangulated these intergenerationally documented clinical observations on two exemplars (Arnica and St. John's Wort) with corresponding listings in multiple qualitative and quantitative sources. A Pragmatic Historical Assessment (PHA) tool was developed and tested as a method to systematically collate the large amount of pharmacological data recorded in these judiciously selected sources. The evidential validity of longstanding professional clinical knowledge could thus be compared with therapeutic indications approved in official and authoritative sources (pharmacopoeias, monographs) and with those supported by contemporary scientific research (randomised-controlled trials [RCTs], experimental research).

RESULTS

There was high congruency between therapeutic indications that are based on repeated empirical observations from professional patient care (empirical evidence), those approved in pharmacopoeias and monographs, and modern scientific evidence based on RCTs. The extensive herbal triangulation confirmed parallel records of all main therapeutic indications of the exemplars across all qualitative and quantitative sources over the past 400 years.

CONCLUSIONS

Historical clinical medical textbooks and contemporary phytotherapeutic equivalents are the key repository of repeatedly evaluated therapeutic plant knowledge. The professional clinical literature proved to be a reliable and verifiable body of empirical evidence that harmonised with contemporary scientific assessments. The newly developed PHA tool provides a coding framework for the systematic collation and evaluation of empirical data on the effectiveness and safety of TPMs. It is suggested as a feasible and efficient tool to extend evidence typologies that substantiate therapeutic claims for TPMs as part of an evidence-based regulatory framework that formally integrates these medically and culturally important therapeutics.

Einleitung

Traditionelle pflanzliche Arzneimittel sind aus Pflanzen gewonnene Heilmittel, die gemäß langjähriger medizinischer Praxis zubereitet und angewendet werden. Weltweit sind sie in der primären und präventiven Gesundheitsversorgung weit verbreitet. Die Weltgesundheitsorganisation (WHO) ruft in ihrer Traditional Medicine Strategy 2014–2023 die Mitgliedstaaten dazu auf, regulatorische Rahmenbedingungen zu schaffen, welche den formellen Beitrag traditioneller Therapeutika in den nationalen Gesundheitssystemen fördern. Der Nachweis von Wirksamkeit und Sicherheit ist von zentraler Bedeutung für die regulatorische Integration traditioneller pflanzlicher Arzneimittel, doch das angebliche Fehlen solcher “Nachweise“ ist eine der Hürden für die vollständige Integration. Daraus ergibt sich die gesundheitspolitische Frage, wie man therapeutische Anwendungsgebiete pflanzlicher Heilmittel systematisch evaluieren kann, wenn die vorliegende Evidenz überwiegend auf deren historischer und aktueller klinischen Verwendung beruht, also empirischer Natur ist. In dieser Arbeit wird eine neue Methode mitsamt veranschaulichenden Beispielen vorgestellt.

Methoden

Unser Forschungsansatz beruhte auf einer longitudinalen, vergleichenden Textanalyse von Standard-Lehrwerken der europäischen medizinischen Fachliteratur ausgehend von der frühen Neuzeit (1588/1664) bis heute. Die über Generationen dokumentierten klinischen Beobachtungen wurden anhand von zwei Beispielen (Arnika and Johanniskraut) mit den diesbezüglichen Angaben in unterschiedlichen qualitativen und quantitativen Quellen trianguliert. Ein Pragmatisch‐Historisches Auswertungstool (PHA) wurde als Methode entwickelt und getestet, um die großen Mengen der in diesen kritisch ausgewählten Quellen enthaltenen pharmakologischen Daten systematisch zu erfassen. Die Evidenzvalidität des langjährigen klinischen Fachwissens konnte so mit den therapeutischen Anwendungsgebieten verglichen werden, die in offiziellen und autoritativen Quellen (Pharmakopöen, Monografien) zugelassen sind, sowie mit denjenigen, die durch zeitgenössische wissenschaftliche Forschung gestützt werden (randomisierte kontrollierte Studien [RCTs], experimentelle Forschung).

Ergebnisse

Es bestand ein hohes Maß an Kongruenz zwischen den therapeutischen Anwendungsgebieten, welche auf wiederholte empirische Beobachtung aus der professionellen Patientenversorgung beruhen (empirische Evidenz), den zugelassenen Indikationen in Pharmakopöen und Monographien sowie der aktuellen wissenschaftlichen Evidenz basierend auf klinischen Studien. Die umfassende pflanzenbezogene Triangulation bestätigte parallele Aufzeichnungen aller wesentlichen Anwendungsgebiete der untersuchten Beispiele in allen qualitativen und quantitativen Quellen über die letzten 400 Jahre hinweg.

Schlussfolgerungen

Historische Lehrbücher für klinische Medizin und zeitgenössische phytotherapeutische Äquivalente sind die wichtigsten Quellen von wiederholt evaluiertem therapeutischem Wissen zu Heilpflanzen. Die klinische Fachliteratur erwies sich als zuverlässiger und verifizierbarer Korpus empirischer Evidenz, der mit aktuellen wissenschaftlichen Untersuchungen übereinstimmte. Das neu entwickelte PHA-Verfahren bietet ein Kodierungs‐Instrument für das systematische Erfassen und Auswerten empirischer Daten zur Wirksamkeit und Sicherheit von traditionellen pflanzlichen Arzneimitteln. Das PHA‐Verfahren wird als praktikables und effizientes Instrument zur Erweiterung der Evidenz‐Typologien empfohlen, indem es therapeutische Indikationen für traditionelle pflanzliche Arzneimittel untermauern kann, so dass diese medizinisch und kulturell wichtigen Therapeutika in einen evidenz-basierten regulatorischen Rahmen integriert werden können.

The challenge of determining appropriate care in the era of patient-centered care and rising health care costs

Objective

To expand the appropriateness of care methodology to include patient preferences and resource utilization, and the impact of care appropriateness on patient outcomes.

Data Sources/Study Setting

Primary data from expert panels, focus groups, chiropractors, chiropractic patients with chronic low back pain (CLBP) and chronic cervical pain (CCP), and from internet “workers” via crowdsourcing. Study setting is a cluster sample of 125 chiropractic clinics from six US regions.

Study Design

This multicomponent methods study includes analysis of longitudinal data on patient outcomes, preferences, CLBP and CCP symptoms and healthcare utilization.

Data Collection/Extraction Methods

Data were collected bi-weekly for 3 months via online surveys that included both new and legacy measures, including PROMIS and CAHPS.

Principle Findings

Appropriateness panels generated ratings for 1800 CLBP and 744 CCP indications which will be applied to patient charts. Data from 2025 patients are being analyzed.

Conclusions

Patient-centered care is a significant policy initiative but translating it into policy that has been clinician and research-expert based, poses significant methodological issues. Nonetheless, we make the case that patient preferences, self-reported outcomes, and financial burden should be considered in the evaluation of the appropriateness of healthcare.

Researching the Appropriateness of Care in the Complementary and Integrative Health Professions: Part I

OBJECTIVES

The purpose of this article is to report on the Center of Excellence for Research on Complementary and Alternative Medicine at RAND Corporation. The overall project examined the appropriateness of chiropractic spinal manipulation and mobilization for chronic low back pain and chronic cervical pain using the RAND and University of California Los Angeles Appropriateness Method, including patient preferences and costs, to acknowledge the importance of patient-centered care in clinical decision-making.

METHODS

This article is a narrative summary of the overall project and its inter-related components (ie, 4 Research Project Grants and 2 centers), including the Data Collection Core, whose activities and learning will be the subject of a following series of methods articles.

RESULTS

The project team faced many challenges in accomplishing data collection goals. The processes we developed to overcome barriers may be of use to other researchers and for practitioners who may want to participate in such studies in complementary and integrative health, which previously was known as complementary and alternative medicine.

CONCLUSION

For this large, complex, successful project, we gathered online survey data, collected charts, and abstracted chart data from thousands of chiropractic patients. The present article delineates the challenges and lessons that were learned during this project so that others may gain from the authors' experience. This information may be of use to future research that collects data from independent practitioners and their patients because it provides what is needed to be successful in such studies and may encourage participation.

Why Prospectively Randomized Clinical Trials Have Been Rare in Reproductive Medicine and Will Remain So?

There is almost unanimity that modern medicine should be "evidence based." In this context, lack of prospectively randomized clinical trials (RCTs) is widely lamented in reproductive medicine. Some leading voices, indeed, increasingly suggest that only RCT-based clinical conclusions should be integrated into clinical practice, since lower levels of evidence are inadequate. We have argued that reproductive medicine requires special considerations because, like clinical oncology, fertility treatments (especially in older women) are time dependent. Unlike clinical oncology, reproductive medicine, however, does not receive substantial financial research support from government or industry and, at least in the United States, has, therefore, to be primarily funded via patient revenues. Given a 50% chance of receiving placebo, infertility patients are, understandably, reluctant to fund their own RCTs. We here selectively review this subject, contrasting opposing opinions recently published in the literature by a prominent reproductive scientist and one of the world's leading experts on evidence-based medicine. Placing these recent publications into the evolving context of infertility practice, as also addressed in this journal in recent publications, we conclude that objective reasons explain why relatively few RCTs are performed in reproductive medicine and predict that this will not change in the foreseeable future. Reproductive medicine, therefore, has to find ways to develop satisfactory clinical evidence in other ways, satisfying patients' rights to easy access to potentially beneficial medical treatments with low costs and low risks. The RCTs should be reserved for relatively high risk and/or high cost treatments.

Televised medical talk shows--what they recommend and the evidence to support their recommendations: a prospective observational study

OBJECTIVE

To determine the quality of health recommendations and claims made on popular medical talk shows.

DESIGN

Prospective observational study.

SETTING

Mainstream television media.

SOURCES

Internationally syndicated medical television talk shows that air daily (The Dr Oz Show and The Doctors).

INTERVENTIONS

Investigators randomly selected 40 episodes of each of The Dr Oz Show and The Doctors from early 2013 and identified and evaluated all recommendations made on each program. A group of experienced evidence reviewers independently searched for, and evaluated as a team, evidence to support 80 randomly selected recommendations from each show.

MAIN OUTCOMES MEASURES

Percentage of recommendations that are supported by evidence as determined by a team of experienced evidence reviewers. Secondary outcomes included topics discussed, the number of recommendations made on the shows, and the types and details of recommendations that were made.

RESULTS

We could find at least a case study or better evidence to support 54% (95% confidence interval 47% to 62%) of the 160 recommendations (80 from each show). For recommendations in The Dr Oz Show, evidence supported 46%, contradicted 15%, and was not found for 39%. For recommendations in The Doctors, evidence supported 63%, contradicted 14%, and was not found for 24%. Believable or somewhat believable evidence supported 33% of the recommendations on The Dr Oz Show and 53% on The Doctors. On average, The Dr Oz Show had 12 recommendations per episode and The Doctors 11. The most common recommendation category on The Dr Oz Show was dietary advice (39%) and on The Doctors was to consult a healthcare provider (18%). A specific benefit was described for 43% and 41% of the recommendations made on the shows respectively. The magnitude of benefit was described for 17% of the recommendations on The Dr Oz Show and 11% on The Doctors. Disclosure of potential conflicts of interest accompanied 0.4% of recommendations.

CONCLUSIONS

Recommendations made on medical talk shows often lack adequate information on specific benefits or the magnitude of the effects of these benefits. Approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. Potential conflicts of interest are rarely addressed. The public should be skeptical about recommendations made on medical talk shows. Additional details of methods used and changes made to study protocol.

Placebo use in the United kingdom: results from a national survey of primary care practitioners

Objectives

Surveys in various countries suggest 17% to 80% of doctors prescribe ‘placebos’ in routine practice, but prevalence of placebo use in UK primary care is unknown.

Methods

We administered a web-based questionnaire to a representative sample of UK general practitioners. Following surveys conducted in other countries we divided placebos into ‘pure’ and ‘impure’. ‘Impure’ placebos are interventions with clear efficacy for certain conditions but are prescribed for ailments where their efficacy is unknown, such as antibiotics for suspected viral infections. ‘Pure’ placebos are interventions such as sugar pills or saline injections without direct pharmacologically active ingredients for the condition being treated. We initiated the survey in April 2012. Two reminders were sent and electronic data collection closed after 4 weeks.

Results

We surveyed 1715 general practitioners and 783 (46%) completed our questionnaire. Our respondents were similar to those of all registered UK doctors suggesting our results are generalizable. 12% (95% CI 10 to 15) of respondents used pure placebos while 97% (95% CI 96 to 98) used impure placebos at least once in their career. 1% of respondents used pure placebos, and 77% (95% CI 74 to 79) used impure placebos at least once per week. Most (66% for pure, 84% for impure) respondents stated placebos were ethical in some circumstances.

Conclusion and implications

Placebo use is common in primary care but questions remain about their benefits, harms, costs, and whether they can be delivered ethically. Further research is required to investigate ethically acceptable and cost-effective placebo interventions.

How Much of CAM Is Based on Research Evidence?

The aim of this article is to provide a preliminary estimate of how much CAM is evidence-based. For this purpose, I calculated the percentage of 685 treatment/condition pairings evaluated in the "Desktop Guide to Complementary and Alternative Medicine" which are supported by sound data. The resulting figure was 7.4%. For a range of reasons, it might be a gross over-estimate. Further investigations into this subject are required to arrive at more representative figures.

Questioning the methodologic superiority of 'placebo' over 'active' controlled trials

A resilient issue in research ethics is whether and when a placebo-controlled trial (PCT) is justified if it deprives research subjects of a recognized treatment. The clinicians' moral duty to provide the best available care seems to require the use of 'active' controlled trials (ACTs) that use an established treatment as a control whenever such a therapy is available. In another regard, ACTs are supposedly methodologically inferior to PCTs. Hence, the moral duty of the clinical researcher to use the best methods will favor PCTs. In this target article, I analyze the three reasons for believing that ACTs are inferior to PCTs namely: 1) ACTs lack 'assay sensitivity'; 2) ACTs do not measure absolute effect size; and 3) ACTs require more participants; and I contend that none are acceptable. Consequently the tension between clinical and research ethics dissolves: the moral duty of the clinician to avoid PCTs is unopposed by methodological considerations.

When and how should new therapies become routine clinical practice?

The process by which new therapies enter clinical practice is frequently suboptimal. Often, ideas for new therapies are generated by clinical observations or laboratory studies; therapies based on those ideas may enter clinical practice without any further scrutiny. As a consequence, some ineffective practices become widespread. This article proposes a six-stage protocol for the implementation of new therapies. Hypotheses about therapy based on preclinical research should be subject to clinical exploration and pilot studies prior to rigorous assessment with randomised clinical trials. If randomised clinical trials suggest that the intervention produces clinically important effects, further randomised studies can be conducted to refine the intervention. New interventions should not be recommended, or included in teaching curricula, or taught in continuing education courses until their effectiveness has been demonstrated in high-quality randomised clinical trials.

Evidence Based Complementary and Alternative Medicine: Promises and Problems

OBJECTIVE

The present paper examines the experience of establishing a center for evidence-based complementary and alternative medicine (EBCAM) practice. It examines both the difficulties and the challenges of doing research to establish EBCAM. The paper also examines the political context of the demand for evidence- based practice (EBP) for CAM.

IMPLEMENTATION

A center for EBCAM was funded for 3 years within the Southern California Evidence-Based Practice Center by the National Center for CAM and administered by the Agency for Health Research Quality. This experience provides the basis for this paper.

RESULTS

While the experience of creating an EBM Center for CAM has shown that much work can be accomplished by applying standard methods of EBP medicine, it also highlights the weaknesses of such an agenda. Many standard research methods are simply not applicable to CAM, and even where they are, effectiveness is a much more important means of assessing CAM than simply efficacy. Researchers however, must be conscious of the political motivations behind much of the demand for EBCAM. Where such demands are coming from allopathic medicine, they clearly form a continuing part of medical opposition to CAM and may be intended to perpetuate the dominance of the biomedical paradigm in healthcare. The challenge for CAM is to recognize the limitations of EBP but not to throw the 'baby out with the bathwater'. There is much in EBP that clearly should be emulated by the CAM community but only where it is appropriate.

Evidence-based medicine: discrepancy between perceived competence and actual performance among graduating medical students

Since at the time of graduation from medical school physicians are expected to demonstrate adequate professional competence including mastery of critical appraisal skills, we conducted a preliminary, cross-sectional, web-based study to examine the extent to which fourth year medical students in the US are competent in core areas of evidence-based medicine (EBM). Using self-assessment instruments, subjects (n = 150) were asked to demonstrate their ability to understand the practical meaning of key methodological and data analysis constructs as they relate to patient care, to rate their perceived competence in core areas of EBM and to disclose their attitudes toward critical appraisal of the literature and EBM. The mean score in our cohort was 55% suggesting that students may have knowledge gaps that interfere with their ability to critically appraise the medical literature. There was an apparent chasm between subjects' perceived competence and their actual performance on the assessment instrument. These findings, if corroborated in larger studies, (1) suggest that better education in EBM is needed so as to avoid the possibility that patient care may inadvertently be jeopardized; and (2) cast doubt on the use of self-assessed knowledge as a proxy for actual skills with respect to EBM and medical decision-making.