Rating the appropriateness of coronary angiography--do practicing physicians agree with an expert panel and with each other?

Expert consensus established around flexible, individualized migraine treatment utilizing a modified Delphi panel

OBJECTIVE

To characterize treatment decision-making processes and formalize consensus regarding key factors headache specialists consider in treatment decisions for patients with migraine, considering novel therapies.

BACKGROUND

Migraine therapies have long been subject to binary classification, acute versus preventive, due to limitations of available drugs. The emergence of novel therapies that can be used more flexibly creates an opportunity to rethink this binary classification. To determine the role of these novel therapies in treatment, it is critical to understand whether existing guidelines reflect clinical practice and to establish consensus around factors driving management.

METHODS

A three-round modified Delphi process was conducted with migraine clinical experts. Round 1 consisted of an online questionnaire; Round 2 involved an online discussion of aggregated Round 1 results; and Round 3 allowed participants to revise Round 1 responses, incorporating Round 2 insights. Questions elicited likelihood ratings (0 = highly unlikely to 100 = highly likely), rankings, and estimates on treatment decision-making.

RESULTS

Nineteen experts completed three Delphi rounds. Experts strongly agreed on definitions for "acute" (median = 100, inter-quartile range [IQR] = 5) and "preventive" treatment (median = 90, IQR = 15), but noted a need for treatment customization for patients (median = 100, IQR = 6). Experts noted certain aspects of guidelines may no longer apply based on established tolerability and efficacy of newer acute and preventive agents (median = 91, IQR = 17). Further, experts agreed on a treatment category referred to as "situational prevention" (or "short-term prevention") for patients with reliable and predictable migraine triggers (median = 100, IQR = 10) or time-limited periods when headache avoidance is important (median = 100, IQR = 12).

CONCLUSIONS

Using the modified Delphi method, a panel of migraine experts identified the importance of customizing treatment for people with migraine and the utility of "situational prevention," given the ability of new treatment options to meet this need and the potential to clinically identify patients and time periods when this approach would add value.

A Delphi Consensus Approach for Difficult-to-Treat Patients with Severe Hemophilia A without Inhibitors

Introduction

Over the past decade, there has been an increase in novel therapeutic options to treat hemophilia A. It is still unclear how these novel treatments are used in the management of patients with hemophilia A, particularly those with challenging clinical scenarios who are typically excluded in clinical trials.

Purpose

This study aimed to understand the areas of consensus and disagreement among hematologists regarding the preferences toward therapeutic approaches for difficult-to-treat patients with severe hemophilia A without inhibitors.

Patients and Methods

During February-June 2020, a three-round modified Delphi study was conducted to generate consensus among 13 US experts in the field of hemophilia. Experts were asked about their preferences toward therapeutic options for patients with challenging clinical situations, including age-related morbidities (eg, myocardial infarction, joint arthropathy), increasing demand for high-impact physical activities, early onset osteoporosis, and newborns with hemophilia A. Consensus was defined as ≥75% agreement between the panelists.

Results

Consensus was reached on many, but not all cases, leaving uncertainty about appropriateness of therapeutic approaches for some patients where clinical evidence is not available or driven by physicians' or patients' preferences toward therapeutic options. A majority of panelists preferred FVIII replacement therapy rather than emicizumab prophylaxis for the challenging cases presented due to established evidence on safety, efficacy, and level of bleed protection for FVIII treatment.

Conclusion

Recommendations emerging from this study may help guide practicing hematologists in the management of challenging hemophilia A cases. Future studies are needed to address treatment options in the clinical cases where no consensus was reached.

Understanding the evolution of coverage policies for prophylaxis treatments of hemophilia A without inhibitors: a payer Delphi panel

BACKGROUND: The landscape for hemophilia A prophylaxis is rapidly expanding from factor VIII replacement therapy to include novel treatments such as nonfactor replacement therapies that may enhance coagulation (e.g., emicizumab) or inhibit anticoagulant pathways (e.g., fitusiran and concizumab). For payers, this expansion presents challenges in balancing well-established treatments with new options that cost more and have lesser known real-world safety and efficacy. OBJECTIVE: To evaluate likely coverage practices for hemophilia A prophylaxis therapies among U.S. payers given evolving real-world data on safety and efficacy. METHODS: A 3-round modified Delphi process was conducted with representatives of U.S. commercial health plans who had considerable expertise in managing populations of patients with hemophilia. Round 1 consisted of an online questionnaire; round 2 involved an online discussion about the aggregated results from round 1; and round 3 allowed participants to revise their responses from round 1 based on insights gained during round 2. Questions elicited ratings, rankings, and estimates on access restrictions based on given safety and efficacy information for hemophilia A prophylaxis therapies. Consensus was reached if ≥ 74% of panelists (14 of 19) were within 1 SD of the median group estimate during round 3. RESULTS: 19 Payers participated in the research. Among them, 94% dealt with commercial insurance, 94% with Medicare, and 81% with Medicaid; 79% had spent ≥ 5 years in their current role. Panelists reported limited access restrictions on hemophilia A prophylaxis therapies; the most common restrictions were prior authorization (n = 16, 84%) and quantity level limits (n = 13, 67%). Tiering and step therapy were reported by 7 respondents (39%). Respondents agreed that there was an 80% median likelihood that ≥ 9 additional patients with any safety event (e.g., thrombotic event, death) per year would trigger access restrictions, with the median likelihood of restrictions increasing to 95% for another ≥ 10 patients with safety events per year. Respondents also agreed that > 5 thrombotic events requiring treatment per patient per year would have a 98% median likelihood of leading to access restrictions and that ≥ 5 years of real-world safety and efficacy data would be highly likely (95% median likelihood) to affect coverage decisions. Noncoverage was highly unlikely (ranked fifth or sixth of 6 by 14 respondents), as was no restriction-coverage parity (ranked sixth of 6 by 10 respondents). All else being equal, cost continues to affect access policies, with respondents agreeing that a 13%-30% difference in net cost may lead to preferred formulary treatment for a drug with superior efficacy and noninferior safety, inferior efficacy and noninferior safety, or noninferior efficacy and inferior safety. CONCLUSIONS: Payers prefer treatments with well-understood efficacy, safety, and cost over newer treatments with uncertain long-term effects. Relatively unrestricted access to legacy and new hemophilia A prophylaxis will likely continue unless additional real-world safety concerns or major cost differences emerge. DISCLOSURES: Financial support for this study was provided by Takeda Pharmaceutical Company, which was involved in study concept and design. Graf, Tuly, Harley, and Pednekar are employees of PRECISIONheor, a research consultancy to the health and life sciences industries that was contracted by Takeda to conduct this study and write the manuscript. Batt served as a consultant on this project through PRECISIONheor.

Beyond visual acuity: Patient-relevant assessment measures of visual function in retinal diseases

PURPOSE

To identify patient-reported outcomes (PROs) and other clinical outcome measures (contrast sensitivity (CS), low-luminance visual acuity (LLVA) and reading acuity or reading speed (RA-RS)), relevant to patients with age-related macular degeneration (AMD) or diabetic retinopathy (DR), which would be recommended for use in clinical practice.

METHODS

The RAND/UCLA Appropriateness Method, based on the synthesis of the scientific evidence and the collective judgment of an expert panel using the two-round Delphi method, was applied. The evidence synthesis was performed by searching for articles on outcome measures for AMD and/or DR published between 2005 and 2018 in English or Spanish. The expert panel consisted of 14 Spanish ophthalmologists, who rated the recommendation degree for each outcome measure on a scale of 1 (extremely irrelevant) to 9 (maximum relevance). The recommended outcome measures were established according to the panel median score and the level of the panelists' agreement.

RESULTS

Through the evidence search, 33 PRO-specific questionnaires (21 for visual function, six for AMD, three for DR, one for AMD and DR) and two treatment satisfaction questionnaires (one on AMD and one on DR) were identified. In addition, 21 methods were found for measuring CS, five for LLVA, and nine for RA-RS. According to the panel ratings, 11 of the 64 outcome measures evaluated for AMD, and seven of the 61 evaluated for DR were recommended.

CONCLUSION

The AMD and DR outcome measures recommended will help ophthalmologists choose the outcome measure most appropriate for their patients. Furthermore, the use of PROs will contribute to shifting clinical practice towards patient-centered medicine.

Guidelines in urology: Lights and shadows

Clinical practice guidelines have become increasingly important in the decision-making process in many clinical conditions and have been recognized as key instruments to improve the quality of care. On the one hand, guidelines represent a good tool for improving patient's outcome, and on the other hand, the adherence to guidelines and good practice recommendations is mandatory to reduce the risk of legal disputes. A recent revision by the Italian Parliament of the legal system that rules the responsibilities of health professionals and health care safety stimulated all clinicians to improve their adhesion to clinical guidelines. It is justified by the high-quality level obtained in the recent years by the international guidelines. In the recent years, a revision of the clinical guideline development has been done. In particular, the European Association of Urology Guideline Office changed the "Guidelines for Guidelines," and a rigorous development process has been established. A clinical recommendation is produced after a rigorous methodological process using an analysis of all published clinical trials, and the expert opinion is not yet considered. For oncological guidelines, the adherence to the producing process is easily feasible due to the high number of clinical trials; for non-oncological guidelines, instead, the small number of clinical trials could represent a problem for obtaining recommendation based on rigorous methodology. Here, on the basis of these considerations, we aim to discuss the lights and the shadows of the clinical applicability of guidelines in urology.

The impact of patient preferences and costs on the appropriateness of spinal manipulation and mobilization for chronic low back pain and chronic neck pain

BACKGROUND

Although the delivery of appropriate healthcare is an important goal, the definition of what constitutes appropriate care is not always agreed upon. The RAND/UCLA Appropriateness Method is one of the most well-known and used approaches to define care appropriateness from the clinical perspective-i.e., that the expected effectiveness of a treatment exceeds its expected risks. However, patient preferences (the patient perspective) and costs (the healthcare system perspective) are also important determinants of appropriateness and should be considered.

METHODS

We examined the impact of including information on patient preferences and cost on expert panel ratings of clinical appropriateness for spinal mobilization and manipulation for chronic low back pain and chronic neck pain.

RESULTS

The majority of panelists thought patient preferences were important to consider in determining appropriateness and that their inclusion could change ratings, and half thought the same about cost. However, few actually changed their appropriateness ratings based on the information presented on patient preferences regarding the use of these therapies and their costs. This could be because the panel received information on average patient preferences for spinal mobilization and manipulation whereas some panelists commented that appropriateness should be determined based on the preferences of individual patients. Also, because these therapies are not expensive, their ratings may not be cost sensitive. The panelists also generally agreed that preferences and costs would only impact their ratings if the therapies were considered clinically appropriate.

CONCLUSIONS

This study found that the information presented on patient preferences and costs for spinal mobilization and manipulation had little impact on the rated appropriateness of these therapies for chronic low back pain and chronic neck pain. Although it was generally agreed that patient preferences and costs were important to the appropriateness of M/M for CLBP and CNP, it seems that what would be most important were the preferences of the individual patient, not patients in general, and large cost differentials.

Development of Quality Indicators for the Care of Women with Abnormal Uterine Bleeding by Primary Care Providers in the Veterans Health Administration

BACKGROUND

Abnormal uterine bleeding (AUB) is a common women's health complaint. However, the quality of primary care (PC) management of AUB is unknown. Our objective was to develop quality indicators for Veterans Health Administration (VA) PC assessment and management of AUB.

METHODS

We drafted candidate indicators based on comprehensive review of the scientific literature, including published consensus guidelines. Then, we convened a national panel of nine experts including PC providers, obstetrician-gynecologists, VA policy stakeholders, and quality measurement experts, and used a modified Delphi panel process. First, panelists individually rated 19 candidate indicators, using 9-point scales, on three metrics: consistency with established guidelines, importance to women's health, and reliability of measurement from VA electronic health records. Panelists then discussed the indicators. Finally, panelists re-rated revised indicators using the same metrics. Indicators were selected if final median ratings were ≥7 on each 9-point scale, without dispersion in ratings.

RESULTS

Eighteen indicators were selected. Three focused on assessing need for emergency care (e.g., profuse bleeding or pregnancy). Three addressed ascertaining key aspects of the medical history (e.g., endometrial cancer risk). Two addressed performing a physical examination (e.g., pelvic examination). Six addressed indications for diagnostic studies and specialty care referrals, (e.g., transvaginal ultrasound examination). Four addressed initiation of treatment and counseling (e.g., hormone therapy).

CONCLUSIONS

We developed quality indicators for PC assessment and management of AUB that span reproductive and postmenopausal life phases. Applying these indicators in VA and other health systems with integrated electronic health records can assess need for, and effects of, AUB quality improvement programs.

Defining and Rating the Effectiveness of Enabling Services Using a Multi-stakeholder Expert Panel Approach

The Affordable Care Act provides opportunities to reimburse non-medical enabling services that promote the delivery of medical care for patients with social barriers. However, limited evidence exists to guide delivery of these services. We addressed this gap by convening community health center patients, providers, and other stakeholders in two panels that developed a framework for defining and evaluating these services. We adapted a group consensus method where the panelists rated services for effectiveness in increasing access to, use, and understanding of medical care. Panelists defined six broad categories, 112 services, and 21 variables including the type of provider delivering the service. We identified 16 highest-rated services and found that the service provider's level of training affected effectiveness for some but not all services. In a field with little evidence, these findings provide guidance to decision-makers for the targeted spread of services that enable patients to overcome social barriers to care.

Should obtaining a preoperative audiogram before tympanostomy tube placement be used as a quality metric? A survey of pediatric otolaryngologists

INTRODUCTION

Obtaining a preoperative audiogram prior to tympanostomy tube placement is recommended by the American Academy of Otolaryngology-Head and Neck Surgery clinical practice guideline (CPG): Tympanostomy tubes in Children, and this process measure is also used as a quality metric by payers. However, whether audiograms should be mandated in cases of tube placement for both chronic otitis media with effusion (COME) and recurrent acute otitis media (RAOM) is controversial. The objective of this study is to determine reports of practice patterns of pediatric otolaryngologists regarding obtaining audiograms before and after tympanostomy tube placement and opinions regarding utility of CPGs and use of this process measure as a quality metric.

METHODS

A 16-question cross-sectional survey of American Society of Pediatric Otolaryngology (ASPO) members was conducted. Per ASPO policy, no repeated requests or other enhanced response techniques were permitted. Independent t-tests for proportions were used to compare responses.

RESULTS

127 pediatric otolaryngologists completed the survey (response rate 26.9%). Nearly 70% of respondents reported being in practice for >10 years. 74% of respondents reported obtaining preoperative audiograms "always" or "most of the time" for COME, vs. 56.7% for RAOM (p < 0.0001). 76% agreed that obtaining a preoperative audiogram was representative of high quality for COME, vs. 52% for RAOM (p < 0.0001). 12% of respondents "completely agreed" that compliance with all aspects of CPGs represented high quality, while 68.8% responded that they somewhat agreed.

CONCLUSION

There is no consensus among pediatric otolaryngologists regarding the necessity of a preoperative audiogram in tympanostomy tube placement, especially for RAOM. Further evidence demonstrating the benefit of preoperative audiogram obtainment should be developed prior to inclusion as a guideline recommendation and as a quality metric.

Indications for Use of Damage Control Surgery in Civilian Trauma Patients: A Content Analysis and Expert Appropriateness Rating Study

OBJECTIVES

To characterize and evaluate indications for use of damage control (DC) surgery in civilian trauma patients.

BACKGROUND

Although DC surgery may improve survival in select, severely injured patients, the procedure is associated with significant morbidity, suggesting that it should be used only when appropriately indicated.

METHODS

Two investigators used an abbreviated grounded theory method to synthesize indications for DC surgery reported in peer-reviewed articles between 1983 and 2014 into a reduced number of named, content-characteristic codes representing unique indications. An international panel of trauma surgery experts (n = 9) then rated the appropriateness (expected benefit-to-harm ratio) of the coded indications for use in surgical practice.

RESULTS

The 1107 indications identified in the literature were synthesized into 123 unique pre- (n = 36) and intraoperative (n = 87) indications. The panel assessed 101 (82.1%) of these indications to be appropriate. The indications most commonly reported and assessed to be appropriate included pre- and intraoperative hypothermia (median temperature <34°C), acidosis (median pH <7.2), and/or coagulopathy. Others included 5 different injury patterns, inability to control bleeding by conventional methods, administration of a large volume of packed red blood cells (median >10 units), inability to close the abdominal wall without tension, development of abdominal compartment syndrome during attempted abdominal wall closure, and need to reassess extent of bowel viability.

CONCLUSIONS

This study identified a comprehensive list of candidate indications for use of DC surgery. These indications provide a practical foundation to guide surgical practice while studies are conducted to evaluate their impact on patient care and outcomes.

Influence of Clinical and Economical Factors on the Expert Rating of Appropriateness of Preoperative Use of Recombinant Erythropoietin in Elective Orthopedic Surgery Patients

To evaluate the relative impact of clinical factors and health care environment resources on the expert ratings of appropriateness of preoperative erythropoietin in elective orthopedic surgery, the authors analyzed 6905 individual votes on 496 clinical scenarios from 14 experts, applying a multivariate logistic model. Sixty-six percent of the indications were appropriate when resource constraints (RC) were not considered and 53% when they were, resulting in a drop in the median vote of 2 points on a 9-point scale (P < 0.05). Initial hemoglobin level, expected perioperative blood loss, and RC were by far the most significant contributors to the model (P < 0.01), but other factors (i.e., clinical specialty of the expert, prior history of transfusion reactions, patient age, cardiovascular disease, anemia of chronic disease) also contributed significantly (P< 0.01). For assessing appropriateness of care, this study confirms the need for detailed clinical scenarios and a multidisciplinary panel carefully selected to reflect those involved in the interventions under consideration.

Testing an Adapted Modified Delphi Method

Background. The Affordable Care Act incentivizes health systems for better meeting patient needs, but often guidance about patient preferences for particular health services is limited. All too often vulnerable patient populations are excluded from these decision-making settings. A community-based participatory approach harnesses the in-depth knowledge of those experiencing barriers to health care. Method. We made three modifications to the RAND-UCLA appropriateness method, a modified Delphi approach, involving patients, adding an advisory council group to characterize existing knowledge in this little studied area, and using effectiveness rather than “appropriateness” as the basis for rating. As a proof of concept, we tested this method by examining the broadly delivered but understudied nonmedical services that community health centers provide. Results. This method created discrete, new knowledge about these services by defining 6 categories and 112 unique services and by prioritizing among these services based on effectiveness using a 9-point scale. Consistent with the appropriateness method, we found statistical convergence of ratings among the panelists. Discussion. Challenges include time commitment and adherence to a clear definition of effectiveness of services. This diverse stakeholder engagement method efficiently addresses gaps in knowledge about the effectiveness of health care services to inform population health management.

Acceptance of inflammatory bowel disease treatment recommendations based on appropriateness ratings: do practicing gastroenterologists agree with experts?

BACKGROUND

Appropriateness criteria for the treatment of Crohn’s disease (CD) and ulcerative colitis (UC) have been developed by expert panels. Little is known about the acceptance of such recommendations by care providers. The aim was to explore how treatment decisions of practicing gastroenterologists differ from those of experts, using a vignette case study and a focus group.

METHODS

Seventeen clinical vignettes were drawn from clinical indications evaluated by the expert panel. A vignette case questionnaire asking for treatment options in 9 or 10 clinical situations was submitted to 26 practicing gastroenterologists. For each vignette case, practitioners’ answers on treatments deemed appropriate were compared with panel decisions. Qualitative analysis was performed on focus group discussion to explore acceptance and divergence reasons.

RESULTS

Two hundred thirty-nine clinical vignettes were completed, 98 for CD and 141 for UC.Divergence between proposed treatments and panel recommendations was more frequent for CD (34%) than for UC (27%). Among UC clinical vignettes, the main divergences with the panel were linked to 5-aminosalicylate (5-ASA) failure assessment and to situations in which stopping treatment was the main decision. For CD, the propositions of care providers diverged from the panel in mild to moderate active disease, for which practitioners were more prone to an accelerated step-up than the panel’s recommendations.

CONCLUSIONS

In about one-third of vignette cases, inflammatory bowel disease treatment propositions made by practicing gastroenterologists diverged from expert recommendations. Practicing gastroenterologists may experience difficulty in applying recommendations in daily practice.

The effect of clinical decision support for advanced inpatient imaging

PURPOSE

To examine the effect of integrating point-of-care clinical decision support (CDS) using the ACR Appropriateness Criteria (AC) into an inpatient computerized provider order entry (CPOE) system for advanced imaging requests.

METHODS

Over 12 months, inpatient CPOE requests for nuclear medicine, CT, and MRI were processed by CDS to generate an AC score using provider-selected data from pull-down menus. During the second 6-month period, AC scores were displayed to ordering providers, and acknowledgement was required to finalize a request. Request AC scores and percentages of requests not scored by CDS were compared among primary care providers (PCPs) and specialists, and by years in practice of the responsible physician of record.

RESULTS

CDS prospectively generated a score for 26.0% and 30.3% of baseline and intervention requests, respectively. The average AC score increased slightly for all requests (7.2 ± 1.6 versus 7.4 ± 1.5; P < .001), for PCPs (6.9 ± 1.9 versus 7.4 ± 1.6; P < .001), and minimally for specialists (7.3 ± 1.6 versus 7.4 ± 1.5; P < .001). The percentage of requests lacking sufficient structured clinical information to generate an AC score decreased for all requests (73.1% versus 68.9%; P < .001), for PCPs (78.0% versus 71.7%; P < .001), and for specialists (72.9% versus 69.1%; P < .001).

CONCLUSIONS

Integrating CDS into inpatient CPOE slightly increased the overall AC score of advanced imaging requests as well as the provision of sufficient structured data to automatically generate AC scores. Both effects were more pronounced in PCPs compared with specialists.

Establishing a quality measurement system for cancer care in Japan

Ensuring the quality of care is a major objective of cancer control policy. The Cancer Control Act 2006 placed responsibility on the Japanese government to maintain the quality of cancer care nationwide. To function as centers providing high-quality care, designated cancer care hospitals (397 hospitals as of April 2012) were instituted nationwide. Although they meet the structural standards, such as the presence of radiation equipment and palliative care teams, it remains unclear whether the designation has led to appropriate provision of care and optimal patient outcomes. A national system to examine the processes and outcomes of cancer care is under development. In 2007 and 2008, the Japanese Association of Clinical Cancer Centers publicly disclosed the 5-year survival of their member facilities with strict data quality standards, including sufficient follow-up of patients' vital status. The network of designated cancer care hospitals will follow this lead to provide a national outcome monitoring system. The processes of care have also been addressed by a government-funded research project. With the collaboration of clinical experts, 206 quality indicators have been developed for five major cancers in Japan (breast, colorectal, liver, lung and stomach) and palliative care. Each indicator described the target patients and standards of care for the patients, the provision of which was considered an aspect of quality. In 2012, the Cancer Registry Chapter of the Association of Prefectural Designated Cancer Care Hospitals instituted quality measurement using these indicators. These activities will soon lead to effective quality monitoring and improvement in Japan.

Quality indicators of clinical cancer care (QC3) in colorectal cancer

OBJECTIVES

Assessing the quality of cancer care (QoCC) has become increasingly important to providers, regulators and purchasers of care worldwide. The aim of this study was to develop evidence-based quality indicators (QIs) for colorectal cancer (CRC) to be applied in a population-based setting.

DESIGN

A comprehensive evidence-based literature search was performed to identify the initial list of QIs, which were then selected and developed using a two-step-modified Delphi process involving two multidisciplinary expert panels with expertise in CRC care, quality of care and epidemiology.

SETTING

The QIs of the clinical cancer care (QC3) population-based project, which involves all the public and private hospitals and clinics present on the territory of Canton Ticino (South Switzerland).

PARTICIPANTS

Ticino Cancer Registry, The Colorectal Cancer Working Group (CRC-WG) and the external academic Advisory Board (AB).

MAIN OUTCOME MEASURES

Set of QIs which encompass the whole diagnostic-treatment process of CRC.

RESULTS

Of the 149 QIs that emerged from 181 sources of literature, 104 were selected during the in-person meeting of CRC-WG. During the Delphi process, CRC-WG shortened the list to 89 QI. AB finally validated 27 QIs according to the phase of care: diagnosis (N=6), pathology (N=3), treatment (N=16) and outcome (N=2).

CONCLUSIONS

Using the validated Delphi methodology, including a literature review of the evidence and integration of expert opinions from local clinicians and international experts, we were able to develop a list of QIs to assess QoCC for CRC. This will hopefully guarantee feasibility of data retrieval, as well as acceptance and translation of QIs into the daily clinical practice to improve QoCC. Moreover, evidence-based selected QIs allow one to assess immediate changes and improvements in the diagnostic-therapeutic process that could be translated into a short-term benefit for patients with a possible gain both in overall and disease-free survival.

Appraisal of WHO guidelines in maternal health using the AGREE II assessment tool

In 2007, the World Health Organization (WHO) received a criticism for a lack of transparency and systematic methods in the development of guidelines, which were at that time perceived as substantially driven by expert opinion. In this paper we assessed the quality of maternal and perinatal health guidelines developed since then. We used the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool to evaluate the quality of methodological rigour and transparency of four different WHO guidelines published between 2007 and 2011. Our findings showed high scores among the most recent guidelines on maternal and perinatal health suggesting higher quality. However, there is still potential for improvement, especially in including different stakeholder views, transparency of guidelines regarding the role of the funding body and presentation of the guideline document.

Telemedicine: technology mediated service relationship, encounter, or something else?

PURPOSE

Service interactions between service providers and health care consumers happen daily in health care organizations, and can occur face-to-face or through mediating technology. We use the demanding and rich environment of telemedicine to better understand the nature of the real time service-encounter interactions among the human and technology actors engaged in the process and to inform telemedicine providers about key factors to consider in telemedicine design.

METHODS

We conducted a case study of medical video conferencing (MVC) for the delivery of patient healthcare (a form of telemedicine) using multiple data collection and analysis techniques involving a range of telemedicine stakeholders.

RESULTS

The research reveals that telemedicine requires a new kind of service relationship, an Advanced Encounter, with unique relationships between the telemedicine service providers, presenters, patients, and technology. Seven facilitating factors for the Advanced Encounter of telemedicine are identified and discussed, including the telemedicine servicescape: a set of supporting structures that are critical to telemedicine success.

CONCLUSIONS

Key contributions are a deep understanding of the relationships between telemedicine actors, and the organizational actions needed to deploy a technology-mediated telemedicine service.

Cancer risks associated with external radiation from diagnostic imaging procedures

The 600% increase in medical radiation exposure to the US population since 1980 has provided immense benefit, but increased potential future cancer risks to patients. Most of the increase is from diagnostic radiologic procedures. The objectives of this review are to summarize epidemiologic data on cancer risks associated with diagnostic procedures, describe how exposures from recent diagnostic procedures relate to radiation levels linked with cancer occurrence, and propose a framework of strategies to reduce radiation from diagnostic imaging in patients. We briefly review radiation dose definitions, mechanisms of radiation carcinogenesis, key epidemiologic studies of medical and other radiation sources and cancer risks, and dose trends from diagnostic procedures. We describe cancer risks from experimental studies, future projected risks from current imaging procedures, and the potential for higher risks in genetically susceptible populations. To reduce future projected cancers from diagnostic procedures, we advocate the widespread use of evidence-based appropriateness criteria for decisions about imaging procedures; oversight of equipment to deliver reliably the minimum radiation required to attain clinical objectives; development of electronic lifetime records of imaging procedures for patients and their physicians; and commitment by medical training programs, professional societies, and radiation protection organizations to educate all stakeholders in reducing radiation from diagnostic procedures.

Clinical factors associated with the non-utilization of an anaesthesia incident reporting system

BACKGROUND

Incident reporting is a widely recommended method to measure undesirable events in anaesthesia. Under-utilization is a major weakness of voluntary incident reporting systems. Little is known about factors influencing reporting practices, particularly the clinical environment, anaesthesia team composition, severity of the incident, and perceived risk of litigation. The purpose of this study was to assess each of these, using an existing anaesthesia database.

METHODS

We performed a retrospective cohort study and analysed 46 207 surgical patients. We used multivariate analysis to identify factors associated with the non-utilization of the reporting system.

RESULTS

We found that in 7022 (15.1%) of the procedures performed, the incident reporting system was not used. Factors associated with the non-use of the system were regional anaesthesia/local anaesthesia, odds ratio (OR) 1.64 [95% confidence interval (CI) 1.03-2.62], emergency procedures OR 1.15 (95% CI: 1.05-1.27), and a consultant anaesthetist working without a trainee, OR 1.71 (95% CI: 1.03-2.82). In contrast, factors such as longer duration of surgery, OR 0.85 (95% CI: 0.76-0.94), the presence of a senior anaesthesia trainee, OR 0.86 (95% CI: 0.81-0.92), and the occurrence of severe complications with a high risk of litigation (i.e. death, nerve injuries) were less associated with a non-use of the reporting system, OR 0.65 (95% CI: 0.44-0.97). Team composition and time of day had no measurable impact on reporting practices.

CONCLUSIONS

Clinical factors play a significant role in the utilization of an anaesthesia incident reporting system and more particularly, severity of complications and higher liability risks which appear more as incentives than barriers to incident reporting.

The current state of evidence-based protocols in wound care

BACKGROUND

Clinical practice guidelines have blossomed in the last 10 years in medicine as well as wound care. The physician practicing wound care and attempting to use published clinical practice guidelines may, however, have difficulty judging the quality of these guidelines given legitimate concerns that many aspects of clinical practice guidelines are not being properly addressed.

METHODS

Guidelines were located using the National Guideline Clearinghouse Web site, PubMed, and the Cochrane database for reviews on diabetic foot ulcers, venous ulcers, and pressure ulcers. The Appraisal of Guidelines for Research and Evaluation instrument was used to evaluate guidelines.

RESULTS

Search engines returned many irrelevant guidelines. Many guidelines would be difficult to evaluate by clinicians not versed in guideline evaluation and were cumbersome in format or were presented more as reference works. Too little attention is focused on such issues as clarity of presentation, consideration of multidisciplinary panels, stakeholder involvement, validity, testing, settings, resources required, cost impact, methods of addressing guideline implementation, and a means of tracking important criteria for feedback once the guideline is in the field. The venous and diabetic ulcer guidelines that were formally evaluated scored poorly in many of these areas despite using relatively sound methods for gathering and evaluating the evidence. Only the developers of one guideline made a commitment for regular update.

CONCLUSION

Although progress has been made in regard to wound care clinical practice guidelines, much more work will be required before such guidelines are highly accepted by wound care clinicians.

Lessons learned in the development of process quality indicators for cancer care in Japan

In Japan, attention has increasingly focused on ensuring the quality of care, particularly in the area of cancer care. The 2006 Basic Cancer Control Act reinforced efforts to ensure the quality of cancer care in a number of sectors, including the role of government in ensuring quality. We initiated a government-funded research project to develop quality indicators to measure the quality of care for five major cancers (breast, lung, stomach, colorectal, and liver cancer) in Japan, and palliative care for cancers in general. While we successfully developed a total of 206 quality indicators, a number of issues have been raised regarding the concepts and methodologies used to measure quality. Examples include the choice between measuring the process of care versus the outcome of care; the degree to which the process-outcome link should be confirmed in real-world measurement; handling of exceptional cases; interpretation of measurement results between quality of care versus quality of documentation; creation of summary scores; and the optimal number of quality indicators for measurement considering the trade-off between the measurement validity versus resource limitations. These and other issues must be carefully considered when attempting to measure quality of care, and although many appear to have no correct answer, continuation of the project requires that a decision nevertheless be made. Future activities in this project, which is still ongoing, should focus on the further exploration of these problems.

Evaluation and revision of checklists for screening facilities and municipal governmental programs for gastric cancer and colorectal cancer screening in Japan

OBJECTIVE

To evaluate the appropriateness of current checklists created by a governmental committee to assess screening programs run by municipal governments and service provider facilities for gastric and colorectal cancer, and to accumulate expert opinions to provide insights aimed at the next revision.

METHODS

We convened an expert panel that consisted of physicians nominated by regional offices of the Japanese Society for Gastrointestinal Cancer Screening and radiology technicians nominated by the technician chapter of the society. The panel rated the appropriateness of each checklist item on a scale of 1-9 (1, extremely inappropriate; 9, extremely appropriate) twice, between which they had a face-to-face discussion meeting. During the process they were allowed to propose modifications and additions to the items.

RESULTS

In the first round of rating, the panelists rated all 57 and 56 checklists items for gastric and colorectal cancer, respectively, as appropriate based on an acceptance rule determined a priori. During the process of the face-to-face discussion, however, the panel proposed modifications to 23 (40%) and 22 (39%) items, respectively, and the addition of 27 new items each. After integrating overlapping items and rating again for appropriateness, 66 and 64 items, respectively, were accepted as the revised checklist set.

CONCLUSIONS

The expert panel considered current checklists for colorectal and gastric cancer-screening programs and facilities to be suitable. Their proposals for a new set of checklist items will help further improve the checklists.

The impact of cardiac CT on the appropriate utilization of catheter coronary angiography

The purpose of this study was to evaluate the impact of computed tomography coronary angiography (CTCA) on the appropriate utilization of catheter angiography (CA). This observational trial analyzed all patients undergoing CA in 2006 and 2007 in one hospital. In 2007, patients having a low to intermediate cardiovascular risk and suspicion of coronary artery disease (CAD) and those with suspicion of progression of known organic heart disease (OHD) underwent CTCA either prior to CA or as the sole imaging modality. Appropriate utilization of CA was defined as: (1) percentage of patients showing normal or non-significant findings at CA, (2) percentage of self-referred patients to CA, and (3) percentage of patients with known OHD undergoing CA without immediate operative or interventional consequences. Use of CTCA resulted in a significant drop in the percentage of CA examinations in patients with suspected CAD showing normal or non-significant findings (19% in 2006, 10% in 2007, P < 0.001). The percentage of self-referred CA significantly dropped (29% in 2006, 10% in 2007, P < 0.001). CT ruled-out CAD in 74/151 (49%) patients, obviating subsequent CA. During a follow-up of 15 +/- 4 months, CA and percutaneous interventions was considered necessary in 2/74 patients. CT ruled-out progression of known OHD in 53/60 (90%) patients, while one patient underwent CA and percutaneous intervention during a follow-up period of 16 +/- 4 months. No reduction of CA examinations without immediate consequences was found in patients with known OHD (13% in 2006, 27% in 2007). In patients with suspicion of CAD, CTCA improved the appropriate utilization of CA without jeopardizing patient safety, along with a decrease of self-referred patients for CA. CTCA did not influence the appropriate utilization of CA in patients with known OHD.

Does managed care affect quality? Appropriateness, referral patterns, and outcomes of carotid endarterectomy

This was a population-based observational study to assess the impact of managed care (MC) on several dimensions of quality of surgical care among Medicare beneficiaries undergoing carotid endarterectomies (CEAs) (N = 9308) in New York. Clinical data were abstracted from medical charts to assess appropriateness and deaths or strokes within 30 days of surgery. Differences in patients, appropriateness, and outcomes were compared using chi-square tests; risk-adjusted outcomes were compared using regression. Fee-For-Service (FFS, N = 8691) and MC (N = 897) CEA patients had similar indications for surgery, perioperative risk, and comorbidities. There were no differences in inappropriateness between FFS and MC (8.6% vs 8.4%). MC patients were less likely to use a high-volume surgeon (20.1% vs 13.5%) or hospital (20.5% vs 13.0%, P < .05). There were no differences in risk-adjusted rates of death or stroke (OR = 0.97; 95% CI = 0.69-1.37). Medicare MC plans did not have a positive impact on inappropriateness, referral patterns, or outcomes of CEA.

Appropriateness of healthcare interventions: Concepts and scoping of the published literature

Objectives:This report is a scoping review of the literature with the objective of identifying definitions, conceptual models and frameworks, as well as the methods and range of perspectives, for determining appropriateness in the context of healthcare delivery.

Methods:To lay groundwork for future, intervention-specific research on appropriateness, this work was carried out as a scoping review of published literature since 1966. Two reviewers, with two screens using inclusion/exclusion criteria based on the objective, focused the research and articles chosen for review.

Results:The first screen examined 2,829 abstracts/titles, with the second screen examining 124 full articles, leaving 37 articles deemed highly relevant for data extraction and interpretation. Appropriateness is defined largely in terms of net clinical benefit to the average patient and varies by service and setting. The most widely used method to assess appropriateness of healthcare services is the RAND/UCLA Model. There are many related concepts such as medical necessity and small-areas variation.

Conclusions:A broader approach to determining appropriateness for healthcare interventions is possible and would involve clinical, patient and societal perspectives.

Developing Clinical Recommendations for Breast, Colorectal, and Lung Cancer Adjuvant Treatments Using the GRADE System: A Study From the Programma Ricerca e Innovazione Emilia Romagna Oncology Research Group

Purpose

In the area of anticancer drugs, the legitimate search for effective interventions can be jeopardized by the strong pressure for accelerated approval, which may hinder the full assessment of their benefit-risk profile. We aimed to produce drug-specific recommendations using an explicit approach that separates the judgments on quality of evidence from the judgment about strength of recommendations.

Materials and Methods

We used the GRADE (Grades of Recommendation, Assessment, Development, and Evaluation) system to develop recommendations for the use of specific anticancer drugs/regimens; 12 clinical questions relevant to adjuvant treatment of breast (three), colorectal (four) and lung (five) cancer have been assessed by multidisciplinary panels supported by a group of methodologists.

Results

For nine of 12 questions, recommendations were produced (one strong and six weak in favor and one weak and one strong against the index treatment); for the remaining three questions no specific course of action could be recommended. The perceived benefits to risk balance of the treatment was the most important and statistically significant (P < .01) predictor of panels’ recommendations and of their strength, whereas panelists’ personal (age, sex) and professional (specialty) characteristics were not statistically associated.

Conclusion

Because the GRADE system sets out an explicit process going from evaluation of the quality of evidence and benefit-risk profile to the judgment of the strength of recommendations, in this experience, it proved very useful to combine methodologic rigor with the interdisciplinary participation that is important in the definition of evidence based clinical policies.

Fostering knowledge exchange between researchers and decision-makers: exploring the effectiveness of a mixed-methods approach

OBJECTIVES

Knowledge exchange is thought to enhance research utilization by decision-makers but there is little guidance on appropriate methods. This study evaluated the effectiveness of a research planning exercise utilizing technical (ranking, rating) and interpretive strategies (interdisciplinary workshop deliberation).

METHODS

Participants were surveyed to establish research priorities and professional roles. Observation was used to examine actual contribution and outcomes. Semi-structured interviews with participants elicited perceived outcomes, commitment, contribution and learning. Survey data was reported with summary statistics. Transcripts were analyzed qualitatively.

RESULTS

Stakeholders were satisfied with the overall process, gaps in research were prioritized, and research questions were proposed, but anticipated intermediate or lateral outcomes were not achieved. Identifying differing perspectives and establishing relationships were unanticipated outcomes. Barriers included group dynamics, lack of clarity on objectives and processes, and minimal experience or interest in interpretive activities.

CONCLUSIONS

A conceptual framework for evaluating factors influencing knowledge exchange outcomes had not been previously investigated. Strategies for overcoming identified barriers include better facilitation, involving a critical volume of non-clinicians, in-person sharing of background information, and incentives for decision-makers. Further research is required to examine the effectiveness of different forms of knowledge exchange, and the degree to which they are currently being practiced.

The quality of communication about older patients between hospital physicians and general practitioners: a panel study assessment

Background

Optimal care of patients is dependent on good professional interaction between general practitioners and general hospital physicians. In Norway this is mainly based upon referral and discharge letters. The main objectives of this study were to assess the quality of the written communication between physicians and to estimate the number of patients that could have been treated at primary care level instead of at a general hospital.

Methods

This study comprised referral and discharge letters for 100 patients above 75 years of age admitted to orthopaedic, pulmonary and cardiological departments at the city general hospital in Trondheim, Norway. The assessments were done using a Delphi technique with two expert panels, each with one general hospital specialist, one general practitioner and one public health nurse using a standardised evaluation protocol with a visual analogue scale (VAS). The panels assessed the quality of the description of the patient's actual medical condition, former medical history, signs, medication, Activity of Daily Living (ADL), social network, need of home care and the benefit of general hospital care.

Results

While information in the referral letters on actual medical situation, medical history, symptoms, signs and medications was assessed to be of high quality in 84%, 39%, 56%, 56% and 39%, respectively, the corresponding information assessed to be of high quality in discharge letters was for actual medical situation 96%, medical history 92%, symptoms 60%, signs 55% and medications 82%. Only half of the discharge letters had satisfactory information on ADL. Some two-thirds of the patients were assessed to have had large health benefits from the general hospital care in question. One of six patients could have been treated without a general hospital admission. The specialists assessed that 77% of the patients had had a large benefit from the general hospital care; however, the general practitioners assessment was only 59%. One of four of the discharge letters did not describe who was responsible for follow-up care.

Conclusion

In this study from one general hospital both referral and discharge letters were missing vital medical information, and referral letters to such an extent that it might represent a health hazard for older patients. There was also low consensus between health professionals at primary and secondary level of what was high benefit of care for older patients at a general hospital.

Use of evidence in WHO recommendations

BACKGROUND

WHO regulations, dating back to 1951, emphasise the role of expert opinion in the development of recommendations. However, the organisation's guidelines, approved in 2003, emphasise the use of systematic reviews for evidence of effects, processes that allow for the explicit incorporation of other types of information (including values), and evidence-informed dissemination and implementation strategies. We examined the use of evidence, particularly evidence of effects, in recommendations developed by WHO departments.

METHODS

We interviewed department directors (or their delegates) at WHO headquarters in Geneva, Switzerland, and reviewed a sample of the recommendation-containing reports that were discussed in the interviews (as well as related background documentation). Two individuals independently analysed the interviews and reviewed key features of the reports and background documentation.

FINDINGS

Systematic reviews and concise summaries of findings are rarely used for developing recommendations. Instead, processes usually rely heavily on experts in a particular specialty, rather than representatives of those who will have to live with the recommendations or on experts in particular methodological areas.

INTERPRETATION

Progress in the development, adaptation, dissemination, and implementation of recommendations for member states will need leadership, the resources necessary for WHO to undertake these processes in a transparent and defensible way, and close attention to the current and emerging research literature related to these processes.

Clinical presentation of myocardial infarction contributes to lower use of coronary angiography in patients with chronic kidney disease

Patients with chronic kidney disease (CKD) have high mortality following myocardial infarction (MI), but are less likely to undergo coronary angiography than those without CKD. Whether this phenomenon is explained by differences in the presentation of MI or by bias against performing coronary angiography in patients with CKD is unclear. We examined the clinical presentation of 1876 elderly patients who presented with MI and categorized them by estimated glomerular filtration rate: >60 ml/min (no/mild CKD), 30-60 ml/min (CKD Stage 3) or <30 ml/min (CKD Stage 4/5). Compared with patients with no/mild CKD, patients with CKD Stage 3 or Stage 4/5 had more comorbidity, greater prior nursing home use, and higher frequency of conduction abnormalities or anterior infarction. By contrast, peak creatinine kinase-MB fraction (CK-MB) concentrations were lower and ST-elevation MI was less common in patients with CKD Stage 3 or Stage 4/5. In univariate analyses, patients with CKD Stage 4/5 (odds ratio (OR)=0.34, 95% confidence interval (CI): 0.23-0.50) or Stage 3 (OR=0.57, 95% CI: 0.45-0.73) were markedly less likely to undergo angiography than subjects with no/mild CKD. After multivariable adjustment, the association of CKD Stage 3 with the use of coronary angiography was attenuated (OR=0.78, 95% CI: 0.60-1.03), but CKD Stage 4/5 remained strongly associated with lower use (OR=0.52, 95% CI: 0.34-0.80). Clinical features of MI are different in patients with and without CKD and may partly explain the low use of angiography in patients with CKD Stage 3. However, the clinical features of MI do not account for its underuse in MI patients with CKD Stages 4/5. Whether reduced use of angiography in patients with advanced CKD is justified must be evaluated in formal risk-benefit analyses.

Improving the use of research evidence in guideline development: 3. Group composition and consultation process

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the third of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

Objective

In this review we address the composition of guideline development groups and consultation processes during guideline development.

Methods

We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Key questions and answers

What should be the composition of a WHO-panel that is set up to develop recommendations?

The existing empirical evidence suggests that panel composition has an impact on the content of the recommendations that are made. There is limited research evidence to guide the exact composition of a panel. Based on logical arguments and the experience of other organisations we recommend the following:

• Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers.

• Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing.

• Groups should include or have access to content experts.

• To work well a group needs an effective leader, capable of guiding the group in terms of the task and process, and capable of facilitating collaboration and balanced contribution from all of the group members.

• Because many group members will not be familiar with the methods and processes that are used in developing recommendations, groups should be offered training and support to help ensure understanding and facilitate active participation.

What groups should be consulted when a panel is being set up?

We did not identify methodological research that addressed this question, but based on logical arguments and the experience of other organisations we recommend that as many relevant stakeholder groups as practical should be consulted to identify suitable candidates with an appropriate mix of perspectives, technical skills and expertise, as well as to obtain a balanced representation with respect to regions and gender.

What methods should WHO use to ensure appropriate consultations?

We did not find any references that addressed issues related to this question. Based on logical arguments and the experience of other organisations we believe that consultations may be desirable at several stages in the process of developing guidelines or recommendations, including:

• Identifying and setting priorities for guidelines and recommendations

• Commenting on the scope of the guidelines or recommendations

• Commenting on the evidence that is used to inform guidelines or recommendations

• Commenting on drafts of the guidelines or recommendations

• Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations.

• Key stakeholder organisations should be contacted directly whenever possible.

• Consultation processes should be transparent and should encourage feedback from interested parties.

Geographic variation in the appropriate use of cesarean delivery

There is enormous geographic variation in the use of cesarean delivery: For births over 2,500 grams, adjusted cesarean rates vary fourfold between low- and high-use areas. Even for births under 2,500 grams, high-use counties have rates that are double those of low-use ones. Higher cesarean rates are only partially explained by patient characteristics but are greatly influenced by nonmedical factors such as provider density, the capacity of the local health care system, and malpractice pressure. Areas with higher usage rates perform the intervention in medically less appropriate populations-that is, relatively healthier births-and do not see improvements in maternal or neonatal mortality.

A comparison of two consensus methods for classifying morbidities in a single professional group showed the same outcomes

OBJECTIVE

To investigate whether consensus differs when reached by the Nominal or the Delphi method.

STUDY DESIGN AND SETTING

Seventeen general practices from North Staffordshire, England were randomly allocated to Delphi (postal feedback only) or Nominal group (also had group discussion). General practitioners classified 56 morbidities according to four scales of severity (chronicity, time course, health care use, patient impact) in two consensus rounds. Consensus outcomes were assessed by between-group comparison of severity scores at baseline and follow-up rounds, and consensus process by within-group change in the variance of severity scores between the two rounds.

RESULTS

Consensus rounds were completed by 21 out of 35 Nominal GPs and 23 out of 43 Delphi GPs. Baseline scores for three of the four severity scales were significantly higher for Nominal compared to Delphi GPs, but there were no differences at follow-up. Between the two rounds, variance reduced within the Nominal and Delphi group, respectively, by 61% and 35% (chronicity), 40% and 62% (time course), 42% and 36% (health care use), and 19% and 38% (patient impact).

CONCLUSION

The Nominal and Delphi methods did not result in different outcomes and we conclude that either method can be used in health services research.

Development of measures of the quality of emergency department care for children using a structured panel process

BACKGROUND

Performance measures are essential components of public reporting and quality improvement. To date, few such measures exist to provide a comprehensive assessment of the quality of emergency department services for children.

OBJECTIVES

Our goal was to use a systematic process to develop measures of emergency department care for children (0-19 years) that are (1) based on research evidence and expert opinion, (2) representative of a range of conditions treated in most emergency departments, (3) related to links between processes and outcomes, and (4) feasible to measure.

METHODS

We presented a panel of providers and managers data from emergency department use to identify common conditions across levels of patient acuity, which could be targets for quality improvement. We used a structured panel process informed by a literature review to (1) identify condition-specific links between processes of care and defined outcomes and (2) select indicators to assess these process-outcome links. We determined the feasibility of calculating these indicators using an administrative data set of emergency department visits for Ontario, Canada.

RESULTS

The panel identified 18 clinical conditions for indicator development and 61 condition-specific links between processes of care and outcomes. After 2 rounds of ratings, the panel defined 68 specific clinical indicators for the following conditions: adolescent mental health problems, ankle injury, asthma, bronchiolitis, croup, diabetes, fever, gastroenteritis, minor head injury, neonatal jaundice, seizures, and urinary tract infections. Visits for these conditions account for 23% of all pediatric emergency department use. Using an administrative data set, we were able to calculate 19 indicators, covering 9 conditions, representing 20% of all emergency department visits by children.

CONCLUSIONS

Using a structured panel process, data on emergency department use, and literature review, it was possible to define indicators of emergency department care for children. The feasibility of these indicators will depend on the availability of high-quality data.

Development of ovarian cancer surgery quality indicators using a modified Delphi approach

OBJECTIVE

Little performance measurement has been undertaken in the area of oncology, particularly for surgery which is a pivotal event in the continuum of cancer care. This work was conducted to develop indicators of quality ovarian cancer surgery using a modified three-step Delphi approach.

METHODS

A multidisciplinary panel, comprised of surgical and methodologic co-chairs, nine surgeons, one medical oncologist, one radiation oncologist, a nurse, and a pathologist, reviewed potential indicators extracted from the medical literature through two consecutive rounds of rating followed by consensus discussion. The panel then prioritized the indicators selected in the previous two rounds.

RESULTS

Of 33 possible indicators that emerged from 41 selected articles, 14 were prioritized by the panel as benchmarks for assessing the quality of surgical care. The 14 indicators represent three levels of measurement (provincial/regional, hospital, individual provider) across several phases of care (diagnosis, surgery, pathology, and adjuvant therapy), as well as broad measures of access and outcomes. Some of the indicators selected by the panel were also recommended as standards of care by national initiatives in other countries.

CONCLUSIONS

A systematic evidence- and consensus-based approach was used to develop quality indicators of ovarian cancer care, with a focus on pre-, peri-, and postoperative care as well as outcomes, that are applicable in any jurisdiction.

An experimental study of determinants of group judgments in clinical guideline development

BACKGROUND

Clinical guidelines for improving the quality of care are a familiar part of clinical practice. Formal consensus methods such as the nominal group technique are often used as part of guideline development, but little is known about factors that affect the statements produced by nominal groups, and on their consistency with the research evidence.

METHODS

Cognitive behavioural therapy, behavioural therapy, brief psychodynamic interpersonal therapy, and antidepressants for irritable bowel syndrome, chronic fatigue syndrome, and chronic back pain were selected for study. 16 nominal groups in a factorial design allowed comparison of GP-only with mixed groups of GPs and specialists, provision of a literature review with no provision, and ratings made in the context of realistic or ideal levels of health-care resources. Participants rated appropriateness independently, and again after a facilitated meeting. Audiotapes of four group discussions were analysed.

FINDINGS

There was agreement with the research evidence for 51% of 192 scenarios. Agreement was more likely if the group was GP-only, if a literature review was provided, or if the evidence was in accordance with clinicians' beliefs. Assumptions about the level of resources available had no impact. Clinical and social cues had mixed effects, irrespective of the research evidence. Qualitative analysis showed the modifying effect of clinical experience and beliefs about research evidence.

INTERPRETATION

Guidelines cannot be based on data alone; judgment is unavoidable. The nominal group technique is a method of eliciting and aggregating judgments in a transparent and structured way. It can provide important information on levels of agreement between experts. However, conclusions can be at odds with the published literature. If they are, reasons need to be explicit.

An audit tool for assessing the appropriateness of carotid endarterectomy

Background

To update appropriateness ratings for carotid endarterectomy using the best clinical evidence and to develop a tool to audit the procedure's use.

Methods

A nine-member expert panel drawn from all the Canadian Specialist societies that are involved in the care of patients with carotid artery disease, used the RAND Appropriateness Methodology to rate scenarios where carotid endarterectomy may be performed. A 9-point rating scale was used that permits the categorization of the use of carotid endarterectomy as appropriate, uncertain, or inappropriate. A descriptive analysis was undertaken of the final results of the panel meeting. A database and code were then developed to rate all carotid endarterectomies performed in a Western Canadian Health region from 1997 to 2001.

Results

All scenarios for severe symptomatic stenosis (70–99%) were determined to be appropriate. The ratings for moderate symptomatic stenosis (50–69%) ranged from appropriate to inappropriate. It was never considered appropriate to perform endarterectomy for mild stenosis (0–49%) or for chronic occlusions. Endarterectomy for asymptomatic carotid disease was thought to be of uncertain benefit at best. The majority of indications for the combination of endarterectomy either prior to, or at time of coronary artery bypass grafting were inappropriate. The audit tool classified 98.0% of all cases.

Conclusions

These expert panel ratings, based on the best evidence currently available, provide a comprehensive and updated guide to appropriate use of carotid endarterectomy. The resulting audit tool can be downloaded by readers from the Internet and immediately used for hospital audits of carotid endarterectomy appropriateness.

Reliability of clinical guideline development using mail-only versus in-person expert panels

BACKGROUND

Clinical practice guidelines quickly become outdated. One reason they might not be updated as often as needed is the expense of collecting expert judgment regarding the evidence. The RAND-UCLA Appropriateness Method is one commonly used method for collecting expert opinion. We tested whether a less expensive, mail-only process could substitute for the standard in-person process normally used.

METHODS

We performed a 4-way replication of the appropriateness panel process for coronary revascularization and hysterectomy, conducting 3 panels using the conventional in-person method and 1 panel entirely by mail. All indications were classified as inappropriate or not (to evaluate overuse), and coronary revascularization indications were classified as necessary or not (to evaluate underuse). Kappa statistics were calculated for the comparison in ratings from the 2 methods.

RESULTS

Agreement beyond chance between the 2 panel methods ranged from moderate to substantial. The kappa statistic to detect overuse was 0.57 for coronary revascularization and 0.70 for hysterectomy. The kappa statistic to detect coronary revascularization underuse was 0.76. There were no cases in which coronary revascularization was considered inappropriate by 1 method, but necessary or appropriate by the other. Three of 636 (0.5%) hysterectomy cases were categorized as inappropriate by 1 method but appropriate by the other.

CONCLUSIONS

The reproducibility of the overuse and underuse assessments from the mail-only compared with the conventional in-person conduct of expert panels in this application was similar to the underlying reproducibility of the process. This suggests a potential role for updating guidelines using an expert judgment process conducted entirely through the mail.

Race and Gender Disparities in Rates of Cardiac Revascularization

BACKGROUND

Numerous studies have documented substantial differences by race and gender in the use of coronary artery bypass graft surgery and percutaneous coronary angioplasty. However, few studies have examined whether these differences reflect problems in quality of care.

METHOD

We selected a random sample stratified by gender, race, and income of 5026 Medicare beneficiaries aged 65 to 75 who underwent inpatient coronary angiography during 1991 to 1992 in 1 of 5 states. We compared the frequency of 2 problems in quality by race and gender: underuse or the failure to receive a clinically indicated revascularization procedure and receipt of revascularization when it was not clinically indicated. We used 2 independent sets of criteria developed by the RAND Corporation and the American College of Cardiology/American Hospital Association (ACC/AHA). We also examined survival of the cohort through March 31, 1994.

RESULTS

Revascularization procedures were clinically indicated more frequently among whites than blacks and among men than women. Failure to receive revascularization when it was indicated was more common among blacks than among whites (40% vs. 23-24%, depending on the criteria, both P<0.001) but similar among men and women (25% vs. 22-24%, P>0.05). Racial disparities remained similar after adjusting for patient and hospital characteristics. Among patients rated inappropriate, use of procedures was greater for whites than blacks using RAND criteria (10.5% vs. 5.8%, P<0.01) and greater for men than for women (14.2% vs. 5.3% by RAND criteria, P=0.001; 8.2% vs. 4.0%% by ACC/AHA criteria, P=0.04). After multivariate adjustment, the disparities for race and gender remained similar and were statistically significant using RAND criteria. Mortality rates tended to validate our appropriateness criteria for underuse.

CONCLUSIONS

Racial differences in procedure use reflect higher rates of clinical appropriateness among whites, greater underuse among blacks, and more frequent revascularization when it was not clinically indicated among whites. Underuse is associated with higher mortality. In contrast, men had higher rates of clinical appropriateness and were more likely to receive revascularization when it was not clinically indicated. There was no evidence of greater underuse among women.