Assessing the need to update prevention guidelines: a comparison of two methods

Designing tailored maintenance strategies for systematic reviews and clinical practice guidelines using the Portfolio Maintenance by Test-Treatment (POMBYTT) framework

BACKGROUND

Organizations face diverse contexts and requirements when updating and maintaining their portfolio, or pool, of systematic reviews or clinical practice guidelines they need to manage. We aimed to develop a comprehensive, theoretical framework that might enable the design and tailoring of maintenance strategies for portfolios containing systematic reviews and guidelines.

METHODS

We employed a conceptual approach combined with a literature review. Components of the diagnostic test-treatment pathway used in clinical healthcare were transferred to develop a framework specifically for systematic review and guideline portfolio maintenance strategies.

RESULTS

We developed the Portfolio Maintenance by Test-Treatment (POMBYTT) framework comprising diagnosis, staging, management, and monitoring components. To illustrate the framework's components and their elements, we provided examples from both a clinical healthcare test-treatment pathway and a clinical practice guideline maintenance scenario. Additionally, our literature review provided possible examples for the elements in the framework, such as detection variables, detection tests, and detection thresholds. We furthermore provide three example strategies using the framework, of which one was based on living recommendations strategies.

CONCLUSIONS

The developed framework might support the design of maintenance strategies that could contain multiple options besides updating to manage a portfolio (e.g. withdrawing and archiving), even in the absence of the target condition. By making different choices for variables, tests, test protocols, indications, management options, and monitoring, organizations might tailor their maintenance strategy to suit specific contexts and needs. The framework's elements could potentially aid in the design by being explicit about the operational aspects of maintenance strategies. This might also be helpful for end-users and other stakeholders of systematic reviews and clinical practice guidelines.

An adapted 'Ottawa' method allowed assessing the need to update topic areas within clinical practice guidelines

OBJECTIVE

To adapt and evaluate a method for assessing the need to update guideline topic areas involving multiple recommendations.

STUDY DESIGN AND SETTING

The 'Ottawa method' uses literature signals to determine changes in evidence that trigger a need to update individual guideline questions. We adapted the Ottawa method to include a process for aggregating updating signals by topic area (e.g., resuscitation) and tested this method using the German guideline on the treatment of patients with severe/multiple injuries. This involved a focused systematic evaluation of current evidence to identify updating signals, and classifying the need to update for each topic area. Then, we surveyed the guideline group online about the modified method.

RESULTS

We conducted focused literature searches for 37 topic areas and screened a mean of 97 abstracts per topic area in 2021. The need to update was high for eight (21.6%), intermediate for eight (21.6%), and low for 21 topic areas (56.8%) based on updating signals. The survey response rate was 56% (24/43). Most guideline group members (94%, 16/17 responders) would use the Ottawa method again, but their comments identified some weaknesses.

CONCLUSION

The modified Ottawa method is a suitable, efficient tool to generate evidence-based updating signals for guideline topic areas involving multiple recommendations. Further fine-tuning is recommended.

WSES guidelines updates

The World Society of Emergency Surgery promotes training and continuing medical education in the field of emergency surgery and trauma. One of the most important activities of the society is the development of guidelines. The debate about the process of developing and updating guidelines is very active with no clear consensus and different policies among scientific societies. The present commentary provides the position of the World Society of Emergency Surgery on guideline development process and their update.

A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study

OBJECTIVES

The objective of this study was to test and evaluate a new decision-making process on the need for updating within the update of a German clinical practice guideline (CPG).

STUDY DESIGN AND SETTING

The pilot study comprised (1) limited searches in Pubmed to identify new potentially relevant evidence, (2) an online survey among the members of the CPG group to assess the need for update, and (3) a consensus conference for determination and prioritization of guideline sections with a high need for update. Subsequently, we conducted a second online survey to evaluate the procedure.

RESULTS

The searches resulted in 902 abstracts that were graded as new potentially relevant evidence. Twenty five of 39 members of the CPG group (64%) participated in the online survey. Seventy six percent of those took part in the second online survey. The evaluation study found on average a grade of support of the procedure regarding the determination of the need for update of 3.65 (standard deviation: 0.76) on a likert scale with 1 = "no support" to 5 = "very strong support."

CONCLUSION

The conducted procedure presents a systematic approach for assessing whether and to what extent a CPG requires updating and enables setting priorities for which particular guideline section to update within a CPG.

Updated clinical guidelines experience major reporting limitations

BACKGROUND

The Checklist for the Reporting of Updated Guidelines (CheckUp) was recently developed. However, so far, no systematic assessment of the reporting of updated clinical guidelines (CGs) exists. We aimed to examine (1) the completeness of reporting the updating process in CGs and (2) the inter-observer reliability of CheckUp.

METHODS

We conducted a systematic assessment of the reporting of the updating process in a sample of updated CGs using CheckUp. We performed a systematic search to identify updated CGs published in 2015, developed by a professional society, reporting a systematic review of the evidence, and containing at least one recommendation. Three reviewers independently assessed the CGs with CheckUp (16 items). We calculated the median score per item, per domain, and overall, converting scores to a 10-point scale. Multiple linear regression analyses were used to identify differences according to country, type of organisation, scope, and health topic of updated CGs. We calculated the intraclass coefficient (ICC) and 95% confidence interval (95% CI) for domains and overall score.

RESULTS

We included in total 60 updated CGs. The median domain score on a 10-point scale for presentation was 5.8 (range 1.7 to 10), for editorial independence 8.3 (range 3.3 to 10), and for methodology 5.7 (range 0 to 10). The median overall score on a 10-point scale was 6.3 (range 3.1 to 10). Presentation and justification items at recommendation level (respectively reported by 27 and 38% of the CGs) and the methods used for the external review and implementing changes in practice were particularly poorly reported (both reported by 38% of the CGs). CGs developed by a European or international institution obtained a statistically significant higher overall score compared to North American or Asian institutions (p = 0.014). Finally, the agreement among the reviewers on the overall score was excellent (ICC 0.88, 95% CI 0.75 to 0.95).

CONCLUSIONS

The reporting of updated CGs varies considerably with significant room for improvement. We recommend using CheckUp to assess the updating process in updated CGs and as a blueprint to inform methods and reporting strategies in updating.

Continuous surveillance of a pregnancy clinical guideline: an early experience

BACKGROUND

To date there is no consensus about the optimal strategy for keeping clinical guidelines (CGs) up-to-date. The aims of this study were (1) to develop a continuous surveillance and updating strategy for CGs and (2) to test the strategy in a specific CG.

METHODS

The main steps were as follows: (1) recruiting members for the CG Updating Working Group, (2) mapping the CG, (3) identifying new evidence from the CG Updating Working Group, (4) designing and running restricted literature searches, (5) reviewing drugs and medical devices alerts, (6) screening and assessing the new evidence, (7) reviewing and, if necessary, modifying clinical questions and recommendations, and (8) updating the CG document.

RESULTS

The Pregnancy CG Updating Working Group consisted of 29 members, including clinicians, patients and caregivers, and clinical guideline methodology experts. We selected 69 clinical questions (123 recommendations) from the "Assistance during pregnancy" section. For the first update cycle (32-month duration), 9710 references were identified. Of these, 318 were pertinent, 289 were relevant, and 55 were classified as potential key references. For the second and third update cycles (6-month duration each), 2160 and 2010 references were retrieved, respectively. The continuous surveillance and updating strategy has not yet been completely implemented.

CONCLUSIONS

Further resources are needed in updating the CG field, both for implementing updating strategies and for developing methodological research.

Reporting Items for Updated Clinical Guidelines: Checklist for the Reporting of Updated Guidelines (CheckUp)

BACKGROUND

Scientific knowledge is in constant development. Consequently, regular review to assure the trustworthiness of clinical guidelines is required. However, there is still a lack of preferred reporting items of the updating process in updated clinical guidelines. The present article describes the development process of the Checklist for the Reporting of Updated Guidelines (CheckUp).

METHODS AND FINDINGS

We developed an initial list of items based on an overview of research evidence on clinical guideline updating, the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument, and the advice of the CheckUp panel (n = 33 professionals). A multistep process was used to refine this list, including an assessment of ten existing updated clinical guidelines, interviews with key informants (response rate: 54.2%; 13/24), a three-round Delphi consensus survey with the CheckUp panel (33 participants), and an external review with clinical guideline methodologists (response rate: 90%; 53/59) and users (response rate: 55.6%; 10/18). CheckUp includes 16 items that address (1) the presentation of an updated guideline, (2) editorial independence, and (3) the methodology of the updating process. In this article, we present the methodology to develop CheckUp and include as a supplementary file an explanation and elaboration document.

CONCLUSIONS

CheckUp can be used to evaluate the completeness of reporting in updated guidelines and as a tool to inform guideline developers about reporting requirements. Editors may request its completion from guideline authors when submitting updated guidelines for publication. Adherence to CheckUp will likely enhance the comprehensiveness and transparency of clinical guideline updating for the benefit of patients and the public, health care professionals, and other relevant stakeholders.

Designing a rapid response program to support evidence-informed decision-making in the Americas region: using the best available evidence and case studies

Background

The objective of this work was to inform the design of a rapid response program to support evidence-informed decision-making in health policy and practice for the Americas region. Specifically, we focus on the following: (1) What are the best methodological approaches for rapid reviews of the research evidence? (2) What other strategies are needed to facilitate evidence-informed decision-making in health policy and practice? and (3) How best to operationalize a rapid response program?

Methods

The evidence used to inform the design of a rapid response program included (i) two rapid reviews of methodological approaches for rapid reviews of the research evidence and strategies to facilitate evidence-informed decision-making, (ii) supplementary literature in relation to the “shortcuts” that could be considered to reduce the time needed to complete rapid reviews, (iii) four case studies, and (iv) supplementary literature to identify additional operational issues for the design of the program.

Results

There is no agreed definition of rapid reviews in the literature and no agreed methodology for conducting them. Better reporting of rapid review methods is needed. The literature found in relation to shortcuts will be helpful in choosing shortcuts that maximize timeliness while minimizing the impact on quality. Evidence for other strategies that can be used concurrently to facilitate the uptake of research evidence, including evidence drawn from rapid reviews, is presented. Operational issues that need to be considered in designing a rapid response program include the implications of a “user-pays” model, the importance of recruiting staff with the right mix of skills and qualifications, and ensuring that the impact of the model on research use in decision-making is formally evaluated.

Conclusions

When designing a new rapid response program, greater attention needs to be given to specifying the rapid review methods and reporting these in sufficient detail to allow a quality assessment. It will also be important to engage in other strategies to facilitate the uptake of the rapid reviews and to evaluate the chosen model in order to make refinements and add to the evidence base for evidence-informed decision-making.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-016-0472-9) contains supplementary material, which is available to authorized users.

Updating contextualized clinical practice guidelines on stroke rehabilitation and low back pain management using a novel assessment framework that standardizes decisions

BACKGROUND

Clinical practice guidelines need to be regularly updated with current literature in order to remain relevant. This paper reports on the approach taken by the Philippine Academy of Rehabilitation Medicine (PARM). This dovetails with its writing guide, which underpinned its foundational work in contextualizing guidelines for stroke and low back pain (LBP) in 2011.

METHODS

Working groups of Filipino rehabilitation physicians and allied health practitioners met to reconsider and modify, where indicated, the 'typical' Filipino patient care pathways established in the foundation guidelines. New clinical guidelines on stroke and low back pain which had been published internationally in the last 3 years were identified using a search of electronic databases. The methodological quality of each guideline was assessed using the iCAHE Guideline Quality Checklist, and only those guidelines which provided full text references, evidence hierarchy and quality appraisal of the included literature, were included in the PARM update. Each of the PARM-endorsed recommendations was then reviewed, in light of new literature presented in the included clinical guidelines. A novel standard updating approach was developed based on the criteria reported by Johnston et al. (Int J Technol Assess Health Care 19(4):646-655, 2003) and then modified to incorporate wording from the foundational PARM writing guide. The new updating tool was debated, pilot-tested and agreed upon by the PARM working groups, before being applied to the guideline updating process.

RESULTS

Ten new guidelines on stroke and eleven for low back pain were identified. Guideline quality scores were moderate to good, however not all guidelines comprehensively linked the evidence body underpinning recommendations with the literature. Consequently only five stroke and four low back pain guidelines were included. The modified PARM updating guide was applied by all working groups to ensure standardization of the wording of updated recommendations and the underpinning evidence bases.

CONCLUSIONS

The updating tool provides a simple, standard and novel approach that incorporates evidence hierarchy and quality, and wordings of recommendations. It could be used efficiently by other guideline updaters particularly in developing countries, where resources for guideline development and updates are limited. When many people are involved in guideline writing, there is always the possibility of 'slippage' in use of wording and interpretation of evidence. The PARM updating tool provides a mechanism for maintaining a standard process for guideline updating processes that can be followed by clinicians with basic training in evidence-based practice principles.

Efficiency of pragmatic search strategies to update clinical guidelines recommendations

Background

A major challenge in updating clinical guidelines is to efficiently identify new, relevant evidence. We evaluated the efficiency and feasibility of two new approaches: the development of restrictive search strategies using PubMed Clinical Queries for MEDLINE and the use of the PLUS (McMaster Premium Literature Service) database.

Methods

We evaluated a random sample of recommendations from a national guideline development program and identified the references that would potentially trigger an update (key references) using an exhaustive approach.

We designed restrictive search strategies using the minimum number of Medical Subject Headings (MeSH) terms and text words required from the original exhaustive search strategies and applying broad and narrow filters. We developed PLUS search strategies, matching Medical Subject Headings (MeSH) and Systematized Nomenclature of Medicine (SNOMED) terms with guideline topics. We compared the number of key references retrieved by these approaches with those retrieved by the exhaustive approach.

Results

The restrictive approach retrieved 68.1 % fewer references than the exhaustive approach (12,486 versus 39,136), and identified 89.9 % (62/69) of key references and 88 % (22/25) of recommendation updates. The use of PLUS retrieved 88.5 % fewer references than the exhaustive approach (4,486 versus 39,136) and identified substantially fewer key references (18/69, 26.1 %) and fewer recommendation updates (10/25, 40 %).

Conclusions

The proposed restrictive approach is a highly efficient and feasible method to identify new evidence that triggers a recommendation update. Searching only in the PLUS database proved to be a suboptimal approach and suggests the need for topic-specific tailoring.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0058-2) contains supplementary material, which is available to authorized users.

Updating Clinical Practice Guidelines: How Do We Stay Current?

Clinical practice guidelines (CPGs) are created to address quality improvement opportunities for all clinicians, optimize patient care, promote effective diagnosis and therapy, and reduce harmful or unnecessary variations in patient care. Tremendous resources are invested in creating CPGs we can trust. Once a guideline is created, however, how is it maintained to reflect the most up-to-date clinical evidence? This article reviews protocols for reviewing and maintaining CPGs with particular attention to the protocols established by the American Academy of Otolaryngology-Head and Neck Surgery.

The validity of recommendations from clinical guidelines: a survival analysis

BACKGROUND

Clinical guidelines should be updated to maintain their validity. Our aim was to estimate the length of time before recommendations become outdated.

METHODS

We used a retrospective cohort design and included recommendations from clinical guidelines developed in the Spanish National Health System clinical guideline program since 2008. We performed a descriptive analysis of references, recommendations and resources used, and a survival analysis of recommendations using the Kaplan-Meier method.

RESULTS

We included 113 recommendations from 4 clinical guidelines with a median of 4 years since the most recent search (range 3.9-4.4 yr). We retrieved 39 136 references (range 3343-14 787) using an exhaustive literature search, 668 of which were related to the recommendations in our sample. We identified 69 (10.3%) key references, corresponding to 25 (22.1%) recommendations that required updating. Ninety-two percent (95% confidence interval 86.9-97.0) of the recommendations were valid 1 year after their development. This probability decreased at 2 (85.7%), 3 (81.3%) and 4 years (77.8%).

INTERPRETATION

Recommendations quickly become outdated, with 1 out of 5 recommendations being out of date after 3 years. Waiting more than 3 years to review a guideline is potentially too long.

Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks

Background

Updating clinical practice guidelines (CPGs) is a crucial process for maintaining the validity of recommendations. Methodological handbooks should provide guidance on both developing and updating CPGs. However, little is known about the updating guidance provided by these handbooks.

Methods

We conducted a systematic review to identify and describe the updating guidance provided by CPG methodological handbooks and included handbooks that provide updating guidance for CPGs. We searched in the Guidelines International Network library, US National Guidelines Clearinghouse and MEDLINE (PubMed) from 1966 to September 2013. Two authors independently selected the handbooks and extracted the data. We used descriptive statistics to analyze the extracted data and conducted a narrative synthesis.

Results

We included 35 handbooks. Most handbooks (97.1%) focus mainly on developing CPGs, including variable degrees of information about updating. Guidance on identifying new evidence and the methodology of assessing the need for an update is described in 11 (31.4%) and eight handbooks (22.8%), respectively. The period of time between two updates is described in 25 handbooks (71.4%), two to three years being the most frequent (40.0%). The majority of handbooks do not provide guidance for the literature search, evidence selection, assessment, synthesis, and external review of the updating process.

Conclusions

Guidance for updating CPGs is poorly described in methodological handbooks. This guidance should be more rigorous and explicit. This could lead to a more optimal updating process, and, ultimately to valid trustworthy guidelines.

Partial updating of clinical practice guidelines often makes more sense than full updating: a systematic review on methods and the development of an updating procedure

OBJECTIVES

To conduct a systematic review of the methods used to determine when and how to update clinical practice guidelines (CPGs) and develop a procedure for updating CPGs.

STUDY DESIGN AND SETTING

We searched MEDLINE, Embase, and the Cochrane Methodology Register for methodological publications on updating CPGs. Guideline development manuals were obtained from the Web sites of guideline-developing organizations. Using the information obtained from these records, a procedure for updating CPGs was developed.

RESULTS

A total of 5,116 journal articles were screened, and seven articles met the criteria for inclusion. Forty-seven manuals were included; of these, eight included details about the methods used to update the guidelines. Most of the included publications focused on assessing whether the CPGs needed updating and not on how to update them. The developed procedure includes a systematic monitoring system and a scheduled process for updating the CPGs, which includes guidance on how to determine the type and scope of an update.

CONCLUSION

Partial updating often makes more sense than updating the whole CPG because topics and recommendations differ in terms of the need for updating. Guideline developers should implement a systematic updating procedure that includes an ongoing monitoring system that is appropriate for the nature of the guideline topics and the capabilities of the developers.

Measuring quality of care: considering conceptual approaches to quality indicator development and evaluation

OBJECTIVE

In this article, we describe one approach for developing and evaluating quality indicators.

STUDY DESIGN AND SETTING

We focus on describing different conceptual approaches to quality indicator development, review one approach for developing quality indicators, outline how to evaluate quality indicators once developed, and discuss quality indicator maintenance.

RESULTS

The key steps for developing quality indicators include specifying a clear goal for the indicators; using methodologies to incorporate evidence, expertise, and patient perspectives; and considering contextual factors and logistics of implementation. The Strategic Framework Board and the National Quality Measure Clearinghouse have developed criteria for evaluating quality indicators that complement traditional psychometric evaluations. Optimal strategies for quality indicator maintenance and dissemination have not been determined, but experiences with clinical guideline maintenance may be informative.

CONCLUSION

For quality indicators to effectively guide quality improvement efforts, they must be developed, evaluated, maintained, and implemented using rigorous evidence-informed practices.

Strategies to assess the validity of recommendations: a study protocol

Background

Clinical practice guidelines (CPGs) become quickly outdated and require a periodic reassessment of evidence research to maintain their validity. However, there is little research about this topic. Our project will provide evidence for some of the most pressing questions in this field: 1) what is the average time for recommendations to become out of date?; 2) what is the comparative performance of two restricted search strategies to evaluate the need to update recommendations?; and 3) what is the feasibility of a more regular monitoring and updating strategy compared to usual practice?. In this protocol we will focus on questions one and two.

Methods

The CPG Development Programme of the Spanish Ministry of Health developed 14 CPGs between 2008 and 2009. We will stratify guidelines by topic and by publication year, and include one CPG by strata.

We will develop a strategy to assess the validity of CPG recommendations, which includes a baseline survey of clinical experts, an update of the original exhaustive literature searches, the identification of key references (reference that trigger a potential recommendation update), and the assessment of the potential changes in each recommendation.

We will run two alternative search strategies to efficiently identify important new evidence: 1) PLUS search based in McMaster Premium LiteratUre Service (PLUS) database; and 2) a Restrictive Search (ReSe) based on the least number of MeSH terms and free text words needed to locate all the references of each original recommendation.

We will perform a survival analysis of recommendations using the Kaplan-Meier method and we will use the log-rank test to analyse differences between survival curves according to the topic, the purpose, the strength of recommendations and the turnover. We will retrieve key references from the exhaustive search and evaluate their presence in the PLUS and ReSe search results.

Discussion

Our project, using a highly structured and transparent methodology, will provide guidance of when recommendations are likely to be at risk of being out of date. We will also assess two novel restrictive search strategies which could reduce the workload without compromising rigour when CPGs developers check for the need of updating.

Strategies for monitoring and updating clinical practice guidelines: a systematic review

Background

Scientific knowledge is in constant change. The flow of new information requires a frequent re-evaluation of the available research results. Clinical practice guidelines (CPGs) are not exempted from this phenomenon and need to be kept updated to maintain the validity of their recommendations. The objective of our review is to systematically identify, describe and assess strategies for monitoring and updating CPGs.

Study design and setting

We conducted a systematic review of studies evaluating one or more methods of updating (with or without monitoring) CPGs or recommendations. We searched MEDLINE (PubMed) and The Cochrane Methodology Register (The Cochrane Library) from 1966 to June 2012. Additionally, we hand-searched reference lists of the included studies and the Guidelines International Network book of abstracts. If necessary, we contacted study authors to obtain additional information.

Results

We included a total of eight studies. Four evaluated if CPGs were out of date, three updated CPGs, and one continuously monitored and updated CPGs. The most detailed reported phase of the process was the identification of new evidence. As opposed to studies updating guidelines, studies evaluating if CPGs were out of date applied restricted searches. Only one study compared a restricted versus an exhaustive search suggesting that a restricted search is sufficient to assess recommendations’ Validity. One study analyzed the survival time of CPGs and suggested that these should be reassessed every three years.

Conclusions

There is limited evidence about the optimal strategies for monitoring and updating clinical practice guidelines. A restricted search is likely to be sufficient to monitor new evidence and assess the need to update, however, more information is needed about the timing and type of search. Only the exhaustive search strategy has been assessed for the update of CPGs. The development and evaluation of more efficient strategies is needed to improve the timeliness and reduce the burden of maintaining the validity of CPGs.

Developing clinical practice guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the emerging issues of enhancing guideline implementability and accounting for comorbid conditions in guideline development

Clinical practice guidelines are one of the foundations of efforts to improve health care. In 1999, we authored a paper about methods to develop guidelines. Since it was published, the methods of guideline development have progressed both in terms of methods and necessary procedures and the context for guideline development has changed with the emergence of guideline clearing houses and large scale guideline production organisations (such as the UK National Institute for Health and Clinical Excellence). It therefore seems timely to, in a series of three articles, update and extend our earlier paper. In this third paper we discuss the issues of: reviewing, reporting, and publishing guidelines; updating guidelines; and the two emerging issues of enhancing guideline implementability and how guideline developers should approach dealing with the issue of patients who will be the subject of guidelines having co-morbid conditions.

Speed of updating online evidence based point of care summaries: prospective cohort analysis

OBJECTIVE

To evaluate the ability of international point of care information summaries to update evidence relevant to medical practice.

DESIGN

Prospective cohort bibliometric analysis.

SETTING

Top five point of care information summaries (Clinical Evidence, EBMGuidelines, eMedicine, Dynamed, UpToDate) ranked for coverage of medical conditions, editorial quality, and evidence based methodology.

MAIN OUTCOME MEASURES

From June 2009 to May 2010 we measured the incidence of research findings relating to potentially eligible newsworthy evidence. As samples, we chose systematic reviews rated as relevant by international research networks (such as, Evidence-Based Medicine, ACP Journal Club, and the Cochrane Collaboration). Every month we assessed whether each sampled review was cited in at least one chapter of the five summaries. The cumulative updating rate was analysed with Kaplan-Meier curves. Results From April to December 2009, 128 reviews were retrieved; 53% (68) from the literature surveillance journals and 47% (60) from the Cochrane Library. At nine months, Dynamed had cited 87% of the sampled reviews, while the other summaries had cited less than 50%. The updating speed of Dynamed clearly led the others. For instance, the hazard ratios for citations in EBM Guidelines and Clinical Evidence versus the top performer were 0.22 (95% confidence interval 0.17 to 0.29) and 0.03 (0.01 to 0.05). Conclusions Point of care information summaries include evidence relevant to practice at different speeds. A qualitative analysis of updating mechanisms is needed to determine whether greater speed corresponds to more appropriate incorporation of new information.

[Guidelines and guideline research. Overview and state of guideline implementation in medical rehabilitation]

"Guideline research" is understood here to be the area of healthcare research that deals with the methodology and evaluation of processes for developing, implementing, assessing, and evaluating guidelines. For example, guideline research deals with the selection of adequate implementation strategies and the proof of the effectiveness and efficiency of the implementation of guidelines. The current data situation does not allow any reliable statements to be made about whether and to what extent the health-related outcome for patients is actually improved by implementing guidelines. For medical rehabilitation, there are a number of guidelines which appear to be just as good with respect to quality as the guidelines for acute medicine. However, there is a need for improvement in the quality criteria for methods. Rehab-related guidelines and therapy standards can be found in the AWMF guidelines, in the program for national healthcare guidelines, and in the rehabilitation quality assurance of the German Pension Fund. Guideline research in rehabilitation should concentrate on topics that arise from the specific situation of this area of healthcare, e.g., development of interdisciplinary guidelines, applicability of guidelines for multiple diseases, or possibility of evidence-based guidelines for complex interventions.

Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program

OBJECTIVES

To review the current knowledge and efforts on updating systematic reviews (SRs) as applied to comparative effectiveness reviews (CERs).

STUDY DESIGN AND SETTING

This article outlines considerations for updating CERs by including a definition of the updating process, describing issues around assessing whether to update, and providing general guidelines for the update process. Key points to consider include (1) identifying when to update CERs, (2) how to update CERs, and (3) how to present, report, and interpret updated results in CERs.

RESULTS

Currently, there is little information about what proportion of SRs needs updating. Similarly, there is no consensus on when to initiate updating and how best to carry it out.

CONCLUSION

CERs need to be regularly updated as new evidence is produced. Lack of attention to updating may lead to outdated and sometimes misleading conclusions that compromise health care and policy decisions. The article outlines several specific goals for future research, one of them being the development of efficient guideline for updating CERs applicable across evidence-based practice centers.

Supported local implementation of clinical guidelines in psychiatry: a two-year follow-up

BACKGROUND

The gap between evidence-based guidelines for clinical care and their use in medical settings is well recognized and widespread. Only a few implementation studies of psychiatric guidelines have been carried out, and there is a lack of studies on their long-term effects.The aim of this study was to measure compliance to clinical guidelines for treatment of patients with depression and patients with suicidal behaviours, two years after an actively supported implementation.

METHODS

Six psychiatric clinics in Stockholm, Sweden, participated in an implementation of the guidelines. The guidelines were actively implemented at four of them, and the other two only received the guidelines and served as controls. The implementation activities included local implementation teams, seminars, regular feedback, and academic outreach visits. Compliance to guidelines was measured using quality indicators derived from the guidelines. At baseline, measurements of quality indicators, part of the guidelines, were abstracted from medical records in order to analyze the gap between clinical guidelines and current practice. On the basis of this, a series of seminars was conducted to introduce the guidelines according to local needs. Local multidisciplinary teams were established to monitor the process. Data collection took place after 6, 12, and 24 months and a total of 2,165 patient records were included in the study.

RESULTS

The documentation of the quality indicators improved from baseline in the four clinics with an active implementation, whereas there were no changes, or a decline, in the two control clinics. The increase was recorded at six months, and persisted over 12 and 24 months.

CONCLUSIONS

Compliance to the guidelines increased after active implementation and was sustained over the two-year follow-up. These results indicate that active local implementation of clinical guidelines involving clinicians can change behaviour and maintain compliance.

Strategies in assessing the need for updating evidence-based guidelines for six clinical topics: an exploration of two search methodologies

BACKGROUND

Because of the expense of updating practice guidelines, recent attention has focused on approaches that can reliably assess any updating required. Shekelle et al. (Journal of the American Medical Association 2001, 286, 1461-7) proposed using limited literature searches with expert involvement to reduce resources used in assessing whether a guideline needs updating.

OBJECTIVES

This study compared Shekelle's method and the traditional systematic review method regarding comprehensiveness and effort.

METHODS

Two research teams translated critical key questions on screening test treatments and outcomes to Medical Subjects Headings (MeSH) and search strategies. They refined Shekelle's method over three iterations, seeking greater efficiency. Using both methods independently, teams assessed the need to update six topics from the 1996 Guide to Clinical Preventive Services (US Preventive Services Task Force). Outcomes included completeness of study identification, importance of missed studies and effort involved.

RESULTS

The revised review approach produced fewer citations than the traditional approach and saved time, identifying fewer eligible studies than the traditional approach. None of the studies missed was rated important by the experts consulted.

CONCLUSIONS

The revised review approach provides an acceptable method for judging whether a guideline requires updating. Librarians were an integral part of the research process that streamlined the searches.

An approach to measure compliance to clinical guidelines in psychiatric care

BACKGROUND

The aim of this study was to measure six months compliance to Swedish clinical guidelines in psychiatric care after an active supported implementation process, using structured measures derived from the guidelines.

METHODS

In this observational study four psychiatric clinics each participated in active implementation of the clinical guidelines for the assessment and treatment of depression and guidelines for assessment and treatment of patients with suicidal behaviours developed by The Stockholm Medical Advisory Board for Psychiatry. The implementation programme included seminars, local implementation teams, regular feedback and academic visits. Additionally two clinics only received the guidelines and served as controls. Compliance to guidelines was measured using indicators, which operationalised requirements of preferred clinical practice. 725 patient records were included, 365 before the implementation and 360 six months after.

RESULTS

Analyses of indicators registered showed that the actively implementing clinics significantly improved their compliance to the guidelines. The total score differed significantly between implementation clinics and control clinics for management of depression (mean scores 9.5 (1.3) versus 5.0 (1.5), p < 0.001) as well as for the management of suicide (mean scores 8.1 (2.3) versus 4.5 (1.9), p < 0.001). No changes were found in the control clinics and only one of the OR was significant.

CONCLUSION

Compliance to clinical guidelines measured by process indicators of required clinical practice was enhanced by an active implementation.

When and how to update systematic reviews

BACKGROUND

Systematic reviews are most helpful if they are up-to-date. We did a systematic review of strategies and methods describing when and how to update systematic reviews.

OBJECTIVES

To identify, describe and assess strategies and methods addressing: 1) when to update systematic reviews and 2) how to update systematic reviews.

SEARCH STRATEGY

We searched MEDLINE (1966 to December 2005), PsycINFO, the Cochrane Methodology Register (Issue 1, 2006), and hand searched the 2005 Cochrane Colloquium proceedings.

SELECTION CRITERIA

We included methodology reports, updated systematic reviews, commentaries, editorials, or other short reports describing the development, use, or comparison of strategies and methods for determining the need for updating or updating systematic reviews in healthcare.

DATA COLLECTION AND ANALYSIS

We abstracted information from each included report using a 15-item questionnaire. The strategies and methods for updating systematic reviews were assessed and compared descriptively with respect to their usefulness, comprehensiveness, advantages, and disadvantages.

MAIN RESULTS

Four updating strategies, one technique, and two statistical methods were identified. Three strategies addressed steps for updating and one strategy presented a model for assessing the need to update. One technique discussed the use of the "entry date" field in bibliographic searching. Statistical methods were cumulative meta-analysis and predicting when meta-analyses are outdated.

AUTHORS' CONCLUSIONS

Little research has been conducted on when and how to update systematic reviews and the feasibility and efficiency of the identified approaches is uncertain. These shortcomings should be addressed in future research.

A systematic review identified few methods and strategies describing when and how to update systematic reviews

OBJECTIVE

Systematic reviews (SRs) are convenient summaries of evidence for health care practitioners. They form a basis for clinical practice guidelines and suggest directions for new research. SRs are most helpful if they are current; however, most of them are not being updated. This SR summarizes strategies and methods describing when and how to update SRs.

STUDY DESIGN AND SETTING

We searched MEDLINE (1966 to December 2005), PsycINFO, the Cochrane Methodology Register, and the 2005 Cochrane Colloquium proceedings to identify records describing when and how to update SRs in health care.

RESULTS

Four updating strategies, one technique, and two statistical methods were identified. Three strategies addressed steps for updating, and one strategy presented a model for assessing the need to update. One technique discussed the use of the "entry date" field in bibliographic searching. The statistical methods were cumulative meta-analysis and a test for detecting outdated meta-analyses with statistically nonsignificant results.

CONCLUSION

Little research has been conducted on when and how to update SRs in contrast to other methodological areas of conducting SRs (e.g., publication bias, variance imputation). The feasibility and efficiency of the identified approaches is uncertain. More research is needed to develop pragmatic and efficient methodologies for updating SRs.

Critical look at the clinical practice guidelines for allergic rhinitis

Allergic rhinitis (AR) is a major health concern and numerous guidelines have been developed to standardize and to improve the management of this disease. As in many other areas of medicine, the methodology of the AR guidelines has evolved from opinion-based to evidence-based medicine. Although evidence-based medicine has many benefits, it also has limitations and cannot cancel the value of the individual clinical expertise. More important than the methodology of guideline development is the efficacy of guidelines to change patient and physician behaviour and to improve clinical outcomes. At present, however, studies on the effectiveness of guidelines are few. The International Consensus on Rhinitis from 1994 is the only guideline for AR that has been assessed for its effects on health outcomes. Furthermore, there is a lack of valid and reliable instruments to assess physician's and patient's attitude towards and compliance with guideline recommendations. There is no single effective way to ensure the use of guidelines into practice, but a carefully developed and multifaceted dissemination and implementation strategy and targeting and adapting guideline recommendations to the local and individual level are key elements. The final and most important step of putting guidelines into practice occurs at the level of the patient. Patients should be considered as effective partners in health care. Education of the patient and efforts to change patient's behaviour can maximize compliance, increase satisfaction and optimize health outcomes.

Improving the use of research evidence in guideline development: 16. Evaluation

BACKGROUND

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

OBJECTIVES

We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations.

METHODS

We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

KEY QUESTIONS AND ANSWERS

Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. A checklist, such as the AGREE instrument, should be used. The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research.

Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence

BACKGROUND

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the eighth of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

OBJECTIVES

We reviewed the literature on the synthesis and presentation of research evidence, focusing on four key questions.

METHODS

We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

KEY QUESTIONS AND ANSWERS

We found two reviews of instruments for critically appraising systematic reviews, several studies of the importance of using extensive searches for reviews and determining when it is important to update reviews, and consensus statements about the reporting of reviews that informed our answers to the following questions. How should existing systematic reviews be critically appraised? Because preparing systematic reviews can take over a year and require capacity and resources, existing reviews should be used when possible and updated, if needed. Standard criteria, such as A MeaSurement Tool to Assess Reviews (AMSTAR), should be used to critically appraise existing systematic reviews, together with an assessment of the relevance of the review to the questions being asked. When and how should WHO undertake or commission new reviews? Consideration should be given to undertaking or commissioning a new review whenever a relevant, up-to-date review of good quality is not available. When time or resources are limited it may be necessary to undertake rapid assessments. The methods that are used to do these assessments should be reported, including important limitations and uncertainties and explicit consideration of the need and urgency of undertaking a full systematic review. Because WHO has limited capacity for undertaking systematic reviews, reviews will often need to be commissioned when a new review is needed. Consideration should be given to establishing collaborating centres to undertake or support this work, similar to what some national organisations have done. How should the findings of systematic reviews be summarised and presented to committees responsible for making recommendations? Concise summaries (evidence tables) of the best available evidence for each important outcome, including benefits, harms and costs, should be presented to the groups responsible for making recommendations. These should include an assessment of the quality of the evidence and a summary of the findings for each outcome. The full systematic reviews, on which the summaries are based, should also be available to both those making recommendations and users of the recommendations. What additional information is needed to inform recommendations and how should this information be synthesised with information about effects and presented to committees? Additional information that is needed to inform recommendations includes factors that might modify the expected effects, need (prevalence, baseline risk or status), values (the relative importance of key outcomes), costs and the availability of resources. Any assumptions that are made about values or other factors that may vary from setting to setting should be made explicit. For global guidelines that are intended to inform decisions in different settings, consideration should be given to using a template to assist the synthesis of information specific to a setting with the global evidence of the effects of the relevant interventions.