Second-order peer review of the medical literature for clinical practitioners

Five Domains of a Conceptual Framework of Continuing Professional Development

ABSTRACT

Continuing professional development (CPD) for health professionals involves efforts at improving health of individuals and the population through educational activities of health professionals who previously attained a recognized level of acceptable proficiency (licensure). However, those educational activities have inconsistently improved health care outcomes of patients. We suggest a conceptual change of emphasis in designing CPD to better align it with the goals of improving health care value for patients through the dynamic incorporation of five distinct domains to be included in learning activities. We identify these domains as: (1) identifying, appraising, and learning new information [New Knowledge]; (2) ongoing practicing of newly or previously acquired skills to maintain expertise [New Skills and Maintenance]; (3) sharing and transfer of new learning for the health care team which changes their practice [Teams]; (4) analyzing data to identify problems and drive change resulting in improvements in the health care system and patient outcomes [Quality Improvement]; and (5) promoting population health and prevention of disease [Prevention]. We describe how these five domains can be integrated into a comprehensive conceptual framework of CPD, supported by appropriate learning theories that align with the goals of the health care delivery system. Drawing on these distinct but interrelated areas of CPD will help organizers and directors of learning events to develop their activities to meet the goals of learners and the health care system.

A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation

Background

A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence.

Objective

The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature.

Methods

Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores.

Results

Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022.

Conclusions

The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching.

International Registered Report Identifier (IRRID)

DERR1-10.2196/29398

Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers

OBJECTIVE

Our aim was to develop an efficient search strategy for prognostic studies and clinical prediction guides (CPGs), optimally balancing sensitivity and precision while independent of MeSH terms, as relying on them may miss the most current literature.

MATERIALS AND METHODS

We combined 2 Hedges-based search strategies, modified to remove MeSH terms for overall prognostic studies and CPGs, and ran the search on 269 journals. We read abstracts from a random subset of retrieved references until ≥ 20 per journal were reviewed and classified them as positive when fulfilling standardized quality criteria, thereby assembling a standard dataset used to calibrate the search strategy. We determined performance characteristics of our new search strategy against the Hedges standard and performance characteristics of published search strategies against the standard dataset.

RESULTS

Our search strategy retrieved 16 089 references from 269 journals during our study period. One hundred fifty-four journals yielded ≥ 20 references and ≥ 1 prognostic study or CPG. Against the Hedges standard, the new search strategy had sensitivity/specificity/precision/accuracy of 84%/80%/2%/80%, respectively. Existing published strategies tested against our standard dataset had sensitivities of 36%-94% and precision of 5%-10%.

DISCUSSION

We developed a new search strategy to identify overall prognosis studies and CPGs independent of MeSH terms. These studies are important for medical decision-making, as they identify specific populations and individuals who may benefit from interventions.

CONCLUSION

Our results may benefit literature surveillance and clinical guideline efforts, as our search strategy performs as well as published search strategies while capturing literature at the time of publication.

An evidence rating service provided valid correlates of the clinical importance of medical articles and journals

OBJECTIVES

The objective of this study was to determine reliability and validity of McMaster PLUS measures of scientific merit and clinical importance of articles in medical journals.

STUDY DESIGN AND SETTING

Analytic survey of peer-reviewed medical journals was carried out. Articles were qualified for inclusion by meeting (1) scientific criteria and (2) a clinical importance rating threshold. Included articles were sent as e-mail alerts to physicians according to their clinical interests. Internal measures included the number of high-quality, clinically important studies published in source journals and response to alerts. For external validation, we correlated internal measures with the Journal Impact Factor (JIF) and citation in DynaMed Plus (DMP).

RESULTS

We evaluated 34,232 articles from 57 journals. Inclusion criteria were met by 2,638 articles (7.71%). The number of qualifying articles per journal was correlated with the number of articles with high clinical importance ratings (r 0.96, P < 0.001), article alert clicks (r 0.86, P < 0.001), and DMP citations (r 0.99, P < 0.001). Correlation was lower with the JIF (r 0.68, P < 0.01).

CONCLUSIONS

Measures of scientific merit and clinical importance of medical journal articles were strongly correlated with each other, less so with JIFs. Journals varied widely by these measures but, generally, few articles were both scientifically sound and clinically important.

Uma proposta metodológica para a elaboração de revisões clínicas

As revisões clínicas são sumários de evidências com escopo amplo e orientação à prática profissional, ocupando um dos mais altos níveis da pirâmide da assistência à saúde baseada em evidências. Com base nessa pirâmide, refletimos sobre as instruções aos autores de periódicos em medicina de família e comunidade para propor instruções para a escrita de revisões clínicas. As instruções abrangem a busca, avaliação, síntese e análise das evidências clínicas, além do planejamento e redação da revisão. Esperamos que estas instruções ajudem a elevar a quantidade e qualidade das revisões clínicas em medicina de família e comunidade no país.

A review of publication bias in the gastroenterology literature

In systematic reviews and meta-analyses, publication bias is particularly problematic, given that combining only statistically significant outcomes is likely to overestimate the true effect of an intervention since non-significant findings have been omitted. We examined practices for evaluating publication bias from gastroenterology literature. We performed a PubMed search to identify systematic reviews published in American Journal of Gastroenterology, Gut, and Gastroenterology from 2005 to 2015. Of the 304 found, 215 studies were eligible for inclusion based on relevant study characteristics. There were 190 systematic reviews which used at least one method to evaluate publication bias and/or included ten or more primary studies. There were 115/190 (60.53%) systematic reviews which used at least one method to evaluate publication bias. Most (105/115, 91.27%) qualified reviews used at least one method to evaluate publication bias and 78/115 (67.83%) used a combination of methods. The most common methods were funnel plot (100/115, 86.96%), Egger's regression (67/115, 58.26%), and Begg's (28/115, 24.35%). Of the 115 reviews that performed evaluations, 26 (22.61%) conducted these analyses with fewer than ten primary studies, and a minority (24/115, 20.87%) reached the conclusion that publication bias was present in their work. While methods to assess publication bias were frequently noted among qualified systematic reviews, these methods are limited in value and could be improved by incorporating approaches that assess the degree of publication bias severity.

Progress in evidence-based medicine: a quarter century on

In response to limitations in the understanding and use of published evidence, evidence-based medicine (EBM) began as a movement in the early 1990s. EBM's initial focus was on educating clinicians in the understanding and use of published literature to optimise clinical care, including the science of systematic reviews. EBM progressed to recognise limitations of evidence alone, and has increasingly stressed the need to combine critical appraisal of the evidence with patient's values and preferences through shared decision making. In another progress, EBM incorporated and further developed the science of producing trustworthy clinical practice guidelines pioneered by investigators in the 1980s. EBM's enduring contributions to clinical medicine include placing the practice of medicine on a solid scientific basis, the development of more sophisticated hierarchies of evidence, the recognition of the crucial role of patient values and preferences in clinical decision making, and the development of the methodology for generating trustworthy recommendations.

McMaster Optimal Aging Portal: an evidence-based database for geriatrics-focused health professionals

Objective

The objective of this work was to provide easy access to reliable health information based on good quality research that will help health care professionals to learn what works best for seniors to stay as healthy as possible, manage health conditions and build supportive health systems. This will help meet the demands of our aging population that clinicians provide high quality care for older adults, that public health professionals deliver disease prevention and health promotion strategies across the life span, and that policymakers address the economic and social need to create a robust health system and a healthy society for all ages.

Results

The McMaster Optimal Aging Portal’s (Portal) professional bibliographic database contains high quality scientific evidence about optimal aging specifically targeted to clinicians, public health professionals and policymakers. The database content comes from three information services: McMaster Premium LiteratUre Service (MacPLUS™), Health Evidence™ and Health Systems Evidence. The Portal is continually updated, freely accessible online, easily searchable, and provides email-based alerts when new records are added. The database is being continually assessed for value, usability and use. A number of improvements are planned, including French language translation of content, increased linkages between related records within the Portal database, and inclusion of additional types of content. While this article focuses on the professional database, the Portal also houses resources for patients, caregivers and the general public, which may also be of interest to geriatric practitioners and researchers.

Identifying existing Choosing Wisely recommendations of high relevance and importance to hematology

Choosing Wisely (CW) is a medical stewardship initiative led by the American Board of Internal Medicine Foundation in collaboration with professional medical societies in the United States. In an effort to learn from and leverage the work of others, the American Society of Hematology CW Task Force developed a method to identify and prioritize CW recommendations from other medical societies of high relevance and importance to patients with blood disorders and their physicians. All 380 CW recommendations were reviewed and assessed for relevance and importance. Relevance was assessed using the MORE(TM) relevance scale. Importance was assessed with regard to six guiding principles: harm avoidance, evidence, aggregate cost, relevance, frequency and impact. Harm avoidance was considered the most important principle. Ten highly relevant and important recommendations were identified from a variety of professional societies. Recommendations focused on decreasing unnecessary imaging, blood work, treatments and transfusions, as well as on increasing collaboration across disciplines and considering value when recommending treatments. Many CW recommendations have relevance beyond the society of origin. The methods developed by the ASH CW Task Force could be easily adapted by other Societies to identify additional CW recommendations of relevance and importance to their fields. Am. J. Hematol. 91:787-792, 2016. © 2016 Wiley Periodicals, Inc.

'Peer review' for scientific manuscripts: Emerging issues, potential threats, and possible remedies

Reviewers play a vital role in ensuring quality control of scientific manuscripts published in any journal. The traditional double blind peer review, although a time-tested method, has come under increasing criticism in the face of emerging trends in the review process with the primary concern being the delays in completion of the review process. Other issues are the inability to detect errors/fraud, lack of transparency, lack of reliability, potential for bias, potential for unethical practices, lack of objectivity, inconsistencies amongst reviewers, lack of recognition and motivation of reviewers. Alternative options to classical peer review being propagated are: open review, immediate self-publication using preprint servers, nonselective review focusing primarily on the scientific content, and post-publication review. These alternative review processes, however, may suffer from the inability to validate quality control. In addition, anecdotal instances of peer review frauds are being reported more often than earlier. Suggested means to ensure quality of peer review process includes:(a) each journal to have its own database of reviewers, (b) verification of email IDs of reviewers provided by authors along with details of their institutions, (c) ensure credibility of reviewers before requesting for review, (d) check for plagiarism at the editorial level, (e) editors to distinguish between a good review from a possible biased/bad review, and (f) give recognition for reviewers once in a year. To conclude, quickness of review and publication should not dictate the scientific publication process at the cost of quality of contents.

[Is the age of causality over?]

For the assessment of diagnostic and therapeutic interventions a sound scientific base has been developed during the last twenty years. Under the headline of Evidence-based Medicine nowadays a comprehensive set of tools is offered which can be used to assess the benefit and the risk of medical interventions. The overarching rule which evolved for the grading of evidence from studies is to maximize the protection against bias. Despite this coherent approach, there is still controversy that is regularly mainly sparked by the dominant position of randomized controlled trials. Observational studies and registries are deemed to be more relevant because they provide results that are produced under "everyday conditions". These controversial discussions often show a lack of orientation, as they do without the explicit naming of scientific criteria for the evaluation and to a large extent rely on common sense. That the latter may not be a good guide for assessments in the medical field is known from numerous studies. For unbiased assessments the rigorous use of basic scientific principles is the only way. To express doubt and question these principles requires a scientific basis itself. The alternative is to move away from the established scientific foundation. The path to a "new" scientific paradigm is currently dominated by a discussion under the buzzword Big Data. Defined by the three V's of Variety, Velocity and Volume, a potential of the unlimited analysis of data is envisioned, for which there is currently no validation and whose logical foundations are extremely doubtful. The demand must be reaffirmed that instead of promises strict validation criteria be followed for the evaluation of all interventions in medicine, particularly in view of these developments.

Brief Resolved Unexplained Events (Formerly Apparent Life-Threatening Events) and Evaluation of Lower-Risk Infants

This is the first clinical practice guideline from the American Academy of Pediatrics that specifically applies to patients who have experienced an apparent life-threatening event (ALTE). This clinical practice guideline has 3 objectives. First, it recommends the replacement of the term ALTE with a new term, brief resolved unexplained event (BRUE). Second, it provides an approach to patient evaluation that is based on the risk that the infant will have a repeat event or has a serious underlying disorder. Finally, it provides management recommendations, or key action statements, for lower-risk infants. The term BRUE is defined as an event occurring in an infant younger than 1 year when the observer reports a sudden, brief, and now resolved episode of ≥1 of the following: (1) cyanosis or pallor; (2) absent, decreased, or irregular breathing; (3) marked change in tone (hyper- or hypotonia); and (4) altered level of responsiveness. A BRUE is diagnosed only when there is no explanation for a qualifying event after conducting an appropriate history and physical examination. By using this definition and framework, infants younger than 1 year who present with a BRUE are categorized either as (1) a lower-risk patient on the basis of history and physical examination for whom evidence-based recommendations for evaluation and management are offered or (2) a higher-risk patient whose history and physical examination suggest the need for further investigation and treatment but for whom recommendations are not offered. This clinical practice guideline is intended to foster a patient- and family-centered approach to care, reduce unnecessary and costly medical interventions, improve patient outcomes, support implementation, and provide direction for future research. Each key action statement indicates a level of evidence, the benefit-harm relationship, and the strength of recommendation.

Is There a Consensus when Physicians Evaluate the Relevance of Retrieved Systematic Reviews?

BACKGROUND

A significant challenge associated with practicing evidence-based medicine is to provide physicians with relevant clinical information when it is needed. At the same time it appears that the notion of relevance is subjective and its perception is affected by a number of contextual factors.

OBJECTIVES

To assess to what extent physicians agree on the relevance of evidence in the form of systematic reviews for a common set of patient cases, and to identify possible contextual factors that influence their perception of relevance.

METHODS

A web-based survey was used where pediatric emergency physicians from multiple academic centers across Canada were asked to evaluate the relevance of systematic reviews retrieved automatically for 14 written case vignettes (paper patients). The vignettes were derived from prospective data describing pediatric patients with asthma exacerbations presenting at the emergency department. To limit the cognitive burden on respondents, the number of reviews associated with each vignette was limited to three.

RESULTS

Twenty-two academic emergency physicians with varying years of clinical practice completed the survey. There was no consensus in their evaluation of relevance of the retrieved reviews and physicians' assessments ranged from very relevant to irrelevant evidence, with the majority of evaluations being somewhere in the middle. This indicates that the study participants did not share a notion of relevance uniformly. Further analysis of commentaries provided by the physicians allowed identifying three possible contextual factors: expected specificity of evidence (acute vs chronic condition), the terminology used in the systematic reviews, and the micro environment of clinical setting.

CONCLUSIONS

There is no consensus among physicians with regards to what constitutes relevant clinical evidence for a given patient case. Subsequently, this finding suggests that evidence retrieval systems should allow for deep customization with regards to physician's preferences and contextual factors, including differences in the micro environment of each clinical setting.

A process evaluation accompanying an attempted randomized controlled trial of an evidence service for health system policymakers

BACKGROUND

We developed an evidence service that draws inputs from Health Systems Evidence (HSE), which is a comprehensive database of research evidence about governance, financial and delivery arrangements within health systems and about implementation strategies relevant to health systems. Our goal was to evaluate whether, how and why a 'full-serve' evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care as compared to a 'self-serve' evidence service.

METHODS

We attempted to conduct a two-arm, 10-month randomized controlled trial (RCT), along with a follow-up qualitative process evaluation, but we terminated the RCT when we failed to reach our recruitment target. For the qualitative process evaluation we modified the original interview guide to allow us to explore the (1) factors influencing participation in the trial; (2) usage of HSE, factors explaining usage patterns, and strategies to increase usage; (3) participation in training workshops and use of other supports; and (4) views about and experiences with key HSE features.

RESULTS

We terminated the RCT given our 15% recruitment rate. Six factors were identified by those who had agreed to participate in the trial as encouraging their participation: relevance of the study to participants' own work; familiarity with the researchers; personal view of the importance of using research evidence in policymaking; academic background; support from supervisors; and participation of colleagues. Most reported that they never, infrequently or inconsistently used HSE and suggested strategies to increase its use, including regular email reminders and employee training. However, only two participants indicated that employee training, in the form of a workshop about finding and using research evidence, had influenced their use of HSE. Most participants found HSE features to be intuitive and helpful, although registration/sign-in and some page formats (particularly the advanced search page and detailed search results page) discouraged their use or did not optimize the user experience.

CONCLUSIONS

The qualitative findings informed a re-design of HSE, which allows users to more efficiently find and use research evidence about how to strengthen or reform health systems or in how to get cost-effective programs, services and drugs to those who need them. Our experience with RCT recruitment suggests the need to consider changing the unit of allocation to divisions instead of individuals within divisions, among other lessons.

TRIAL REGISTRATION

This protocol for this study is published in Implementation Science and registered with ClinicalTrials.gov ( HHS/FHS REB 10-267 ).

Blood Culture in Evaluation of Pediatric Community-Acquired Pneumonia: A Systematic Review and Meta-analysis

BACKGROUND AND OBJECTIVE

Current guidelines strongly recommend collection of blood cultures (BCs) in children requiring hospitalization for presumed moderate to severe bacterial community-acquired pneumonia (CAP). Our objective was to systematically review the international pediatric literature to evaluate how often BCs are positive in hospitalized children with CAP, identify the most commonly isolated pathogens, and determine the impact of positive BCs on clinical management.

METHODS

We identified articles in PubMed and Scopus published from January 1970 through December 2013 that addressed BCs in children with CAP. We extracted total number of BCs collected and prevalence of positive BCs and used meta-regression to evaluate whether subgroups had any impact on prevalence.

RESULTS

Meta-analysis showed that the overall prevalence of positive BCs was 5.14% (95% confidence interval 3.61-7.28). Studies focusing on severe CAP had a significant effect on prevalence (P=.008), at 9.89% (95% CI 6.79-14.19) compared with 4.17% (95% confidence interval 2.79-6.18) for studies not focusing on severe CAP. The most commonly isolated organisms were Streptococcus pneumoniae (76.7%) followed by Haemophilus influenzae (3.1%) and Staphylococcus aureus (2.1%). Contaminants accounted for 14.7%. Only 3 studies reported on BC-driven change in management, with contrasting findings.

CONCLUSIONS

BCs in pediatric CAP identified organisms in only a small percentage of patients, predominantly S. pneumoniae. False-positive BC rates can be substantial. The 3 studies that examined BC-driven changes in management had conflicting results. This systematic review was limited by heterogeneous case definitions, which may overestimate the true prevalence of positive BCs in hospitalized children.

Efficiency of pragmatic search strategies to update clinical guidelines recommendations

Background

A major challenge in updating clinical guidelines is to efficiently identify new, relevant evidence. We evaluated the efficiency and feasibility of two new approaches: the development of restrictive search strategies using PubMed Clinical Queries for MEDLINE and the use of the PLUS (McMaster Premium Literature Service) database.

Methods

We evaluated a random sample of recommendations from a national guideline development program and identified the references that would potentially trigger an update (key references) using an exhaustive approach.

We designed restrictive search strategies using the minimum number of Medical Subject Headings (MeSH) terms and text words required from the original exhaustive search strategies and applying broad and narrow filters. We developed PLUS search strategies, matching Medical Subject Headings (MeSH) and Systematized Nomenclature of Medicine (SNOMED) terms with guideline topics. We compared the number of key references retrieved by these approaches with those retrieved by the exhaustive approach.

Results

The restrictive approach retrieved 68.1 % fewer references than the exhaustive approach (12,486 versus 39,136), and identified 89.9 % (62/69) of key references and 88 % (22/25) of recommendation updates. The use of PLUS retrieved 88.5 % fewer references than the exhaustive approach (4,486 versus 39,136) and identified substantially fewer key references (18/69, 26.1 %) and fewer recommendation updates (10/25, 40 %).

Conclusions

The proposed restrictive approach is a highly efficient and feasible method to identify new evidence that triggers a recommendation update. Searching only in the PLUS database proved to be a suboptimal approach and suggests the need for topic-specific tailoring.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0058-2) contains supplementary material, which is available to authorized users.

How do I find a point-of-care answer to my clinical question?

Emergency physicians often need point-of-care access to current, valid information to guide patient management. Most emergency physicians do not work in a hospital with a computerized decision support system that prompts and provides them with information to answer their clinical questions. Searching for answers to clinical questions online, especially those related to diagnosis and treatment, can be challenging, in part because determining the validity and clinical applicability of the results of individual studies is beyond the time constraints of most emergency physicians. This article describes currently available point-of-care sources of evidence-based information to answer clinical questions and provides the access information for each.

An automated clinical alert system for newly-diagnosed atrial fibrillation

Objective

Clinical decision support systems that notify providers of abnormal test results have produced mixed results. We sought to develop, implement, and evaluate the impact of a computer-based clinical alert system intended to improve atrial fibrillation stroke prophylaxis, and identify reasons providers do not implement a guideline-concordant response.

Materials and Methods

We conducted a cohort study with historical controls among patients at a tertiary care hospital. We developed a decision rule to identify newly-diagnosed atrial fibrillation, automatically notify providers, and direct them to online evidence-based management guidelines. We tracked all notifications from December 2009 to February 2010 (notification period) and applied the same decision rule to all patients from December 2008 to February 2009 (control period). Primary outcomes were accuracy of notification (confirmed through chart review) and prescription of warfarin within 30 days.

Results

During the notification period 604 notifications were triggered, of which 268 (44%) were confirmed as newly-diagnosed atrial fibrillation. The notifications not confirmed as newly-diagnosed involved patients with no recent electrocardiogram at our institution. Thirty-four of 125 high-risk patients (27%) received warfarin in the notification period, compared with 34 of 94 (36%) in the control period (odds ratio, 0.66 [95% CI, 0.37–1.17]; p = 0.16). Common reasons to not prescribe warfarin (identified from chart review of 151 patients) included upcoming surgical procedure, choice to use aspirin, and discrepancy between clinical notes and the medication record.

Conclusions

An automated system to identify newly-diagnosed atrial fibrillation, notify providers, and encourage access to management guidelines did not change provider behaviors.

A stronger post-publication culture is needed for better science

Hilda Bastian considers post-publication commenting and the cultural changes that are needed to better capture this intellectual effort. Please see later in the article for the Editors' Summary.

Increasing the quantity and quality of searching for current best evidence to answer clinical questions: protocol and intervention design of the MacPLUS FS Factorial Randomized Controlled Trials

BACKGROUND & AIMS

Finding current best evidence for clinical decisions remains challenging. With 3,000 new studies published every day, no single evidence-based resource provides all answers or is sufficiently updated. McMaster Premium LiteratUre Service--Federated Search (MacPLUS FS) addresses this issue by looking in multiple high quality resources simultaneously and displaying results in a one-page pyramid with the most clinically useful at the top. Yet, additional logistical and educational barriers need to be addressed to enhance point-of-care evidence retrieval. This trial seeks to test three innovative interventions, among clinicians registered to MacPLUS FS, to increase the quantity and quality of searching for current best evidence to answer clinical questions.

METHODS & DESIGN

In a user-centered approach, we designed three interventions embedded in MacPLUS FS: (A) a web-based Clinical Question Recorder; (B) an Evidence Retrieval Coach composed of eight short educational videos; (C) an Audit, Feedback and Gamification approach to evidence retrieval, based on the allocation of 'badges' and 'reputation scores.' We will conduct a randomized factorial controlled trial among all the 904 eligible medical doctors currently registered to MacPLUS FS at the hospitals affiliated with McMaster University, Canada. Postgraduate trainees (n=429) and clinical faculty/staff (n=475) will be randomized to each of the three following interventions in a factorial design (AxBxC). Utilization will be continuously recorded through clinicians’ accounts that track logins and usage, down to the level of individual keystrokes. The primary outcome is the rate of searches per month per user during the six months of follow-up. Secondary outcomes, measured through the validated Impact Assessment Method questionnaire, include: utility of answers found (meeting clinicians’ information needs), use (application in practice), and perceived usefulness on patient outcomes.

DISCUSSION

Built on effective models for the point-of-care teaching, these interventions approach evidence retrieval as a clinical skill. If effective, they may offer the opportunity to enhance it for a large audience, at low cost, providing better access to relevant evidence across many top EBM resources in parallel.

TRIAL REGISTRATION

ClinicalTrials.Gov NCT02038439.

Best Evidence in Emergency Medicine (BEEM) rater scores correlate with publications' future citations

BACKGROUND

The "BEEM" (best evidence in emergency medicine) rater scale was created for emergency physicians (EPs) to evaluate the physician-derived clinical relevance score of recently published, emergency medicine (EM)-related studies. BEEM therefore is designed to help make EPs aware of studies most likely to confirm or change current clinical practice.

OBJECTIVES

The objective was to validate the BEEM rater score as a predictor of literature citation, using a bibliometric construct of clinical relevance to EM based on author-, document-, and journal-level measures (first and last author h-indices, number of authors including corporate and group authors, citations from date of publication to 2011, and journal impact factor scores) and study characteristics (design, category, and sample size).

METHODS

Each month from 2007 through 2012, approximately 200 EPs from around the world voluntarily reviewed the titles and conclusions of recently published EM-related studies identified by BEEM faculty via the McMaster Health Information Research Unit. Using the BEEM rater scale, a reliable seven-item instrument that evaluates the clinical relevance of studies, raters independently assigned BEEM scores to approximately 10 to 20 articles each month. Two investigators independently abstracted the bibliometric indices for these articles. A citation rate for each article was calculated by dividing the Thomson Reuters Web of Science (WoS) total citation count by the number of years in publication. BEEM rater scores were correlated with the citation rate using Spearman's rho. The performance of the BEEM rater score was assessed for each article using negative binomial regression with composite citation count as the criterion standard, while controlling for other independent bibliometric variables in three models.

RESULTS

The BEEM raters evaluated 605 articles with a mean (±SD) BEEM score of 3.84 (±0.7) and a median BEEM score of 3.85 (interquartile range = 3.38 to 4.30). Articles were primarily therapeutic (59%) and diagnostic (27%), with various designs, including 37% systematic reviews, 32% randomized controlled trials (RCTs), and 30% observational designs. The citation rate and BEEM rater score correlated positively (0.144), while the BEEM rater score and the Journal Citation Report (JCR) impact factor score were minimally correlated (0.053). In the first model, the BEEM rater score significantly predicted WoS citation rate (p < 0.0001) with an odds ratio (OR) of 1.24 (95% confidence interval [CI] = 1.106 to 1.402). In subsequent models adjusting for the JCR impact factor score, the h-indices of the first and last authors, number of authors, and study design, the BEEM rater score was not significant (p = 0.08).

CONCLUSIONS

To the best of our knowledge, the BEEM rater score is the only known measure of clinical relevance. It has a high interrater reliability and face validity and correlates with future citations. Future research should assess this instrument against alternative constructs of clinical relevance.

Twenty years of Cochrane reviews in menstrual disorders and subfertility

The past three decades have seen considerable change in the understanding of clinical research methods. There has been an acceptance that RCTs are the best way of establishing treatment effectiveness and a recognition that, while single studies are useful, pooling knowledge from a complete body of work is likely to provide the best evidence. Advances in methodology have been mirrored by the many advances in the field of reproductive medicine, such as assisted reproduction, assessment of male fertility, ovulation induction and laparoscopic surgery. Together, they have led to welcome improvements in the outcomes of fertility treatments. In particular, systematic reviews have become important tools enabling clinicians and patients to make health-care decisions based on evidence from all the available high-quality studies. The move towards identifying and aggregating the highest quality evidence has been led by the Cochrane Collaboration, which this year celebrates 20 years of preparing and publishing systematic reviews. This paper outlines the achievements, progress and challenges of this enterprise to date, with a particular focus on systematic reviews of reproductive medicine.

Strategies to assess the validity of recommendations: a study protocol

Background

Clinical practice guidelines (CPGs) become quickly outdated and require a periodic reassessment of evidence research to maintain their validity. However, there is little research about this topic. Our project will provide evidence for some of the most pressing questions in this field: 1) what is the average time for recommendations to become out of date?; 2) what is the comparative performance of two restricted search strategies to evaluate the need to update recommendations?; and 3) what is the feasibility of a more regular monitoring and updating strategy compared to usual practice?. In this protocol we will focus on questions one and two.

Methods

The CPG Development Programme of the Spanish Ministry of Health developed 14 CPGs between 2008 and 2009. We will stratify guidelines by topic and by publication year, and include one CPG by strata.

We will develop a strategy to assess the validity of CPG recommendations, which includes a baseline survey of clinical experts, an update of the original exhaustive literature searches, the identification of key references (reference that trigger a potential recommendation update), and the assessment of the potential changes in each recommendation.

We will run two alternative search strategies to efficiently identify important new evidence: 1) PLUS search based in McMaster Premium LiteratUre Service (PLUS) database; and 2) a Restrictive Search (ReSe) based on the least number of MeSH terms and free text words needed to locate all the references of each original recommendation.

We will perform a survival analysis of recommendations using the Kaplan-Meier method and we will use the log-rank test to analyse differences between survival curves according to the topic, the purpose, the strength of recommendations and the turnover. We will retrieve key references from the exhaustive search and evaluate their presence in the PLUS and ReSe search results.

Discussion

Our project, using a highly structured and transparent methodology, will provide guidance of when recommendations are likely to be at risk of being out of date. We will also assess two novel restrictive search strategies which could reduce the workload without compromising rigour when CPGs developers check for the need of updating.

Management of apparent life-threatening events in infants: a systematic review

OBJECTIVE

To determine in patients who are well-appearing and without a clear etiology after an apparent life-threatening event (ALTE): (1) What historical and physical examination features suggest that a child is at risk for a future adverse event and/or serious underlying diagnosis and would, therefore, benefit from testing or hospitalization? and (2) What testing is indicated on presentation and during hospitalization?

STUDY DESIGN

Systematic review of clinical studies, excluding case reports, published from 1970 through 2011 identified using key words for ALTE.

RESULTS

The final analysis was based on 37 studies; 18 prospective observational, 19 retrospective observational. None of the studies provided sufficient evidence to fully address the clinical questions. Risk factors identified from historical and physical examination features included a history of prematurity, multiple ALTEs, and suspected child maltreatment. Routine screening tests for gastroesophageal reflux, meningitis, bacteremia, and seizures are low yield in infants without historical risk factors or suggestive physical examination findings.

CONCLUSION

Some historical and physical examination features can be used to identify risk in infants who are well-appearing and without a clear etiology at presentation, and testing tailored to these risks may be of value. The true risk of a subsequent event or underlying disorder cannot be ascertained. A more precise definition of an ALTE is needed and further research is warranted.

The Clinical Relevance of Information Index (CRII): assessing the relevance of health information to the clinical practice

BACKGROUND

The high volume of health information creates a need for processes and tools to select, evaluate and disseminate relevant information to health professionals in clinical practice.

OBJECTIVES

To introduce an index of the clinical relevance of information and to show that it is different from existing measures.

METHODS

A conceptual model of knowledge translation was developed to explain the need for a new index, whose application was verified by an exploratory study with two (quantitative and qualitative) phases. The Clinical Relevance of Information Index (CRII) was defined employing descriptive statistical analyses of assessments performed by health professionals. The model and the CRII were applied in a primary healthcare context.

RESULTS

The CRII was applied to 4574 relevance assessments of 194 evidence synopses. The assessments were performed by 41 family physicians in 2008. The CRII value of each synopsis was compared with the number of citations received by its corresponding research paper and with the level of evidence of the study, presenting weak correlation with both.

CONCLUSION

The CRII captures aspects of information not considered by other indices. It can be a parameter for information providers, institutions, editors, as well as health and information professionals targeting knowledge translation.

How current are leading evidence-based medical textbooks? An analytic survey of four online textbooks

BACKGROUND

The consistency of treatment recommendations of evidence-based medical textbooks with more recently published evidence has not been investigated to date. Inconsistencies could affect the quality of medical care.

OBJECTIVE

To determine the frequency with which topics in leading online evidence-based medical textbooks report treatment recommendations consistent with more recently published research evidence.

METHODS

Summarized treatment recommendations in 200 clinical topics (ie, disease states) covered in four evidence-based textbooks--UpToDate, Physicians' Information Education Resource (PIER), DynaMed, and Best Practice--were compared with articles identified in an evidence rating service (McMaster Premium Literature Service, PLUS) since the date of the most recent topic updates in each textbook. Textbook treatment recommendations were compared with article results to determine if the articles provided different, new conclusions. From these findings, the proportion of topics which potentially require updating in each textbook was calculated.

RESULTS

478 clinical topics were assessed for inclusion to find 200 topics that were addressed by all four textbooks. The proportion of topics for which there was 1 or more recently published articles found in PLUS with evidence that differed from the textbooks' treatment recommendations was 23% (95% CI 17-29%) for DynaMed, 52% (95% CI 45-59%) for UpToDate, 55% (95% CI 48-61%) for PIER, and 60% (95% CI 53-66%) for Best Practice (χ(2) (3)=65.3, P<.001). The time since the last update for each textbook averaged from 170 days (range 131-209) for DynaMed, to 488 days (range 423-554) for PIER (P<.001 across all textbooks).

CONCLUSIONS

In online evidence-based textbooks, the proportion of topics with potentially outdated treatment recommendations varies substantially.

How well are journal and clinical article characteristics associated with the journal impact factor? a retrospective cohort study

OBJECTIVE

Journal impact factor (JIF) is often used as a measure of journal quality. A retrospective cohort study determined the ability of clinical article and journal characteristics, including appraisal measures collected at the time of publication, to predict subsequent JIFs.

METHODS

Clinical research articles that passed methods quality criteria were included. Each article was rated for relevance and newsworthiness by 3 to 24 physicians from a panel of more than 4,000 practicing clinicians. The 1,267 articles (from 103 journals) were divided 60∶40 into derivation (760 articles) and validation sets (507 articles), representing 99 and 88 journals, respectively. A multiple regression model was produced determining the association of 10 journal and article measures with the 2007 JIF.

RESULTS

Four of the 10 measures were significant in the regression model: number of authors, number of databases indexing the journal, proportion of articles passing methods criteria, and mean clinical newsworthiness scores. With the number of disciplines rating the article, the 5 variables accounted for 61% of the variation in JIF (R(2) = 0.607, 95% CI 0.444 to 0.706, P<0.001).

CONCLUSION

For the clinical literature, measures of scientific quality and clinical newsworthiness available at the time of publication can predict JIFs with 60% accuracy.

McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews

OBJECTIVE

We compared the performance of McMaster Premium LiteratUre Service (PLUS) and Clinical Queries (CQs) to that of the Cochrane Controlled Trials Register, MEDLINE, and EMBASE for locating studies added during an update of reviews.

STUDY DESIGN AND SETTING

A sample of new studies in updated Cochrane systematic reviews was used as a reference standard. Searches were performed for each study in each database. Where a new study was not indexed in PLUS, we examined the effect on the review of excluding the study.

RESULTS

Ninety-eight updated Cochrane reviews were identified. For the 87 reviews with a usable meta-analysis, PLUS contained all new studies for 13 reviews. No statistically significant difference between PLUS and non-PLUS new studies was found when ratio of odds ratios (RORs) were pooled across 39 reviews (ROR(⊕/⊖): 0.99; 95% confidence interval: 0.87-1.14). Thirty-five updated reviews had no new studies indexed in PLUS, but conclusions were seldom altered by addition of new studies.

CONCLUSIONS

PLUS included less than a quarter of the new studies in Cochrane updates, but most reviews appeared unaffected by the omission of these studies. Reviewers should consider adopting PLUS and CQ filters to improve the efficiency of keeping their reviews up to date.

Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians

BACKGROUND

Studies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed "Best Evidence in Emergency Medicine" (BEEM) to ameliorate these problems by searching for, identifying, appraising, and translating potentially practice-changing studies for EPs. An initial step in the BEEM process is the BEEM rater scale, a novel tool for EPs to collectively evaluate the relative clinical relevance of EM-related studies found in more than 120 journals. The BEEM rater process was designed to serve as a clinical relevance filter to identify those studies with the greatest potential to affect EM practice. Therefore, only those studies identified by BEEM raters as having the highest clinical relevance are selected for the subsequent critical appraisal process and, if found methodologically sound, are promoted as the best evidence in EM.

OBJECTIVES

The primary objective was to measure inter-rater reliability (IRR) of the BEEM rater scale. Secondary objectives were to determine the minimum number of EP raters needed for the BEEM rater scale to achieve acceptable reliability and to compare performance of the scale against a previously published evidence rating system, the McMaster Online Rating of Evidence (MORE), in an EP population.

METHODS

The authors electronically distributed the title, conclusion, and a PubMed link for 23 recently published studies related to EM to a volunteer group of 134 EPs. The volunteers answered two demographic questions and rated the articles using one of two randomly assigned seven-point Likert scales, the BEEM rater scale (n = 68) or the MORE scale (n = 66), over two separate administrations. The IRR of each scale was measured using generalizability theory.

RESULTS

The IRR of the BEEM rater scale ranged between 0.90 (95% confidence interval [CI] = 0.86 to 0.93) to 0.92 (95% CI = 0.89 to 0.94) across administrations. Decision studies showed a minimum of 12 raters is required for acceptable reliability of the BEEM rater scale. The IRR of the MORE scale was 0.82 to 0.84.

CONCLUSIONS

The BEEM rater scale is a highly reliable, single-question tool for a small number of EPs to collectively rate the relative clinical relevance within the specialty of EM of recently published studies from a variety of medical journals. It compares favorably with the MORE system because it achieves a high IRR despite simply requiring raters to read each article's title and conclusion.

Choosing health technology assessment and systematic review topics: the development of priority-setting criteria for patients' and consumers' interests

BACKGROUND

The Institute for Quality and Efficiency in Health Care (IQWiG) was established in 2003 by the German parliament. Its legislative responsibilities are health technology assessment, mostly to support policy making and reimbursement decisions. It also has a mandate to serve patients' interests directly, by assessing and communicating evidence for the general public.

OBJECTIVES

To develop a priority-setting framework based on the interests of patients and the general public.

METHODS

A theoretical framework for priority setting from a patient/consumer perspective was developed. The process of development began with a poll to determine level of lay and health professional interest in the conclusions of 124 systematic reviews (194 responses). Data sources to identify patients' and consumers' information needs and interests were identified.

RESULTS

IQWiG's theoretical framework encompasses criteria for quality of evidence and interest, as well as being explicit about editorial considerations, including potential for harm. Dimensions of "patient interest" were identified, such as patients' concerns, information seeking, and use. Rather than being a single item capable of measurement by one means, the concept of "patients' interests" requires consideration of data and opinions from various sources.

CONCLUSIONS

The best evidence to communicate to patients/consumers is right, relevant and likely to be considered interesting and/or important to the people affected. What is likely to be interesting for the community generally is sufficient evidence for a concrete conclusion, in a common condition. More research is needed on characteristics of information that interest patients and consumers, methods of evaluating the effectiveness of priority setting, and methods to determine priorities for disinvestment.

Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review

Background

Some drugs have a narrow therapeutic range and require monitoring and dose adjustments to optimize their efficacy and safety. Computerized clinical decision support systems (CCDSSs) may improve the net benefit of these drugs. The objective of this review was to determine if CCDSSs improve processes of care or patient outcomes for therapeutic drug monitoring and dosing.

Methods

We conducted a decision-maker-researcher partnership systematic review. Studies from our previous review were included, and new studies were sought until January 2010 in MEDLINE, EMBASE, Evidence-Based Medicine Reviews, and Inspec databases. Randomized controlled trials assessing the effect of a CCDSS on process of care or patient outcomes were selected by pairs of independent reviewers. A study was considered to have a positive effect (i.e., CCDSS showed improvement) if at least 50% of the relevant study outcomes were statistically significantly positive.

Results

Thirty-three randomized controlled trials were identified, assessing the effect of a CCDSS on management of vitamin K antagonists (14), insulin (6), theophylline/aminophylline (4), aminoglycosides (3), digoxin (2), lidocaine (1), or as part of a multifaceted approach (3). Cluster randomization was rarely used (18%) and CCDSSs were usually stand-alone systems (76%) primarily used by physicians (85%). Overall, 18 of 30 studies (60%) showed an improvement in the process of care and 4 of 19 (21%) an improvement in patient outcomes. All evaluable studies assessing insulin dosing for glycaemic control showed an improvement. In meta-analysis, CCDSSs for vitamin K antagonist dosing significantly improved time in therapeutic range.

Conclusions

CCDSSs have potential for improving process of care for therapeutic drug monitoring and dosing, specifically insulin and vitamin K antagonist dosing. However, studies were small and generally of modest quality, and effects on patient outcomes were uncertain, with no convincing benefit in the largest studies. At present, no firm recommendation for specific systems can be given. More potent CCDSSs need to be developed and should be evaluated by independent researchers using cluster randomization and primarily assess patient outcomes related to drug efficacy and safety.

Effects of an evidence service on community-based AIDS service organizations' use of research evidence: a protocol for a randomized controlled trial

BACKGROUND

To support the use of research evidence by community-based organizations (CBOs) we have developed 'Synthesized HIV/AIDS Research Evidence' (SHARE), which is an evidence service for those working in the HIV sector. SHARE consists of several components: an online searchable database of HIV-relevant systematic reviews (retrievable based on a taxonomy of topics related to HIV/AIDS and open text search); periodic email updates; access to user-friendly summaries; and peer relevance assessments. Our objective is to evaluate whether this 'full serve' evidence service increases the use of research evidence by CBOs as compared to a 'self-serve' evidence service.

METHODS/DESIGN

We will conduct a two-arm randomized controlled trial (RCT), along with a follow-up qualitative process study to explore the findings in greater depth. All CBOs affiliated with Canadian AIDS Society (n = 120) will be invited to participate and will be randomized to receive either the 'full-serve' version of SHARE or the 'self-serve' version (a listing of relevant systematic reviews with links to records on PubMed and worksheets that help CBOs find and use research evidence) using a simple randomized design. All management and staff from each organization will be provided access to the version of SHARE that their organization is allocated to. The trial duration will be 10 months (two-month baseline period, six-month intervention period, and two month crossover period), the primary outcome measure will be the mean number of logins/month/organization (averaged across the number of users from each organization) between baseline and the end of the intervention period. The secondary outcome will be intention to use research evidence as measured by a survey administered to one key decision maker from each organization. For the qualitative study, one key organizational decision maker from 15 organizations in each trial arm (n = 30) will be purposively sampled. One-on-one semi-structured interviews will be conducted by telephone on their views about and their experiences with the evidence service they received, how helpful it was in their work, why it was helpful (or not helpful), what aspects were most and least helpful and why, and recommendations for next steps.

DISCUSSION

To our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to support CBOs in finding and using research evidence.

TRIAL REGISTRATION

ClinicalTrials.gov: NCT01257724.

Effects of an evidence service on health-system policy makers' use of research evidence: a protocol for a randomised controlled trial

Background

Health-system policy makers need timely access to synthesised research evidence to inform the policy-making process. No efforts to address this need have been evaluated using an experimental quantitative design. We developed an evidence service that draws inputs from Health Systems Evidence, which is a database of policy-relevant systematic reviews. The reviews have been (a) categorised by topic and type of review; (b) coded by the last year searches for studies were conducted and by the countries in which included studies were conducted; (c) rated for quality; and (d) linked to available user-friendly summaries, scientific abstracts, and full-text reports. Our goal is to evaluate whether a "full-serve" evidence service increases the use of synthesized research evidence by policy analysts and advisors in the Ontario Ministry of Health and Long-Term Care (MOHLTC) as compared to a "self-serve" evidence service.

Methods/design

We will conduct a two-arm randomized controlled trial (RCT), along with a follow-up qualitative process study in order to explore the findings in greater depth. For the RCT, all policy analysts and policy advisors (n = 168) in a single division of the MOHLTC will be invited to participate. Using a stratified randomized design, participants will be randomized to receive either the "full-serve" evidence service (database access, monthly e-mail alerts, and full-text article availability) or the "self-serve" evidence service (database access only). The trial duration will be ten months (two-month baseline period, six-month intervention period, and two month cross-over period). The primary outcome will be the mean number of site visits/month/user between baseline and the end of the intervention period. The secondary outcome will be participants' intention to use research evidence. For the qualitative study, 15 participants from each trial arm (n = 30) will be purposively sampled. One-on-one semi-structured interviews will be conducted by telephone on their views about and their experiences with the evidence service they received, how helpful it was in their work, why it was helpful (or not helpful), what aspects were most and least helpful and why, and recommendations for next steps.

Discussion

To our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically designed to support health-system policy makers in finding and using research evidence.

Trial registration

ClinicalTrials.gov: NCT01307228

Supporting the use of health technology assessments in policy making about health systems

Objectives:The objective of this study is to profile the health technology assessments (HTAs) produced in Canada and other selected countries and assess their potential to inform policy making about health systems in jurisdictions other than the ones for which they were produced, and to develop and pilot test prototypes for packaging and assessing the relevance of HTAs for health system managers and policy makers.

Methods:We compiled an inventory of all HTAs that were produced by nine HTA agencies between September 2003 and August 2006; coded the title and abstract of each HTA according to the technologies assessed, methods used, and whether or not context-specific actionable messages were provided; developed a prototype for a structured, decision-relevant HTA summary and for a relevance-assessment form; and pilot-tested the prototypes using semistructured telephone interviews with a purposive sample of Canadian healthcare managers and policy makers.

Results:Our review of the 223 HTAs identified that: (i) 44 HTAs addressed health system arrangements (20 percent); (ii) 205 incorporated a systematic review (92 percent), whereas only 12 incorporated a sociopolitical assessment using explicit methods (5 percent); and (iii) 50 contained context-specific actionable messages (22 percent). Our interviews identified significant support for both the general idea of an HTA summary and the prototype's specific elements, but mixed views about using peer assessments of relevance.

Conclusions:Those involved in supporting the use of HTAs in policy making about health systems may wish to produce structured decision-relevant summaries for their systematic review-containing HTAs to increase the prospects for their HTAs being used outside the jurisdiction for which they were produced.

Surveying the medical literature: five notable articles in general internal medicine from 2008 and 2009

Given the vast and growing volume of medical literature, it is essential to develop reliable strategies for identifying articles of importance and relevance. Here, we summarize 5 notable articles for general internal medicine published in 2008 and 2009. Clinical vignettes are presented to illustrate situations in which each study might apply, and each summary ends with a description of how a physician might use the study findings to resolve the vignette case. Finally, we describe a surveillance strategy that physicians can use to identify articles important to their own practices.

Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) invokes evidence-based medicine (EBM) principles through the practice-based learning core competency. The authors hypothesized that among a representative sample of emergency medicine (EM) residency programs, a wide variability in EBM resident training priorities, faculty expertise expectations, and curricula exists.

OBJECTIVES

The primary objective was to obtain descriptive data regarding EBM practices and expectations from EM physician educators. Our secondary objective was to assess differences in EBM educational priorities among journal club directors compared with non-journal club directors.

METHODS

A 19-question survey was developed by a group of recognized EBM curriculum innovators and then disseminated to Council of Emergency Medicine Residency Directors (CORD) conference participants, assessing their opinions regarding essential EBM skill sets and EBM curricular expectations for residents and faculty at their home institutions. The survey instrument also identified the degree of interest respondents had in receiving a free monthly EBM journal club curriculum.

RESULTS

A total of 157 individuals registered for the conference, and 98 completed the survey. Seventy-seven (77% of respondents) were either residency program directors or assistant/associate program directors. The majority of participants were from university-based programs and in practice at least 5 years. Respondents reported the ability to identify flawed research (45%), apply research findings to patient care (43%), and comprehend research methodology (33%) as the most important resident skill sets. The majority of respondents reported no formal journal club or EBM curricula (75%) and do not utilize structured critical appraisal instruments (71%) when reviewing the literature. While journal club directors believed that resident learners' most important EBM skill is to identify secondary peer-reviewed resources, non-journal club directors identified residents' ability to distinguish significantly flawed research as the key skill to develop. Interest in receiving a free monthly EBM journal club curriculum was widely accepted (89%).

CONCLUSIONS

Attaining EBM proficiency is an expected outcome of graduate medical education (GME) training, although the specific domains of anticipated expertise differ between faculty and residents. Few respondents currently use a formalized curriculum to guide the development of EBM skill sets. There appears to be a high level of interest in obtaining EBM journal club educational content in a structured format. Measuring the effects of providing journal club curriculum content in conjunction with other EBM interventions may warrant further investigation.

Semantic Space models for classification of consumer webpages on metadata attributes

To deal with the quantity and quality issues with online healthcare resources, creating web portals centred on particular health topics and/or communities of users is a strategy to provide access to a reduced corpus of information resources that meet quality and relevance criteria. In this paper we use hyperspace analogue to language (HAL) to model the language use patterns of webpages as Semantic Spaces. We have applied machine learning methods, including support vector machine (SVM), decision forest, and a novel summed similarity measure (SSM) to automatically classify online webpages on their Semantic Space models. We find classification accuracy on metadata attributes to be over 93% for 'medical' versus 'supportive' perspective, over 92% for disease stage of 'early' versus 'advanced', and over 90% for author credentials of 'lay' versus 'clinician' based on webpages of the Breast Cancer Knowledge Online portal. These results indicate that language use patterns can be used to automate such classification with useful levels of accuracy.

Community-based knowledge transfer and exchange: helping community-based organizations link research to action

BACKGROUND

Community-based organizations (CBOs) are important stakeholders in health systems and are increasingly called upon to use research evidence to inform their advocacy, program planning, and service delivery efforts. CBOs increasingly turn to community-based research (CBR) given its participatory focus and emphasis on linking research to action. In order to further facilitate the use of research evidence by CBOs, we have developed a strategy for community-based knowledge transfer and exchange (KTE) that helps CBOs more effectively link research evidence to action. We developed the strategy by: outlining the primary characteristics of CBOs and why they are important stakeholders in health systems; describing the concepts and methods for CBR and for KTE; comparing the efforts of CBR to link research evidence to action to those discussed in the KTE literature; and using the comparison to develop a framework for community-based KTE that builds on both the strengths of CBR and existing KTE frameworks.

DISCUSSION

We find that CBR is particularly effective at fostering a climate for using research evidence and producing research evidence relevant to CBOs through community participation. However, CBOs are not always as engaged in activities to link research evidence to action on a larger scale or to evaluate these efforts. Therefore, our strategy for community-based KTE focuses on: an expanded model of 'linkage and exchange' (i.e., producers and users of researchers engaging in a process of asking and answering questions together); a greater emphasis on both producing and disseminating systematic reviews that address topics of interest to CBOs; developing a large-scale evidence service consisting of both 'push' efforts and efforts to facilitate 'pull' that highlight actionable messages from community relevant systematic reviews in a user-friendly way; and rigorous evaluations of efforts for linking research evidence to action.

SUMMARY

Through this type of strategy, use of research evidence for CBO advocacy, program planning, and service delivery efforts can be better facilitated and continually refined through ongoing evaluations of its impact.

Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?

Background

Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).

Methodology/Principal Findings

We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19–0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12–0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01).

Conclusions/Significance

Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review

Background

Computerized clinical decision support systems are information technology-based systems designed to improve clinical decision-making. As with any healthcare intervention with claims to improve process of care or patient outcomes, decision support systems should be rigorously evaluated before widespread dissemination into clinical practice. Engaging healthcare providers and managers in the review process may facilitate knowledge translation and uptake. The objective of this research was to form a partnership of healthcare providers, managers, and researchers to review randomized controlled trials assessing the effects of computerized decision support for six clinical application areas: primary preventive care, therapeutic drug monitoring and dosing, drug prescribing, chronic disease management, diagnostic test ordering and interpretation, and acute care management; and to identify study characteristics that predict benefit.

Methods

The review was undertaken by the Health Information Research Unit, McMaster University, in partnership with Hamilton Health Sciences, the Hamilton, Niagara, Haldimand, and Brant Local Health Integration Network, and pertinent healthcare service teams. Following agreement on information needs and interests with decision-makers, our earlier systematic review was updated by searching Medline, EMBASE, EBM Review databases, and Inspec, and reviewing reference lists through 6 January 2010. Data extraction items were expanded according to input from decision-makers. Authors of primary studies were contacted to confirm data and to provide additional information. Eligible trials were organized according to clinical area of application. We included randomized controlled trials that evaluated the effect on practitioner performance or patient outcomes of patient care provided with a computerized clinical decision support system compared with patient care without such a system.

Results

Data will be summarized using descriptive summary measures, including proportions for categorical variables and means for continuous variables. Univariable and multivariable logistic regression models will be used to investigate associations between outcomes of interest and study specific covariates. When reporting results from individual studies, we will cite the measures of association and p-values reported in the studies. If appropriate for groups of studies with similar features, we will conduct meta-analyses.

Conclusion

A decision-maker-researcher partnership provides a model for systematic reviews that may foster knowledge translation and uptake.

Competency-based continuing professional development

Competence is traditionally viewed as the attainment of a static set of attributes rather than a dynamic process in which physicians continuously use their practice experiences to "progress in competence" toward the attainment of expertise. A competency-based continuing professional development (CPD) model is premised on a set of learning competencies that include the ability to (a) use practice information to identify learning priorities and to develop and monitor CPD plans; (b) access information sources for innovations in development and new evidence that may potentially be integrated into practice; (c) establish a personal knowledge management system to store and retrieve evidence and to select and manage learning projects; (d) construct questions, search for evidence, and record and track conclusions for practice; and (e) use tools and processes to measure competence and performance and develop action plans to enhance practice. Competency-based CPD emphasizes self-directed learning processes and promotes the role of assessment as a professional expectation and obligation. Various approaches to defining general competencies for practice require the creation of specific performance metrics to be meaningful and relevant to the lifelong learning strategies of physicians. This paper describes the assumptions, advantages, and challenges of establishing a CPD system focused on competencies that improve physician performance and the quality and safety of patient care. Implications for competency-based CPD are discussed from an individual and organizational perspective, and a model to bridge the transition from residency to practice is explored.

Where is the wisdom? I--a conceptual history of evidence-based medicine

Rationale Evidence-based medicine (EBM) has been acclaimed as a major advance in medical science, but criticized as a proposed alternative model for the practice and teaching of medicine. Ambiguity regarding the proper role of the contributions of EBM within the fabric of medicine and health care has contributed to this discrepancy. Aims and objectives We undertook a critical review of the history of the EBM movement, beginning with its origins in the 1970s and continuing through this century. We drew upon the results of an independent project that rationalized the EBM domain from the perspective of educational evaluation and assessment. We considered the content of EBM in relationship to the propositions and promises embodied in advocacy publications. Results EBM emerged in the context of the explosion of biomedical information in the decade preceding public access to the Internet in the mid-1990s and drew upon the independently derived 'information literacy' formula developed by information scientists during the 1980s. The critically important content and achievements of EBM are fully explained within the confines of the information literacy model. The thesis that EBM offers an alternative paradigm for individualized health care, asserted in the advocacy literature, is not supported by published models of evidence-based clinical practice. Conclusion A critical historical review of the origins, content and development of the EBM movement proposes that full integration of the fruits of the movement into routine clinical care remains a conceptual and practical challenge.

Self-reported priorities and resources of academic emergency physicians for the maintenance of clinical competence: a pilot study

OBJECTIVES

Medical licensing bodies and professional colleges require their members to maintain a broad spectrum of knowledge, skills and attitudes, which, when taken together, define a competent emergency physician (EP). The objectives of this pilot study were: 1) to determine the resources used by academic EPs to maintain competence and 2) to determine academic EPs' learning priorities.

METHODS

Using a modified Dillman method, we surveyed EPs from 2 Canadian academic tertiary health sciences centres.

RESULTS

Thirty-seven (68.5%) of 54 EPs responded. Of those responding, 14 (37.8%) attended grand rounds 3 times or more annually, and 34 (91.7%) attended a medical conference or course at least once annually. Thirty-three (89.2%) respondents read journal articles at least once monthly, with 22 (59.5%) of those reading synopses of original articles. Twenty-three (62.1%) received clinical updates via email, and 11 (29.7%) subscribed to an audio journal or podcast of reviews of original research. Among the CanMEDS roles, Medical Expert, Scholar and Manager were selected as top professional development priorities by more than one-third of respondents. The topics that were not selected as priorities by respondents included patient communication and charting (Communicator); conflict resolution skills and teamwork abilities (Collaborator); advocate for patient and promote health in emergency department populations (Health Advocate) and ethical conflict resolution (Professional).

CONCLUSION

The results of this pilot study suggest that in order to maintain clinical competence in emergency medicine, traditional formats of professional development (e.g., grand rounds, print media and original research) are being substituted for independent study, online media and reviews of original research. This study also suggests a strong preference for Medical Expert topics, while Professional, Health Advocate, Collaborator and Communicator topics are not a reported priority for professional development.

Can the highly cited psychiatric paper be predicted early?

OBJECTIVE

Predicting the impact of any research article on its scientific discipline is often viewed as requiring the passage of time. A recent BMJ article, however, reported that an article's citation rate at 2 years could be predicted by data available 3 weeks following publication. The question remains as to whether establishing a citation trajectory at an early stage holds for psychiatric publications, given the low percentage of psychiatric articles in their analysis. The aim of the current article was to critically examine this area of the scientific literature.

METHOD

Data were collected from the Institute for Scientific Information on scientific papers published in January/February 2006, in the top 30 psychiatric journals. Analyses examined the comparative impact of early citation numbers and several predictors identified in the BMJ article.

RESULTS

Only two BMJ variables (a larger number of references per article and larger number of authors) predicted higher citations at 2 years in the principal analysis. Citation counts at 1, 3, 6 and 12 months predicted citations at 2 years, with increasing success over time, and such citation counts were distinctly superior to the quantified variables in the previous study.

CONCLUSIONS

It appears doubtful that data available at 3 weeks after publication for psychiatric articles are useful in predicting citation counts at 2 years. The trajectory of citation counts for a psychiatric article becomes more apparent with time.