Answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed

Question answering systems for health professionals at the point of care-a systematic review

OBJECTIVES

Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement.

MATERIALS AND METHODS

We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology, and forward and backward citations on February 7, 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems.

RESULTS

We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians.

DISCUSSION

While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.

Effective web-based clinical practice guidelines resources: recommendations from a mixed methods usability study

BACKGROUND

Clinical practice guidelines (CPG) are an important knowledge translation resource to help clinicians stay up to date about relevant clinical knowledge. Effective communication of guidelines, including format, facilitates its implementation. Despite the digitalization of healthcare, there is little literature to guide CPG website creation for effective dissemination and implementation. Our aim was to assess the effectiveness of the content and format of the Diabetes Canada CPG website, and use our results to inform recommendations for other CPG websites.

METHODS

Fourteen clinicians (family physicians, nurses, pharmacists, and dieticians) in diabetes care across Canada participated in this mixed-methods study (questionnaires, usability testing and interviews). Participants "thought-aloud" while completing eight usability tasks on the CPG website. Outcomes included task success rate, completion time, click per tasks, resource used, paths, search attempts and success rate, and error types. Participants were then interviewed.

RESULTS

The Diabetes Canada CPG website was found to be usable. Participants had a high task success rate of 79% for all tasks and used 144 (standard deviation (SD) = 152) seconds and 4.6 (SD = 3.9) clicks per task. Interactive tools were most frequently used compared to full guidelines and static tools. Misinterpretation accounted for 48% of usability errors. Participants overall found the website intuitive, with effective content and design elements.

CONCLUSION

Different versions of CPG information (e.g. interactive tools, quick reference guide, static tools) can help answer clinical questions more quickly. Effective web design should be assessed during CPG website creation for effective guideline dissemination and implementation.

The Development History and Research Tendency of Medical Informatics: Topic Evolution Analysis

BACKGROUND

Medical informatics has attracted the attention of researchers worldwide. It is necessary to understand the development of its research hot spots as well as directions for future research.

OBJECTIVE

The aim of this study is to explore the evolution of medical informatics research topics by analyzing research articles published between 1964 and 2020.

METHODS

A total of 56,466 publications were collected from 27 representative medical informatics journals indexed by the Web of Science Core Collection. We identified the research stages based on the literature growth curve, extracted research topics using the latent Dirichlet allocation model, and analyzed topic evolution patterns by calculating the cosine similarity between topics from the adjacent stages.

RESULTS

The following three research stages were identified: early birth, early development, and rapid development. Medical informatics has entered the fast development stage, with literature growing exponentially. Research topics in medical informatics can be classified into the following two categories: data-centered studies and people-centered studies. Medical data analysis has been a research hot spot across all 3 stages, and the integration of emerging technologies into data analysis might be a future hot spot. Researchers have focused more on user needs in the last 2 stages. Another potential hot spot might be how to meet user needs and improve the usability of health tools.

CONCLUSIONS

Our study provides a comprehensive understanding of research hot spots in medical informatics, as well as evolution patterns among them, which was helpful for researchers to grasp research trends and design their studies.

A study on how using an interactive multimedia e-book improves teachers' ability to teach evidence-based medicine depending on their seniority

BACKGROUND

Teaching evidence-based medicine (EBM) is not an easy task. The role of the electronic book (e-book) is a useful supplement to traditional methods for improving skills. Our aim is to use an interactive e-book or PowerPoint to evaluate instructors' teaching effects on EBM.

METHODS

Our study group was introduced to learning EBM using an interactive e-book available on the Internet, while the control group used a PowerPoint presentation. We adopted the Modified Fresno test to assess EBM skills both before and after their learning. EBM teaching sessions via e-book or PowerPoint were 20-30 min long, followed by students' feedback. We adopted Student's t-test to compare teachers' evaluation of their EBM skills prior to the class and the students' assessment of the teachers' instruction. We also adopted repeated measures ANCOVA to compare teachers' evaluation of their EBM skills using the Fresno test both before and after the class.

RESULTS

We observed no difference regarding EBM skills between the two groups prior to their experimental learning, which was assessed by the Modified Fresno test. After learning, physicians in the study group ranked higher in choosing a case to explain which kind of research design was used for the study type of the question and explaining their choice (P = 0.024) as assessed by the post-test to pre-test Fresno test. Teaching effect was better in the e-book group than in the control group for the items, "I am satisfied with this lesson," "The teaching was of high quality," "This was a good teaching method," and "It aroused my interest in EBM." However, no differences were observed between the two groups in physicians who had more than 10 years' experience.

CONCLUSIONS

The use of interactive e-books in clinical teaching can enhance a teacher's EBM skills, though not in more senior physicians. This may suggest that teaching methodology and activities differ for teachers' varying years of experience.

Successes, lessons and opportunities: 15-year follow-up of an integrated evidence-based medicine curriculum

In 2005, the Department of Family Medicine at the University of Alberta introduced an evidence-based practice curriculum into the 2-year Family Medicine Residency Program. The curriculum was based on best available evidence, had multiple components and was comprehensive in its approach. It prioritised preappraised summary evidence over in-depth evidence appraisal. This paper describes the lessons learnt over the past 15 years including components that were eventually discontinued. We also discuss additions to the programme including the development of accessible, preappraised, summarised resources. We review the difficulties associated with evaluation and the incorporation of evidence-based practice into all aspects of residency training. Future directions are discussed including the incorporation of shared decision-making at the point of care.

Is Electronic Health Literacy Associated with Learning Outcomes among Medical Students in the First Clinical Year?: A Cross-Sectional Study

Medical students tend to use the internet as a primary resource when seeking health information. This study aims to assess the patterns of internet use, eHL level, and learning outcomes with eHL among medical students at Chiang Mai University. A cross-sectional study was conducted among 88 medical students in the first clinical year. The eHL level was determined using the Thai version of the electronic Health Literacy Scale or eHEALS. The patient case report scores were obtained representing the learning outcome. Linear regression was used to identify factors influencing their eHL level and case report scores. Students recognized the importance and usefulness of the internet. The mean eHEALS score was 33.45. There was a lower degree of agreement on questions regarding internet usage, having skills to evaluate the resources, and confidence in using health information to make health decisions. The eHEALS score had no statistically significant association with most variables and case report scores, but with the longer time of internet use (p-value = 0.014). Although medical students perceived that they have high eHL levels, they report lower confidence in using the information. Including critical thinking skills for electronic health information in the medical curriculum could be useful.

UpToDate versus DynaMed: a cross-sectional study comparing the speed and accuracy of two point-of-care information tools

Objective:

To compare the accuracy, time to answer, user confidence, and user satisfaction between UpToDate and DynaMed (formerly DynaMed Plus), which are two popular point-of-care information tools.

Methods:

A crossover study was conducted with medical residents in obstetrics and gynecology and family medicine at the University of Toronto in order to compare the speed and accuracy with which they retrieved answers to clinical questions using UpToDate and DynaMed. Experiments took place between February 2017 and December 2019. Following a short tutorial on how to use each tool and completion of a background survey, participants attempted to find answers to two clinical questions in each tool. Time to answer each question, the chosen answer, confidence score, and satisfaction score were recorded for each clinical question.

Results:

A total of 57 residents took part in the experiment, including 32 from family medicine and 25 from obstetrics and gynecology. Accuracy in clinical answers was equal between UpToDate (average 1.35 out of 2) and DynaMed (average 1.36 out of 2). However, time to answer was 2.5 minutes faster in UpToDate compared to DynaMed. Participants were also more confident and satisfied with their answers in UpToDate compared to DynaMed.

Conclusions:

Despite a preference for UpToDate and a higher confidence in responses, the accuracy of clinical answers in UpToDate was equal to those in DynaMed. Previous exposure to UpToDate likely played a major role in participants' preferences. More research in this area is recommended.

How searching under time pressure impacts clinical decision making

Objective:

Clinicians encounter many questions during patient encounters that they cannot answer. While search systems (e.g., PubMed) can help clinicians find answers, clinicians are typically busy and report that they often do not have sufficient time to use such systems. The objective of this study was to assess the impact of time pressure on clinical decisions made with the use of a medical literature search system.

Design:

In stage 1, 109 final-year medical students and practicing clinicians were presented with 16 clinical questions that they had to answer using their own knowledge. In stage 2, the participants were provided with a search system, similar to PubMed, to help them to answer the same 16 questions, and time pressure was simulated by limiting the participant's search time to 3, 6, or 9 minutes per question.

Results:

Under low time pressure, the correct answer rate significantly improved by 32% when the participants used the search system, whereas under high time pressure, this improvement was only 6%. Also, under high time pressure, participants reported significantly lower confidence in the answers, higher perception of task difficulty, and higher stress levels.

Conclusions:

For clinicians and health care organizations operating in increasingly time-pressured environments, literature search systems become less effective at supporting accurate clinical decisions. For medical search system developers, this study indicates that system designs that provide faster information retrieval and analysis, rather than traditional document search, may provide more effective alternatives.

Using a Collaborative, Virtual Discussion Platform to Mobilize Oncologic Expertise for the COVID-19 Pandemic

PURPOSE

COVID-19 is a rapidly emerging worldwide pandemic that has drastically changed health care across the United States. Oncology patients are especially vulnerable. Novel point-of-care resources may be useful to rapidly disseminate peer-reviewed information from oncology experts nationwide. We describe our initial experience with distributing this information through a private, curated, virtual collaboration question-and-answer (Q&A) platform for oncologists.

METHODS

The Q&A database was queried for a 2-month period from March 12 to May 12, 2020. We collected the total number of views and unique viewers for the questions. We classified the questions according to their emphasis (practice management, clinical management, both) and disease type across radiation oncology, medical oncology, gynecologic oncology, and pediatric oncology.

RESULTS

Seventy-nine questions were approved, 67 of which were answered and generated 49,494 views with 5,148 unique viewers. Most discussions covered clinical management, with breast cancer being the most active disease site. Ten questions covered pediatric oncology and gynecologic oncology. Forty-seven percent of the 11,010 users of the platform visited the website during the 2-month period.

CONCLUSION

Discussions on the Q&A platform reached a substantial number of oncologists throughout the nation and may help oncologists to modify their treatment in real time with the rapidly evolving COVID-19 pandemic.

A Mobile Medical Knowledge Dissemination Platform (HeadToToe): Mixed Methods Study

BACKGROUND

Finding readily accessible, high-quality medical references can be a challenging task. HeadToToe is a mobile platform designed to allow easy and quick access to sound, up-to-date, and validated medical knowledge and guidance. It provides easy access to essential clinical medical content in the form of documents, videos, clinical scores, and other formats for the day-to-day access and use by medical students and physicians during their pre- and postgraduate education.

OBJECTIVE

The aim of this paper is to describe the architecture, user interface, and potential strengths and limitations of an innovative knowledge dissemination platform developed at the University of Geneva, Switzerland. We also report preliminary results from a user-experience survey and usage statistics over a selected period.

METHODS

The dissemination platform consists of a smartphone app. Through an administration interface, content is managed by senior university and hospital staff. The app includes the following sections: (1) main section of medical guidance, organized by clinical field; (2) checklists for history-taking and clinical examination, organized by body systems; (3) laboratory section with frequently used lab values; and (4) favorites section. Each content item is programmed to be available for a given duration as defined by the content's author. Automatic notifications signal the author when the content is about to expire, hence, promoting its timely updating and reducing the risk of using obsolete content. In the background, a third-party statistical collecting tool records anonymous utilization statistics.

RESULTS

We launched the final version of the platform in March 2019, both at the Faculty of Medicine at the University of Geneva and at the University Hospital of Geneva in Switzerland. A total of 622 students at the university and 613 health professionals at the hospital downloaded the app. Two-thirds of users at both institutions had an iOS device. During the practical examination period (ie, May 2019) there was a significant increase in the number of active users (P=.003), user activity (P<.001), and daily usage time (P<.001) among medical students. In addition, there were 1086 clinical skills video views during this period compared to a total of 484 in the preceding months (ie, a 108% increase). On a 10-point Likert scale, students and physicians rated the app with mean scores of 8.2 (SD 1.9) for user experience, 8.1 (SD 2.0) for usefulness, and 8.5 (SD 1.8) for relevance of content. In parallel, postgraduate trainees viewed more than 6000 documents during the first 3 months after the implementation in the Division of Neurology at our institution.

CONCLUSIONS

HeadToToe is an educator-driven, mobile dissemination platform, which provides rapid and user-friendly access to up-to-date medical content and guidance. The platform was given high ratings for user experience, usefulness, and content quality and was used more often during the exam period. This suggests that the platform could be used as tool for exam preparation.

Development of a contemporary evidence-based practice workshop for health professionals with a focus on pre-appraised evidence and shared decision-making: a before-after pilot study

Shared decision-making (SDM) has emerged as a key skill to assist clinicians in applying evidence-based practice (EBP). We aimed to develop and pilot a new approach to teaching EBP, which focuses on teaching knowledge and skills about SDM and pre-appraised evidence. We designed a half-day workshop, informed by an international consensus on EBP core competencies and invited practicing clinicians to participate. Skills in SDM and communicating evidence were assessed by audio-recording consultations between clinicians and standardised patients (immediately pre-workshop and post-workshop). These were rated by two independent assessors using the OPTION (Observing Patient Involvement, 0 to 100 points) and ACEPP (Assessing Communication about Evidence and Patient Preferences, 0 to 5 points) tools. Participants also completed a feedback questionnaire (9 Likert scale and four open-ended questions). Fourteen clinicians participated. Skills in SDM and communicating research evidence improved from pre-workshop to post-workshop (mean increase in OPTION score=5.5, 95% CI 1.0 to 9.9; increase in ACEPP score=0.5, 95% CI 0.02 to 1.06). Participant feedback was positive, with most indicating 'agree' or 'strongly agree' to the questions. A contemporary approach to teaching clinicians EBP, with a focus on SDM and pre-appraised evidence, was feasible, perceived as useful, and showed modest improvements in skills. Results should be interpreted cautiously because of the small study size and pre-post design.

Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis

Background

Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use.

Objective

The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes.

Methods

We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies.

Results

Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I²>50%) across studies. When compared with other resources—7 studies, not amenable to meta-analytic pooling—the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource.

Conclusions

Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.

Information seeking behavior and awareness among physicians regarding drug information centers in Saudi Arabia

Background:

The role of Drug Information Center (DIC) in a health-care setting has increased tremendously owing to the high influx of pharmaceutical molecules that pose serious challenges to physicians. DIC promotes rational prescribing behavior among physicians, leading to better patient outcome.

Objectives:

This study aimed to explore information-seeking behaviors and awareness of physicians regarding DIC services in the Kingdom of Saudi Arabia.

Methods:

A cross-sectional study was conducted among physicians working in government and private sectors between June to November 2018 by using an 18-item electronic anonymous questionnaire. Descriptive and inferential statistics were performed using IBM SPSS (Version 21). A P-value of <0.05 was taken as the level of significance between responses.

Results:

In total, 500 questionnaires were distributed among the included hospitals, and only 254 physicians (response rate: 50.8%), including 193 males (76%), participated in the study. The majority of participants (n = 83, 32.7%) had more than ten years of experience, and many of the respondents (n=131) worked as residents. Most of the physicians (62.9%) were aware of their institutional DIC. UpToDate was the most preferred drug information database among physicians. Regarding the improvement required in the DIC services, most of the physicians (23.6%) opined that the contact details should be available in all clinical wards.

Conclusions:

Only 10% of the respondents were not aware of the presence of DIC at their institution. The UpToDate online drug information database was the most frequently used database by the physicians. Our findings showed that there is a need for conducting educational programs for physicians regarding DIC services. Such an attempt can increase the frequency of drug-related queries and promote patient safety.

Impact of a Search Engine on Clinical Decisions Under Time and System Effectiveness Constraints: Research Protocol

BACKGROUND

Many clinical questions arise during patient encounters that clinicians are unable to answer. An evidence-based medicine approach expects that clinicians will seek and apply the best available evidence to answer clinical questions. One commonly used source of such evidence is scientific literature, such as that available through MEDLINE and PubMed. Clinicians report that 2 key reasons why they do not use search systems to answer questions is that it takes too much time and that they do not expect to find a definitive answer. So, the question remains about how effectively scientific literature search systems support time-pressured clinicians in making better clinical decisions. The results of this study are important because they can help clinicians and health care organizations to better assess their needs with respect to clinical decision support (CDS) systems and evidence sources. The results and data captured will contribute a significant data collection to inform the design of future CDS systems to better meet the needs of time-pressured, practicing clinicians.

OBJECTIVE

The purpose of this study is to understand the impact of using a scientific medical literature search system on clinical decision making. Furthermore, to understand the impact of realistic time pressures on clinicians, we vary the search time available to find clinical answers. Finally, we assess the impact of improvements in search system effectiveness on the same clinical decisions.

METHODS

In this study, 96 practicing clinicians and final year medical students are presented with 16 clinical questions which they must answer without access to any external resource. The same questions are then represented to the clinicians; however, in this part of the study, the clinicians can use a scientific literature search engine to find evidence to support their answers. The time pressures of practicing clinicians are simulated by limiting answer time to one of 3, 6, or 9 min per question. The correct answer rate is reported both before and after search to assess the impact of the search system and the time constraint. In addition, 2 search systems that use the same user interface, but which vary widely in their search effectiveness, are employed so that the impact of changes in search system effectiveness on clinical decision making can also be assessed.

RESULTS

Recruiting began for the study in June 2018. As of the April 4, 2019, there were 69 participants enrolled. The study is expected to close by May 30, 2019, with results to be published in July.

CONCLUSIONS

All data collected in this study will be made available at the University of Queensland's UQ eSpace public data repository.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/12803.

Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features

OBJECTIVES

Finding recent clinical studies that warrant changes in clinical practice ("high impact" clinical studies) in a timely manner is very challenging. We investigated a machine learning approach to find recent studies with high clinical impact to support clinical decision making and literature surveillance.

METHODS

To identify recent studies, we developed our classification model using time-agnostic features that are available as soon as an article is indexed in PubMed®, such as journal impact factor, author count, and study sample size. Using a gold standard of 541 high impact treatment studies referenced in 11 disease management guidelines, we tested the following null hypotheses: (1) the high impact classifier with time-agnostic features (HI-TA) performs equivalently to PubMed's Best Match sort and a MeSH-based Naïve Bayes classifier; and (2) HI-TA performs equivalently to the high impact classifier with both time-agnostic and time-sensitive features (HI-TS) enabled in a previous study. The primary outcome for both hypotheses was mean top 20 precision.

RESULTS

The differences in mean top 20 precision between HI-TA and three baselines (PubMed's Best Match, a MeSH-based Naïve Bayes classifier, and HI-TS) were not statistically significant (12% vs. 3%, p = 0.101; 12% vs. 11%, p = 0.720; 12% vs. 25%, p = 0.094, respectively). Recall of HI-TA was low (7%).

CONCLUSION

HI-TA had equivalent performance to state-of-the-art approaches that depend on time-sensitive features. With the advantage of relying only on time-agnostic features, the proposed approach can be used as an adjunct to help clinicians identify recent high impact clinical studies to support clinical decision-making. However, low recall limits the use of HI-TA for literature surveillance.

Interactive Visual Displays for Interpreting the Results of Clinical Trials: Formative Evaluation With Case Vignettes

Background

At the point of care, evidence from randomized controlled trials (RCTs) is underutilized in helping clinicians meet their information needs.

Objective

To design interactive visual displays to help clinicians interpret and compare the results of relevant RCTs for the management of a specific patient, and to conduct a formative evaluation with physicians comparing interactive visual versus narrative displays.

Methods

We followed a user-centered and iterative design process succeeded by development of information display prototypes as a Web-based application. We then used a within-subjects design with 20 participants (8 attendings and 12 residents) to evaluate the usability and problem-solving impact of the information displays. We compared subjects’ perceptions of the interactive visual displays versus narrative abstracts.

Results

The resulting interactive visual displays present RCT results side-by-side according to the Population, Intervention, Comparison, and Outcome (PICO) framework. Study participants completed 19 usability tasks in 3 to 11 seconds with a success rate of 78% to 100%. Participants favored the interactive visual displays over narrative abstracts according to perceived efficiency, effectiveness, effort, user experience and preference (all P values <.001).

Conclusions

When interpreting and applying RCT findings to case vignettes, physicians preferred interactive graphical and PICO-framework-based information displays that enable direct comparison of the results from multiple RCTs compared to the traditional narrative and study-centered format. Future studies should investigate the use of interactive visual displays to support clinical decision making in care settings and their effect on clinician and patient outcomes.

Evidence-based mental health and e-learning: a guide for clinicians

The internet provides access to what is often a bewildering array of medical knowledge on mental health, some (but not all) of which is evidence based. As well as information for clinicians, there has been a dramatic increase in the variety and quality of information available for patients and carers. In this article we discuss the advantages and limitations of the types of information available, with suggested sites and strategies for assessing their relative merits.

Automatic identification of high impact articles in PubMed to support clinical decision making

OBJECTIVES

The practice of evidence-based medicine involves integrating the latest best available evidence into patient care decisions. Yet, critical barriers exist for clinicians' retrieval of evidence that is relevant for a particular patient from primary sources such as randomized controlled trials and meta-analyses. To help address those barriers, we investigated machine learning algorithms that find clinical studies with high clinical impact from PubMed®.

METHODS

Our machine learning algorithms use a variety of features including bibliometric features (e.g., citation count), social media attention, journal impact factors, and citation metadata. The algorithms were developed and evaluated with a gold standard composed of 502 high impact clinical studies that are referenced in 11 clinical evidence-based guidelines on the treatment of various diseases. We tested the following hypotheses: (1) our high impact classifier outperforms a state-of-the-art classifier based on citation metadata and citation terms, and PubMed's® relevance sort algorithm; and (2) the performance of our high impact classifier does not decrease significantly after removing proprietary features such as citation count.

RESULTS

The mean top 20 precision of our high impact classifier was 34% versus 11% for the state-of-the-art classifier and 4% for PubMed's® relevance sort (p=0.009); and the performance of our high impact classifier did not decrease significantly after removing proprietary features (mean top 20 precision=34% vs. 36%; p=0.085).

CONCLUSION

The high impact classifier, using features such as bibliometrics, social media attention and MEDLINE® metadata, outperformed previous approaches and is a promising alternative to identifying high impact studies for clinical decision support.

Implementing tools to support evidence-based practice: a survey and brief intervention study of the National Elf Service across Oxford Health NHS Foundation Trust

BACKGROUND

Technology and the internet has enabled rapid access to research but most mental health professionals do not have time to keep up with the vast and growing scientific literature. Secondary information sources, such as the National Elf Service (NES), aim to summarise the most important and up-to-date research to improve mental health professionals' access to information to support evidence-based medicine (EBM).

OBJECTIVE

To explore mental health professionals' attitudes towards evidence-based practice and methods used to keep up-to-date with research. To promote use of a digital evidence-based platform (the National Elf Service), assess its use and explore its potential to impact clinical practice.

METHODS

Baseline and follow-up surveys were distributed among staff of 5 adult mental health community teams and 2 early intervention services (n=331) in Oxford Health Foundation Trust (OHFT) prior to and following an intervention raising awareness of the National Elf Service.

FINDINGS

Of 133 baseline survey responders, the majority of staff reported their clinical practice was informed by evidence, mostly using existing clinical guidelines and online resources. Few had used the National Elf Service. 122 staff members completed the follow-up survey. Postintervention, 42 staff members indicated they had used the National Elf Service (compared with 13 preintervention) and that it had improved access to research. Lack of time was most often the barrier restricting evidence-based practice.

CONCLUSIONS

Mental health professionals are engaged with EBM and those that used the National Elf Service felt it did, or could have the potential to impact on their clinical practice.

CLINICAL IMPLICATIONS

Barriers and challenges to implement EBM more widely suggest targeted efforts should be made to embed evidence-based practice into the working culture.

Context-sensitive decision support (infobuttons) in electronic health records: a systematic review

OBJECTIVE

Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes.

METHODS

We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information.

RESULTS

Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes.

CONCLUSIONS

Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.

Qualitative study of physicians' varied uses of biomedical research in the USA

OBJECTIVE

To investigate the nature of physicians' use of research evidence in experimental conditions of open access to inform training and policy.

DESIGN

This qualitative study was a component of a larger mixed-methods initiative that provided 336 physicians with relatively complete access to research literature via PubMed and UpToDate, for 1 year via an online portal, with their usage recorded in web logs. Using a semistructured interview protocol, a subset of 38 physician participants were interviewed about their use of research articles in general and were probed about their reasons for accessing specific articles as identified through their web logs. Transcripts were analysed using a general inductive approach.

SETTING

Physician participants were recruited from and registered in the USA.

PARTICIPANTS

38 physicians from 16 US states, engaged in 22 medical specialties, possessing more than 1 year of experience postresidency training participated.

RESULTS

26 participants attested to the value of consulting research literature within the context of the study by making reference to their roles as clinicians, educators, researchers, learners, administrators and advocates. The physicians reported previously encountering what they experienced as a prohibitive paywall barrier to the research literature and other frustrations with the nature of information systems, such as the need for passwords.

CONCLUSIONS

The findings, against the backdrop of growing open access to biomedical research, indicate that a minority of physicians, at least initially, is likely to seek out and use research and do so in a variety of common roles. Physicians' use of research in these roles has not traditionally been part of their training or part of the considerations for open access policies. The findings have implications for educational and policy initiatives directed towards increasing the effectiveness of this access to and use of research in improving the quality of healthcare.

Towards Evidence-based Precision Medicine: Extracting Population Information from Biomedical Text using Binary Classifiers and Syntactic Patterns

Precision Medicine is an emerging approach for prevention and treatment of disease that considers individual variability in genes, environment, and lifestyle for each person. The dissemination of individualized evidence by automatically identifying population information in literature is a key for evidence-based precision medicine at the point-of-care. We propose a hybrid approach using natural language processing techniques to automatically extract the population information from biomedical literature. Our approach first implements a binary classifier to classify sentences with or without population information. A rule-based system based on syntactic-tree regular expressions is then applied to sentences containing population information to extract the population named entities. The proposed two-stage approach achieved an F-score of 0.81 using a MaxEnt classifier and the rule- based system, and an F-score of 0.87 using a Nai've-Bayes classifier and the rule-based system, and performed relatively well compared to many existing systems. The system and evaluation dataset is being released as open source.

Classification of clinically useful sentences in clinical evidence resources

Most patient care questions raised by clinicians can be answered by online clinical knowledge resources. However, important barriers still challenge the use of these resources at the point of care.

OBJECTIVE

To design and assess a method for extracting clinically useful sentences from synthesized online clinical resources that represent the most clinically useful information for directly answering clinicians' information needs.

MATERIALS AND METHODS

We developed a Kernel-based Bayesian Network classification model based on different domain-specific feature types extracted from sentences in a gold standard composed of 18 UpToDate documents. These features included UMLS concepts and their semantic groups, semantic predications extracted by SemRep, patient population identified by a pattern-based natural language processing (NLP) algorithm, and cue words extracted by a feature selection technique. Algorithm performance was measured in terms of precision, recall, and F-measure.

RESULTS

The feature-rich approach yielded an F-measure of 74% versus 37% for a feature co-occurrence method (p<0.001). Excluding predication, population, semantic concept or text-based features reduced the F-measure to 62%, 66%, 58% and 69% respectively (p<0.01). The classifier applied to Medline sentences reached an F-measure of 73%, which is equivalent to the performance of the classifier on UpToDate sentences (p=0.62).

CONCLUSIONS

The feature-rich approach significantly outperformed general baseline methods. This approach significantly outperformed classifiers based on a single type of feature. Different types of semantic features provided a unique contribution to overall classification performance. The classifier's model and features used for UpToDate generalized well to Medline abstracts.

Advances in website information resources to aid in clinical practice

The World Wide Web, which has been widely implemented for roughly two decades, is humankind's most impressive effort to aggregate and organize knowledge to date. The medical community was slower to embrace the Internet than others, but the majority of clinicians now use it as part of their everyday practice. For the practicing oncologist, there is a daunting quantity of information to master. For example, a new article relating to cancer is added to the MEDLINE database approximately every 3 minutes. Fortunately, Internet resources can help organize the deluge of information into useful knowledge. This manuscript provides an overview of resources related to general medicine, oncology, and social media that will be of practical use to the practicing oncologist. It is clear from the vast size of the Internet that we are all life-long learners, and the challenge is to acquire "just-in-time" information so that we can provide the best possible care to our patients. The resources that we have presented in this article should help the practicing oncologist continue along the path of transforming information to knowledge to wisdom.

Classification of Clinically Useful Sentences in MEDLINE

OBJECTIVE

In a previous study, we investigated a sentence classification model that uses semantic features to extract clinically useful sentences from UpToDate, a synthesized clinical evidence resource. In the present study, we assess the generalizability of the sentence classifier to Medline abstracts.

METHODS

We applied the classification model to an independent gold standard of high quality clinical studies from Medline. Then, the classifier trained on UpToDate sentences was optimized by re-retraining the classifier with Medline abstracts and adding a sentence location feature.

RESULTS

The previous classifier yielded an F-measure of 58% on Medline versus 67% on UpToDate. Re-training the classifier on Medline improved F-measure to 68%; and to 76% (p<0.01) after adding the sentence location feature.

CONCLUSIONS

The classifier's model and input features generalized to Medline abstracts, but the classifier needed to be retrained on Medline to achieve equivalent performance. Sentence location provided additional contribution to the overall classification performance.

Designing evidence-based medicine training to optimize the transfer of skills from the classroom to clinical practice: applying the four component instructional design model

Evidence-based medicine (EBM) skills, although taught in medical schools around the world, are not optimally practiced in clinical environments because of multiple barriers, including learners' difficulty transferring EBM skills learned in the classroom to clinical practice. This lack of skill transfer may be partially due to the design of EBM training. To facilitate the transfer of EBM skills from the classroom to clinical practice, the authors explore one instructional approach, called the Four Component Instructional Design (4C/ID) model, to guide the design of EBM training. On the basis of current cognitive psychology, including cognitive load theory, the premise of the 4C/ID model is that complex skills training, such as EBM training, should include four components: learning tasks, supportive information, procedural information, and part-task practice. The combination of these four components can inform the creation of complex skills training that is designed to avoid overloading learners' cognitive abilities; to facilitate the integration of the knowledge, skills, and attitudes needed to execute a complex task; and to increase the transfer of knowledge to new situations. The authors begin by introducing the 4C/ID model and describing the benefits of its four components to guide the design of EBM training. They include illustrative examples of educational practices that are consistent with each component and that can be applied to teaching EBM. They conclude by suggesting that medical educators consider adopting the 4C/ID model to design, modify, and/or implement EBM training in classroom and clinical settings.

eHealth 2015 Special Issue: Impact of Electronic Health Records on the Completeness of Clinical Documentation Generated during Diabetic Retinopathy Consultations

BACKGROUND

Two years ago, the Diabetic Retinopathy (DRP) and Traumatology clinic of the Department of Ophthalmology and Optometrics at the Medical University of Vienna, Austria switched from paper-based to electronic health records. A customized electronic health record system (EHR-S) was implemented.

OBJECTIVES

To assess the completeness of information documented electronically compared with manually during patient visits.

METHODS

The Preferred Practice Pattern for Diabetic Retinopathy published by the American Academy of Ophthalmology was distilled into a list of medical features grouped into categories to be assessed and documented during the management of patients with DRP. The last seventy paper-based records and all electronic records generated since the switch were analyzed and graded for the presence of features on the list and the resulting scores compared.

RESULTS

In all categories, clinical documentation was more complete in the EHR group.

CONCLUSIONS

In our setting, the implementation of an EHR-S showed a statistically significant positive impact on documentation completeness.

In an Age of Open Access to Research Policies: Physician and Public Health NGO Staff Research Use and Policy Awareness

INTRODUCTION

Through funding agency and publisher policies, an increasing proportion of the health sciences literature is being made open access. Such an increase in access raises questions about the awareness and potential utilization of this literature by those working in health fields.

METHODS

A sample of physicians (N=336) and public health non-governmental organization (NGO) staff (N=92) were provided with relatively complete access to the research literature indexed in PubMed, as well as access to the point-of-care service UpToDate, for up to one year, with their usage monitored through the tracking of web-log data. The physicians also participated in a one-month trial of relatively complete or limited access.

RESULTS

The study found that participants' research interests were not satisfied by article abstracts alone nor, in the case of the physicians, by a clinical summary service such as UpToDate. On average, a third of the physicians viewed research a little more frequently than once a week, while two-thirds of the public health NGO staff viewed more than three articles a week. Those articles were published since the 2008 adoption of the NIH Public Access Policy, as well as prior to 2008 and during the maximum 12-month embargo period. A portion of the articles in each period was already open access, but complete access encouraged a viewing of more research articles.

CONCLUSION

Those working in health fields will utilize more research in the course of their work as a result of (a) increasing open access to research, (b) improving awareness of and preparation for this access, and (c) adjusting public and open access policies to maximize the extent of potential access, through reduction in embargo periods and access to pre-policy literature.

Information-Seeking Behaviors of Medical Students: A Cross-Sectional Web-Based Survey

BACKGROUND

Medical students face an information-rich environment in which retrieval and appraisal strategies are increasingly important.

OBJECTIVE

To describe medical students' current pattern of health information resource use and characterize their experience of instruction on information search and appraisal.

METHODS

We conducted a cross-sectional web-based survey of students registered in the four-year MD Program at Dalhousie University (Halifax, Nova Scotia, and Saint John, New Brunswick, sites), Canada. We collected self-reported data on information-seeking behavior, instruction, and evaluation of resources in the context of their medical education. Data were analyzed using descriptive statistics.

RESULTS

Surveys were returned by 213 of 462 eligible students (46.1%). Most respondents (165/204, 80.9%) recalled receiving formal instruction regarding information searches, but this seldom included nontraditional tools such as Google (23/107, 11.1%), Wikipedia, or social media. In their daily practice, however, they reported heavy use of these tools, as well as EBM summaries. Accessibility, understandability, and overall usefulness were common features of highly used resources. Students identified challenges managing information and/or resource overload and source accessibility.

CONCLUSIONS

Medical students receive instruction primarily on searching and assessing primary medical literature. In their daily practice, however, they rely heavily on nontraditional tools as well as EBM summaries. Attention to appropriate use and appraisal of nontraditional sources might enhance the current EBM curriculum.

Comparing image search behaviour in the ARRS GoldMiner search engine and a clinical PACS/RIS

Information search has changed the way we manage knowledge and the ubiquity of information access has made search a frequent activity, whether via Internet search engines or increasingly via mobile devices. Medical information search is in this respect no different and much research has been devoted to analyzing the way in which physicians aim to access information. Medical image search is a much smaller domain but has gained much attention as it has different characteristics than search for text documents. While web search log files have been analysed many times to better understand user behaviour, the log files of hospital internal systems for search in a PACS/RIS (Picture Archival and Communication System, Radiology Information System) have rarely been analysed. Such a comparison between a hospital PACS/RIS search and a web system for searching images of the biomedical literature is the goal of this paper. Objectives are to identify similarities and differences in search behaviour of the two systems, which could then be used to optimize existing systems and build new search engines. Log files of the ARRS GoldMiner medical image search engine (freely accessible on the Internet) containing 222,005 queries, and log files of Stanford's internal PACS/RIS search called radTF containing 18,068 queries were analysed. Each query was preprocessed and all query terms were mapped to the RadLex (Radiology Lexicon) terminology, a comprehensive lexicon of radiology terms created and maintained by the Radiological Society of North America, so the semantic content in the queries and the links between terms could be analysed, and synonyms for the same concept could be detected. RadLex was mainly created for the use in radiology reports, to aid structured reporting and the preparation of educational material (Lanlotz, 2006) [1]. In standard medical vocabularies such as MeSH (Medical Subject Headings) and UMLS (Unified Medical Language System) specific terms of radiology are often underrepresented, therefore RadLex was considered to be the best option for this task. The results show a surprising similarity between the usage behaviour in the two systems, but several subtle differences can also be noted. The average number of terms per query is 2.21 for GoldMiner and 2.07 for radTF, the used axes of RadLex (anatomy, pathology, findings, …) have almost the same distribution with clinical findings being the most frequent and the anatomical entity the second; also, combinations of RadLex axes are extremely similar between the two systems. Differences include a longer length of the sessions in radTF than in GoldMiner (3.4 and 1.9 queries per session on average). Several frequent search terms overlap but some strong differences exist in the details. In radTF the term "normal" is frequent, whereas in GoldMiner it is not. This makes intuitive sense, as in the literature normal cases are rarely described whereas in clinical work the comparison with normal cases is often a first step. The general similarity in many points is likely due to the fact that users of the two systems are influenced by their daily behaviour in using standard web search engines and follow this behaviour in their professional search. This means that many results and insights gained from standard web search can likely be transferred to more specialized search systems. Still, specialized log files can be used to find out more on reformulations and detailed strategies of users to find the right content.

Information-seeking behavior during residency is associated with quality of theoretical learning, academic career achievements, and evidence-based medical practice: a strobe-compliant article

Data regarding knowledge acquisition during residency training are sparse. Predictors of theoretical learning quality, academic career achievements and evidence-based medical practice during residency are unknown. We performed a cross-sectional study on residents and attending physicians across several residency programs in 2 French faculties of medicine. We comprehensively evaluated the information-seeking behavior (I-SB) during residency using a standardized questionnaire and looked for independent predictors of theoretical learning quality, academic career achievements, and evidence-based medical practice among I-SB components using multivariate logistic regression analysis. Between February 2013 and May 2013, 338 fellows and attending physicians were included in the study. Textbooks and international medical journals were reported to be used on a regular basis by 24% and 57% of the respondents, respectively. Among the respondents, 47% refer systematically (4.4%) or frequently (42.6%) to published guidelines from scientific societies upon their publication. The median self-reported theoretical learning quality score was 5/10 (interquartile range, 3-6; range, 1-10). A high theoretical learning quality score (upper quartile) was independently and strongly associated with the following I-SB components: systematic reading of clinical guidelines upon their publication (odds ratio [OR], 5.55; 95% confidence interval [CI], 1.77-17.44); having access to a library that offers the leading textbooks of the specialty in the medical department (OR, 2.45, 95% CI, 1.33-4.52); knowledge of the specialty leading textbooks (OR, 2.12; 95% CI, 1.09-4.10); and PubMed search skill score ≥5/10 (OR, 1.94; 95% CI, 1.01-3.73). Research Master (M2) and/or PhD thesis enrolment were independently and strongly associated with the following predictors: PubMed search skill score ≥5/10 (OR, 4.10; 95% CI, 1.46-11.53); knowledge of the leading medical journals of the specialty (OR, 3.33; 95% CI, 1.32-8.38); attending national and international academic conferences and meetings (OR, 2.43; 95% CI, 1.09-5.43); and using academic theoretical learning supports several times a week (OR, 2.23; 95% CI, 1.11- 4.49). This study showed weaknesses in the theoretical learning framework during residency. I-SB was independently associated with quality of academic theoretical learning, academic career achievements, and the use of evidence-based medicine in everyday clinical practice.

STUDY REGISTRATION

CNIL No.1797639.

Increasing the quantity and quality of searching for current best evidence to answer clinical questions: protocol and intervention design of the MacPLUS FS Factorial Randomized Controlled Trials

BACKGROUND & AIMS

Finding current best evidence for clinical decisions remains challenging. With 3,000 new studies published every day, no single evidence-based resource provides all answers or is sufficiently updated. McMaster Premium LiteratUre Service--Federated Search (MacPLUS FS) addresses this issue by looking in multiple high quality resources simultaneously and displaying results in a one-page pyramid with the most clinically useful at the top. Yet, additional logistical and educational barriers need to be addressed to enhance point-of-care evidence retrieval. This trial seeks to test three innovative interventions, among clinicians registered to MacPLUS FS, to increase the quantity and quality of searching for current best evidence to answer clinical questions.

METHODS & DESIGN

In a user-centered approach, we designed three interventions embedded in MacPLUS FS: (A) a web-based Clinical Question Recorder; (B) an Evidence Retrieval Coach composed of eight short educational videos; (C) an Audit, Feedback and Gamification approach to evidence retrieval, based on the allocation of 'badges' and 'reputation scores.' We will conduct a randomized factorial controlled trial among all the 904 eligible medical doctors currently registered to MacPLUS FS at the hospitals affiliated with McMaster University, Canada. Postgraduate trainees (n=429) and clinical faculty/staff (n=475) will be randomized to each of the three following interventions in a factorial design (AxBxC). Utilization will be continuously recorded through clinicians’ accounts that track logins and usage, down to the level of individual keystrokes. The primary outcome is the rate of searches per month per user during the six months of follow-up. Secondary outcomes, measured through the validated Impact Assessment Method questionnaire, include: utility of answers found (meeting clinicians’ information needs), use (application in practice), and perceived usefulness on patient outcomes.

DISCUSSION

Built on effective models for the point-of-care teaching, these interventions approach evidence retrieval as a clinical skill. If effective, they may offer the opportunity to enhance it for a large audience, at low cost, providing better access to relevant evidence across many top EBM resources in parallel.

TRIAL REGISTRATION

ClinicalTrials.Gov NCT02038439.

Speed and accuracy of a point of care web-based knowledge resource for clinicians: a controlled crossover trial

Background

Effective knowledge translation at the point of care requires that clinicians quickly find correct answers to clinical questions, and that they have appropriate confidence in their answers. Web-based knowledge resources can facilitate this process.

Objective

The objective of our study was to evaluate a novel Web-based knowledge resource in comparison with other available Web-based resources, using outcomes of accuracy, time, and confidence.

Methods

We conducted a controlled, crossover trial involving 59 practicing clinicians. Each participant answered questions related to two clinical scenarios. For one scenario, participants used a locally developed Web-based resource, and for the second scenario, they used other self-selected Web-based resources. The local knowledge resource (“AskMayoExpert”) was designed to provide very concise evidence-based answers to commonly asked clinical questions. Outcomes included time to a correct response with at least 80% confidence (primary outcome), accuracy, time, and confidence.

Results

Answers were more often accurate when using the local resource than when using other Web-based resources, with odds ratio 6.2 (95% CI 2.6-14.5; P<.001) when averaged across scenarios. Time to find an answer was faster, and confidence in that answer was consistently higher, for the local resource (P<.001). Overconfidence was also less frequent with the local resource. In a time-to-event analysis, the chance of responding correctly with at least 80% confidence was 2.5 times greater when using the local resource than with other resources (95% CI 1.6-3.8; P<.001).

Conclusions

Clinicians using a Web-based knowledge resource designed to provide quick, concise answers at the point of care found answers with greater accuracy and confidence than when using other self-selected Web-based resources. Further study to improve the design and implementation of knowledge resources may improve point of care learning.

Access of primary and secondary literature by health personnel in an academic health center: implications for open access

PURPOSE

The research sought to ascertain the types and quantity of research evidence accessed by health personnel through PubMed and UpToDate in a university medical center over the course of a year in order to better estimate the impact that increasing levels of open access to biomedical research can be expected to have on clinical practice in the years ahead.

METHODS

Web log data were gathered from the 5,042 health personnel working in the Stanford University Hospitals (SUH) during 2011. Data were analyzed for access to the primary literature (abstracts and full-text) through PubMed and UpToDate and to the secondary literature, represented by UpToDate (research summaries), to establish the frequency and nature of literature consulted.

RESULTS

In 2011, SUH health personnel accessed 81,851 primary literature articles and visited UpToDate 110,336 times. Almost a third of the articles (24,529) accessed were reviews. Twenty percent (16,187) of the articles viewed were published in 2011.

CONCLUSION

When it is available, health personnel in a clinical care setting frequently access the primary literature. While further studies are needed, this preliminary finding speaks to the value of the National Institutes of Health public access policy and the need for medical librarians and educators to prepare health personnel for increasing public access to medical research.

Features of effective medical knowledge resources to support point of care learning: a focus group study

OBJECTIVE

Health care professionals access various information sources to quickly answer questions that arise in clinical practice. The features that favorably influence the selection and use of knowledge resources remain unclear. We sought to better understand how clinicians select among the various knowledge resources available to them, and from this to derive a model for an effective knowledge resource.

METHODS

We conducted 11 focus groups at an academic medical center and outlying community sites. We included a purposive sample of 50 primary care and subspecialist internal medicine and family medicine physicians. We transcribed focus group discussions and analyzed these using a constant comparative approach to inductively identify features that influence the selection of knowledge resources.

RESULTS

We identified nine features that influence users' selection of knowledge resources, namely efficiency (with sub-features of comprehensiveness, searchability, and brevity), integration with clinical workflow, credibility, user familiarity, capacity to identify a human expert, reflection of local care processes, optimization for the clinical question (e.g., diagnosis, treatment options, drug side effect), currency, and ability to support patient education. No single existing resource exemplifies all of these features.

CONCLUSION

The influential features identified in this study will inform the development of knowledge resources, and could serve as a framework for future research in this field.

Should we Google it? Resource use by internal medicine residents for point-of-care clinical decision making

PURPOSE

To determine which resources residents use at the point-of-care (POC) for decision making, the drivers for selection of these resources, and how residents use Google/Google Scholar to answer clinical questions at the POC.

METHOD

In January 2012, 299 residents from three internal medicine residencies were sent an electronic survey regarding resources used for POC decision making. Resource use frequency and factors influencing choice were determined using descriptive statistics. Binary logistic regression analysis was performed to determine relationships between the independent variables.

RESULTS

A total of 167 residents (56%) responded; similar numbers responded at each level of training. Residents most frequently reported using UpToDate and Google at the POC at least daily (85% and 63%, respectively), with speed and trust in the quality of information being the primary drivers of selection. Google, used by 68% of residents, was used primarily to locate Web sites and general information about diseases, whereas Google Scholar, used by 30% of residents, tended to be used for treatment and management decisions or locating a journal article.

CONCLUSIONS

The findings suggest that internal medicine residents use UpToDate most frequently, followed by consultation with faculty and the search engines Google and Google Scholar; speed, trust, and portability are the biggest drivers for resource selection; and time and information overload appear to be the biggest barriers to resources such as Ovid MEDLINE. Residents frequently used Google and may benefit from further training in information management skills.

Contemporary approaches for evidence-based pharmacotherapy

Evidence-based medicine (EBM) is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. This article provides simple ways to practice everyday EBM by (1) asking patient-specific clinical questions, (2) finding the most relevant, best evidence, (3) critically appraising evidence to ensure validity, (4) applying evidence to practice, and (5) using available tools to stay abreast of relevant and valid evidence as it becomes available.

Being better clinicians: an acronym to excellence

Out of an exhaustive review of the literature on the essential attributes of exemplary clinicians, five main characteristics were derived. Summarized by a useful acronym-SOAPS, these attributes include maintaining a Systematic method; employing Observation and listening; Accessing databases; emphasizing a Personal empathic attitude and Sharing information and decisions with the patient and family. Universally lauded as these concepts are, the evidence demonstrates that they are often neglected, with a negative impact on the quality of care and patient outcomes. The article focuses on using the SOAPS acronym as a mental checklist in teaching sessions and patient encounters to improve adherence to these key concepts.

Emergency department physician internet use during clinical encounters

OBJECTIVE

This study explored the Internet log files from emergency department workstations to determine search patterns, compared them to discharge diagnoses, and the emergency medicine curriculum as a way to quantify physician search behaviors.

METHODS

The log files from the computers from January 2006 to March 2010 were mapped to the EM curriculum and compared to discharge diagnoses to explore search terms and website usage by physicians and students.

RESULTS

Physicians in the ED averaged 1.35 searches per patient encounter using Google.com and UpToDate.com 83.9% of the time. The most common searches were for drug information (23.1%) by all provider types. The majority of the websites utilized were in the third tier evidence level for evidence-based medicine (EBM).

CONCLUSION

We have shown a need for a readily accessible drug knowledge base within the EMR for decision support as well as easier access to first and second tier EBM evidence.

Comparing patient characteristics, type of intervention, control, and outcome (PICO) queries with unguided searching: a randomized controlled crossover trial

BACKGROUND

Translating a question into a query using patient characteristics, type of intervention, control, and outcome (PICO) should help answer therapeutic questions in PubMed searches. The authors performed a randomized crossover trial to determine whether the PICO format was useful for quick searches of PubMed.

METHODS

Twenty-two residents and specialists working at the Radboud University Nijmegen Medical Centre were trained in formulating PICO queries and then presented with a randomized set of questions derived from Cochrane reviews. They were asked to use the best query possible in a five-minute search, using standard and PICO queries. Recall and precision were calculated for both standard and PICO queries.

RESULTS

Twenty-two physicians created 434 queries using both techniques. Average precision was 4.02% for standard queries and 3.44% for PICO queries (difference nonsignificant, t(21) = -0.56, P = 0.58). Average recall was 12.27% for standard queries and 13.62% for PICO queries (difference nonsignificant, t(21) = -0.76, P = 0.46).

CONCLUSIONS

PICO queries do not result in better recall or precision in time-limited searches. Standard queries containing enough detail are sufficient for quick searches.

Sensitivity and predictive value of 15 PubMed search strategies to answer clinical questions rated against full systematic reviews

Background

Clinicians perform searches in PubMed daily, but retrieving relevant studies is challenging due to the rapid expansion of medical knowledge. Little is known about the performance of search strategies when they are applied to answer specific clinical questions.

Objective

To compare the performance of 15 PubMed search strategies in retrieving relevant clinical trials on therapeutic interventions.

Methods

We used Cochrane systematic reviews to identify relevant trials for 30 clinical questions. Search terms were extracted from the abstract using a predefined procedure based on the population, interventions, comparison, outcomes (PICO) framework and combined into queries. We tested 15 search strategies that varied in their query (PIC or PICO), use of PubMed’s Clinical Queries therapeutic filters (broad or narrow), search limits, and PubMed links to related articles. We assessed sensitivity (recall) and positive predictive value (precision) of each strategy on the first 2 PubMed pages (40 articles) and on the complete search output.

Results

The performance of the search strategies varied widely according to the clinical question. Unfiltered searches and those using the broad filter of Clinical Queries produced large outputs and retrieved few relevant articles within the first 2 pages, resulting in a median sensitivity of only 10%–25%. In contrast, all searches using the narrow filter performed significantly better, with a median sensitivity of about 50% (all P < .001 compared with unfiltered queries) and positive predictive values of 20%–30% (P < .001 compared with unfiltered queries). This benefit was consistent for most clinical questions. Searches based on related articles retrieved about a third of the relevant studies.

Conclusions

The Clinical Queries narrow filter, along with well-formulated queries based on the PICO framework, provided the greatest aid in retrieving relevant clinical trials within the 2 first PubMed pages. These results can help clinicians apply effective strategies to answer their questions at the point of care.

Use of UpToDate and outcomes in US hospitals

BACKGROUND

Computerized clinical knowledge mana-gement systems hold enormous potential for improving quality and efficiency. However, their impact on clinical practice is not well known.

OBJECTIVE

To examine the impact of UpToDate on outcomes of care.

DESIGN

Retrospective study.

SETTING

National sample of US inpatient hospitals.

PATIENTS

Fee-for-service Medicare beneficiaries.

INTERVENTION

Adoption of UpToDate in US hospitals.

MEASUREMENT

Risk-adjusted lengths of stay, mortality rates, and quality performance.

RESULTS

We found that patients admitted to hospitals using UpToDate had shorter lengths of stay than patients admitted to non-UpToDate hospitals overall (5.6 days vs 5.7 days; P < 0.001) and among 6 prespecified conditions (range, -0.1 to -0.3 days; P < 0.001 for each). Further, patients admitted to UpToDate hospitals had lower risk-adjusted mortality rate for 3 of the 6 conditions (range, -0.1% to -0.6% mortality reduction; P < 0.05). Finally, hospitals with UpToDate had better quality performance for every condition on the Hospital Quality Alliance metrics. In subgroup analyses, we found that it was the smaller hospitals and the non-teaching hospitals where the benefits of the UpToDate seemed most pronounced, compared to the larger, teaching institutions where the benefits of UpToDate seemed small or nonexistent.

CONCLUSIONS

We found a very small but consistent association between use of UpToDate and reduced length of stay, lower risk-adjusted mortality rates, and better quality performance, at least in the smaller, non-teaching institutions. These findings may suggest that computerized tools such as UpToDate could be helpful in improving care.

To compare PubMed Clinical Queries and UpToDate in teaching information mastery to clinical residents: a crossover randomized controlled trial

Purpose

To compare PubMed Clinical Queries and UpToDate regarding the amount and speed of information retrieval and users' satisfaction.

Method

A cross-over randomized trial was conducted in February 2009 in Tehran University of Medical Sciences that included 44 year-one or two residents who participated in an information mastery workshop. A one-hour lecture on the principles of information mastery was organized followed by self learning slide shows before using each database. Subsequently, participants were randomly assigned to answer 2 clinical scenarios using either UpToDate or PubMed Clinical Queries then crossed to use the other database to answer 2 different clinical scenarios. The proportion of relevantly answered clinical scenarios, time to answer retrieval, and users' satisfaction were measured in each database.

Results

Based on intention-to-treat analysis, participants retrieved the answer of 67 (76%) questions using UpToDate and 38 (43%) questions using PubMed Clinical Queries (P<0.001). The median time to answer retrieval was 17 min (95% CI: 16 to 18) using UpToDate compared to 29 min (95% CI: 26 to 32) using PubMed Clinical Queries (P<0.001). The satisfaction with the accuracy of retrieved answers, interaction with UpToDate and also overall satisfaction were higher among UpToDate users compared to PubMed Clinical Queries users (P<0.001).

Conclusions

For first time users, using UpToDate compared to Pubmed Clinical Querries can lead to not only a higher proportion of relevant answer retrieval within a shorter time, but also a higher users' satisfaction. So, addition of tutoring pre-appraised sources such as UpToDate to the information mastery curricula seems to be highly efficient.

Developing and using a rubric for evaluating evidence-based medicine point-of-care tools

OBJECTIVE

The research sought to establish a rubric for evaluating evidence-based medicine (EBM) point-of-care tools in a health sciences library.

METHODS

The authors searched the literature for EBM tool evaluations and found that most previous reviews were designed to evaluate the ability of an EBM tool to answer a clinical question. The researchers' goal was to develop and complete rubrics for assessing these tools based on criteria for a general evaluation of tools (reviewing content, search options, quality control, and grading) and criteria for an evaluation of clinical summaries (searching tools for treatments of common diagnoses and evaluating summaries for quality control).

RESULTS

Differences between EBM tools' options, content coverage, and usability were minimal. However, the products' methods for locating and grading evidence varied widely in transparency and process.

CONCLUSIONS

As EBM tools are constantly updating and evolving, evaluation of these tools needs to be conducted frequently. Standards for evaluating EBM tools need to be established, with one method being the use of objective rubrics. In addition, EBM tools need to provide more information about authorship, reviewers, methods for evidence collection, and grading system employed.

Type of evidence behind point-of-care clinical information products: a bibliometric analysis

BACKGROUND

Point-of-care (POC) products are widely used as information reference tools in the clinical setting. Although usability, scope of coverage, ability to answer clinical questions, and impact on health outcomes have been studied, no comparative analysis of the characteristics of the references, the evidence for the content, in POC products is available.

OBJECTIVE

The objective of this study was to compare the type of evidence behind five POC clinical information products.

METHODS

This study is a comparative bibliometric analysis of references cited in monographs in POC products. Five commonly used products served as subjects for the study: ACP PIER, Clinical Evidence, DynaMed, FirstCONSULT, and UpToDate. The four clinical topics examined to identify content in the products were asthma, hypertension, hyperlipidemia, and carbon monoxide poisoning. Four indicators were measured: distribution of citations, type of evidence, product currency, and citation overlap. The type of evidence was determined based primarily on the publication type found in the MEDLINE bibliographic record, as well as the Medical Subject Headings (MeSH), both assigned by the US National Library of Medicine. MeSH is the controlled vocabulary used for indexing articles in MEDLINE/PubMed.

RESULTS

FirstCONSULT had the greatest proportion of references with higher levels of evidence publication types such as systematic review and randomized controlled trial (137/153, 89.5%), although it contained the lowest total number of references (153/2330, 6.6%). DynaMed had the largest total number of references (1131/2330, 48.5%) and the largest proportion of current (2007-2009) references (170/1131, 15%). The distribution of references cited for each topic varied between products. For example, asthma had the most references listed in DynaMed, Clinical Evidence, and FirstCONSULT, while hypertension had the most references in UpToDate and ACP PIER. An unexpected finding was that the rate of citation overlap was less than 1% for each topic across all five products.

CONCLUSIONS

Differences between POC products are revealed by examining the references cited in the monographs themselves. Citation analysis extended to include key content indicators can be used to compare the evidence levels of the literature supporting the content found in POC products.

AskHERMES: An online question answering system for complex clinical questions

OBJECTIVE

Clinical questions are often long and complex and take many forms. We have built a clinical question answering system named AskHERMES to perform robust semantic analysis on complex clinical questions and output question-focused extractive summaries as answers.

DESIGN

This paper describes the system architecture and a preliminary evaluation of AskHERMES, which implements innovative approaches in question analysis, summarization, and answer presentation. Five types of resources were indexed in this system: MEDLINE abstracts, PubMed Central full-text articles, eMedicine documents, clinical guidelines and Wikipedia articles.

MEASUREMENT

We compared the AskHERMES system with Google (Google and Google Scholar) and UpToDate and asked physicians to score the three systems by ease of use, quality of answer, time spent, and overall performance.

RESULTS

AskHERMES allows physicians to enter a question in a natural way with minimal query formulation and allows physicians to efficiently navigate among all the answer sentences to quickly meet their information needs. In contrast, physicians need to formulate queries to search for information in Google and UpToDate. The development of the AskHERMES system is still at an early stage, and the knowledge resource is limited compared with Google or UpToDate. Nevertheless, the evaluation results show that AskHERMES' performance is comparable to the other systems. In particular, when answering complex clinical questions, it demonstrates the potential to outperform both Google and UpToDate systems.

CONCLUSIONS

AskHERMES, available at http://www.AskHERMES.org, has the potential to help physicians practice evidence-based medicine and improve the quality of patient care.