Evaluation of PubMed filters used for evidence-based searching: validation using relative recall

Assessing research methodologies used to evaluate inequalities in end-of-life cancer care research: a scoping review protocol

INTRODUCTION

To provide equitable cancer care at the end of life, it is essential to first understand the evidence underpinning the existence of unequal cancer outcomes. Study design, measurement and analytical decisions made by researchers are a function of their social systems, academic training, values and biases, which influence both the findings and interpretation of whether inequalities or inequities exist. Methodological choices can lead to results with different implications for research and policy priorities, including where supplementary programmes and services are offered and for whom. The objective of this scoping review is to provide an overview of the methods, including study design, measures and statistical approaches, used in quantitative and qualitative observational studies of health equity in end-of-life cancer care, and to consider how these methods align with recommended approaches for studying health equity questions.

METHODS AND ANALYSIS

This scoping review follows Arksey and O'Malley's expanded framework for scoping reviews. We will systematically search Medline, Embase, CINAHL and PsycINFO electronic databases for quantitative and qualitative studies that examined equity stratifiers in relation to end-of-life cancer care and/or outcomes published in English or French between 2010 and 2021. Two authors will independently review all titles, abstracts and full texts to determine which studies meet the inclusion criteria. Data from included full-text articles will be extracted into a data form that will be developed and piloted by the research team. Extracted information will be summarised quantitatively and qualitatively.

ETHICS AND DISSEMINATION

No ethics approval is required for this scoping review. Results will be disseminated to researchers examining questions of health equity in cancer care through scientific publication and presentation at relevant conferences.

Patient-based benefit-risk assessment of medicines: development, refinement, and validation of a content search strategy to retrieve relevant studies

Introduction:

Poor indexing and inconsistent use of terms and keywords may prevent efficient retrieval of studies on the patient-based benefit-risk assessment (BRA) of medicines. We aimed to develop and validate an objectively derived content search strategy containing generic search terms that can be adapted for any search for evidence on patient-based BRA of medicines for any therapeutic area.

Methods:

We used a robust multistep process to develop and validate the content search strategy: (1) we developed a bank of search terms derived from screening studies on patient-based BRA of medicines in various therapeutic areas, (2) we refined the proposed content search strategy through an iterative process of testing sensitivity and precision of search terms, and (3) we validated the final search strategy in PubMed by firstly using multiple sclerosis as a case condition and secondly computing its relative performance versus a published systematic review on patient-based BRA of medicines in rheumatoid arthritis.

Results:

We conceptualized a final search strategy to retrieve studies on patient-based BRA containing generic search terms grouped into two domains, namely the patient and the BRA of medicines (sensitivity 84%, specificity 99.4%, precision 20.7%). The relative performance of the content search strategy was 85.7% compared with a search from a published systematic review of patient preferences in the treatment of rheumatoid arthritis. We also developed a more extended filter, with a relative performance of 93.3% when compared with a search from a published systematic review of patient preferences in lung cancer.

Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance

The power of language to modify the reader's perception of interpreting biomedical results cannot be underestimated. Misreporting and misinterpretation are pressing problems in randomized controlled trials (RCT) output. This may be partially related to the statistical significance paradigm used in clinical trials centered around a P value below 0.05 cutoff. Strict use of this P value may lead to strategies of clinical researchers to describe their clinical results with P values approaching but not reaching the threshold to be "almost significant." The question is how phrases expressing nonsignificant results have been reported in RCTs over the past 30 years. To this end, we conducted a quantitative analysis of English full texts containing 567,758 RCTs recorded in PubMed between 1990 and 2020 (81.5% of all published RCTs in PubMed). We determined the exact presence of 505 predefined phrases denoting results that approach but do not cross the line of formal statistical significance (P < 0.05). We modeled temporal trends in phrase data with Bayesian linear regression. Evidence for temporal change was obtained through Bayes factor (BF) analysis. In a randomly sampled subset, the associated P values were manually extracted. We identified 61,741 phrases in 49,134 RCTs indicating almost significant results (8.65%; 95% confidence interval (CI): 8.58% to 8.73%). The overall prevalence of these phrases remained stable over time, with the most prevalent phrases being "marginally significant" (in 7,735 RCTs), "all but significant" (7,015), "a nonsignificant trend" (3,442), "failed to reach statistical significance" (2,578), and "a strong trend" (1,700). The strongest evidence for an increased temporal prevalence was found for "a numerical trend," "a positive trend," "an increasing trend," and "nominally significant." In contrast, the phrases "all but significant," "approaches statistical significance," "did not quite reach statistical significance," "difference was apparent," "failed to reach statistical significance," and "not quite significant" decreased over time. In a random sampled subset of 29,000 phrases, the manually identified and corresponding 11,926 P values, 68,1% ranged between 0.05 and 0.15 (CI: 67. to 69.0; median 0.06). Our results show that RCT reports regularly contain specific phrases describing marginally nonsignificant results to report P values close to but above the dominant 0.05 cutoff. The fact that the prevalence of the phrases remained stable over time indicates that this practice of broadly interpreting P values close to a predefined threshold remains prevalent. To enhance responsible and transparent interpretation of RCT results, researchers, clinicians, reviewers, and editors may reduce the focus on formal statistical significance thresholds and stimulate reporting of P values with corresponding effect sizes and CIs and focus on the clinical relevance of the statistical difference found in RCTs.

Topical Review: The Top Five Peripheral Vision Tools in Sport

SIGNIFICANCE

This study summarizes the empirical evidence on the use of peripheral vision for the most-researched peripheral vision tools in sports.

The objective of this review was to explain if and how the tools can be used to investigate peripheral vision usage and how empirical findings with these vision tools might be transferred to sports situations. The data sources used in this study were Scopus, ScienceDirect, and PubMed. We additionally searched the manufacturers' Web pages and used Google Scholar to find full texts that were not available elsewhere. Studies were included if they were published in a peer-reviewed journal, were written in English language, and were conducted in a sports context. From the 10 searched tools, we included the 5 tools with most published studies. In our topical search, we identified 93 studies for the five most-used peripheral vision tools. Surprisingly, none of these studies used eye-tracking methods to control for the use of peripheral vision. Best "passive" control is achieved by tools using (foveal) secondary tasks (Dynavision D2 and Vienna Test System). Best transfer to sports tasks is expected for tools demanding action responses (FitLight, Dynavision D2). Tools are likely to train peripheral monitoring (NeuroTracker), peripheral reaction time (Dynavision D2, Vienna Test System), or peripheral preview (FitLight), whereas one tool did not show any link to peripheral vision processes (Nike SPARQ Vapor Strobe).

Evaluation of Risks of Bias in Addiction Medicine Randomized Controlled Trials

AIMS

Perhaps the most important step when designing and conducting randomized controlled trials (RCTs) in addiction is to put methodological safeguards in place to minimize the likelihood for bias to affect trial outcomes. In this study, we applied the revised Cochrane risk of bias tool (ROB 2) to RCTs of drug, alcohol or tobacco interventions.

METHODS

We searched for trials published in 15 addiction medicine journals over a 7-year period. Our primary endpoint is the risk of bias of included studies. We conducted a sensitivity analysis of publicly funded trials.

RESULTS

Overall, included RCTs were most often at high risk of bias per our judgments (244/487, 50.1%). However, significant proportions of included RCTs were at low risk of bias (123/487, 25.3%) or some concerns for bias (120/497, 24.6%). RCTs with behavioral modification interventions (19/44, 43.2%) and alcohol interventions (80/150, 53.3%) had the highest proportion of high-risk judgments. In a sensitivity analysis of publicly funded RCTs), 195/386 (50.5%) were at high risk of bias.

CONCLUSIONS

Approximately half of included drug, alcohol or tobacco RCTs in our sample were judged to be at high risk of bias with the most common reason being a lack of proper blinding or proper description of blinding. Key action items to reduce bias in future addiction RCTs include adequate randomization, blinding and inclusion of a trial registry number and protocol.

Evaluation of selective outcome reporting and trial registration practices among addiction clinical trials

BACKGROUND AND AIMS

Selective outcome reporting occurs when trialists pre-specify primary and secondary outcomes during trial planning but alter the definitions in the published report. Here, we investigate selective outcome reporting in published addiction randomized controlled trials (RCTs) and evaluate whether particular funding sources are associated with an increased likelihood of selective outcome reporting.

DESIGN

We conducted a cross-sectional study of published addiction clinical trials. A PubMed search was performed to identify RCTs in addiction journals from 2013 to 2017. Included studies used a randomized design to address one of the following topics: (1) drug, alcohol and tobacco addiction prevention, (2) stabilization following excessive use of a substance, (3) relapse prevention or (4) recovery maintenance.

SETTING

Single-center, medical research institution.

PARTICIPANTS

Our sample included 162 RCTs that were prospectively registered with a clearly defined primary outcome.

MEASUREMENT

We extracted the following items from addiction RCTs: journal, funding source, trial registry number (if included), sample size, dates of subject enrollment, whether primary and secondary outcomes were denoted, all published outcomes, P-value for all outcomes and whether authors mentioned any deviations from the trial protocol as it related to RCT outcomes.

FINDINGS

In total, 47 of 162 RCTs (29.0%) had at least one major discrepancy between the trial registry and published RCT. Overall, these 47 RCTs included 54 major discrepancies. The most common major discrepancy was demotion of a primary registered outcome (19/54, 35.2%). The majority of RCTs (132/162, 81.5%) were funded from public sources. Additionally, 166 RCTs were excluded from our sample because registration could not be confirmed.

CONCLUSIONS

There is evidence suggestive of selective outcome reporting in addiction randomized controlled trials (RCTs). The most common major discrepancies pertained to the primary outcome.

Using the CONSORT statement to evaluate the completeness of reporting of addiction randomised trials: a cross-sectional review

OBJECTIVES

Evaluate the completeness of reporting of addiction randomised controlled trials (RCTs) using the Consolidated Standards of Reporting Trials (CONSORT) statement.

SETTING

Not applicable.

PARTICIPANTS

RCTs identified using a PubMed search of 15 addiction journals and a 5-year cross-section.

OUTCOME MEASURES

Completeness of reporting.

RESULTS

Our analysis of 394 addiction RCTs found that the mean number of CONSORT items reported was 19.2 (SD 5.2), out of a possible 31. Twelve items were reported in <50% of RCTs; similarly, 12 items were reported in >75% of RCTs. Journal endorsement of CONSORT was found to improve the number of CONSORT items reported.

CONCLUSIONS

Poor reporting quality may prohibit readers from critically appraising the methodological quality of addiction trials. We recommend journal endorsement of CONSORT since our study and those previous have shown that CONSORT endorsement improves the quality of reporting.

A systematic review of interventions to retain chronically ill occupationally active employees in work: can findings be transferred to cancer survivors?

Background: Interventions supporting occupationally active cancer survivors to retain work and prevent adverse work outcomes, beyond return to work, are scarce. As lessons may be learned from interventions that have been evaluated in working employees with other chronic diseases than cancer, the objective of this review was to summarize the characteristics of these interventions. Material and methods: Studies were identified through computerized PubMed, EMBASE and PsycINFO searches, without any language or year of publication restrictions. Randomized controlled trials were included if they evaluated the effectiveness of interventions to retain chronically ill occupationally active employees in work. Two authors independently extracted data from each study and assessed the risk of bias. Results: The search identified 536 unique studies, of which 18 met the inclusion criteria. All included studies had a low risk of bias. (Psycho-)educational interventions for chronically ill employees to retain work were evaluated in two studies, physical interventions in three studies, vocational/work-related interventions in five studies, and multidisciplinary interventions in eight studies. Vocational/work-related and multidisciplinary interventions, and the involvement of professional trainers, showed the most promising effects in retaining employees. However, small sample sizes may have caused imprecise effect estimates. Conclusion: Based on studies focusing on occupationally active employees with other chronic diseases than cancer, it is advised that working cancer survivors should be offered tailored interventions, by skilled trainers, to sustain their employability. Shared goal setting, with relevant stakeholders, and vocational components should be included, potentially as part of a multidisciplinary intervention.

A review of recent publication trends from top publishing countries

BACKGROUND

Evidence-based medicine relies on current best evidence from the medical literature, the patient's history, and the clinician's own experience to provide the best care for patients. Systematic reviews and meta-analysis are considered the highest levels of evidence for informing clinical decisions. Recently, reports have shown an increase in the number but a decrease in quality of meta-analysis publications. We reviewed publication trends and determined the countries with the most journal articles and types of publications in PubMed from 1995 to 2015.

METHODS

We examined journal entries in PubMed from 1995 to 2015 from top publishing countries for total number of publications and citations in core clinical journals and in specific publication types (systematic reviews, meta-analysis, randomized controlled trials).

RESULTS

Yearly, only 30 countries generated 94.6% of all publications and 98.1% of core clinical journals worldwide. All publication types increased but with a significant increase in meta-analysis publications from China. Collaborative and co-authored papers among the 30 countries also showed an increasing trend.

CONCLUSION

The USA leads in all publication citations and specific publication types, except for meta-analysis where China publishes more. Collaborative publishing among international collaborators is also increasing.

Optimal search strategies for identifying moderators and predictors of treatment effects in PubMed

BACKGROUND

Treatment effects differ across patients. To guide selection of treatments for patients, it is essential to acknowledge these differences and identify moderators or predictors. Our aim was to generate optimal search strategies (commonly known as filters) for PubMed to retrieve papers identifying moderators and predictors of treatment effects.

METHODS

Six journals were hand-searched for articles on moderators or predictors. Selected articles were randomly allocated to a development and validation set. Search terms were extracted from the development set and tested for their performance. Search filters were created from combinations of these terms and tested in the validation set.

RESULTS

Of 4407 articles, 198 were considered to be relevant. The most sensitive filter in the development set '("Epidemiologic Methods" [MeSH] OR assign* OR control*[tiab] OR trial*[tiab]) AND therapy*[sh]' yielded in the validation set a sensitivity of 89% [88%-90%] and a specificity of 80% [79%-82%].

CONCLUSIONS

The search filters created in this study can help to efficiently retrieve evidence on moderators and predictors of treatment effect. Testing of the filters in multiple domains should reveal robustness across disciplines. These filters can facilitate the retrieval of evidence on moderators and predictors of treatment effects, helping the implementation of stratified or personalised health care.

Development and use of a content search strategy for retrieving studies on patients' views and preferences

BACKGROUND

Identifying scientific literature addressing patients' views and preferences is complex due to the wide range of studies that can be informative and the poor indexing of this evidence. Given the lack of guidance we developed a search strategy to retrieve this type of evidence.

METHODS

We assembled an initial list of terms from several sources, including the revision of the terms and indexing of topic-related studies and, methods research literature, and other relevant projects and systematic reviews. We used the relative recall approach, evaluating the capacity of the designed search strategy for retrieving studies included in relevant systematic reviews for the topic. We implemented in practice the final version of the search strategy for conducting systematic reviews and guidelines, and calculated search's precision and the number of references needed to read (NNR).

RESULTS

We assembled an initial version of the search strategy, which had a relative recall of 87.4% (yield of 132/out of 151 studies). We then added some additional terms from the studies not initially identified, and re-tested this improved version against the studies included in a new set of systematic reviews, reaching a relative recall of 85.8% (151/out of 176 studies, 95% CI 79.9 to 90.2). This final version of the strategy includes two sets of terms related with two domains: "Patient Preferences and Decision Making" and "Health State Utilities Values". When we used the search strategy for the development of systematic reviews and clinical guidelines we obtained low precision values (ranging from 2% to 5%), and the NNR from 20 to 50.

CONCLUSIONS

This search strategy fills an important research gap in this field. It will help systematic reviewers, clinical guideline developers, and policy-makers to retrieve published research on patients' views and preferences. In turn, this will facilitate the inclusion of this critical aspect when formulating heath care decisions, including recommendations.

Locating sex- and gender-specific data in health promotion research: evaluating the sensitivity and precision of published filters

Objective

This study explored the effectiveness of search filters in identifying sex- and gender-specific data in health promotion studies that are indexed in MEDLINE.

Methods

Literature searches were conducted to identify studies on patient or consumer attitudes and behaviors toward colorectal cancer screening, nutritional labeling, and influenza vaccination. Publications reporting sex- or gender-specific outcome data constituted the gold standards for this study. The sensitivity and precision of previously published gender-specific filters, as well as individual filter component terms, were calculated and compared with values identified in prior studies.

Results

The sensitivity and precision of published sex or gender filters varied across topics. Sensitivity values ranged from 14.3% to 92.5%, while precision varied from 17.9% to 51.4%. These filters were less sensitive and less precise in their identification of relevant studies than has been reported in previous studies. Further, while the MEDLINE Medical Subject Headings (MeSH) term “Sex Factors” achieved the greatest average precision (59.3%) of any individual filter term, the MEDLINE check tag “Female” returned the highest average sensitivity (90.1%), with an average precision of 25.0% across topics.

Conclusions

Although search filters can facilitate the identification of research evidence to enable decision making, variability in study abstracting and indexing can limit the generalizability and usability of these filters. This potential for variability should be considered when deciding to incorporate a search filter into any literature search. This research highlights the importance of this awareness when developing strategies for searching the published literature and the potential value of supplementing database searching with other methods of study identification.

Use of Expansion Cohorts in Phase I Trials and Probability of Success in Phase II for 381 Anticancer Drugs

Purpose: Evaluate the association between the use of phase I expansion cohorts (ECs) and drug performance in phase II as well as time to approval by the FDA.Experimental Design: We performed a systematic search of MEDLINE for single-agent dose-finding adult oncology phase I trials published in 2006 to 2011 and subsequent phase II trials. Successful phase II trials were those that met their primary endpoints. Dates of approval were obtained from the Drugs@FDA website in April 2014. A logistic regression model was used to determine the associations between variables and success in phase II.Results: We identified 533 phase I trials evaluating 381 drugs; 112 drugs had at least one phase I trial with an expansion cohort. Phase I trials with expansion cohorts of two to 20 patients were associated with a higher rate of successful phase II trials than those with no expansion cohort [48% vs. 27%; OR, 2.1; 95% confidence interval (CI), 1.1-4.0; P = 0.037]. Phase II success rates were the same for expansion cohort with two to 20 and more than 20 patients (48% vs. 52%). Other positive associations were disease-specific trials (OR, 1.7; 95% CI, 1.0-2.9; P = 0.037), industry sponsorship (OR, 2.9; 95% CI, 1.5-5.7; P = 0.0024), and response rate of 6% to 20% (OR, 2.89; 95% CI, 1.6-5.2; P = 0.0007). Drugs tested in phase I trials with expansion cohorts had a higher rate of 5-year approval (19% vs. 5%; HR, 4.4; 95% CI, 2.2-8.8; P < 0.001).Conclusions: The use of expansion cohorts in phase I trials was associated with success of subsequent phase II trials. However, confounders may play a role in this association. Clin Cancer Res; 23(15); 4020-6. ©2017 AACR.

Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry

OBJECTIVES

To summarise logistical aspects of recently completed systematic reviews that were registered in the International Prospective Register of Systematic Reviews (PROSPERO) registry to quantify the time and resources required to complete such projects.

DESIGN

Meta-analysis.

DATA SOURCES AND STUDY SELECTION

All of the 195 registered and completed reviews (status from the PROSPERO registry) with associated publications at the time of our search (1 July 2014).

DATA EXTRACTION

All authors extracted data using registry entries and publication information related to the data sources used, the number of initially retrieved citations, the final number of included studies, the time between registration date to publication date and number of authors involved for completion of each publication. Information related to funding and geographical location was also recorded when reported.

RESULTS

The mean estimated time to complete the project and publish the review was 67.3 weeks (IQR=42). The number of studies found in the literature searches ranged from 27 to 92 020; the mean yield rate of included studies was 2.94% (IQR=2.5); and the mean number of authors per review was 5, SD=3. Funded reviews took significantly longer to complete and publish (mean=42 vs 26 weeks) and involved more authors and team members (mean=6.8 vs 4.8 people) than those that did not report funding (both p<0.001).

CONCLUSIONS

Systematic reviews presently take much time and require large amounts of human resources. In the light of the ever-increasing volume of published studies, application of existing computing and informatics technology should be applied to decrease this time and resource burden. We discuss recently published guidelines that provide a framework to make finding and accessing relevant literature less burdensome.

Evaluating the MEDLINE Core Clinical Journals filter: data-driven evidence assessing clinical utility

RATIONALE, AIMS AND OBJECTIVES

MEDLINE offers the Core Clinical Journals filter to limit to clinically useful journals. To determine its effectiveness for searching and patient-centric decision making, this study compared literature used for Morning Report in Internal Medicine with journals in the filter.

METHOD

An EndNote library with references answering 327 patient-related questions during Morning Report from 2007 to 2012 was exported to a file listing variables including designated Core Clinical Journal, Impact Factor, date used and medical subject. Bradford's law of scattering was applied ranking the journals and reflecting their clinical utility. Recall (sensitivity) and precision of the Core Morning Report journals and non-Core set was calculated. This study applied bibliometrics to compare the 628 articles used against these criteria to determine journals impacting decision making.

RESULTS

Analysis shows 30% of clinically used articles are from the Core Clinical Journals filter and 16% of the journals represented are Core titles. When Bradford-ranked, 55% of the top 20 journals are Core. Articles <5 years old furnish 63% of sources used. Among the 63 Morning Report subjects, 55 have <50% precision and 41 have <50% recall including 37 subjects with 0% precision and 0% recall.

CONCLUSIONS

Low usage of publications within the Core Clinical Journals filter indicates less relevance for hospital-based care. The divergence from high-impact medicine titles suggests clinically valuable journals differ from academically important titles. With few subjects demonstrating high recall or precision, the MEDLINE Core Clinical Journals filter may require a review and update to better align with current clinical needs.

Value of databases other than medline for rapid health technology assessments

OBJECTIVES

The objective of this study was to explore the degree to which databases other than MEDLINE contribute studies relevant for inclusion in rapid health technology assessments (HTA).

METHODS

We determined the extent to which the clinical, economic, and social studies included in twenty-one full and four rapid HTAs published by three Canadian HTA agencies from 2007 to 2012 were indexed in MEDLINE. Other electronic databases, including EMBASE, were then searched, in sequence, to assess whether or not they indexed studies not found in MEDLINE. Assessment topics ranged from purely clinical (e.g., drug-eluting stents) to those with broader social implications (e.g., spousal violence).

RESULTS

MEDLINE contributed the majority of studies in all but two HTA reports, indexing a mean of 89.6 percent of clinical studies across all HTAs, and 88.3 percent of all clinical, economic, and social studies in twenty-four of twenty-five HTAs. While EMBASE contributed unique studies to twenty-two of twenty-five HTAs, three rapid HTAs did not include any EMBASE studies. In some instances, PsycINFO and CINAHL contributed as many, if not more, non-MEDLINE studies than EMBASE.

CONCLUSIONS

Our findings highlight the importance of assessing the topic-specific relative value of including EMBASE, or more specialized databases, in HTA search protocols. Although MEDLINE continues to be a key resource for HTAs, the time and resource limitations inherent in the production of rapid HTAs require that researchers carefully consider the value and limitations of other information sources to identify relevant studies.

Evidence-based pediatric orthopaedics: an introduction, part I

Evidence-based medicine is a relatively new and sometimes controversial concept when applied to pediatric orthopaedics. This article provides pediatric orthopaedists with some basics to help them understand and apply evidence-based medicine to their clinical practice.

Sensitivity and predictive value of 15 PubMed search strategies to answer clinical questions rated against full systematic reviews

Background

Clinicians perform searches in PubMed daily, but retrieving relevant studies is challenging due to the rapid expansion of medical knowledge. Little is known about the performance of search strategies when they are applied to answer specific clinical questions.

Objective

To compare the performance of 15 PubMed search strategies in retrieving relevant clinical trials on therapeutic interventions.

Methods

We used Cochrane systematic reviews to identify relevant trials for 30 clinical questions. Search terms were extracted from the abstract using a predefined procedure based on the population, interventions, comparison, outcomes (PICO) framework and combined into queries. We tested 15 search strategies that varied in their query (PIC or PICO), use of PubMed’s Clinical Queries therapeutic filters (broad or narrow), search limits, and PubMed links to related articles. We assessed sensitivity (recall) and positive predictive value (precision) of each strategy on the first 2 PubMed pages (40 articles) and on the complete search output.

Results

The performance of the search strategies varied widely according to the clinical question. Unfiltered searches and those using the broad filter of Clinical Queries produced large outputs and retrieved few relevant articles within the first 2 pages, resulting in a median sensitivity of only 10%–25%. In contrast, all searches using the narrow filter performed significantly better, with a median sensitivity of about 50% (all P < .001 compared with unfiltered queries) and positive predictive values of 20%–30% (P < .001 compared with unfiltered queries). This benefit was consistent for most clinical questions. Searches based on related articles retrieved about a third of the relevant studies.

Conclusions

The Clinical Queries narrow filter, along with well-formulated queries based on the PICO framework, provided the greatest aid in retrieving relevant clinical trials within the 2 first PubMed pages. These results can help clinicians apply effective strategies to answer their questions at the point of care.

Retrieval of diagnostic and treatment studies for clinical use through PubMed and PubMed's Clinical Queries filters

OBJECTIVE

Clinical Queries filters were developed to improve the retrieval of high-quality studies in searches on clinical matters. The study objective was to determine the yield of relevant citations and physician satisfaction while searching for diagnostic and treatment studies using the Clinical Queries page of PubMed compared with searching PubMed without these filters.

MATERIALS AND METHODS

Forty practicing physicians, presented with standardized treatment and diagnosis questions and one question of their choosing, entered search terms which were processed in a random, blinded fashion through PubMed alone and PubMed Clinical Queries. Participants rated search retrievals for applicability to the question at hand and satisfaction.

RESULTS

For treatment, the primary outcome of retrieval of relevant articles was not significantly different between the groups, but a higher proportion of articles from the Clinical Queries searches met methodologic criteria (p=0.049), and more articles were published in core internal medicine journals (p=0.056). For diagnosis, the filtered results returned more relevant articles (p=0.031) and fewer irrelevant articles (overall retrieval less, p=0.023); participants needed to screen fewer articles before arriving at the first relevant citation (p<0.05). Relevance was also influenced by content terms used by participants in searching. Participants varied greatly in their search performance.

DISCUSSION

Clinical Queries filtered searches returned more high-quality studies, though the retrieval of relevant articles was only statistically different between the groups for diagnosis questions.

CONCLUSION

Retrieving clinically important research studies from Medline is a challenging task for physicians. Methodological search filters can improve search retrieval.

Risk factors for bladder cancer: challenges of conducting a literature search using PubMed

The objective of this study was to assess the risk factors for bladder cancer using PubMed articles from January 2000 to December 2009. The study also aimed to describe the challenges encountered in the methodology of a literature search for bladder cancer risk factors using PubMed. Twenty-six categories of risk factors for bladder cancer were identified using the National Cancer Institute Web site and the Medical Subject Headings (MeSH) Web site. A total of 1,338 PubMed searches were run using the term "urinary bladder cancer" and a risk factor term (e.g., "cigarette smoking") and were screened to identify 260 articles for final analysis. The search strategy had an overall precision of 3.42 percent, relative recall of 12.64 percent, and an F-measure of 5.39 percent. Although search terms derived from MeSH had the highest overall precision and recall, the differences did not reach significance, which indicates that for generalized, free-text searches of the PubMed database, the searchers' own terms are generally as effective as MeSH terms.

Development of a heart failure filter for Medline: an objective approach using evidence-based clinical practice guidelines as an alternative to hand searching

BACKGROUND

Heart failure is a highly debilitating syndrome with a poor prognosis primarily affecting the elderly. Clinicians wanting timely access to heart failure evidence to provide optimal patient care can face many challenges in locating this evidence. This study developed and validated a search filter of high clinical utility for the retrieval of heart failure articles in OvidSP Medline.

METHODS

A Clinical Advisory Group was established to advise study investigators. The study set of 876 relevant articles from four heart failure clinical practice guidelines was divided into three datasets: a Term Identification Set, a Filter Development Set, and a Filter Validation Set. A further validation set (the Cochrane Validation Set) was formed using studies included in Cochrane heart failure systematic reviews. Candidate search terms were identified via word frequency analysis. The filter was developed by creating combinations of terms and recording their performance in retrieving items from the Filter Development Set. The filter's recall was then validated in both the Filter Validation Set and the Cochrane Validation Set. A precision estimate was obtained post-hoc by running the filter in Medline and screening the first 200 retrievals for relevance to heart failure.

RESULTS

The four-term filter achieved a recall of 96.9% in the Filter Development Set; 98.2% in the Filter Validation Set; and 97.8% in the Cochrane Validation Set. Of the first 200 references retrieved by the filter when run in Medline, 150 were deemed relevant and 50 irrelevant. The post-hoc precision estimate was therefore 75%.

CONCLUSIONS

This study describes an objective method for developing a validated heart failure filter of high recall performance and then testing its precision post-hoc. Clinical practice guidelines were found to be a feasible alternative to hand searching in creating a gold standard for filter development. Guidelines may be especially appropriate given their clinical utility. A validated heart failure filter is now available to support health professionals seeking reliable and efficient access to the heart failure literature.

Identifying nurse staffing research in Medline: development and testing of empirically derived search strategies with the PubMed interface

Background

The identification of health services research in databases such as PubMed/Medline is a cumbersome task. This task becomes even more difficult if the field of interest involves the use of diverse methods and data sources, as is the case with nurse staffing research. This type of research investigates the association between nurse staffing parameters and nursing and patient outcomes. A comprehensively developed search strategy may help identify nurse staffing research in PubMed/Medline.

Methods

A set of relevant references in PubMed/Medline was identified by means of three systematic reviews. This development set was used to detect candidate free-text and MeSH terms. The frequency of these terms was compared to a random sample from PubMed/Medline in order to identify terms specific to nurse staffing research, which were then used to develop a sensitive, precise and balanced search strategy. To determine their precision, the newly developed search strategies were tested against a) the pool of relevant references extracted from the systematic reviews, b) a reference set identified from an electronic journal screening, and c) a sample from PubMed/Medline. Finally, all newly developed strategies were compared to PubMed's Health Services Research Queries (PubMed's HSR Queries).

Results

The sensitivities of the newly developed search strategies were almost 100% in all of the three test sets applied; precision ranged from 6.1% to 32.0%. PubMed's HSR queries were less sensitive (83.3% to 88.2%) than the new search strategies. Only minor differences in precision were found (5.0% to 32.0%).

Conclusions

As with other literature on health services research, nurse staffing studies are difficult to identify in PubMed/Medline. Depending on the purpose of the search, researchers can choose between high sensitivity and retrieval of a large number of references or high precision, i.e. and an increased risk of missing relevant references, respectively. More standardized terminology (e.g. by consistent use of the term "nurse staffing") could improve the precision of future searches in this field. Empirically selected search terms can help to develop effective search strategies. The high consistency between all test sets confirmed the validity of our approach.

The publication echo: effects of retrieving literature in PubMed by year of publication

OBJECTIVES

In PubMed search forms, the publication date refers to both the date of electronic and printed publication. This fact is documented in PubMed, but difficult to anticipate by the users and can provoke misinterpretations of search results. The Technical Note aims at systematically investing the effect (referred to as the publication echo), clarifying onset and extent of the publication echo, and comments on its impact.

METHODS

Papers with ambiguous publication dates are systematically retrieved and a trend analysis with seasonal decomposition on monthly publication data is performed.

RESULTS

First doubled search results were found for 1999, their number since then rapidly increasing. Up to 17.6% of all articles of a year are found to be published electronically and in print, which can be before or afterwards. Maximum delay between the two dates is three years, except for one singular publication, where it is five years. Publication trends are exponential and linear when considering echoed and echo-cleaned data, respectively.

CONCLUSIONS

As a conclusion, we suggest using a query formulation that unambiguously retrieves literature from PubMed by the date of publication.