Can electronic search engines optimize screening of search results in systematic reviews: an empirical study

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

INTRODUCTION

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

METHODS

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

RESULTS

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

DISCUSSION

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

CONCLUSIONS

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

Using text mining for study identification in systematic reviews: a systematic review of current approaches

BACKGROUND

The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities.

METHODS

Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings.

RESULTS

The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall).

CONCLUSIONS

Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in 'live' reviews. The use of text mining as a 'second screener' may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.

Using text mining for study identification in systematic reviews: a systematic review of current approaches

BACKGROUND

The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities.

METHODS

Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings.

RESULTS

The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall).

CONCLUSIONS

Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in 'live' reviews. The use of text mining as a 'second screener' may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.

A systematic review of methods for studying consumer health YouTube videos, with implications for systematic reviews

Background. YouTube is an increasingly important medium for consumer health information – with content provided by healthcare professionals, government and non-government organizations, industry, and consumers themselves. It is a rapidly developing area of study for healthcare researchers. We examine the methods used in reviews of YouTube consumer health videos to identify trends and best practices.

Methods and Materials. Published reviews of consumer-oriented health-related YouTube videos were identified through PubMed. Data extracted from these studies included type of journal, topic, characteristics of the search, methods of review including number of reviewers and method to achieve consensus between reviewers, inclusion and exclusion criteria, characteristics of the videos reported, ethical oversight, and follow-up.

Results. Thirty-three studies were identified. Most were recent and published in specialty journals. Typically, these included more than 100 videos, and were examined by multiple reviewers. Most studies described characteristics of the videos, number of views, and sometime characteristics of the viewers. Accuracy of portrayal of the health issue under consideration was a common focus.

Conclusion. Optimal transparency and reproducibility of studies of YouTube health-related videos can be achieved by following guidance designed for systematic review reporting, with attention to several elements specific to the video medium. Particularly when seeking to replicate consumer viewing behavior, investigators should consider the method used to select search terms, and use a snowballing rather than a sequential screening approach. Discontinuation protocols for online screening of relevance ranked search results is an area identified for further development.

Titles versus titles and abstracts for initial screening of articles for systematic reviews

Background

There is no consensus on whether screening titles alone or titles and abstracts together is the preferable strategy for inclusion of articles in a systematic review.

Methods

Two methods of screening articles for inclusion in a systematic review were compared: titles first versus titles and abstracts simultaneously. Each citation found in MEDLINE or Embase was reviewed by two physician reviewers for prespecified criteria: the citation included (1) primary data; (2) the exposure of interest; and (3) the outcome of interest.

Results

There were 2965 unique citations. The titles first strategy resulted in an immediate rejection of 2558 (86%) of the records after reading the title alone, requiring review of 239 titles and abstracts, and subsequently 176 full text articles. The simultaneous titles and abstracts review led to rejection of 2782 citations (94%) and review of 183 full text articles. Interreviewer agreement to include an article for full text review using the titles-first screening strategy was 89%–94% (kappa = 0.54) and 96%–97% (kappa = 0.56) for titles and abstracts combined. The final systematic review included 13 articles, all of which were identified by both screening strategies (yield 100%, burden 114%). Precision was higher in the titles and abstracts method (7.1% versus 3.2%) but recall was the same (100% versus 100%), leading to a higher F-measure for the titles and abstracts approach (0.1327 versus 0.0619).

Conclusion

Screening via a titles-first approach may be more efficient than screening titles and abstracts together.

Basics of systematic reviews and meta-analyses for the nephrologist

Renal practitioners are expected to apply the best available evidence from rigorous scientific research to clinical decision-making and also for policy-making for those involved. Advances in information technology and unprecedented access to data have simplified the process for the search of best available evidence to guide practice. However, it is challenging to cope with the increasing volume of publications in nephrology and other areas of medicine. Accordingly, systematic reviews and meta-analysis have greatly facilitated best practice and effective clinical decision-making. Conducting a systematic review/meta-analysis involves a number of steps that start with protocol development and research question formulation, design and study selection criteria, followed by retrieval of potentially relevant studies, selection of those studies to be included and evaluation of a study's risk of bias. Systematic reviews and meta-analyses have both strengths and weaknesses. Many of the perceived limitations of meta-analysis are not inherent in the methodology, but actually represent deficits in the conduct or reporting of individual primary studies. With the continuous proliferation of published renal clinical studies, such publications will continue to be an important resource for clinicians and researchers in nephrology. It is therefore important for nephrologists to keep abreast of developments in this field, which requires some knowledge about how these studies are conducted, reported and how to appraise them for application to clinical practice or policy-making.