Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects

Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial

OBJECTIVE

To evaluate the impact of adding librarians and information specialists (LIS) as methodological peer reviewers to the formal journal peer review process on the quality of search reporting and risk of bias in systematic review searches in the medical literature.

DESIGN

Pragmatic two-group parallel randomised controlled trial.

SETTING

Three biomedical journals.

PARTICIPANTS

Systematic reviews and related evidence synthesis manuscripts submitted to The BMJ, BMJ Open and BMJ Medicine and sent out for peer review from 3 January 2023 to 1 September 2023. Randomisation (allocation ratio, 1:1) was stratified by journal and used permuted blocks (block size=4). Of 2670 manuscripts sent to peer review during study enrollment, 400 met inclusion criteria and were randomised (62 The BMJ, 334 BMJ Open, 4 BMJ Medicine). 76 manuscripts were revised and resubmitted in the intervention group and 90 in the control group by 2 January 2024.

INTERVENTIONS

All manuscripts followed usual journal practice for peer review, but those in the intervention group had an additional (LIS) peer reviewer invited.

MAIN OUTCOME MEASURES

The primary outcomes are the differences in first revision manuscripts between intervention and control groups in the quality of reporting and risk of bias. Quality of reporting was measured using four prespecified PRISMA-S items. Risk of bias was measured using ROBIS Domain 2. Assessments were done in duplicate and assessors were blinded to group allocation. Secondary outcomes included differences between groups for each individual PRISMA-S and ROBIS Domain 2 item. The difference in the proportion of manuscripts rejected as the first decision post-peer review between the intervention and control groups was an additional outcome.

RESULTS

Differences in the proportion of adequately reported searches (4.4% difference, 95% CI: -2.0% to 10.7%) and risk of bias in searches (0.5% difference, 95% CI: -13.7% to 14.6%) showed no statistically significant differences between groups. By 4 months post-study, 98 intervention and 70 control group manuscripts had been rejected after peer review (13.8% difference, 95% CI: 3.9% to 23.8%).

CONCLUSIONS

Inviting LIS peer reviewers did not impact adequate reporting or risk of bias of searches in first revision manuscripts of biomedical systematic reviews and related review types, though LIS peer reviewers may have contributed to a higher rate of rejection after peer review.

TRIAL REGISTRATION NUMBER

Open Science Framework: https://doi.org/10.17605/OSF.IO/W4CK2.

Integrating Medical Librarians in Evidence-Based Medical Practice in Africa: A Survey of Current Practices and Challenges

Librarians' involvement in Evidence-Based Medical Practice (EBMP) has been widely reported from the Global North. The cross-sectional study designed a survey to investigate how African medical librarians integrate into EBMP. The respondents comprised medical librarians from 12 African countries. Findings revealed that African medical librarians are mostly involved in EBMP activities related to resource use, management, and evidence dissemination. The leading EBMP tools reportedly used or promoted by the librarians include UpToDate and Cochrane Library, while the leading challenges encountered in offering support for EBMP are related to skill deficiency, poor funding, and poor internet connectivity.

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study

OBJECTIVES

To determine the reproducibility of biomedical systematic review search strategies.

STUDY DESIGN AND SETTING

A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.

RESULTS

The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.

CONCLUSION

Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Clinical utility of overviews on adverse events of pharmacological interventions

BACKGROUND

Overviews (i.e., systematic reviews of systematic reviews, meta-reviews, umbrella reviews) are a relatively new type of evidence synthesis. Among others, one reason to conduct an overview is to investigate adverse events (AEs) associated with a healthcare intervention. Overviews aim to provide easily accessible information for healthcare decision-makers including clinicians. We aimed to evaluate the clinical utility of overviews investigating AEs.

METHODS

We used a sample of 27 overviews exclusively investigating drug-related adverse events published until 2021 identified in a prior project. We defined clinical utility as the extent to which overviews are perceived to be useful in clinical practice. Each included overview was assigned to one of seven pharmacological experts with expertise on the topic of the overview. The clinical utility and value of these overviews were determined using a self-developed assessment tool. This included four open-ended questions and a ranking of three clinical utility statements completed by clinicians. We calculated frequencies for the ranked clinical utility statements and coded the answers to the open-ended questions using an inductive approach.

RESULTS

The overall agreement with the provided statements was high. According to the assessments, 67% of the included overviews generated new knowledge. In 93% of the assessments, the overviews were found to add value to the existing literature. The overviews were rated as more useful than the individual included systematic reviews (SRs) in 85% of the assessments. The answers to the open-ended questions revealed two key aspects of clinical utility in the included overviews. Firstly, it was considered useful that they provide a summary of available evidence (e.g., along with additional assessments, or across different populations, or in different settings that have not been evaluated together in the included SRs). Secondly, it was found useful if overviews conducted a new meta-analysis to answer specific research questions that had not been answered previously.

CONCLUSIONS

Overviews on drug-related AEs are considered valuable for clinical practice by clinicians. They can make available evidence on AEs more accessible and provide a comprehensive view of available evidence. As the role of overviews evolves, investigations such as this can identify areas of value.

Paperfetcher: A tool to automate handsearching and citation searching for systematic reviews

Systematic reviews are vital instruments for researchers to understand broad trends in a field and synthesize evidence on the effectiveness of interventions in addressing specific issues. The quality of a systematic review depends critically on having comprehensively surveyed all relevant literature on the review topic. In addition to database searching, handsearching is an important supplementary technique that helps increase the likelihood of identifying all relevant studies in a literature search. Traditional handsearching requires reviewers to manually browse through a curated list of field-specific journals and conference proceedings to find articles relevant to the review topic. This manual process is not only time-consuming, laborious, costly, and error-prone due to human fatigue, but it also lacks replicability due to its cumbersome manual nature. To address these issues, this paper presents a free and open-source Python package and an accompanying web-app, Paperfetcher, to automate the retrieval of article metadata for handsearching. With Paperfetcher's assistance, researchers can retrieve article metadata from designated journals within a specified time frame in just a few clicks. In addition to handsearching, it also incorporates a beta version of citation searching in both forward and backward directions. Paperfetcher has an easy-to-use interface, which allows researchers to download the metadata of retrieved studies as a list of DOIs or as an RIS file to facilitate seamless import into systematic review screening software. To the best of our knowledge, Paperfetcher is the first tool to automate handsearching with high usability and a multi-disciplinary focus.

Specific-class Skin Side-effects of Drugs Might Compromise Blinding in Randomized Controlled Trials

Abstract is missing (Short communication)

Assessing the Reporting of Harms in Systematic Reviews Focused on the Therapeutic and Cosmetic Uses of Botulinum Toxin

BACKGROUND AND OBJECTIVE

The expanding use of botulinum toxin (BoNT) in medical practice demonstrates the need to highlight whether there is adequate information regarding its safety profile. The aim of our study was to identify completeness of harms reporting for BoNT treatment within systematic reviews (SRs), assess quality of SRs using the AMSTAR-2 tool, and determine the degree of overlap among primary studies within each SR.

METHODS

On May 31, 2022, we searched Embase, Epistemonikos, MEDLINE, and the Cochrane Database of Systematic Reviews for SRs on BoNT therapy. Screening and data extraction were performed in a masked, duplicate fashion. AMSTAR-2 was used to evaluate the methodological quality of included SRs. Corrected covered area (CCA) was calculated for SR dyads.

RESULTS

Of the 90 included SRs, we found that 70 completed less than 50% of harms items. The most reported items were BoNT as a favorable intervention (73/90, 81.1%) and harms as a primary outcome (72/90, 80.0%). The least reported items were grades and severity scales used to classify harms (8/90, 8.9%) and number of treatment discontinuations in each arm (10/90, 11.1%). Eighty-three SRs were rated "critically low" (83/90, 92.2%), while 5 SRs were rated "high" (5/90, 5.6%) via AMSTAR-2 tool. Significant associations were found between completion of harms reporting and: (1) a "critically low" appraisal on AMSTAR-2 tool (p = 0.0060) and (2) whether harms was reported as a primary outcome (p = 0.0001). The total CCA overlap was determined to be 0.8%.

CONCLUSION

Our results demonstrate that harms are underreported within BoNT SRs. Because healthcare professionals often refer to SRs to guide clinical decision making, it is important to continue to explore shortcomings among BoNT literature in future studies.

Reporting Quality of Oral TCM Systematic Reviews Based on the PRISMA Harms Checklist from 2013 to 2020

Background

Systematic reviews focusing on the effectiveness of different kinds of healthcare interventions have been widely published, but there were few guidelines for reporting safety concerns before 2016. The PRISMA harms checklist, which was published in 2016, can standardize reporting quality.

Objectives

To evaluate the safety information reporting quality of oral traditional Chinese medicine (TCM) in systematic reviews before and after the PRISMA harms checklist was published and to explore factors associated with better reporting.

Methods

We searched PubMed, the Cochrane Library, and Embase to identify all systematic reviews using oral TCM as interventions published before (from 2013 to 2015) and after (from 2017 to 2020) the PRISMA harms checklist was published. We used the PRISMA harms checklist to assess the quality of reporting of the safety information to included systematic reviews.

Results

In total, 200 systematic reviews were sampled from eligible studies published between 2013 and 2020. Reviews from 2016 were excluded. Scores on the PRISMA harms checklist (23 items) ranged from 0 to 12. A systematic reviews published after 2016 had better reporting quality compared with studies published before 2016 with regard to the title (P=0.03), results of individual studies (P=0.016), and risk of bias across studies (P=0.043). In all included systematic reviews of our study, the state conclusion in coherence with review findings was reported adequately with the proportion of adherence at 95%; other items had a reporting proportion ranging from 0% to 57%. The four essential reporting items of the PRISMA harms checklist also had a low reporting quality ranging from 0% to 4%.

Conclusions

Oral TCM systematic reviews reported inadequate safety information before and after the PRISMA harms checklist was published. This survey suggested that the PRISMA harms checklist should be recommended more to both original research and systematic review authors.

Harms in Systematic Reviews Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin

OBJECTIVE

We compared methods used with current recommendations for synthesizing harms in systematic reviews and meta-analyses (SRMAs) of gabapentin.

STUDY DESIGN & SETTING

We followed recommended systematic review practices. We selected reliable SRMAs of gabapentin (i.e., met a pre-defined list of methodological criteria) that assessed at least one harm. We extracted and compared methods in four areas: pre-specification, searching, analysis, and reporting. Whereas our focus in this paper is on the methods used, Part 2 examines the results for harms across reviews.

RESULTS

We screened 4320 records and identified 157 SRMAs of gabapentin, 70 of which were reliable. Most reliable reviews (51/70; 73%) reported following a general guideline for SRMA conduct or reporting, but none reported following recommendations specifically for synthesizing harms. Across all domains assessed, review methods were designed to address questions of benefit and rarely included the additional methods that are recommended for evaluating harms.

CONCLUSION

Approaches to assessing harms in SRMAs we examined are tokenistic and unlikely to produce valid summaries of harms to guide decisions. A paradigm shift is needed. At a minimal, reviewers should describe any limitations to their assessment of harms and provide clearer descriptions of methods for synthesizing harms.

Harms in Systematic Reviews Paper 1: An introduction to research on harms

OBJECTIVE

Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews.

STUDY DESIGN AND SETTING

Commentary describing challenges with assessing harm.

RESULTS

Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing.

CONCLUSION

The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.

Improving peer review of systematic reviews by involving librarians and information specialists: protocol for a randomized controlled trial

BACKGROUND

Problems continue to exist with the reporting quality and risk of bias in search methods and strategies in systematic reviews and related review types. Peer reviewers who are not familiar with what is required to transparently and fully report a search may not be prepared to review the search components of systematic reviews, nor may they know what is likely to introduce bias into a search. Librarians and information specialists, who have expertise in searching, may offer specialized knowledge that would help improve systematic review search reporting and lessen risk of bias, but they are underutilized as methodological peer reviewers.

METHODS

This study will evaluate the effect of adding librarians and information specialists as methodological peer reviewers on the quality of search reporting and risk of bias in systematic review searches. The study will be a pragmatic randomized controlled trial using 150 systematic review manuscripts submitted to BMJ and BMJ Open as the unit of randomization. Manuscripts that report on completed systematic reviews and related review types and have been sent for peer review are eligible. For each manuscript randomized to the intervention, a librarian/information specialist will be invited as an additional peer reviewer using standard practices for each journal. First revision manuscripts will be assessed in duplicate for reporting quality and risk of bias, using adherence to 4 items from PRISMA-S and assessors' judgements on 4 signaling questions from ROBIS Domain 2, respectively. Identifying information from the manuscripts will be removed prior to assessment.

DISCUSSION

The primary outcomes for this study are quality of reporting as indicated by differences in the proportion of adequately reported searches in first revision manuscripts between intervention and control groups and risk of bias as indicated by differences in the proportions of first revision manuscripts with high, low, and unclear bias. If the intervention demonstrates an effect on search reporting or bias, this may indicate a need for journal editors to work with librarians and information specialists as methodological peer reviewers.

TRIAL REGISTRATION

Open Science Framework. Registered on June 17, 2021, at https://doi.org/10.17605/OSF.IO/W4CK2 .

Reporting of the methodological quality of search strategies in orthodontic quantitative systematic reviews

BACKGROUND

This study aimed to assess the reporting of the methodological quality of search strategies undertaken in orthodontic quantitative systematic reviews (SRs) and hence their reproducibility.

MATERIALS AND METHODS

A search of a single electronic database (Medline via PubMed) was undertaken to identify interventional orthodontic SRs with meta-analysis published within a 10-year period. The Cochrane Library of Systematic Reviews was also sourced. Full articles were reviewed by two assessors against the eligibility criteria. The reporting quality of each search strategy was assessed using a previously validated checklist with a score of 1 or 2 given for each of the eight items. Cumulative totals were calculated. Guided by previous research, the authors agreed the following cut-offs to categorize the overall level of quality: 8-10 (poor), 10-12 (fair), and greater than 13 (good).

RESULTS

A total of 127 SRs were analysed. The overall median quality score for the reporting of the search strategy was 14 [interquartile range (IQR): 13-15]. Cochrane SRs and those originating in Europe received higher aggregate scores, whereas no difference was evident based on Prospero registration. The continent of the corresponding author predicated the overall score. Non-Cochrane reviews achieved lower overall scores compared to Cochrane reviews (-1.0, 95% confidence interval: -1.65, -0.34, P = 0.003). The most frequently searched database was EMBASE (N = 93) and the median number of authors was 5 (IQR 4-6). Authors of 26.8% of SRs searched the grey literature. Language restrictions were applied to the search strategies of 88 (69.3%) SRs.

CONCLUSIONS

The reporting quality of search strategies undertaken in orthodontic SRs is at a good level but differences between Cochrane and non-Cochrane reviews currently exist. The reporting of searching of the grey literature and application of no language restrictions can be improved.

The effect of librarian involvement on the quality of systematic reviews in dental medicine

OBJECTIVES

To determine whether librarian or information specialist authorship is associated with better reproducibility of the search, at least three databases searched, and better reporting quality in dental systematic reviews (SRs).

METHODS

SRs from the top ten dental research journals (as determined by Journal Citation Reports and Scimago) were reviewed for search quality and reproducibility by independent reviewers using two Qualtrics survey instruments. Data was reviewed for all SRs based on reproducibility and librarian participation and further reviewed for search quality of reproducible searches.

RESULTS

Librarians were co-authors in only 2.5% of the 913 included SRs and librarians were mentioned or acknowledged in only 9% of included SRs. Librarian coauthors were associated with more reproducible searches, higher search quality, and at least three databases searched. Although the results indicate librarians are associated with improved SR quality, due to the small number of SRs that included a librarian, results were not statistically significant.

CONCLUSION

Despite guidance from organizations that produce SR guidelines recommending the inclusion of a librarian or information specialist on the review team, and despite evidence showing that librarians improve the reproducibility of searches and the reporting of methodology in SRs, librarians are not being included in SRs in the field of dental medicine. The authors of this review recommend the inclusion of a librarian on SR teams in dental medicine and other fields.

Systematic reviews: Structure, form and content

This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the search. It also covers systematic review reporting standards such as PRISMA-P and PRISMA, critical appraisal and tools and resources to support the review and ensure it is conducted efficiently and effectively. Finally, it summarizes the requirements when screening search results for inclusion in the review, and the statistical synthesis of included studies' findings.

Quality of Literature Searches Published in Leading Neurosurgical Journals: A Review of Reviews

BACKGROUND

There is mounting evidence that the search strategies upon which systematic reviews (SRs) are based frequently contain errors are incompletely reported or insensitive.

OBJECTIVE

To appraise the quality of search strategies in the 10 leading specialty neurosurgical journals and identify factors associated with superior searches.

METHODS

This research-on-research study systematically surveyed SRs published in the 10 leading neurosurgical journals between 01/10/2017 and 31/10/2019. All SRs were eligible for assessment using a predefined coding manual that was adapted from the preferred reporting items for systematic reviews and meta-analyses (PRISMA), a measurement tool to assess systematic reviews (AMSTAR), and Cochrane Collaboration guidelines. The PubMed interface was used to search the MEDLINE database, which was supplemented by individual journal searches. Descriptive statistics were utilized to identify factors associated with improved search strategies.

RESULTS

A total of 633 articles were included and contained a median of 19.00 (2.00-1654.00) studies. Less than half (45.97%) of included search strategies were considered to be reproducible. Aggregated reporting score was positively associated with in-text reference to reporting guideline adherence (τb = 0.156, P < .01). The number of articles retrieved by a search (τb = 0.11, P < .01) was also associated with the reporting of a reproducible search strategy.

CONCLUSION

This study demonstrates that the search strategies used in neurosurgical SRs require improvement. In addition to increasing awareness of reporting standards, we propose that this be achieved by the incorporation of PRISMA and other guidelines into article submission and peer-review processes. This may lead to the conduct of more informative SRs, which may result in improved clinician decision-making and patient outcomes.

Using Embase as a supplement to PubMed in Cochrane reviews differed across fields

OBJECTIVE

Medline/PubMed is often first choice for health science researchers when doing literature searches. However, Medline/PubMed does not cover the health science research literature equally well across specialties. Embase is often considered an important supplement to Medline/PubMed in health sciences. The present study analyzes the coverage of Embase as a supplement to PubMed, and the aim of the study is to investigate if searching Embase can compensate for low PubMed retrieval.

STUDY DESIGN AND SETTING

The population in this study is all the included studies in all Cochrane reviews from 2012 to 2016 across the 53 Cochrane groups. The analyses were performed using two units of analysis (study and publication). We are examining the coverage in Embase of publications and studies not covered by PubMed (25,119 publications and 9,420 studies).

RESULTS

The results showed that using Embase as a supplement to PubMed resulted in a coverage of 66,994 publications out of 86,167 and a coverage rate of 77.7, 95% CI [75.05, 80.45] of all the included publications. Embase combined with PubMed covered 48,326 out of 54,901 studies and thus had a coverage rate of 88.0%, 95% CI [86.2, 89.9] of studies. The results also showed that supplementing PubMed with Embase increased coverage of included publications by 6.8 percentage points, and the coverage of studies increased by 5.5 percentage points. Substantial differences were found across and within review groups over time.

CONCLUSION

The included publications and studies in some groups are covered considerably better by supplementing with Embase, whereas in other groups, the difference in coverage is negligible. However, due to the variation over time, one should be careful predicting the benefit from supplementing PubMed with Embase to retrieve relevant publications to include in a review.

New challenges and solutions for information retrieval: information specialists as “bricklayers” in the health technology assessment framework

We are pleased to present this information retrieval issue. Our goal is to highlight the variety of tasks and organizational structures of information retrieval within heath technology assessment (HTA). This special issue was planned and organized by the HTAi Information Retrieval Group (IRG). The choice of publications in this issue reflects the versatility and high competence in information retrieval methodology that is needed as an information specialist. Furthermore, it provides insights into the daily challenges faced by information specialists.

Risk of Adverse Outcomes in Females Taking Oral Creatine Monohydrate: A Systematic Review and Meta-Analysis

Creatine Monohydrate (CrM) is a dietary supplement routinely used as an ergogenic aid for sport and training, and as a potential therapeutic aid to augment different disease processes. Despite its increased use in recent years, studies reporting potential adverse outcomes of CrM have been mostly derived from male or mixed sex populations. A systematic search was conducted, which included female participants on CrM, where adverse outcomes were reported, with meta-analysis performed where appropriate. Six hundred and fifty-six studies were identified where creatine supplementation was the primary intervention; fifty-eight were female only studies (9%). Twenty-nine studies monitored for adverse outcomes, with 951 participants. There were no deaths or serious adverse outcomes reported. There were no significant differences in total adverse events, (risk ratio (RR) 1.24 (95% CI 0.51, 2.98)), gastrointestinal events, (RR 1.09 (95% CI 0.53, 2.24)), or weight gain, (mean difference (MD) 1.24 kg pre-intervention, (95% CI -0.34, 2.82)) to 1.37 kg post-intervention (95% CI -0.50, 3.23)), in CrM supplemented females, when stratified by dosing regimen and subject to meta-analysis. No statistically significant difference was reported in measures of renal or hepatic function. In conclusion, mortality and serious adverse events are not associated with CrM supplementation in females. Nor does the use of creatine supplementation increase the risk of total adverse outcomes, weight gain or renal and hepatic complications in females. However, all future studies of creatine supplementation in females should consider surveillance and comprehensive reporting of adverse outcomes to better inform participants and health professionals involved in future trials.

A systematic survey showed important limitations in the methods for assessing drug safety among systematic reviews

OBJECTIVES

This study aimed to examine the design, conduct, and analysis of systematic reviews assessing drug safety through a cross-sectional survey.

STUDY DESIGN AND SETTING

We searched PubMed to identity systematic reviews published in the Cochrane Database of Systematic Reviews and Core Clinical Journals indexed in 2015 and randomly sampled systematic reviews assessing drug effects at a 1:1 ratio of Cochrane and non-Cochrane reviews. Teams of two investigators independently conducted study screening and collected data, using prespecified, standardized questionnaires. In addition to general information, we collected details about the planning and analyses of safety outcomes.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Most reviews searched PubMed/MEDLINE (n = 117, 97.5%), EMBASE (n = 105, 87.5%), and Cochrane CENTRAL (n = 110, 91.7%) and conducted independent and duplicate study selection (n = 98, 81.7%), risk of bias assessment (n = 105, 87.5%), and data collection (n = 105, 87.5%). Only nine (7.5%) reviews clearly defined safety outcomes, and seven (5.8%) defined a primary safety outcome; none stated whether the primary safety outcome was predefined. Among the 80 reviews that pooled the primary dichotomous safety data across studies, less than half (41%, n = 33) conducted subgroup analysis to explore for sources of heterogeneity or reported a GRADE assessment for the overall quality of evidence. Cochrane reviews were more likely to provide a study protocol (100% vs. 23.3%; P < 0.001), involve methodologists (53.3% vs. 20.0%; P < 0.001), and report a GRADE assessment for the primary safety outcome (70.6% vs. 19.6%; P < 0.001).

CONCLUSION

Our findings highlighted areas for improved planning and analysis in the assessment of drug safety among systematic reviews. Cochrane reviews were superior to non-Cochrane reviews; however, most reviews did not prespecify their safety outcomes or methods for analysis, explore sources of heterogeneity among pooled effects, or assess the overall quality of evidence with the GRADE approach.

A Computational Approach for Objectively Derived Systematic Review Search Strategies

Searching literature for a systematic review begins with a manually constructed search strategy by an expert information specialist. The typical process of constructing search strategies is often undocumented, ad-hoc, and subject to individual expertise, which may introduce bias in the systematic review. A new method for objectively deriving search strategies has arisen from information specialists attempting to address these shortcomings. However, this proposed method still presents a number of manual, ad-hoc interventions, and trial-and-error processes, potentially still introducing bias into systematic reviews. Moreover, this method has not been rigorously evaluated on a large set of systematic review cases, thus its generalisability is unknown. In this work, we present a computational adaptation of this proposed objective method. Our adaptation removes the human-in-the-loop processes involved in the initial steps of creating a search strategy for a systematic review; reducing bias due to human factors and increasing the objectivity of the originally proposed method. Our proposed computational adaptation further enables a formal and rigorous evaluation over a large set of systematic reviews. We find that our computational adaptation of the original objective method provides an effective starting point for information specialists to continue refining. We also identify a number of avenues for extending and improving our adaptation to further promote supporting information specialists.

What are Systematic Reviews?

Systematic reviews are a type of review that uses repeatable analytical methods to collect secondary data and analyse it. Systematic reviews are a type of evidence synthesis which formulate research questions that are broad or narrow in scope, and identify and synthesize data that directly relate to the systematic review question. While some people might associate ‘systematic review’ with 'meta-analysis', there are multiple kinds of review which can be defined as ‘systematic’ which do not involve a meta-analysis. Some systematic reviews critically appraise research studies, and synthesize findings qualitatively or quantitatively. Systematic reviews are often designed to provide an exhaustive summary of current evidence relevant to a research question. For example, systematic reviews of randomized controlled trials are an important way of informing evidence-based medicine, and a review of existing studies is often quicker and cheaper than embarking on a new study. While systematic reviews are often applied in the biomedical or healthcare context, they can be used in other areas where an assessment of a precisely defined subject would be helpful. Systematic reviews may examine clinical tests, public health interventions, environmental interventions, social interventions, adverse effects, qualitative evidence syntheses, methodological reviews, policy reviews, and economic evaluations. An understanding of systematic reviews and how to implement them in practice is highly recommended for professionals involved in the delivery of health care, public health and public policy.

Benchmarking veterinary librarians' participation in systematic reviews and scoping reviews

OBJECTIVES

The objectives of this study were to benchmark roles that veterinary librarians at universities and colleges play in systematic reviews (SRs) and scoping reviews that are conducted by faculty and students at their institutions, to benchmark the level of training that veterinary librarians have in conducting SRs, to identify barriers to their participation in SRs, and to identify other types of literature reviews that veterinary librarians participate in.

METHODS

Sixty veterinary librarians in universities and colleges in Canada, the United States, England, Scotland, Ireland, Australia, and New Zealand were surveyed online about their roles and training in conducting SRs, barriers to participation in SRs, and participation in other types of literature reviews.

RESULTS

Veterinary librarians' highest participation was at an advising level in traditional librarian roles as question formulator, database selector, search strategy developer, and reference manager. Most respondents reported pretty good to extensive training in traditional roles and no or some training in less traditional roles. Sixty percent of respondents received few or no requests to participate in SRs, and only half of respondents had participated in SRs as a review team member. Sixty percent of respondents stated that their libraries had no policies regarding librarian roles and participation in SRs.

CONCLUSIONS

The surveyed veterinary librarians participated in SRs to a lesser degree than human health sciences librarians, experienced low demand from veterinary faculty and students to participate in SRs, and participated as review team members at significantly lower rates than human health sciences librarians. The main barriers to participation in SRs were lack of library policies, insufficient training, and lack of time.

The reporting of safety among drug systematic reviews was poor before the implementation of the PRISMA harms checklist

OBJECTIVES

To examine, through a cross-sectional survey, how well safety information was reported among drug systematic reviews predating PRISMA harms checklist and explore factors associated with better reporting.

STUDY DESIGN AND SETTING

We searched PubMed to identify all systematic reviews published in the Cochrane Database of Systematic Review or the core clinical journals in 2015, one year before the PRISMA harms checklist was published. We randomly selected, in a 1:1 ratio, Cochrane and non-Cochrane systematic reviews assessing drug effects (including both efficacy and safety). We used the PRISMA harms checklist published in 2016 to assess the quality of reporting of drug safety information. Multivariable linear regression analyses were used to explore the association of six prespecified variables with more complete reporting of PRISMA harms items.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Scores on the PRISMA harms checklist (23 items) were low (median 4, [first, third quartile: 2, 6]), with no difference between Cochrane and non-Cochrane reviews (4.5 [2, 7] vs. 4 [2.5, 5]; P = 0.29). Among all eligible reviews, only one item (i.e., state conclusions in coherence with the review findings) was reported adequately (proportion of adherence 81.6%); proportion of reporting for other items ranged from 1.7% to 68.3%. The four essential reporting items from PRISMA harms checklist were also poorly complied (proportion of adherence ranged from 1.7% to 9.2%). Multivariable linear regression analyses found no significant associations between any study characteristic and reporting on the PRISMA harms, likely because of limited variability in scores across studies.

CONCLUSIONS

The reporting of safety information was poor both for Cochrane and non-Cochrane drug systematic reviews predating PRISMA harms checklist. The findings suggested a strong need to use the PRISMA harms checklist for reporting safety among drug systematic reviews.

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

BACKGROUND

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence. Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

METHOD

A literature review. Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through 'pearl growing', citation chasing, a search of PubMed using the systematic review methods filter, and the authors' topic knowledge. The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of 'key stages' in the process of literature searching.

RESULTS

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

CONCLUSIONS

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations

BACKGROUND

Systematic reviews and meta-analyses of observational studies are frequently performed, but no widely accepted guidance is available at present. We performed a systematic scoping review of published methodological recommendations on how to systematically review and meta-analyse observational studies.

METHODS

We searched online databases and websites and contacted experts in the field to locate potentially eligible articles. We included articles that provided any type of recommendation on how to conduct systematic reviews and meta-analyses of observational studies. We extracted and summarised recommendations on pre-defined key items: protocol development, research question, search strategy, study eligibility, data extraction, dealing with different study designs, risk of bias assessment, publication bias, heterogeneity, statistical analysis. We summarised recommendations by key item, identifying areas of agreement and disagreement as well as areas where recommendations were missing or scarce.

RESULTS

The searches identified 2461 articles of which 93 were eligible. Many recommendations for reviews and meta-analyses of observational studies were transferred from guidance developed for reviews and meta-analyses of RCTs. Although there was substantial agreement in some methodological areas there was also considerable disagreement on how evidence synthesis of observational studies should be conducted. Conflicting recommendations were seen on topics such as the inclusion of different study designs in systematic reviews and meta-analyses, the use of quality scales to assess the risk of bias, and the choice of model (e.g. fixed vs. random effects) for meta-analysis.

CONCLUSION

There is a need for sound methodological guidance on how to conduct systematic reviews and meta-analyses of observational studies, which critically considers areas in which there are conflicting recommendations.

It takes longer than you think: librarian time spent on systematic review tasks

INTRODUCTION

The authors examined the time that medical librarians spent on specific tasks for systematic reviews (SRs): interview process, search strategy development, search strategy translation, documentation, deliverables, search methodology writing, and instruction. We also investigated relationships among the time spent on SR tasks, years of experience, and number of completed SRs to gain a better understanding of the time spent on SR tasks from time, staffing, and project management perspectives.

METHODS

A confidential survey and study description were sent to medical library directors who were members of the Association of Academic Health Sciences Libraries as well as librarians serving members of the Association of American Medical Colleges or American Osteopathic Association.

RESULTS

Of the 185 participants, 143 (77%) had worked on an SR within the last 5 years. The number of SRs conducted by participants during their careers ranged from 1 to 500, with a median of 5. The major component of time spent was on search strategy development and translation. Average aggregated time for standard tasks was 26.9 hours, with a median of 18.5 hours. Task time was unrelated to the number of SRs but was positively correlated with years of SR experience.

CONCLUSION

The time required to conduct the librarian's discrete tasks in an SR varies substantially, and there are no standard time frames. Librarians with more SR experience spent more time on instruction and interviews; time spent on all other tasks varied widely. Librarians also can expect to spend a significant amount of their time on search strategy development, translation, and writing.

Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms

OBJECTIVES

An overview of systematic reviews (OoSRs) is a study designed to synthesize multiple evidence from existing systematic reviews on a specific domain. The aim of this paper was to offer a pilot version checklist with Preferred Reporting Items for OoSRs (PRIO-harms) to promote a more balanced reporting of benefits and harms in OoSRs of health care interventions.

STUDY DESIGN AND SETTING

The included items were developed by combining key features from health care OoSRs designs with recommendations from statements of other relevant checklists and pertinent methodological review articles. Two raters independently used the PRIO-harms checklist to assess a sample of 20 OoSRs.

RESULTS

The PRIO-harms tool consists of a 27-item (56 (sub-)items in total) checklist and is accompanied by a five-stage process flow diagram (identification, screening, eligibility, inclusion, and separation of relevant studies). The mean interrater reliability (Gwet's AC1 statistic) between reviewers was 0.90 (95% confidence interval: 0.88, 0.92) indicating a very good agreement.

CONCLUSION

The PRIO-harms tool can be used in every OoSRs that addresses health care interventions. This instrument will assist overview authors to improve completeness and transparency of research reporting with emphasis on harms. However, it might benefit from critical review and further validation from experts and research teams that produce OoSRs.

Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines

Objective

Complete, accurate reporting of systematic reviews facilitates assessment of how well reviews have been conducted. The primary objective of this study was to examine compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for literature search reporting and to examine the completeness, bias, and reproducibility of the searches in these reviews from what was reported. The second objective was to examine reporting of the credentials and contributions of those involved in the search process.

Methods

A sample of systematic reviews or meta-analyses published in veterinary journals between 2011 and 2015 was obtained by searching PubMed. Reporting in the full text of each review was checked against certain PRISMA checklist items.

Results

Over one-third of reviews (37%) did not search the CAB Abstracts database, and 9% of reviews searched only 1 database. Over two-thirds of reviews (65%) did not report any search for grey literature or stated that they excluded grey literature. The majority of reviews (95%) did not report a reproducible search strategy.

Conclusions

Most reviews had significant deficiencies in reporting the search process that raise questions about how these searches were conducted and ultimately cast serious doubts on the validity and reliability of reviews based on a potentially biased and incomplete body of literature. These deficiencies also highlight the need for veterinary journal editors and publishers to be more rigorous in requiring adherence to PRISMA guidelines and to encourage veterinary researchers to include librarians or information specialists on systematic review teams to improve the quality and reporting of searches.

Impact of librarians on reporting of the literature searching component of pediatric systematic reviews

OBJECTIVE

A critical element in conducting a systematic review is the identification of studies. To date, very little empirical evidence has been reported on whether the presence of a librarian or information professional can contribute to the quality of the final product. The goal of this study was to compare the reporting rigor of the literature searching component of systematic reviews with and without the help of a librarian.

METHOD

Systematic reviews published from 2002 to 2011 in the twenty highest impact factor pediatrics journals were collected from MEDLINE. Corresponding authors were contacted via an email survey to determine if a librarian was involved, the role that the librarian played, and functions that the librarian performed. The reviews were scored independently by two reviewers using a fifteen-item checklist.

RESULTS

There were 186 reviews that met the inclusion criteria, and 44% of the authors indicated the involvement of a librarian in conducting the systematic review. With the presence of a librarian as coauthor or team member, the mean checklist score was 8.40, compared to 6.61 (p<0.001) for reviews without a librarian.

CONCLUSIONS

Findings indicate that having a librarian as a coauthor or team member correlates with a higher score in the literature searching component of systematic reviews.

Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study

Background

A high-quality search strategy is considered an essential component of systematic reviews but many do not contain reproducible search strategies. It is unclear if low reproducibility spans medical disciplines, is affected by librarian/search specialist involvement or has improved with increased awareness of reporting guidelines.

Objectives

To examine the reporting of search strategies in systematic reviews published in Pediatrics, Surgery or Cardiology journals in 2012 and determine rates and predictors of including a reproducible search strategy.

Methods

We identified all systematic reviews published in 2012 in the ten highest impact factor journals in Pediatrics, Surgery and Cardiology. Each search strategy was coded to indicate what elements were reported and whether the overall search was reproducible. Reporting and reproducibility rates were compared across disciplines and we measured the influence of librarian/search specialist involvement, discipline or endorsement of a reporting guideline on search reproducibility.

Results

272 articles from 25 journals were included. Reporting of search elements ranged widely from 91% of articles naming search terms to 33% providing a full search strategy and 22% indicating the date the search was executed. Only 22% of articles provided at least one reproducible search strategy and 13% provided a reproducible strategy for all databases searched in the article. Librarians or search specialists were reported as involved in 17% of articles. There were strong disciplinary differences on the reporting of search elements. In the multivariable analysis, only discipline (Pediatrics) was a significant predictor of the inclusion of a reproducible search strategy.

Conclusions

Despite recommendations to report full, reproducible search strategies, many articles still do not. In addition, authors often report a single strategy as covering all databases searched, further decreasing reproducibility. Further research is needed to determine how disciplinary culture may encourage reproducibility and the role that journal editors and peer reviewers could play.

Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors

Background

Previous research looking at published systematic reviews has shown that their search strategies are often suboptimal and that librarian involvement, though recommended, is low. Confidence in the results, however, is limited due to poor reporting of search strategies the published articles.

Objectives

To more accurately measure the use of recommended search methods in systematic reviews, the levels of librarian involvement, and whether librarian involvement predicts the use of recommended methods.

Methods

A survey was sent to all authors of English-language systematic reviews indexed in the Database of Abstracts of Reviews of Effects (DARE) from January 2012 through January 2014. The survey asked about their use of search methods recommended by the Institute of Medicine, Cochrane Collaboration, and the Agency for Healthcare Research and Quality and if and how a librarian was involved in the systematic review. Rates of use of recommended methods and librarian involvement were summarized. The impact of librarian involvement on use of recommended methods was examined using a multivariate logistic regression.

Results

1560 authors completed the survey. Use of recommended search methods ranged widely from 98% for use of keywords to 9% for registration in PROSPERO and were generally higher than in previous studies. 51% of studies involved a librarian, but only 64% acknowledge their assistance. Librarian involvement was significantly associated with the use of 65% of recommended search methods after controlling for other potential predictors. Odds ratios ranged from 1.36 (95% CI 1.06 to 1.75) for including multiple languages to 3.07 (95% CI 2.06 to 4.58) for using controlled vocabulary.

Conclusions

Use of recommended search strategies is higher than previously reported, but many methods are still under-utilized. Librarian involvement predicts the use of most methods, but their involvement is under-reported within the published article.

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews

OBJECTIVES

To determine whether librarian and information specialist authorship was associated with better reported systematic review (SR) search quality.

STUDY DESIGN AND SETTING

SRs from high-impact general internal medicine journals were reviewed for search quality characteristics and reporting quality by independent reviewers using three instruments, including a checklist of Institute of Medicine Recommended Standards for the Search Process and a scored modification of the Peer Review of Electronic Search Strategies instrument.

RESULTS

The level of librarian and information specialist participation was significantly associated with search reproducibility from reported search strategies (Χ(2) = 23.5; P < 0.0001). Librarian co-authored SRs had significantly higher odds of meeting 8 of 13 analyzed search standards than those with no librarian participation and six more than those with mentioned librarian participation. One-way ANOVA showed that differences in total search quality scores between all three groups were statistically significant (F2,267 = 10.1233; P < 0.0001).

CONCLUSION

Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.

Adverse events in cardiovascular-related training programs in people with spinal cord injury: a systematic review

CONTEXT

There are anecdotal reports of adverse events (AEs) associated with exercise in people with spinal cord injury (SCI) and consequent concern by people with SCI and their providers about potential risks of exercise. Enumeration of specific events has never been performed and the extent of risk of exercise to people with SCI is not understood.

OBJECTIVE

To systematically review published evidence to identify and enumerate reports of adverse events or AEs associated with training in persons with SCI.

METHODS

Review was limited to peer-reviewed studies published in English from 1970 to 2011: (1) in adults with SCI, (2) evaluating training protocols consisting of repeated sessions over at least 4 weeks to maintain or improve cardiovascular health, (3) including volitional exercise modalities and functional electrical stimulation (FES)-enhanced exercise modalities, and (4) including a specific statement about AEs. Trained reviewers initially identified a total of 145 studies. After further screening, 38 studies were included in the review. Quality of evidence was evaluated using established procedures.

RESULTS

There were no serious AEs reported. There were no common AEs reported across most types of interventions, except for musculoskeletal AEs related to FES walking. There were few AEs in volitional exercise studies.

CONCLUSION

There is no evidence to suggest that cardiovascular exercise done according to guidelines and established safety precautions is harmful. To improve the strength of these conclusions, future publications should include definition of AEs, information about pre-intervention screening, and statements of the nature and extent of AEs.

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000-2010): are the searches clearly explained, systematic and reproducible?

Systematic reviews are an essential tool for researchers, prevention providers and policy makers who want to remain current with the evidence in the field. Systematic review must adhere to strict standards, as the results can provide a more objective appraisal of evidence for making scientific decisions than traditional narrative reviews. An integral component of a systematic review is the development and execution of a comprehensive systematic search to collect available and relevant information. A number of reporting guidelines have been developed to ensure quality publications of systematic reviews. These guidelines provide the essential elements to include in the review process and report in the final publication for complete transparency. We identified the common elements of reporting guidelines and examined the reporting quality of search methods in HIV behavioral intervention literature. Consistent with the findings from previous evaluations of reporting search methods of systematic reviews in other fields, our review shows a lack of full and transparent reporting within systematic reviews even though a plethora of guidelines exist. This review underscores the need for promoting the completeness of and adherence to transparent systematic search reporting within systematic reviews.

Network meta-analyses could be improved by searching more sources and by involving a librarian

OBJECTIVE

Network meta-analyses (NMAs) aim to rank the benefits (or harms) of interventions, based on all available randomized controlled trials. Thus, the identification of relevant data is critical. We assessed the conduct of the literature searches in NMAs.

STUDY DESIGN

Published NMAs were retrieved by searching electronic bibliographic databases and other sources. Two independent reviewers selected studies and five trained reviewers abstracted data regarding literature searches, in duplicate. Search method details were examined using descriptive statistics.

RESULTS

Two hundred forty-nine NMAs were included. Eight used previous systematic reviews to identify primary studies without further searching, and five did not report any literature searches. In the 236 studies that used electronic databases to identify primary studies, the median number of databases was 3 (interquartile range: 3-5). MEDLINE, EMBASE, and Cochrane Central Register of Controlled Trials were the most commonly used databases. The most common supplemental search methods included reference lists of included studies (48%), reference lists of previous systematic reviews (40%), and clinical trial registries (32%). None of these supplemental methods was conducted in more than 50% of the NMAs.

CONCLUSION

Literature searches in NMAs could be improved by searching more sources, and by involving a librarian or information specialist.

Quality of reporting in systematic reviews of adverse events: systematic review

OBJECTIVES

To examine the quality of reporting of harms in systematic reviews, and to determine the need for a reporting guideline specific for reviews of harms.

DESIGN

Systematic review.

DATA SOURCES

Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE).

REVIEW METHODS

Databases were searched for systematic reviews having an adverse event as the main outcome, published from January 2008 to April 2011. Adverse events included an adverse reaction, harms, or complications associated with any healthcare intervention. Articles with a primary aim to investigate the complete safety profile of an intervention were also included. We developed a list of 37 items to measure the quality of reporting on harms in each review; data were collected as dichotomous outcomes ("yes" or "no" for each item).

RESULTS

Of 4644 reviews identified, 309 were systematic reviews or meta-analyses primarily assessing harms (13 from CDSR; 296 from DARE). Despite a short time interval, the comparison between the years of 2008 and 2010-11 showed no difference on the quality of reporting over time (P=0.079). Titles in fewer than half the reviews (proportion of reviews 0.46 (95% confidence interval 0.40 to 0.52)) did not mention any harm related terms. Almost one third of DARE reviews (0.26 (0.22 to 0.31)) did not clearly define the adverse events reviewed, nor did they specify the study designs selected for inclusion in their methods section. Almost half of reviews (n=170) did not consider patient risk factors or length of follow-up when reviewing harms of an intervention. Of 67 reviews of complications related to surgery or other procedures, only four (0.05 (0.01 to 0.14)) reported professional qualifications of the individuals involved. The overall, unweighted, proportion of reviews with good reporting was 0.56 (0.55 to 0.57); corresponding proportions were 0.55 (0.53 to 0.57) in 2008, 0.55 (0.54 to 0.57) in 2009, and 0.57 (0.55 to 0.58) in 2010-11.

CONCLUSION

Systematic reviews compound the poor reporting of harms data in primary studies by failing to report on harms or doing so inadequately. Improving reporting of adverse events in systematic reviews is an important step towards a balanced assessment of an intervention.

The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews

BACKGROUND

The usefulness of Google Scholar (GS) as a bibliographic database for biomedical systematic review (SR) searching is a subject of current interest and debate in research circles. Recent research has suggested GS might even be used alone in SR searching. This assertion is challenged here by testing whether GS can locate all studies included in 21 previously published SRs. Second, it examines the recall of GS, taking into account the maximum number of items that can be viewed, and tests whether more complete searches created by an information specialist will improve recall compared to the searches used in the 21 published SRs.

METHODS

The authors identified 21 biomedical SRs that had used GS and PubMed as information sources and reported their use of identical, reproducible search strategies in both databases. These search strategies were rerun in GS and PubMed, and analyzed as to their coverage and recall. Efforts were made to improve searches that underperformed in each database.

RESULTS

GS' overall coverage was higher than PubMed (98% versus 91%) and overall recall is higher in GS: 80% of the references included in the 21 SRs were returned by the original searches in GS versus 68% in PubMed. Only 72% of the included references could be used as they were listed among the first 1,000 hits (the maximum number shown). Practical precision (the number of included references retrieved in the first 1,000, divided by 1,000) was on average 1.9%, which is only slightly lower than in other published SRs. Improving searches with the lowest recall resulted in an increase in recall from 48% to 66% in GS and, in PubMed, from 60% to 85%.

CONCLUSIONS

Although its coverage and precision are acceptable, GS, because of its incomplete recall, should not be used as a single source in SR searching. A specialized, curated medical database such as PubMed provides experienced searchers with tools and functionality that help improve recall, and numerous options in order to optimize precision. Searches for SRs should be performed by experienced searchers creating searches that maximize recall for as many databases as deemed necessary by the search expert.

Methodological developments in searching for studies for systematic reviews: past, present and future?

The Cochrane Collaboration was established in 1993, following the opening of the UK Cochrane Centre in 1992, at a time when searching for studies for inclusion in systematic reviews was not well-developed. Review authors largely conducted their own searches or depended on medical librarians, who often possessed limited awareness and experience of systematic reviews. Guidance on the conduct and reporting of searches was limited. When work began to identify reports of randomized controlled trials (RCTs) for inclusion in Cochrane Reviews in 1992, there were only approximately 20,000 reports indexed as RCTs in MEDLINE and none indexed as RCTs in Embase. No search filters had been developed with the aim of identifying all RCTs in MEDLINE or other major databases. This presented The Cochrane Collaboration with a considerable challenge in identifying relevant studies.Over time, the number of studies indexed as RCTs in the major databases has grown considerably and the Cochrane Central Register of Controlled Trials (CENTRAL) has become the best single source of published controlled trials, with approximately 700,000 records, including records identified by the Collaboration from Embase and MEDLINE. Search filters for various study types, including systematic reviews and the Cochrane Highly Sensitive Search Strategies for RCTs, have been developed. There have been considerable advances in the evidence base for methodological aspects of information retrieval. The Cochrane Handbook for Systematic Reviews of Interventions now provides detailed guidance on the conduct and reporting of searches. Initiatives across The Cochrane Collaboration to improve the quality inter alia of information retrieval include: the recently introduced Methodological Expectations for Cochrane Intervention Reviews (MECIR) programme, which stipulates 'mandatory' and 'highly desirable' standards for various aspects of review conduct and reporting including searching, the development of Standard Training Materials for Cochrane Reviews and work on peer review of electronic search strategies. Almost all Cochrane Review Groups and some Cochrane Centres and Fields now have a Trials Search Co-ordinator responsible for study identification and medical librarians and other information specialists are increasingly experienced in searching for studies for systematic reviews.Prospective registration of clinical trials is increasing and searching trials registers is now mandatory for Cochrane Reviews, where relevant. Portals such as the WHO International Clinical Trials Registry Platform (ICTRP) are likely to become increasingly attractive, given concerns about the number of trials which may not be registered and/or published. The importance of access to information from regulatory and reimbursement agencies is likely to increase. Cross-database searching, gateways or portals and improved access to full-text databases will impact on how searches are conducted and reported, as will services such as Google Scholar, Scopus and Web of Science. Technologies such as textual analysis, semantic analysis, text mining and data linkage will have a major impact on the search process but efficient and effective updating of reviews may remain a challenge.In twenty years' time, we envisage that the impact of universal social networking, as well as national and international legislation, will mean that all trials involving humans will be registered at inception and detailed trial results will be routinely available to all. Challenges will remain, however, to ensure the discoverability of relevant information in diverse and often complex sources and the availability of metadata to provide the most efficient access to information. We envisage an ongoing role for information professionals as experts in identifying new resources, researching efficient ways to link or mine them for relevant data and managing their content for the efficient production of systematic reviews.

US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews

OBJECTIVES

A key systematic review (SR) methodology is comprehensive searching. The Drug Effectiveness Review Project (DERP) SRs search US Food and Drug Administration (FDA) documents to identify unpublished evidence. This study evaluates the success of those efforts.

STUDY DESIGN AND SETTING

We examined DERP reports published since 2003 for the use of FDA preapproval and postmarketing documents. We categorized evidence as (1) unique unpublished studies, (2) supplemental unpublished data, or (3) FDA postmarketing data analysis. Three reviewers independently assigned predetermined impact categories (e.g., qualitative or quantitative usage, fills gaps, confirms findings, and alters conclusions), resolving disagreements through consensus.

RESULTS

Among 114 DERP reports, 19% included unpublished studies and/or supplemental data and 10% included postmarketing analyses. From 175 preapproval documents, 14% provided eligible unpublished studies and 4.0% supplemental unpublished data that helped confirm previous findings, identify important harms, and fill gaps in knowledge about understudied subpopulations, outcomes, and comparisons. Report conclusion statements changed in 9 of 33 instances of premarketing documents compared with 4 of 12 postmarketing analyses.

CONCLUSIONS

The FDA documents can provide important unpublished evidence for SRs, although in a small proportion of cases. Future research should identify attributes that predict which reviews may benefit most from review of FDA documents.

Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

OBJECTIVE

An increasing amount of research and guidelines has been published on search methodology and the reporting of search strategies in systematic reviews. This research assessed whether this has lead to any improvements in the reporting and quality of searching in systematic reviews of adverse effects.

STUDY DESIGN AND SETTING

All records within Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were scanned for systematic reviews of adverse effects. Data were then extracted on the methods used for information retrieval in these reviews and a descriptive analysis conducted by publication year.

RESULTS

A total of 849 reviews published from 1994 to 2011 met the inclusion criteria. There has been a significant increase (P<0.001) in the number of adverse effects reviews per year from 1994 (n=5) to 2010 (n=104). Some improvements were apparent, such as an increase in the number of databases searched and fewer date and language restrictions applied. However, there has been an increase in reviews limited to data from randomized controlled trials, whereas the reporting of search strategies could still be improved further, with only 9% (74/849) of the reviews reporting reproducible searches.

CONCLUSION

Some improvements in searching systematic reviews of adverse effects are apparent; however, poor reporting of search strategies remains a great obstacle to readers.