An assessment of the quality of current clinical meta-analyses

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study

OBJECTIVES

To determine the reproducibility of biomedical systematic review search strategies.

STUDY DESIGN AND SETTING

A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.

RESULTS

The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.

CONCLUSION

Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Need for Training in Research Methodology Prior to Conducting Systematic Reviews and Meta-Analyses, and the Effectiveness of an Online Training Program: The Global Andrology Forum Model

PURPOSE

Systematic reviews and meta-analyses (SRMAs) are used to generate evidence-based guidelines. Although the number of SRMAs published in the literature has increased dramatically in the last decade, the training and the experience of the researchers performing the SRMAs are usually not explained in the SRMAs' methodology, and this may be a source of bias. Although some studies pointed out the need for quality control of SRMAs and training in proper statistical methods, to the best of our knowledge, no study has reported the importance of training the researchers that conduct the SRMAs. The aim of this study is to describe a training program designed to impart the essential knowledge and skills required for the conduct of an SRMA and to assess the need for, and outcome of, such a training.

MATERIALS AND METHODS

Researchers were trained for use of Scopus, study eligibility, assessment of the quality of evidence (QoE) through the Cambridge Quality Checklist for observational studies, as well as the Cochrane Risk of Bias tool, the Consolidated Standards of Reporting Trials (CONSORT) guidelines, and the Jadad score for randomized controlled trials (RCTs), Population, Intervention, Comparison, Outcome (PICO) questions and data extraction. A total of 35 of them were approved to join a planned SRMA. At the end of the SRMA, they were administered 43 multiple-choice questions (MCQs) on demographics, motivation for participation in the SRMA, self-perceived change in knowledge before and after conducting the SRMA, and self-assessment of performance. The senior researchers then revised the spreadsheet of the SRMA and, based on the mistakes found, organized a training focused on the correct assessment of the study design, where 43 researchers (9 joined midway) and 11 trainees with no experience in conducting SRMA attended. They all were tested through a 5 MCQ assessment that was administered before and after the training. Those scoring poorly were re-trained and re-tested, and only those scoring satisfactorily were admitted to further SRMAs.

RESULTS

Approximately 54.3% of the participants were medical doctors (MD), 31.4% were urologists and 48.6% had previous experience with SRMAs. Joining an international collaborative study was the main motivation, chosen by 19.7% of researchers. The results of the self-perceived change in knowledge showed a significant improvement in the use of Scopus, checklists for QoE, PICO questions, data required to perform a meta-analysis, and critical reading of scientific articles. Also, the majority of the researchers ranked the quality of their work as high. The pre-test results of the 5 MCQ showed a low score, which was not different from that achieved by a group of fresh trainees (median, 2; IQR 1-3 vs. median, 1; IQR, 1-2; p=0.3). Post-training there was significant improvement in both groups (researchers: median, 4; IQR, 3-5 vs. median, 2; IQR, 1-3; p<0.001; trainees: median, 4; IQR, 3-4 vs. median, 1; IQR, 1-2; p=0.02). Out of the 44 researchers, 12 (27.3%) scored poorly (≤3). After re-training, all of them scored satisfactorily (>3) and were admitted to subsequent SRMAs.

CONCLUSIONS

At the end of our model, 100% of researchers participating in this study were validated to be included in a meta-analysis. This validation required the involvement of the MT, two meetings, a self-evaluation survey, and one or two sets of objective tests with explanations and corrections. Our results indicate that even well-trained clinicians are naïve when it comes to the methodology of SRMA. All the researchers performing an SRMA need comprehensive training that must cover each aspect of the SRMA methodology. This paper provides a replicable training program that could be used by other investigators to train the researchers to perform high-quality SRMAs.

Supporting Consensus Statements: Considerations and Recommendations for a Systematic Review Service

Increasingly, a critical eye has been placed on the methodological quality of consensus statements. As expert systematic review (SR) methodologists, librarians are often called on to support consensus statement work. Using the Weill Cornell Medicine Samuel J. Wood Library's SR Service experience as a guide, the aim of this paper is to answer three main questions regarding librarians supporting consensus statement work: (1) What is a consensus statement? (2) What is consensus statement methodology and how does this compare to practice guidelines? (3) What are important and practical points to consider when supporting this kind of request?

Systematic reviews: Structure, form and content

This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the search. It also covers systematic review reporting standards such as PRISMA-P and PRISMA, critical appraisal and tools and resources to support the review and ensure it is conducted efficiently and effectively. Finally, it summarizes the requirements when screening search results for inclusion in the review, and the statistical synthesis of included studies' findings.

Methodological Considerations for Auditory Training Interventions for Adults With Hearing Loss: A Rapid Review

Purpose The aim of this study was to evaluate literature on auditory training published since 2013. Method A rapid review or a streamlined approach to systematically identify and summarize relevant studies was performed. Selected health sciences databases were searched using a search strategy developed with the PICO (population, intervention, comparison, and outcome) framework. Studies eligible for inclusion had older adult participants with hearing loss and utilized technology-based auditory training in laboratory or home settings. Results The study quality of most studies was found to be low to moderate, with concentrations between low and moderate. Major issues were related to study design and reporting, such as the need for blinding and a control group, larger sample sizes, and a follow-up for long-term outcomes of auditory training interventions. Wide variability in training approaches, participant backgrounds (e.g., audiograms, hearing aid use), and outcome measures are also noted. Conclusions Evidence on the effectiveness of auditory training is mixed. Future research should include high-quality randomized controlled trials with representative populations; follow-up periods to study long-term effects; and exploration of behavioral, electrophysiological, and self-reported outcome measures. Recommendations for study designs and methodologies are also discussed.

Systematic Reviews and Meta-Analyses in Cardiac Surgery: Rules of the Road - Part 1

The number of cardiac surgical meta-analyses and systematic reviews published in the last decades has constantly increased, paralleling the exponential growth observed in virtually all other medical fields. Meta-analyses are open to methodological flaws, however, if best practices are not strictly followed. Assessment of the appropriateness of the research question is a crucial first step. Once a protocol has been developed, this should be registered before the work is initiated. The cornerstone of any systematic review or meta-analysis is a rigorous, comprehensive, and most of all reproducible, search that follows a prespecified and clear strategy. Eligibility criteria must be discussed and agreed upon in advance to guide final study selection, which ultimately lays the foundation for subsequent data extraction. In case of missing or partially reported data, the authors of the original papers should be contacted. Adherence to rigorous methodological rules at each of these stages will warrant availability of good quality data for formal statistical analyses. The aim of the first part of this expert review is to discuss the limits and pitfalls of the meta-analytic approach and provide guidance on how to perform trial-level meta-analyses, with particular reference to the identification of an appropriate research question, the definition and registration of the protocol, the search strategy, the study selection, and the data abstraction.