"One more time": why replicating some syntheses of evidence relevant to COVID-19 makes sense

Extent and nature of duplication in PROSPERO using COVID-19-related registrations: a retrospective investigation and survey

OBJECTIVES

During COVID-19, the International Prospective Register of Systematic Reviews (PROSPERO) experienced a surge in registrations for COVID-19-related systematic reviews, and duplication of research questions became apparent. Duplication can waste funding, time and research effort and make policy making more difficult.This project explored the extent of and reasons for duplication of COVID-19-related systematic review registrations in PROSPERO during the pandemic.

DESIGN

Retrospective analysis of COVID-19-related registrations in PROSPERO, and a qualitative survey.

SETTING

PROSPERO was searched for registrations related to four COVID-19 research areas: epidemiology, rehabilitation, transmission and treatments.

METHODS

Records identified were compared using Population, Intervention/Exposure, Comparator, Outcome, Study Design (PICOS) elements of PROSPERO registration forms. Registrations with similar or identical PICOS were evaluated further as 'duplicates'.Authors of 'duplicate' registrations were invited to complete a survey asking whether they searched PROSPERO prior to registration, identified similar reviews and, if so, why they continued with their review.

RESULTS

1054 COVID-19 reviews were registered between March 2020 and January 2021, of which 138 were submitted when at least one similar protocol was already registered in PROSPERO. Duplication was greatest in reviews of COVID-19 treatments; for example, there were 14 similar reviews evaluating the efficacy of hydroxychloroquine.From 138 authors invited to take part in the survey, we received 41 responses. Most respondents said that they identified similar reviews when they searched PROSPERO prior to registration. Main reasons given for 'duplication' were differences in PICOS or planned analyses (n=13), poor quality of previous registrations (n=2) and the need to update evidence (n=3).

CONCLUSIONS

This research highlights that registration of similar and duplicate systematic reviews related to COVID-19 in PROSPERO occurred frequently. Awareness of research waste is required, and initial checking for similar reviews should be embedded within good review practice.

Systematic reviews of convalescent plasma in COVID-19 continue to be poorly conducted and reported: a systematic review

OBJECTIVES

To suggest possible approaches to combatting the impact of the COVID-19 infodemic to prevent research waste in future health emergencies and in everyday research and practice.

STUDY DESIGN AND SETTING

Systematic review. The Epistemonikos database was searched in June 2021 for systematic reviews on the effectiveness of convalescent plasma for COVID-19. Two reviewers independently screened the retrieved references with disagreements resolved by discussion. Data extraction was completed by one reviewer with a proportion checked by a second. We used the Assessment of Multiple Systematic Reviews to assess the quality of conduct and reporting of included reviews.

RESULTS

Fifty one systematic reviews are included with 193 individual studies included within the systematic reviews. There was considerable duplication of effort; multiple reviews were conducted at the same time with inconsistencies in the evidence included. The reviews were of low methodological quality, poorly reported, and did not adhere to preferred reporting items for systematic reviews and meta-analysis guidance.

CONCLUSION

Researchers need to conduct, appraise, interpret, and disseminate systematic reviews better. All in the research community (researchers, peer-reviewers, journal editors, funders, decision makers, clinicians, journalists, and the public) need to work together to facilitate the conduct of robust systematic reviews that are published and communicated in a timely manner, reducing research duplication and waste, increasing transparency and accessibility of all systematic reviews.

Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency

Objective

To investigate the completeness and currency of published systematic reviews of remdesivir for COVID-19 and to compare this with a living guidelines approach.

Study Design and Setting

In this cross-sectional study, we searched Europe PMC on May 20, 2021 for systematic reviews of remdesivir (including preprints, living review updates). Completeness and currency were based on the inclusion of four major randomized trials of remdesivir available at the time of publication of the review (including as preliminary results and preprints).

Results

We included 38 reviews (45 reports), equivalent to a new publication every 9 days. 23 (51%) reports were out of date at the time of publication. Eleven reviews that were current on publication had a median survival time of 10 days (range 4–57). A third of reviews cited other systematic reviews, but only four provided justifications for why another review was necessary. Eight (21%) of the reviews were registered in PROSPERO. The Australian COVID-19 Clinical Evidence Taskforce living guidelines were updated within 14 days for three of the remdesivir trials, and within 28 days for the fourth.

Conclusion

There was considerable duplication of systematic reviews of remdesivir, and half were already out of date at the time of publication.

Methodological assessment of systematic reviews and meta-analyses on COVID-19: A meta-epidemiological study

RATIONALE, AIMS, AND OBJECTIVES

COVID-19 has caused an ongoing public health crisis. Many systematic reviews and meta-analyses have been performed to synthesize evidence for better understanding this new disease. However, some concerns have been raised about rapid COVID-19 research. This meta-epidemiological study aims to methodologically assess the current systematic reviews and meta-analyses on COVID-19.

METHODS

We searched in various databases for systematic reviews with meta-analyses published between 1 January 2020 and 31 October 2020. We extracted their basic characteristics, data analyses, evidence appraisal, and assessment of publication bias and heterogeneity.

RESULTS

We identified 295 systematic reviews on COVID-19. The median time from submission to acceptance was 33 days. Among these systematic reviews, 73.9% evaluated clinical manifestations or comorbidities of COVID-19. Stata was the most used software programme (43.39%). The odds ratio was the most used effect measure (34.24%). Moreover, 28.14% of the systematic reviews did not present evidence appraisal. Among those reporting the risk of bias results, 14.64% of studies had a high risk of bias. Egger's test was the most used method for assessing publication bias (38.31%), while 38.66% of the systematic reviews did not assess publication bias. The I2 statistic was widely used for assessing heterogeneity (92.20%); many meta-analyses had high values of I2 . Among the meta-analyses using the random-effects model, 75.82% did not report the methods for model implementation; among those meta-analyses reporting implementation methods, the DerSimonian-Laird method was the most used one.

CONCLUSIONS

The current systematic reviews and meta-analyses on COVID-19 might suffer from low transparency, high heterogeneity, and suboptimal statistical methods. It is recommended that future systematic reviews on COVID-19 strictly follow well-developed guidelines. Sensitivity analyses may be performed to examine how the synthesized evidence might depend on different methods for appraising evidence, assessing publication bias, and implementing meta-analysis models.

The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

BACKGROUND

Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision-making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions.

METHODS

The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by 1 team each and 2 reviews will be replicated by 15 teams (Study 4).

DISCUSSION

The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.

The mass production of systematic reviews about COVID-19: An analysis of PROSPERO records

OBJECTIVE

This study aimed to assess the characteristics of different designs of systematic reviews (SRs) registered in the International Prospective Register of Systematic Reviews (PROSPERO) about COVID-19.

METHODS

The search was performed in the PROSPERO database using the strategy proposed by the database and considered only human studies. The last date of the search was April 27, 2020. Full text of all records was accessed, and data were extracted by a single researcher, which was further double-checked by another researcher. A descriptive analysis was performed considering record characteristics using tables.

RESULTS

We included 564 records from which the vast majority were registered as SRs (n = 513, 91%). In general, we found poor reporting and missing or confusing information, since 84% of the records (n = 474) did not report the full search that would be adopted, 16% (n = 90) did not report clearly the databases that would be used, and 49.1% (n = 277) did not report the number of primary outcomes. The main focus of most of the records involved clinical, epidemiological, complication, and laboratory characteristics (n = 173, 30.7%) or the treatment of COVID-19 (n = 138, 24.5%).

CONCLUSION

A large number of SRs about COVID-19 have been conducted, and many of the assessed records were poorly reported and would be difficult to replicate. Besides, collected data points to an epidemic of redundant reviews on COVID-19.

Assessment of Duplicate Evidence in Systematic Reviews of Imaging Findings of Children With COVID-19

This cross-sectional study maps a coronavirus research question to illustrate the overlap and shortcomings of the evidence syntheses in this area.

Developing Guidelines Before, During, and After the COVID-19 Pandemic

These authors discuss several challenges and solutions for developers of COVID-19–related guidelines and recommendations.

Rapid Scoping Review of Laparoscopic Surgery Guidelines During the COVID-19 Pandemic and Appraisal Using a Simple Quality Appraisal Tool "EMERGE"

The theoretical danger of virus transmission during laparoscopic surgery (LS) via surgical smoke and laparoscopy gas has led to the formulation of many guidelines during the COVID-19 pandemic. This rapid scoping review of these guidelines was done to assess the quality of their evidence and appraise them for their impact on surgical services from the global south. A simple quality appraisal tool was constructed which can be used to evaluate rapidly emerging guidelines for evidence as well as for the needs of the global south. This rapid scoping review was conducted according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews protocol. Electronic databases were searched with predefined strategy and retrieved papers were screened according to relevant criteria. A simple objective tool to assess the quality of rapidly emerging guidelines including evidence, methodology, ease, resource optimization, geography, and the economy was constructed. Twenty studies met the inclusion criteria. None of the guidelines qualified to be evidence-based clinical practice guidelines as the level of evidence was uniformly rated “low”. A newly constructed tool showed good validation, reliability, and internal consistency. This rapid scoping review found two major research gaps: lack of systematic review of evidence during their development and insufficient weightage of their impact on surgical services from the global south. These significant issues were addressed by constructing a simple and more representative tool for evaluating rapidly emerging guidelines which also gives the rightful importance of their impact on surgical services from the global south.