Most published systematic reviews of remdesivir for COVID-19 were redundant and lacked currency

Assessing the coverage of PubMed, Embase, OpenAlex, and Semantic Scholar for automated single-database searches in living guideline evidence surveillance: a case study of the international polycystic ovary syndrome guidelines 2023

OBJECTIVES

Living guideline maintenance is underpinned by manual approaches toward evidence retrieval, limiting long-term sustainability. Our study aimed to evaluate the feasibility of using only PubMed, Embase, OpenAlex, or Semantic Scholar in automatically retrieving articles that were included in a high-quality international guideline - the 2023 international polycystic ovary syndrome (PCOS) guidelines.

METHODS

The digital object identifiers (DOIs) and PubMed ID (PMIDs) of articles included after full-text screening in the 2023 international PCOS guidelines were extracted. These IDs were used to automatically retrieve article metadata from all tested databases. A title-only search was then conducted on articles that were not initially retrievable. The extent of coverage, and overlap of coverage, was determined for each database. An exploratory analysis of the risk of bias (RoB) of articles that were unretrievable was then conducted for each database.

RESULTS

OpenAlex had the best coverage (98.6%), followed by Semantic Scholar (98.3%), Embase (96.8%), and PubMed (93.0%). However, 90.5% of all articles were retrievable from all four databases. All articles that were not retrievable from OpenAlex and Semantic Scholar were either assessed as medium or high RoB. In contrast, both Embase and PubMed missed articles that were of high quality (low RoB).

CONCLUSION

OpenAlex should be considered a single source for automated evidence retrieval in living guideline development, due to high coverage, and low risk of missing high-quality articles. These insights are being leveraged as part of transitioning the 2023 international PCOS guidelines toward a living format.

A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making

BACKGROUND

The COVID-19 pandemic has had an unprecedented impact in the global research production and has also increased research waste. Living evidence syntheses (LESs) seek to regularly update a body of evidence addressing a specific question. During the COVID-19 pandemic, the production and dissemination of LESs emerged as a cornerstone of the evidence infrastructure. This critical interpretive synthesis answers the questions: What constitutes an LES to support decision-making?; when should one be produced, updated, and discontinued?; and how should one be disseminated?

METHODS

Searches included the Cochrane Library, EMBASE (Ovid), Health Systems Evidence, MEDLINE (Ovid), PubMed, and Web of Science up to 23 April 2024 and included articles that provide any insights on addressing the compass questions on LESs. Articles were selected and appraised, and their insights extracted. An interpretive and iterative coding process was used to identify relevant thematic categories and create a conceptual framework.

RESULTS

Among the 16,630 non-duplicate records identified, 208 publications proved eligible. Most were non-empirical articles, followed by actual LESs. Approximately one in three articles were published in response to the COVID-19 pandemic. The conceptual framework addresses six thematic categories: (1) what is an LES; (2) what methodological approaches facilitate LESs production; (3) when to produce an LES; (4) when to update an LES; (5) how to make available the findings of an LES; and (6) when to discontinue LES updates.

CONCLUSION

LESs can play a critical role in reducing research waste and ensuring alignment with advisory and decision-making processes. This critical interpretive synthesis provides relevant insights on how to better organize the global evidence architecture to support their production.

TRIAL REGISTRATION

PROSPERO registration: CRD42021241875.

Learning health systems and evidence ecosystems: a perspective on the future of evidence-based medicine and evidence-based guideline development

Despite forming the cornerstone of modern clinical practice for decades, implementation of evidence-based medicine at scale remains a crucial challenge for health systems. As a result, there has been a growing need for conceptual models to better contextualise and pragmatize the use of evidence-based medicine, particularly in tandem with patient-centred care. In this commentary, we highlight the emergence of the learning health system as one such model and analyse its potential role in pragmatizing both evidence-based medicine and patient-centred care. We apply the learning health system lens to contextualise the key activity of evidence-based guideline development and implementation, and highlight how current inefficiencies and bottlenecks in the evidence synthesis phase of evidence-based guideline development threaten downstream adherence. Lastly, we introduce the evidence ecosystem as a complementary model to learning health systems, and propose how innovative developments from the evidence ecosystem may be integrated with learning health systems to better enable health impact at speed and scale.

A framework to characterise the reproducibility of meta-analysis results with its application to direct oral anticoagulants in the acute treatment of venous thromboembolism

The number of meta-analyses of aggregate data has dramatically increased due to the facility of obtaining data from publications and the development of free, easy-to-use, and specialised statistical software. Even when meta-analyses include the same studies, their results may vary owing to different methodological choices. Assessment of the replication of meta-analysis provides an example of the variation of effect 'naturally' observed between multiple research projects. Reproducibility of results has mostly been reported using graphical descriptive representations. A quantitative analysis of such results would enable (i) breakdown of the total observed variability with quantification of the variability generated by the replication process and (ii) identification of which variables account for this variability, such as methodological quality or the statistical analysis procedures used. These variables might explain systematic mean differences between results and dispersion of the results. To quantitatively characterise the reproducibility of meta-analysis results, a bivariate linear mixed-effects model was developed to simulate both mean results and their corresponding uncertainty. Results were assigned to several replication groups, those assessing the same studies, outcomes, treatment indication and comparisons classified in the same replication group. A nested random effect structure was used to break down the total variability within each replication group and between these groups to enable calculation of an intragroup correlation coefficient and quantification of reproducibility. Determinants of variability were investigated by modelling both mean and variance parameters using covariates. The proposed model was applied to the example of meta-analyses evaluating direct oral anticoagulants in the acute treatment of venous thromboembolism.

Evidence surveillance for a living clinical guideline: Case study of the Australian stroke guidelines

BACKGROUND

Continual evidence surveillance is an integral feature of living guidelines. The Australian Stroke Guidelines include recommendations on 100 clinical topics and have been 'living' since 2018.

OBJECTIVES

To describe the approach for establishing and evaluating an evidence surveillance system for the living Australian Stroke Guidelines.

METHODS

We developed a pragmatic surveillance system based on an analysis of the searches for the 2017 Stroke Guidelines and evaluated its reliability by assessing the potential impact on guideline recommendations. Search retrieval and screening workload are monitored monthly, together with the frequency of changes to the guideline recommendations.

RESULTS

Evidence surveillance was guided by practical considerations of efficiency and sustainability. A single PubMed search covering all guideline topics, limited to systematic reviews and randomised trials, is run monthly. The search retrieves about 400 records a month of which a sixth are triaged to the guideline panels for further consideration. Evaluations with Epistemonikos and the Cochrane Stroke Trials Register demonstrated the robustness of adopting this more restrictive approach. Collaborating with the guideline team in designing, implementing and evaluating the surveillance is essential for optimising the approach.

CONCLUSION

Monthly evidence surveillance for a large living guideline is feasible and sustainable when applying a pragmatic approach.

Replication of systematic reviews: is it to the benefit or detriment of methodological quality?

OBJECTIVES

To perform an overview of the overlap of systematic reviews (SRs) assessing direct oral anticoagulants and characterize these reviews in terms of bias and methodological quality (PROSPERO: CRD42022316273).

STUDY DESIGN AND SETTING

A PubMed-indexed search was performed from inception to January 31, 2022 to identify SRs evaluating direct oral anticoagulants in patients treated for an acute venous thromboembolism. The risk of bias of these SRs was assessed according to the Risk Of Bias In Systematic reviews tool. Redundancy was defined as overlap in terms of the type of population considered, the interventions compared, and the studies included.

RESULTS

A total of 144 SRs were evaluated, of which 26 (18.1%) were classified as original, 87 (60.4%) as conceptual replications, and 31 (21.5%) as excessive replications. The risk of bias was high in 19 (73.1%) of the original SRs, 65 (74.7%) of the conceptual replications, and 21 (67.7%) of the excessive replications. Compared to the original SRs, the overall methodological quality was not improved in either conceptual or excessive replications.

CONCLUSION

A large number of SRs was classified as replications; a fifth constituted excessive replications. The replications showed no improvement in overall methodological quality compared to the original SRs.

Pragmatic and open science-based solution to a current problem in the reporting of living systematic reviews

Living systematic reviews (LSRs) are an increasingly common approach to keeping reviews up to date, in which new relevant studies are incorporated as they become available, so as to inform healthcare policy and practice in a timely manner. While journal publishers have been exploring the publication of LSRs using different updating and publishing approaches, readers cannot currently assess if the evidence underpinning a published LSR is up to date, as neither the search details, the selection process, nor the list of identified studies is made available between the publication of updates. We describe a new method to transparently report the living evidence surveillance process that occurs between published LSR versions. We use the example of the living Cochrane Review on nirmatrelvir combined with ritonavir (Paxlovid) for preventing and treating COVID-19 to illustrate how this can work in practice. We created a publicly accessible spreadsheet on the Open Science Framework platform, linking to the living Cochrane Review, that details the search and study selection process, enabling readers to track the progress of eligible ongoing or completed studies. Further automation of the evidence surveillance process should be explored.

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study

BACKGROUND

COVID-19 led to a rapid acceleration in the number of systematic reviews. Readers need to know how up to date evidence is when selecting reviews to inform decisions. This cross-sectional study aimed to evaluate how easily the currency of COVID-19 systematic reviews published early in the pandemic could be determined and how up to date these reviews were at the time of publication.

METHODS

We searched for systematic reviews and meta-analyses relevant to COVID-19 added to PubMed in July 2020 and January 2021, including any that were first published as preprints. We extracted data on the date of search, number of included studies, and date first published online. For the search date, we noted the format of the date and where in the review this was reported. A sample of non-COVID-19 systematic reviews from November 2020 served as a comparator.

RESULTS

We identified 246 systematic reviews on COVID-19. In the abstract of these reviews, just over half (57%) reported the search date (day/month/year or month/year) while 43% failed to report any date. When the full text was considered, the search date was missing from 6% of reviews. The median time from last search to publication online was 91 days (IQR 63-130). Time from search to publication was similar for the subset of 15 rapid or living reviews (92 days) but shorter for the 29 reviews published as preprints (37 days). The median number of studies or publications included per review was 23 (IQR 12-40). In the sample of 290 non-COVID SRs, around two-thirds (65%) reported the search date while a third (34%) did not include any date in the abstract. The median time from search to publication online was 253 days (IQR 153-381) and each review included a median of 12 studies (IQR 8-21).

CONCLUSIONS

Despite the context of the pandemic and the need to easily ascertain the currency of systematic reviews, reporting of the search date information for COVID-19 reviews was inadequate. Adherence to reporting guidelines would improve the transparency and usefulness of systematic reviews to users.

Definition, harms, and prevention of redundant systematic reviews

BACKGROUND

Along with other types of research, it has been stated that the extent of redundancy in systematic reviews has reached epidemic proportions. However, it was also emphasized that not all duplication is bad, that replication in research is essential, and that it can help discover unfortunate behaviors of scientists. Thus, the question is how to define a redundant systematic review, the harmful consequences of such reviews, and what we could do to prevent the unnecessary amount of this redundancy.

MAIN BODY

There is no consensus definition of a redundant systematic review. Also, it needs to be defined what amount of overlap between systematic reviews is acceptable and not considered a redundancy. One needs to be aware that it is possible that the authors did not intend to create a redundant systematic review. A new review on an existing topic, which is not an update, is likely justified only when it can be shown that the previous review was inadequate, for example, due to suboptimal methodology. Redundant meta-analyses could have scientific, ethical, and economic questions for researchers and publishers, and thus, they should be avoided, if possible. Potential solutions for preventing redundant reviews include the following: (1) mandatory prospective registration of systematic reviews; (2) editors and peer reviewers rejecting duplicate/redundant and inadequate reviews; (3) modifying the reporting checklists for systematic reviews; (4) developing methods for evidence-based research (EBR) monitoring; (5) defining systematic reviews; (6) defining the conclusiveness of systematic reviews; (7) exploring interventions for the adoption of methodological advances; (8) killing off zombie reviews (i.e., abandoned registered reviews); (9) better prevention of duplicate reviews at the point of registration; (10) developing living systematic reviews; and (11) education of researchers.

CONCLUSIONS

Disproportionate redundancy of the same or very similar systematic reviews can lead to scientific, ethical, economic, and societal harms. While it is not realistic to expect that the creation of redundant systematic reviews can be completely prevented, some preventive measures could be tested and implemented to try to reduce the problem. Further methodological research and development in this field will be welcome.

From standard systematic reviews to living systematic reviews

Systematic reviews (SRs) have become a central tool for evidence-based health care over the last 30 years. The number of SRs being published has increased steadily. However, concerns have been raised regarding the duplication of work, methodological flaws and the currency of many systematic reviews, also in the context of the COVID-19 pandemic. Living systematic reviews (LSRs) offer a new approach to updating systematic reviews, particularly in high-priority research fields that face the challenge of dynamically evolving and sometimes uncertain evidence. Continual updates serve to ensure that LSRs remain current and methodologically rigorous. As a new element of the evidence ecosystem, LSRs can inform living guidelines and recommendations, user-adapted formats, decisions at the patient and system level as well as gaps in primary research.

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study

BACKGROUND

Machine learning (ML) tools exist that can reduce or replace human activities in repetitive or complex tasks. Yet, ML is underutilized within evidence synthesis, despite the steadily growing rate of primary study publication and the need to periodically update reviews to reflect new evidence. Underutilization may be partially explained by a paucity of evidence on how ML tools can reduce resource use and time-to-completion of reviews.

METHODS

This protocol describes how we will answer two research questions using a retrospective study design: Is there a difference in resources used to produce reviews using recommended ML versus not using ML, and is there a difference in time-to-completion? We will also compare recommended ML use to non-recommended ML use that merely adds ML use to existing procedures. We will retrospectively include all reviews conducted at our institute from 1 August 2020, corresponding to the commission of the first review in our institute that used ML.

CONCLUSION

The results of this study will allow us to quantitatively estimate the effect of ML adoption on resource use and time-to-completion, providing our organization and others with better information to make high-level organizational decisions about ML.

Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

OBJECTIVES

The aim of this study is to describe (1) registered and (2) published systematic reviews (SRs) on COVID-19 treatments, and to analyze (3) the proportion of publications among registered SRs and (4) the proportion of registrations among published SRs.

STUDY DESIGN AND SETTING

This meta-research study (CRD42021240423) is part of CEOsys (http://www.covid-evidenz.de/). Two reviewers identified protocols in PROSPERO (registered January 2020 to September 2020) and SRs published as preprint or peer-reviewed article in L·OVE (Living OVerview of the Evidence) COVID-19 (by May 2021). SRs of all types assessing COVID-19 treatments in humans were included.

RESULTS

We included 239 PROSPERO protocols and 346 SRs published in L·OVE. In both samples, the affiliation of the corresponding author with an Asian institution, standard SR as review type, and meta-analysis as synthesis method were the most frequent characteristics. Living SRs made up ≤10%. A total of 71 of 239 (29.7%) PROSPERO protocols were published as SR by February 2022, that is, after at least 17 months of follow-up (25 of 71 as preprints, 35.2%). In L·OVE, 261 of 346 (75.4%) SRs published by May 2021 were not registered in PROSPERO.

CONCLUSION

Overall, one-third PROSPERO protocols were published and three-fourth published SRs were not registered. We strongly encourage authors to register and publish their SRs promptly to reduce research waste and to allocate resources efficiently during the pandemic and beyond.

Extent and nature of duplication in PROSPERO using COVID-19-related registrations: a retrospective investigation and survey

OBJECTIVES

During COVID-19, the International Prospective Register of Systematic Reviews (PROSPERO) experienced a surge in registrations for COVID-19-related systematic reviews, and duplication of research questions became apparent. Duplication can waste funding, time and research effort and make policy making more difficult.This project explored the extent of and reasons for duplication of COVID-19-related systematic review registrations in PROSPERO during the pandemic.

DESIGN

Retrospective analysis of COVID-19-related registrations in PROSPERO, and a qualitative survey.

SETTING

PROSPERO was searched for registrations related to four COVID-19 research areas: epidemiology, rehabilitation, transmission and treatments.

METHODS

Records identified were compared using Population, Intervention/Exposure, Comparator, Outcome, Study Design (PICOS) elements of PROSPERO registration forms. Registrations with similar or identical PICOS were evaluated further as 'duplicates'.Authors of 'duplicate' registrations were invited to complete a survey asking whether they searched PROSPERO prior to registration, identified similar reviews and, if so, why they continued with their review.

RESULTS

1054 COVID-19 reviews were registered between March 2020 and January 2021, of which 138 were submitted when at least one similar protocol was already registered in PROSPERO. Duplication was greatest in reviews of COVID-19 treatments; for example, there were 14 similar reviews evaluating the efficacy of hydroxychloroquine.From 138 authors invited to take part in the survey, we received 41 responses. Most respondents said that they identified similar reviews when they searched PROSPERO prior to registration. Main reasons given for 'duplication' were differences in PICOS or planned analyses (n=13), poor quality of previous registrations (n=2) and the need to update evidence (n=3).

CONCLUSIONS

This research highlights that registration of similar and duplicate systematic reviews related to COVID-19 in PROSPERO occurred frequently. Awareness of research waste is required, and initial checking for similar reviews should be embedded within good review practice.

Lessons from SARS‑CoV‑2 and its variants (Review)

COVID-19 has swept through mainland China by human-to-human transmission. The rapid spread of SARS-CoV-2 and its variants, including the currently prevalent Omicron strain, pose a serious threat worldwide. The present review summarizes epidemiological investigation and etiological analysis of genomic, epidemiological, and pathological characteristics of the original strain and its variants, as well as progress in diagnosis and treatment. Prevention and control measures used during the current Omicron pandemic are discussed to provide further knowledge of SARS-CoV-2.