Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

Research culture influences in health and biomedical research: rapid scoping review and content analysis

BACKGROUND

Research culture is strongly influenced by academic incentives and pressures such as the imperative to publish in academic journals, and can influence the nature and quality of the evidence we produce.

OBJECTIVE

The purpose of this rapid scoping review is to capture the breadth of differential pressures and contributors to current research culture, drawing together content from empirical research specific to the health and biomedical sciences.

STUDY DESIGN AND SETTING

PubMed and Web of Science were searched for empirical studies of influences and impacts on health and biomedical research culture, published between January 2012 and April 2024. Data charting extracted the key findings and relationships in research culture from included papers such as workforce composition; equitable access to research; academic journal trends, incentives, and reproducibility; erroneous research; questionable research practices; biases vested interests; and misconduct. A diverse author network was consulted to ensure content validity of the proposed framework of i) inclusivity, ii) transparency, iii) rigor, and iv) objectivity.

RESULTS

A growing field of studies examining research culture exists ranging from the inclusivity of the scientific workforce, the transparency of the data generated, the rigor of the methods used and the objectivity of the researchers involved. Figurative diagrams are presented to storyboard the links between research culture content and findings.

CONCLUSION

The wide range of research culture influences in the recent literature indicates the need for coordinated and sustained research culture conversations. Core principles in effective research environments should include inclusive collaboration and diverse research workforces, rigorous methodological approaches, transparency, data sharing, and reflection on scientific objectivity.

Yet another problem with systematic reviews: a living review update

BACKGROUND

In February 2023, the Journal of Clinical Epidemiology published 'The Problems with Systematic Reviews: A Living Systematic Review.' In updating this living review for the first time a new problem and several themes relating to research culture have emerged.

METHODS

Literature searches were rerun to identify articles published or indexed between May 2022 and May 2023. Thematic analysis coded articles and problems across four domains of systematic review conduct (1. comprehensive, 2. rigour, 3. transparent, 4. objective).

RESULTS

One hundred fifty-two newly included articles bring the total number of relevant articles to 637. A new problem (the lack of gender diversity of systematic review author teams) brings the total number of problems with systematic reviews up to 68. This update also reveals emerging themes such as: fast science from systematic reviews on COVID-19; the failure of citation of methodological or reporting guidelines to predict high-quality methodological or reporting quality; and the influence of vested interests on systematic review conclusions. These findings coupled with a proliferation of research waste from "me-too" meta-research articles highlighting well-established problems in systematic reviews underscores the need for reforms in research culture to address the incentives for producing and publishing research papers. This update also reports where the identified flaws in systematic reviews affect their conclusions drawing on 77 meta-epidemiological studies from the total 637 included articles. These meta-meta-analytic studies begin the important work of examining which problems threaten the reliability and validity of treatment effects or conclusions derived from systematic reviews.

CONCLUSION

This living review has captured an emerging theme in the published literature relating to the composition of the review author team and highlights a potential effect on the equity reporting of the systematic reviews. We recommend that meta-research endeavors evolve from merely documenting well-established issues to understanding lesser-known problems or consequences to systematic reviews.

Several methods for assessing research waste in reviews with a systematic search: a scoping review

Background

Research waste is present in all study designs and can have significant consequences for science, including reducing the reliability of research findings and contributing to the inefficient use of resources. Estimates suggest that as much as 85% of all biomedical research is wasted. However, it is uncertain how avoidable research waste is assessed in specific types of study designs and what methods could be used to examine different aspects of research waste. We aimed to investigate which methods, systematic reviews, scoping reviews, and overviews of reviews discussing research waste, have used to assess avoidable research waste.

Materials and Methods

We published a protocol in the Open Science Framework prospectively (https://osf.io/2fbp4). We searched PubMed and Embase with a 30-year limit (January 1993-August 2023). The concept examined was how research waste and related synonyms (e.g., unnecessary, redundant, duplicate, etc.) were assessed in reviews with a systematic search: systematic, scoping, or overviews of reviews. We extracted data on the method used in the review to examine for research waste and for which study design this method was applied.

Results

The search identified 4,285 records of which 93 reviews with systematic searches were included. The reviews examined a median of 90 (range 10-6,781) studies, where the study designs most commonly included were randomized controlled trials (48%) and systematic reviews (33%). In the last ten years, the number of reports assessing research waste has increased. More than 50% of examined reviews reported evaluating methodological research waste among included studies, typically using tools such as one of Cochrane Risk of Bias tools (n = 8) for randomized controlled trials or AMSTAR 1 or 2 (n = 12) for systematic reviews. One fourth of reviews assessed reporting guideline adherence to e.g., CONSORT (n = 4) for randomized controlled trials or PRISMA (n = 6) for systematic reviews.

Conclusion

Reviews with systematic searches focus on methodological quality and reporting guideline adherence when examining research waste. However, this scoping review revealed that a wide range of tools are used, which may pose difficulties in comparing examinations and performing meta-research. This review aids researchers in selecting methodologies and contributes to the ongoing discourse on optimizing research efficiency.

Author instructions in biomedical journals infrequently address systematic review reporting and methodology: a cross-sectional study

OBJECTIVES

We aimed to analyze how instructions for authors in journals indexed in MEDLINE address systematic review (SR) reporting and methodology.

STUDY DESIGN AND SETTING

We analyzed instructions for authors in 20% of MEDLINE-indexed journals listed in the online catalog of the National Library of Medicine on July 27, 2021. We extracted data only from the instructions published in English. We extracted data on the existence of instructions for reporting and methodology of SRs.

RESULTS

Instructions from 1,237 journals mentioned SRs in 45% (n = 560) of the cases. Systematic review (SR) registration was mentioned in 104/1,237 (8%) of instructions. Guidelines for reporting SR protocols were found in 155/1,237 (13%) of instructions. Guidelines for reporting SRs were explicitly mentioned in 461/1,237 (37%), whereas the EQUATOR (Enhancing the Quality and Transparency of Health Research) network was referred to in 474/1,237 (38%) of instructions. Less than 2% (n = 20) of instructions mentioned risk of bias and meta-analyses; less than 1% mentioned certainty of evidence assessment, methodological expectations, updating of SRs, overviews of SRs, or scoping reviews.

CONCLUSION

Journals indexed in MEDLINE rarely provide instructions for authors regarding SR reporting and methodology. Such instructions could potentially raise authors' awareness and improve how SRs are prepared and reported.

Clinical and methodological implications for research elements in systematic reviews on COVID-19 treatment were often unstructured and under-reported: a metaresearch study

OBJECTIVES

Numerous systematic reviews (SRs) have been published in the first months of the COVID-19 pandemic and clinical trials were designed rapidly highlighting the importance of informative implications for research (IfRs) sections in SRs. IfR is one item of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 checklist and the Cochrane Handbook suggests considering population, intervention, control, outcome (PICO) and Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) domains when developing IfR. We aimed (1) to assess whether SRs on COVID-19 treatments included any IfR statements and, for SRs with an IfR statement, (2) to examine which elements informed the IfR statement.

STUDY DESIGN AND SETTING

We conducted a metaresearch study based on SRs on COVID-19 treatment identified in the Living OVerview of the Evidence COVID-19 database in May 2021 as part of another research project (CRD42021240423). We defined an IfR statement as at least one sentence that contained at least one bit of information that could be informative for planning future research. We extracted any IfR statements anywhere in the SRs on predefined IfR variables, in particular PICO elements, study design, and concepts underlying GRADE domains. Three authors extracted data independently after piloting the data extraction form. We resolved discrepancies in weekly discussions to ensure a high-quality data extraction.

RESULTS

We included 326 SRs, of which 284 SRs (87.1%) stated IfR. Of these 284 SRs, 201 (70.8%) reported using Preferred Reporting Items for Systematic Reviews and Meta-Analyses and 66 (23.2%) using GRADE. IfR statements (n = 284) addressing PICO were unstructured and commonly reported 'population' (n = 195, 68.7%), 'intervention' (n = 242, 85.2%), and 'outcome' (n = 127, 44.7%) but not 'control' (n = 29, 10.2%). Concepts underlying GRADE domains were infrequently reported in IfR statements of SRs (n = 284): 'risk of bias' (n = 14, 4.9%), 'imprecision' (n = 8, 2.8%), 'inconsistency' (n = 7, 2.5%), 'publication bias' (n = 3, 1.1%), and 'indirectness' (n = 1, 0.4%). Additional IfR elements mentioned in IfR were 'better reporting' of future studies (n = 17, 6.0%) and 'standardization of procedures in clinical trials' (n = 12, 4.2%).

CONCLUSION

Almost 90% of SRs on COVID-19 treatments reported IfR. IfR statements addressing PICO were unstructured across SRs and concepts underlying GRADE were rarely reported to inform IfR. Further work is needed to assess generalizability beyond COVID-19 and to define more precisely which IfR elements should be considered, and how they should be reported in SRs of interventions. Until then, considering PICO elements and concepts underlying GRADE to derive IfR seems to be a sensible starting point.

Evaluation of 'implications for research' statements in systematic reviews of interventions in advanced cancer patients - a meta-research study

BACKGROUND

Implications for research (IfR) sections are an important part of systematic reviews (SRs) to inform health care researchers and policy makers. PRISMA 2020 recommends reporting IfR, while Cochrane Reviews require a separate chapter on IfR. However, it is unclear to what extent SRs discuss IfR. We aimed i) to assess whether SRs include an IfR statement and ii) to evaluate which elements informed IfR statements.

METHODS

We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). As suggested in the Cochrane Handbook, we assessed if the following predefined variables were referred to in IfR statements: patient, intervention, control, outcome (PICO) and study design; concepts underlying Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions.

RESULTS

We included 261 SRs. The majority evaluated a pharmacological intervention (n = 244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR statement (n = 210, 80.5%). IfR statements commonly addressed 'intervention' (n = 121, 57.6%), 'patient ' (n = 113, 53.8%), and 'study design' (n = 107, 51.0%). The most frequent PICO and study design combinations were 'patient and intervention ' (n = 71, 33.8%) and 'patient, intervention and study design ' (n = 34, 16.2%). Concepts underlying GRADE domains were rarely used for informing IfR recommendations: 'risk of bias ' (n = 2, 1.0%), and 'imprecision ' (n = 1, 0.5%), 'inconsistency ' (n = 1, 0.5%). Additional elements informing IfR were considerations on cost effectiveness (n = 9, 4.3%), reporting standards (n = 4, 1.9%), and individual patient data meta-analysis (n = 4, 1.9%).

CONCLUSION

Although about 80% of SRs included an IfR statement, the reporting of PICO elements varied across SRs. Concepts underlying GRADE domains were rarely used to derive IfR. Further work needs to assess the generalizability beyond SRs in advanced cancer patients. We suggest that more specific guidance on which and how IfR elements to report in SRs of interventions needs to be developed. Utilizing PICO elements and concepts underlying GRADE according to the Cochrane Handbook to state IfR seems to be a reasonable approach in the interim.

REGISTRATION

CRD42019134904.

Reducing uncertainty in evidence-based health policy by integrating empirical and theoretical evidence: An EbM+theory approach

BACKGROUND

To reduce their decisional uncertainty, health policy decision-makers rely more often on experts or their intuition than on evidence-based knowledge, especially in times of urgency. However, this practice is unacceptable from an evidence-based medicine (EbM) perspective. Therefore, in fast-changing and complex situations, we need an approach that delivers recommendations that serve decision-makers' needs for urgent, sound and uncertainty-reducing decisions based on the principles of EbM.

AIMS

The aim of this paper is to propose an approach that serves this need by enriching EbM with theory.

MATERIALS AND METHODS

We call this the EbM+theory approach, which integrates empirical and theoretical evidence in a context-sensitive way to reduce intervention and implementation uncertainty.

RESULTS

Within this framework, we propose two distinct roadmaps to decrease intervention and implementation uncertainty: one for simple and the other for complex interventions. As part of the roadmap, we present a three-step approach: applying theory (step 1), conducting mechanistic studies (EbM+; step 2) and conducting experiments (EbM; step 3).

DISCUSSION

This paper is a plea for integrating empirical and theoretical knowledge by combining EbM, EbM+ and theoretical knowledge in a common procedural framework that allows flexibility even in dynamic times. A further aim is to stimulate a discussion on using theories in health sciences, health policy, and implementation.

CONCLUSION

The main implications are that scientists and health politicians - the two main target groups of this paper-should receive more training in theoretical thinking; moreover, regulatory agencies like NICE may think about the usefulness of integrating elements of the EbM+theory approach into their considerations.

Definition, harms, and prevention of redundant systematic reviews

BACKGROUND

Along with other types of research, it has been stated that the extent of redundancy in systematic reviews has reached epidemic proportions. However, it was also emphasized that not all duplication is bad, that replication in research is essential, and that it can help discover unfortunate behaviors of scientists. Thus, the question is how to define a redundant systematic review, the harmful consequences of such reviews, and what we could do to prevent the unnecessary amount of this redundancy.

MAIN BODY

There is no consensus definition of a redundant systematic review. Also, it needs to be defined what amount of overlap between systematic reviews is acceptable and not considered a redundancy. One needs to be aware that it is possible that the authors did not intend to create a redundant systematic review. A new review on an existing topic, which is not an update, is likely justified only when it can be shown that the previous review was inadequate, for example, due to suboptimal methodology. Redundant meta-analyses could have scientific, ethical, and economic questions for researchers and publishers, and thus, they should be avoided, if possible. Potential solutions for preventing redundant reviews include the following: (1) mandatory prospective registration of systematic reviews; (2) editors and peer reviewers rejecting duplicate/redundant and inadequate reviews; (3) modifying the reporting checklists for systematic reviews; (4) developing methods for evidence-based research (EBR) monitoring; (5) defining systematic reviews; (6) defining the conclusiveness of systematic reviews; (7) exploring interventions for the adoption of methodological advances; (8) killing off zombie reviews (i.e., abandoned registered reviews); (9) better prevention of duplicate reviews at the point of registration; (10) developing living systematic reviews; and (11) education of researchers.

CONCLUSIONS

Disproportionate redundancy of the same or very similar systematic reviews can lead to scientific, ethical, economic, and societal harms. While it is not realistic to expect that the creation of redundant systematic reviews can be completely prevented, some preventive measures could be tested and implemented to try to reduce the problem. Further methodological research and development in this field will be welcome.