Meta-research evaluating redundancy and use of systematic reviews when planning new studies in health research: a scoping review

Why is it important to implement meta-research in universities and institutes with medical research activities?

In recent years, there has been a growing concern over questionable practices and a lack of rigor in scientific activities, particularly in health and medical sciences. Universities and research institutes are key players in the development of science, technology, and innovation. Academic institutions, whose primary mission is to generate and disseminate knowledge, bear the responsibility in many parts of the world to act as consultants and guardians of scientific integrity in health research. Then, universities and research institutes must act as guardians of the research and technological development process, utilizing methodological and operational evaluation tools to validate the rigor and quality of medical research. Meta-research is defined as the research of research itself. Some of the most important specific objectives of meta-research include the assessment of research relevance, the evaluation of evidence validity, and the exploration of scientific integrity. A significant portion of evidence in the medical and health sciences literature has been found to be redundant, misleading, or inconsistent. Although this issue is of great importance in global health, discussions about practical and tangible solutions remain fragmented and limited. The aim of this manuscript is to highlight the significance of employing meta-research within universities and research institutes as a tool to monitor scientific rigor and promote responsible practices in medical research.

Use of Digital Health Technologies for Dementia Care: Bibliometric Analysis and Report

BACKGROUND

Dementia is a syndrome that compromises neurocognitive functions of the individual and that is affecting 55 million individuals globally, as well as global health care systems, national economic systems, and family members.

OBJECTIVE

This study aimed to determine the status quo of scientific production on use of digital health technologies (DHTs) to support (older) people living with dementia, their families, and care partners. In addition, our study aimed to map the current landscape of global research initiatives on DHTs on the prevention, diagnosis, treatment, and support of people living with dementia and their caregivers.

METHODS

A bibliometric analysis was performed as part of a systematic review protocol using MEDLINE, Embase, Scopus, Epistemonikos, the Cochrane Database of Systematic Reviews, and Google Scholar for systematic and scoping reviews on DHTs and dementia up to February 21, 2024. Search terms included various forms of dementia and DHTs. Two independent reviewers conducted a 2-stage screening process with disagreements resolved by a third reviewer. Eligible reviews were then subjected to a bibliometric analysis using VOSviewer to evaluate document types, authorship, countries, institutions, journal sources, references, and keywords, creating social network maps to visualize emergent research trends.

RESULTS

A total of 704 records met the inclusion criteria for bibliometric analysis. Most reviews were systematic, with a substantial number covering mobile health, telehealth, and computer-based cognitive interventions. Bibliometric analysis revealed that the Journal of Medical Internet Research had the highest number of reviews and citations. Researchers from 66 countries contributed, with the United Kingdom and the United States as the most prolific. Overall, the number of publications covering the intersection of DHTs and dementia has increased steadily over time. However, the diversity of reviews conducted on a single topic has resulted in duplicated scientific efforts. Our assessment of contributions from countries, institutions, and key stakeholders reveals significant trends and knowledge gaps, particularly highlighting the dominance of high-income countries in this research domain. Furthermore, our findings emphasize the critical importance of interdisciplinary, collaborative teams and offer clear directions for future research, especially in underrepresented regions.

CONCLUSIONS

Our study shows a steady increase in dementia- and DHT-related publications, particularly in areas such as mobile health, virtual reality, artificial intelligence, and sensor-based technologies interventions. This increase underscores the importance of systematic approaches and interdisciplinary collaborations, while identifying knowledge gaps, especially in lower-income regions. It is crucial that researchers worldwide adhere to evidence-based medicine principles to avoid duplication of efforts. This analysis offers a valuable foundation for policy makers and academics, emphasizing the need for an international collaborative task force to address knowledge gaps and advance dementia care globally.

TRIAL REGISTRATION

PROSPERO CRD42024511241; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=511241.

Several methods for assessing research waste in reviews with a systematic search: a scoping review

Background

Research waste is present in all study designs and can have significant consequences for science, including reducing the reliability of research findings and contributing to the inefficient use of resources. Estimates suggest that as much as 85% of all biomedical research is wasted. However, it is uncertain how avoidable research waste is assessed in specific types of study designs and what methods could be used to examine different aspects of research waste. We aimed to investigate which methods, systematic reviews, scoping reviews, and overviews of reviews discussing research waste, have used to assess avoidable research waste.

Materials and Methods

We published a protocol in the Open Science Framework prospectively (https://osf.io/2fbp4). We searched PubMed and Embase with a 30-year limit (January 1993-August 2023). The concept examined was how research waste and related synonyms (e.g., unnecessary, redundant, duplicate, etc.) were assessed in reviews with a systematic search: systematic, scoping, or overviews of reviews. We extracted data on the method used in the review to examine for research waste and for which study design this method was applied.

Results

The search identified 4,285 records of which 93 reviews with systematic searches were included. The reviews examined a median of 90 (range 10-6,781) studies, where the study designs most commonly included were randomized controlled trials (48%) and systematic reviews (33%). In the last ten years, the number of reports assessing research waste has increased. More than 50% of examined reviews reported evaluating methodological research waste among included studies, typically using tools such as one of Cochrane Risk of Bias tools (n = 8) for randomized controlled trials or AMSTAR 1 or 2 (n = 12) for systematic reviews. One fourth of reviews assessed reporting guideline adherence to e.g., CONSORT (n = 4) for randomized controlled trials or PRISMA (n = 6) for systematic reviews.

Conclusion

Reviews with systematic searches focus on methodological quality and reporting guideline adherence when examining research waste. However, this scoping review revealed that a wide range of tools are used, which may pose difficulties in comparing examinations and performing meta-research. This review aids researchers in selecting methodologies and contributes to the ongoing discourse on optimizing research efficiency.

How can meta-research be used to evaluate and improve the quality of research in the field of traditional, complementary, and integrative medicine?

The field of traditional, complementary, and integrative medicine (TCIM) has garnered increasing attention due to its holistic approach to health and well-being. While the quantity of published research about TCIM has increased exponentially, critics have argued that the field faces challenges related to methodological rigour, reproducibility, and overall quality. This article proposes meta-research as one approach to evaluating and improving the quality of TCIM research. Meta-research, also known as research about research, can be defined as "the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives". By systematically evaluating methodological rigour, identifying biases, and promoting transparency, meta-research can enhance the reliability and credibility of TCIM research. Specific topics of interest that are discussed in this article include the following: 1) study design and research methodology, 2) reporting of research, 3) research ethics, integrity, and misconduct, 4) replicability and reproducibility, 5) peer review and journal editorial practices, 6) research funding: grants and awards, and 7) hiring, promotion, and tenure. For each topic, we provide case examples to illustrate meta-research applications in TCIM. We argue that meta-research initiatives can contribute to maintaining public trust, safeguarding research integrity, and advancing evidence based TCIM practice, while challenges include navigating methodological complexities, biases, and disparities in funding and academic recognition. Future directions involve tailored research methodologies, interdisciplinary collaboration, policy implications, and capacity building in meta-research.

Only half of the authors of overviews of exercise-related interventions use some strategy to manage overlapping primary studies-a metaresearch study

OBJECTIVES

The conduct of systematic reviews (SRs) and overviews share several similarities. However, because the unit of analysis for overviews is the SRs, there are some unique challenges. One of the most critical issues to manage when conducting an overview is the overlap of data across the primary studies included in the SRs. This metaresearch study aimed to describe the frequency of strategies to manage the overlap in overviews of exercise-related interventions.

STUDY DESIGN AND SETTING

A systematic search in MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Epistemonikos, and other sources was conducted from inception to June 2022. We included overviews of SRs that considered primary studies and evaluated the effectiveness of exercise-related interventions for any health condition. The overviews were screened by two authors independently, and the extraction was performed by one author and checked by a second. We found 353 overviews published between 2005 and 2022 that met the inclusion criteria.

RESULTS

One hundred and sixty-four overviews (46%) used at least one strategy to visualize, quantify, or resolve overlap, with a matrix (32/164; 20%), absolute frequency (34/164; 21%), and authors' algorithms (24/164; 15%) being the most used methods, respectively. From 2016 onwards, there has been a trend toward increasing the use of some strategies to manage overlap. Of the 108 overviews that used some strategy to resolve the overlap, ie, avoiding double or multiple counting of primary study data, 79 (73%) succeeded. In overviews where no strategies to manage overlap were reported (n = 189/353; 54%), 16 overview authors (8%) recognized this as a study limitation.

CONCLUSION

Although there is a trend toward increasing its use, only half of the authors of the overviews of exercise-related interventions used a strategy to visualize, quantify, or resolve overlap in the primary studies' data. In the future, authors should report such strategies to communicate more valid results.

Use of Evidence-Based Research Approach in RCTs of Acupuncture-Related Therapies for Primary Dysmenorrhea: A Meta-Research

OBJECTIVE

To assess the use of evidence-based research (EBR) approach in randomized controlled trials (RCTs) of acupuncture-related therapies for primary dysmenorrhea (PD).

METHODS

PubMed, EMBASE, Cochrane Library, China National Knowledge Infrastructure, Wanfang Database, Chinese Biomedical Database, and China Science and Technology Journal Database were searched from January 2013 to December 2022 for RCTs of acupuncture on PD. The full text and references of each RCT were read to assess whether systematic reviews (SRs) or other types of studies with similar research questions and end-users' perspectives were cited to justify and design the trial. In addition, the discussion section were analyzed to evaluate whether trials placed the new result in the existing SRs to draw a conclusion. Multivariable logistic regression was used to find variables that associated with 3 aspects of EBR approach: (1) citing clinical studies for justification, (2) citing relevant studies that obtain the perspectives of end users, and (3) citing clinical studies for results discussion.

RESULTS

Of 473 RCTs included, 45.67% (216) of the trials cited relevant similar studies, 21.56% (102) referenced to the studies that collected end-users' perspectives, and 10.99% (52) placed result in the context of the previous research. Few RCTs appropriately applied EBR approach. Among all the included studies, 3.17% (15) of the trials used SRs to inform study questions but none of them used updated SRs with acceptable quality; 1.05% (5) of the trials cited SRs of end-user's perspectives in the justification and design of the study, and only 1 trial added results in existing SR to draw a conclusion. Year of publication, language, funding, registration, ethical approval and number of sites were significantly associated with 1 of the 3 aspects of EBR approach.

CONCLUSIONS

Few RCTs in acupuncture-related therapies for PD used the EBR approach to minimize research redundancy. Researchers, research institutes, funding agencies, ethics committees, journals and peer reviewers in acupuncture should make efforts to use and promote the EBR approach to ensure the value of new trials.

Identifying meta-research with researchers as study subjects: Protocol for a scoping review

BACKGROUND

Meta-research in which researchers are the study subjects can illuminate how to better support researchers and enhance the development of research capacity. Comprehensively compiling the literature in this area can help define best practices for research capacity development and reveal gaps in the literature. However, there are challenges to assessing and synthesizing the breadth of the meta-research literature produced.

METHODS

In this article, we discuss the current barriers to conducting literature reviews on meta-research and strategies to address these barriers. We then outline proposed methods for conducting a scoping review on meta-research with researchers as study subjects.

DISCUSSION

Due to its interdisciplinary nature, broad scope, and difficult to pinpoint terminology, little is known about the state of meta-research with researchers as the study subjects. For this reason, there is a need for a scoping review that will identify research performed in which researchers were the study subjects.

Assessment of redundancy, methodological and reporting quality, and potential discrepancies of results of systematic reviews of early mobilisation of critically ill adults: a meta-research protocol

INTRODUCTION

Several systematic reviews (SRs) have been conducted to determine the effectiveness of early mobilisation in critically ill adults with heterogeneous methodology and results. Redundancy in conducting SRs, unclear justification when leading new SRs or updating, and discordant results of SRs on the same research question may generate research waste that makes it difficult for clinicians to keep up to date with the best available evidence. This meta-research aims to assess the redundancy, methodological and reporting quality, and potential reasons for discordance in the results reported by SRs conducted to determine the effectiveness of early mobilisation in critically ill adult patients.

METHODS AND ANALYSIS

A meta-research of early mobilisation SRs in critically ill adult patients will be conducted. A search of MEDLINE (Ovid), Embase (Ovid), CINAHL (EBSCOhost), Cochrane Library, Epistemonikos and other search resources will be conducted. Two independent reviewers will perform study selection, data extraction and quality appraisal. Discrepancies will be resolved by consensus or a third reviewer. The redundancy of SRs will be assessed by the degree of overlap of primary studies. In addition, the justification for conducting new SRs will be evaluated with the 'Evidence-Based Research' framework. The methodological quality of the SRs will be assessed with the A MeaSurement Tool to Assess systematic Reviews 2 tool, and the quality of the reports through compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. To assess the potential reasons for discordance in the results of the SRs considering divergence in results and their interpretation.

ETHICS AND DISSEMINATION

As meta-research, this study does not involve the participation of people whose rights may be violated. However, this overview will be developed rigorously and systematically to achieve valid and reliable results. The findings of this meta-research study will be presented at conferences and published in a peer-reviewed journal related to rehabilitation, critical care or research methodology.

TRIAL REGISTRATION NUMBER

osf.io/kxwq9.

Evidence-based health policy in Germany: lack of communication and coordination between academia and health authorities?

Health-care decision making should consider the best available evidence, often in the form of systematic reviews (SRs). The number of existing SRs and their overlap make their identification and use difficult. Decision makers often rely on de novo SRs instead of using existing SRs. We describe two cases of duplicate reviews (minimum volume threshold of total knee arthroplasties and lung cancer screening) and one case of duplicate primary data analysis (transcatheter aortic valve implantation). All cases have in common that unintended duplication of research occurred between health authorities and academia, demonstrating a lack of communication and coordination between them.It is important to note that academia and health authorities have different incentives. Academics are often measured by the number of peer-reviewed publications and grants awarded. In contrast, health authorities must comply with laws and are commissioned to deliver a specific report within a defined period of time. Most replication is currently unintended. A solution may be the collaboration of stakeholders commonly referred to as integrated knowledge translation (IKT). The IKT approach means that research is conducted in collaboration with the end users of the research. It requires active collaborations between researchers and decision-makers or knowledge users (clinicians, managers, policy makers) throughout the research process. Wherever cooperation is possible in spite of requirements for independence or confidentiality, legal regulations should facilitate and support collaborative approaches between academia and health authorities.