Challenges of evidence synthesis during the 2020 COVID pandemic: a scoping review

Observational research in epidemic settings: a roadmap to reform

Observational studies are critical tools in clinical research and public health response, but challenges arise in ensuring the data produced by these studies are scientifically robust and socially valuable. Resolving these challenges requires careful attention to prioritising the most valuable research questions, ensuring robust study design, strong data management practices, expansive community engagement, and access and benefit sharing of results and research materials. This paper opens with a discussion of how well-designed observational studies contribute to biomedical evidence and provides examples from across the clinical literature of how these methods generate hypotheses for future research and uncover otherwise unattainable insights by providing examples from across the clinical literature. Then, we present obstacles that remain in ensuring observational studies are optimally designed, conducted and communicated.

Addressing evidence needs during health crises in the province of Quebec (Canada): a proposed action plan for rapid evidence synthesis

BACKGROUND

The COVID-19 pandemic necessitated the rapid availability of evidence to respond in a timely manner to the needs of practice settings and decision-makers in health and social services. Now that the pandemic is over, it is time to put in place actions to improve the capacity of systems to meet knowledge needs in a situation of crisis. The main objective of this project was thus to develop an action plan for the rapid syntheses of evidence in times of health crisis in Quebec (Canada).

METHODS

We conducted a three-phase collaborative research project. First, we carried out a survey with producers and users of rapid evidence syntheses (n = 40) and a group interview with three patient partners to prioritize courses of action. In parallel, we performed a systematic mapping of the literature to identify rapid evidence synthesis initiatives developed during the pandemic. The results of these two phases were used in a third phase, in which we organized a deliberative workshop with 26 producers and users of rapid evidence syntheses to identifying strategies to operationalize priorities. The data collected at each phase were compared to identify common courses of action and integrated to develop an action plan.

RESULTS

A total of 14 specific actions structured into four main axes were identified over the three phases. In axis 1, actions on raising awareness of the importance of evidence-informed decision-making among stakeholders in the health and social services network are presented. Axis 2 includes actions to promote optimal collaboration of key stakeholders in the production of rapid evidence synthesis to support decision-making. Actions advocating the use of a variety of rapid evidence synthesis methodologies known to be effective in supporting decision-making are presented in axis 3. Finally, axis 4 is about actions on the use of effective knowledge translation strategies to promote the use of rapid evidence synthesis products to support decision-making.

CONCLUSIONS

This project led to the development of a collective action plan aimed at preparing the Quebec ecosystem and other similar jurisdictions to meet knowledge needs more effectively in times of health emergency. The implementation of this plan and its evaluation will enable us to continue to fine-tune it.

Paper 4: a systematic review on the use of logic models and frameworks for methodological conduct of evidence synthesis

OBJECTIVES

To identify evidence syntheses of health interventions addressing racial health equity reporting the use of equity-focused frameworks and logic models.

STUDY DESIGN AND SETTING

The search strategy included three sources; a search of three bibliographic databases to identify systematic reviews assessing interventions to improve racial health equity, semistructured interviews with diverse group and a targeted organization website searches (eg, National Institute of Health, United States Preventive Services Task Force) to identify relevant logic models and frameworks. The searches were conducted between January 1, 2020, and January 25, 2023. We used a qualitative approach to identify and describe key characteristics of equity-focused logic models and frameworks used in evidence syntheses.

RESULTS

Of the 153 racial health equity-focused evidence syntheses identified, two explicitly used logic models to describe the intervention mechanism. We identified seven existing health equity frameworks from semistructured interviews and electronic search of key websites that were categorized by stated purpose as providing guidance for 1) research, 2) health policy, 3) digital health-care solutions, and 4) clinical preventive services. Two out of seven frameworks included guidance on integrating frameworks or logic models in evidence synthesis while the majority provided contextual information on how to define or consider race or racism as a structural determinant of health.

CONCLUSION

There is limited use of logic models and frameworks in evidence syntheses addressing racial health equity. There is a need for more applied frameworks providing guidance for framing, conducting and interpreting findings of evidence syntheses addressing racial health equity.

PLAIN LANGUAGE SUMMARY

The goal of this study was to find reviews of health programs that focus on improving racial health equity, and to see if they used special frameworks or models designed to address equity. To do this, we searched three major research databases, conducted interviews with a diverse group of people, and looked at relevant organization websites (like the National Institute of Health and the World Health Organization) between January 2020 and January 2023. We used a qualitative approach to study the key features of these equity-focused frameworks and models. We found 153 reviews focused on racial health equity, but only two of them used logic models to explain the intervention. From interviews and website searches, we identified seven existing health equity frameworks. These were grouped into four categories: research, health policy, digital health care, and clinical preventive services. Only two of these frameworks provided advice on how to use them in evidence reviews, while most focused on understanding how race and racism impact health as a social factor. In conclusion, there is limited use of frameworks and models in reviews about racial health equity. More practical frameworks are needed to help guide the research and interpretation of these reviews.

Leveraging new methodologies for public health crisis management

Evidence-based medicine is critical in public health emergencies, offering a framework for decision-making and adaptive healthcare responses. By relying on up-to-date and reliable evidence, EBM enables healthcare systems to respond quickly to evolving crises and ensures efficient resource allocation. This perspective presents the importance of evidence-based medicine in public health emergencies, emphasizing the need for rapid decision-making and preparedness. It identifies challenges from the COVID-19 pandemic, including barriers to evidence synthesis, and explores innovative solutions, including methodological pluralism and systems thinking. The findings highlight that evidence-based medicine improves health care systems' responsiveness to public health crises, supports the efficient resource allocation, and reinforces the need for flexible strategies that adapt to rapidly evolving information. In particular, the practical implications underscore that, in crisis settings, EBM must expand beyond strict evidence hierarchies to include timely, reasonable, and sometimes intuitive expert judgments, ensuring robust and adaptable responses. In conclusion, while EBM enhances healthcare adaptability and decision-making in emergencies, future responses will benefit from incorporating more diverse and flexible approaches to ensure more resilient and effective public health strategies.

Uncertainty in COVID-19 transmission could undermine our ability to predict long COVID

As SARS-CoV-2 has transitioned from a novel pandemic-causing pathogen into an established seasonal respiratory virus, focus has shifted to post-acute sequelae of COVID-19 (PASC, colloquially 'long COVID'). We use compartmental mathematical models simulating emergence of new variants to help identify key sources of uncertainty in PASC trajectories. Some parameters (such as the duration and equilibrium prevalence of infection, as well as the fraction of infections that develop PASC) matter more than others (such as the duration of immunity and secondary vaccine efficacy against PASC). Even if newer variants carry the same risk of PASC as older types, the dynamics of selection can give rise to greater PASC prevalence. However, identifying plausible PASC prevalence trajectories requires accurate knowledge of the transmission potential of COVID-19 variants in the endemic phase. Precise estimates for secondary vaccine efficacy and duration of immunity will not greatly improve forecasts for PASC prevalence. Researchers involved with Living Evidence Synthesis, or other similar initiatives focused on PASC, are well advised to ascertain primary efficacy against infection, duration of infection and prevalence of active infection in order to facilitate predictions.

Automation tools to support undertaking scoping reviews

OBJECTIVE

This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews.

STUDY DESIGN AND SETTING

The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group.

RESULTS

This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de-duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability.

CONCLUSION

This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high-quality scoping reviews.

Production and use of rapid responses during the COVID-19 pandemic in Quebec (Canada): perspectives from evidence synthesis producers and decision makers

BACKGROUND

The COVID-19 pandemic has required evidence to be made available more rapidly than usual, in order to meet the needs of decision makers in a timely manner. These exceptional circumstances have caused significant challenges for organizations and teams responsible for evidence synthesis. They had to adapt to provide rapid responses to support decision-making. This study aimed to document (1) the challenges and adaptations made to produce rapid responses during the pandemic, (2) their perceived usefulness, reported use and factors influencing their use and (3) the methodological adaptations made to produce rapid responses.

METHODS

A qualitative study was conducted in 2021 with eight organizations in the health and social services system in Quebec (Canada), including three institutes with a provincial mandate. Data collection included focus groups (n = 9 groups in 8 organizations with 64 participants), interviews with decision makers (n = 12), and a document analysis of COVID-19 rapid responses (n = 128). A thematic analysis of qualitative data (objectives 1 and 2) and a descriptive analysis of documents (objective 3) were conducted.

RESULTS

The results highlight the teams and organizations' agility to deal with the many challenges encountered during the pandemic (e.g., increased their workloads, adoption of new technological tools or work processes, improved collaboration, development of scientific monitoring, adaptation of evidence synthesis methodologies and products). The challenge of balancing rigor and speed was reported by teams and organizations. When available at the right time, rapid responses have been reported as a useful tool for informing or justifying decisions in a context of uncertainty. Several factors that may influence their use were identified (e.g., clearly identify needs, interactions with producers, perceived rigor and credibility, precise and feasible recommendations). Certain trends in the methodological approaches used to speed up the evidence synthesis process were identified.

CONCLUSIONS

This study documented rapid responses producers' experiences during the COVID-19 pandemic in Quebec, and decision makers who requested, consulted, or used these products. Potential areas of improvements are identified such as reinforce coordination, improve communication loops, clarify guidelines or methodological benchmarks, and enhance utility of rapid response products for decision makers.

Overcoming the challenges of using automated technologies for public health evidence synthesis

Many organisations struggle to keep pace with public health evidence due to the volume of published literature and length of time it takes to conduct literature reviews. New technologies that help automate parts of the evidence synthesis process can help conduct reviews more quickly and efficiently to better provide up-to-date evidence for public health decision making. To date, automated approaches have seldom been used in public health due to significant barriers to their adoption. In this Perspective, we reflect on the findings of a study exploring experiences of adopting automated technologies to conduct evidence reviews within the public health sector. The study, funded by the European Centre for Disease Prevention and Control, consisted of a literature review and qualitative data collection from public health organisations and researchers in the field. We specifically focus on outlining the challenges associated with the adoption of automated approaches and potential solutions and actions that can be taken to mitigate these. We explore these in relation to actions that can be taken by tool developers (e.g. improving tool performance and transparency), public health organisations (e.g. developing staff skills, encouraging collaboration) and funding bodies/the wider research system (e.g. researchers, funding bodies, academic publishers and scholarly journals).

Health Research Collaborations by Academic Entities: A Systematic Review

PURPOSE

To review the literature on health research collaborations by academic entities and to identify the main phases, components, and concepts of these research collaborations.

METHOD

The authors conducted a systematic review of the literature, searching 4 databases in March 2022 for studies on health research collaboration between an academic entity (individual, group, or institution) and any other entity included. They excluded non-health-related studies and studies in which collaboration was not for the purpose of research. From included studies, reviewers abstracted data about the 4 main phases of research collaborations (initiation, conduct, monitoring, and evaluation) and synthesized their corresponding components and concepts using thematic analysis method.

RESULTS

A total of 59 studies met inclusion criteria. These studies described building research collaborations between an academic entity and other academic entities (n = 29; 49%), communities (n = 28; 47%), industry (n = 7; 12%), and/or governmental entities (n = 4; 7%). Of the 59 studies, 22 addressed 2 phases of collaboration, 20 addressed 3 phases, and 17 addressed all 4 phases. All included studies described at least 1 of the components relevant to the initiation phase and at least 1 relevant to the conduct phase. Team structure was the most common component discussed in relation to the initiation phase (n = 48; 81%), and team dynamics was the most common component discussed in relation to the conduct phase (n = 55; 93%). At least 1 of the components relevant to the monitoring phase was reported in 36 studies, and at least 1 component relevant to the evaluation phase was reported in 28 studies.

CONCLUSIONS

This review provides important information for groups aiming to engage in collaborative research. The synthesized list of collaboration phases and their components can serve as a road map for collaborators at different steps of their research.

Structural under-reporting of informed consent, data handling and sharing, ethical approval, and application of Open Science principles as proxies for study quality conduct in COVID-19 research: a systematic scoping review

BACKGROUND

The COVID-19 pandemic required science to provide answers rapidly to combat the outbreak. Hence, the reproducibility and quality of conducting research may have been threatened, particularly regarding privacy and data protection, in varying ways around the globe. The objective was to investigate aspects of reporting informed consent and data handling as proxies for study quality conduct.

METHODS

A systematic scoping review was performed by searching PubMed and Embase. The search was performed on November 8th, 2020. Studies with hospitalised patients diagnosed with COVID-19 over 18 years old were eligible for inclusion. With a focus on informed consent, data were extracted on the study design, prestudy protocol registration, ethical approval, data anonymisation, data sharing and data transfer as proxies for study quality. For reasons of comparison, data regarding country income level, study location and journal impact factor were also collected.

RESULTS

972 studies were included. 21.3% of studies reported informed consent, 42.6% reported waivers of consent, 31.4% did not report consent information and 4.7% mentioned other types of consent. Informed consent reporting was highest in clinical trials (94.6%) and lowest in retrospective cohort studies (15.0%). The reporting of consent versus no consent did not differ significantly by journal impact factor (p=0.159). 16.8% of studies reported a prestudy protocol registration or design. Ethical approval was described in 90.9% of studies. Information on anonymisation was provided in 17.0% of studies. In 257 multicentre studies, 1.2% reported on data sharing agreements, and none reported on Findable, Accessible, Interoperable and Reusable data principles. 1.2% reported on open data. Consent was most often reported in the Middle East (42.4%) and least often in North America (4.7%). Only one report originated from a low-income country.

DISCUSSION

Informed consent and aspects of data handling and sharing were under-reported in publications concerning COVID-19 and differed between countries, which strains study quality conduct when in dire need of answers.

Mapping reviews, scoping reviews, and evidence and gap maps (EGMs): the same but different- the "Big Picture" review family

Scoping reviews, mapping reviews, and evidence and gap maps are evidence synthesis methodologies that address broad research questions, aiming to describe a bigger picture rather than address a specific question about intervention effectiveness. They are being increasingly used to support a range of purposes including guiding research priorities and decision making. There is however a confusing array of terminology used to describe these different approaches. In this commentary, we aim to describe where there are differences in terminology and where this equates to differences in meaning. We demonstrate the different theoretical routes that underpin these differences. We suggest ways in which the approaches of scoping and mapping reviews may differ in order to guide consistency in reporting and method. We propose that mapping and scoping reviews and evidence and gap maps have similarities that unite them as a group but also have unique differences. Understanding these similarities and differences is important for informing the development of methods used to undertake and report these types of evidence synthesis.

Systematic reviews of convalescent plasma in COVID-19 continue to be poorly conducted and reported: a systematic review

OBJECTIVES

To suggest possible approaches to combatting the impact of the COVID-19 infodemic to prevent research waste in future health emergencies and in everyday research and practice.

STUDY DESIGN AND SETTING

Systematic review. The Epistemonikos database was searched in June 2021 for systematic reviews on the effectiveness of convalescent plasma for COVID-19. Two reviewers independently screened the retrieved references with disagreements resolved by discussion. Data extraction was completed by one reviewer with a proportion checked by a second. We used the Assessment of Multiple Systematic Reviews to assess the quality of conduct and reporting of included reviews.

RESULTS

Fifty one systematic reviews are included with 193 individual studies included within the systematic reviews. There was considerable duplication of effort; multiple reviews were conducted at the same time with inconsistencies in the evidence included. The reviews were of low methodological quality, poorly reported, and did not adhere to preferred reporting items for systematic reviews and meta-analysis guidance.

CONCLUSION

Researchers need to conduct, appraise, interpret, and disseminate systematic reviews better. All in the research community (researchers, peer-reviewers, journal editors, funders, decision makers, clinicians, journalists, and the public) need to work together to facilitate the conduct of robust systematic reviews that are published and communicated in a timely manner, reducing research duplication and waste, increasing transparency and accessibility of all systematic reviews.

Adapt or die: how the pandemic made the shift from EBM to EBM+ more urgent

Evidence-based medicine (EBM's) traditional methods, especially randomised controlled trials (RCTs) and meta-analyses, along with risk-of-bias tools and checklists, have contributed significantly to the science of COVID-19. But these methods and tools were designed primarily to answer simple, focused questions in a stable context where yesterday's research can be mapped more or less unproblematically onto today's clinical and policy questions. They have significant limitations when extended to complex questions about a novel pathogen causing chaos across multiple sectors in a fast-changing global context. Non-pharmaceutical interventions which combine material artefacts, human behaviour, organisational directives, occupational health and safety, and the built environment are a case in point: EBM's experimental, intervention-focused, checklist-driven, effect-size-oriented and deductive approach has sometimes confused rather than informed debate. While RCTs are important, exclusion of other study designs and evidence sources has been particularly problematic in a context where rapid decision making is needed in order to save lives and protect health. It is time to bring in a wider range of evidence and a more pluralist approach to defining what counts as 'high-quality' evidence. We introduce some conceptual tools and quality frameworks from various fields involving what is known as mechanistic research, including complexity science, engineering and the social sciences. We propose that the tools and frameworks of mechanistic evidence, sometimes known as 'EBM+' when combined with traditional EBM, might be used to develop and evaluate the interdisciplinary evidence base needed to take us out of this protracted pandemic. Further articles in this series will apply pluralistic methods to specific research questions.

Lessons learnt: Undertaking rapid reviews on public health and social measures during a global pandemic

Public health and social measures (PHSM) have been central to the COVID‐19 response. Consequently, there has been much pressure on decision‐makers to make evidence‐informed decisions and on researchers to synthesize the evidence regarding these measures. This article describes our experiences, responses and lessons learnt regarding key challenges when planning and conducting rapid reviews of PHSM during the COVID‐19 pandemic. Stakeholder consultations and scoping reviews to obtain an overview of the evidence inform the scope of reviews that are policy‐relevant and feasible. Multiple complementary reviews serve to examine the benefits and harms of PHSM across different populations and contexts. Conceiving reviews of effectiveness as adaptable living reviews helps to respond to evolving evidence needs and an expanding evidence base. An appropriately skilled review team and good planning, coordination and communication ensures smooth and rigorous processes and efficient use of resources. Scientific rigor, the practical implications of PHSM‐related complexity and likely time savings should be carefully weighed in deciding on methodological shortcuts. Making the best possible use of modeling studies represents a particular challenge, and methods should be carefully chosen, piloted and implemented. Our experience raises questions regarding the nature of rapid reviews and regarding how different types of evidence should be considered in making decisions about PHSM during a global pandemic. We highlight the need for readily available protocols for conducting studies on the effectiveness, unintended consequences and implementation of PHSM in a timely manner, as well as the need for rapid review standards tailored to “rapid” versus “emergency” mode reviewing.