A systematic bias assessment of top-cited full-length original clinical investigations related to COVID-19

Hindsight2020: Characterizing Uncertainty in the COVID-19 Scientific Literature

Following emerging, re-emerging, and endemic pathogen outbreaks, the rush to publish and the risk of data misrepresentation, misinterpretation, and even misinformation puts an even greater onus on methodological rigor, which includes revisiting initial assumptions as new evidence becomes available. This study sought to understand how and when early evidence emerges and evolves when addressing different types of recurring pathogen-related questions. By applying claim-matching by means of deep learning Natural Language Processing (NLP) of coronavirus disease 2019 (COVID-19) scientific literature against a set of expert-curated evidence, patterns in timing across different COVID-19 questions-and-answers were identified, to build a framework for characterizing uncertainty in emerging infectious disease (EID) research over time. COVID-19 was chosen as a use case for this framework given the large and accessible datasets curated for scientists during the beginning of the pandemic. Timing patterns in reliably answering broad COVID-19 questions often do not align with general publication patterns, but early expert-curated evidence was generally stable. Because instability in answers often occurred within the first 2 to 6 mo for specific COVID-19 topics, public health officials could apply more conservative policies at the start of future pandemics, to be revised as evidence stabilizes.

Structural under-reporting of informed consent, data handling and sharing, ethical approval, and application of Open Science principles as proxies for study quality conduct in COVID-19 research: a systematic scoping review

BACKGROUND

The COVID-19 pandemic required science to provide answers rapidly to combat the outbreak. Hence, the reproducibility and quality of conducting research may have been threatened, particularly regarding privacy and data protection, in varying ways around the globe. The objective was to investigate aspects of reporting informed consent and data handling as proxies for study quality conduct.

METHODS

A systematic scoping review was performed by searching PubMed and Embase. The search was performed on November 8th, 2020. Studies with hospitalised patients diagnosed with COVID-19 over 18 years old were eligible for inclusion. With a focus on informed consent, data were extracted on the study design, prestudy protocol registration, ethical approval, data anonymisation, data sharing and data transfer as proxies for study quality. For reasons of comparison, data regarding country income level, study location and journal impact factor were also collected.

RESULTS

972 studies were included. 21.3% of studies reported informed consent, 42.6% reported waivers of consent, 31.4% did not report consent information and 4.7% mentioned other types of consent. Informed consent reporting was highest in clinical trials (94.6%) and lowest in retrospective cohort studies (15.0%). The reporting of consent versus no consent did not differ significantly by journal impact factor (p=0.159). 16.8% of studies reported a prestudy protocol registration or design. Ethical approval was described in 90.9% of studies. Information on anonymisation was provided in 17.0% of studies. In 257 multicentre studies, 1.2% reported on data sharing agreements, and none reported on Findable, Accessible, Interoperable and Reusable data principles. 1.2% reported on open data. Consent was most often reported in the Middle East (42.4%) and least often in North America (4.7%). Only one report originated from a low-income country.

DISCUSSION

Informed consent and aspects of data handling and sharing were under-reported in publications concerning COVID-19 and differed between countries, which strains study quality conduct when in dire need of answers.

Irish Media Coverage of COVID-19 Evidence-Based Research Reports From One National Agency

BACKGROUND

How research findings are presented through domestic news can influence behaviour and risk perceptions, particularly during emergencies such as the coronavirus disease 2019 (COVID-19) pandemic. Monitoring media communications to track misinformation and find information gaps is an important component of emergency risk communication. Therefore, this study investigated the traditional media coverage of nine selected COVID-19 evidence-based research reports and associated press releases (PRs) published during the initial phases of the pandemic (April to July 2020) by one national agency.

METHODS

NVivo was used for summative content analysis. 'Key messages' from each research report were proposed and 488 broadcast, print, and online media sources were coded at the phrase level. Manifest content was coded and counted to locate patterns in the data (what and how many) while latent content was analysed to further investigate these patterns (why and how). This included the coding of the presence of political and public health actors in coverage.

RESULTS

Coverage largely did not misrepresent the results of the reports, however, selective reporting and the variability in the use of quotes from governmental and public health stakeholders changed and contextualised results in different manners than perhaps originally intended in the PR. Reports received varying levels of media attention. Coverage focused on more 'human-interest' stories (eg, spread of COVID-19 by children and excess mortality) as opposed to more technical reports (eg, focusing on viral load, antibodies, testing, etc).

CONCLUSION

Our findings provide a case-study of European media coverage of evidence reports produced by a national agency. Results highlighted several strengths and weaknesses of current communication efforts.

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

INTRODUCTION

Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.

METHODS

We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.

RESULTS

After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.

DISCUSSION

The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

How did COVID-19 affect medical and cardiology journals? A pandemic in literature

BACKGROUND AND AIMS

The spreading speed of the COVID-19 pandemic forced the medical community to produce efforts in updating and sharing the evidence about this new disease, trying to preserve the accuracy of the data but at the same time avoiding the potentially harmful delay from discovery to implementation. The aim of our analysis was to assess the impact of the COVID-19 pandemic on medical literature in terms of proportion of COVID-19-related published papers and temporal patterns of publications within a sample of general/internal medicine and cardiology journals.

METHODS

We searched through PubMed scientific papers published from 1 January 2020 to 31 January 2021 about COVID-19 in ten major medical journals, of which five were in general/internal medicine and five in the cardiology field. We analyzed the proportion of COVID-19-related papers, and we examined temporal trends in the number of published papers.

RESULTS

Overall, the proportion of COVID-19-related papers was 18.5% (1986/10 756). This proportion was higher among the five selected general/internal medicine journals, compared with cardiology journals (23.8% vs 9.5%). The vast majority of papers were not original articles; in particular, in cardiology journals, there were 28% 'original articles', 17% 'review articles' and 55.1% 'miscellaneous', compared with 20.2%, 5.1% and 74.7% in general/internal medicine journals, respectively.

CONCLUSIONS

Our analysis highlights the big impact of the COVID-19 pandemic on international scientific literature. General and internal medicine journals were mainly involved, with cardiology journals only at a later time.

Research integrity in times of pandemic

In 1994, Douglas Graham Altman, one of the greatest statisticians of all time, wrote "We need less research, better research, and research done for good reasons". Twenty-seven years ago, Altman pointed out that the system favored unscientific behavior and that "bad science" was easy to publish, highlighting the financial implications of this amount of poorly designed research, with erroneous statistical methods, unrepresentative samples, or fraud. The covid-19 pandemic has once again put clinical research in check. The pressure for urgent responses was unprecedented. Knowledge of the origin of the virus, the transmission dynamics, the pathophysiology of the disease, efficient pharmacological and non-pharmacological measures would be counted in lives - and economies, and in governments.

Is rapid scientific publication also high quality? Bibliometric analysis of highly disseminated COVID-19 research papers

The impact of COVID-19 has underlined the need for reliable information to guide clinical practice and policy. This urgency has to be balanced against disruption to journal handling capacity and the continued need to ensure scientific rigour. We examined the reporting quality of highly disseminated COVID-19 research papers using a bibliometric analysis examining reporting quality and risk of bias (RoB) amongst 250 top scoring Altmetric Attention Score (AAS) COVID-19 research papers between January and April 2020. Method-specific RoB tools were used to assess quality. After exclusions, 84 studies from 44 journals were included. Forty-three (51%) were case series/studies, and only one was an randomized controlled trial. Most authors were from institutions based in China (n = 44, 52%). The median AAS and impact factor was 2015 (interquartile range [IQR] 1,105-4,051.5) and 12.8 (IQR 5-44.2) respectively. Nine studies (11%) utilized a formal reporting framework, 62 (74%) included a funding statement, and 41 (49%) were at high RoB. This review of the most widely disseminated COVID-19 studies highlights a preponderance of low-quality case series with few research papers adhering to good standards of reporting. It emphasizes the need for cautious interpretation of research and the increasingly vital responsibility that journals have in ensuring high-quality publications.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation (PEACHPIE): A systematic review of study design and evidence strength

Introduction:

Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature.

Methods:

We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.

Results:

After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.

Discussion:

The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.