Preprints Involving Medical Research-Do the Benefits Outweigh the Challenges?

The changing roles of scientific journals

After centuries of relative stability, the scientific publishing world has undergone tremendous disruption and change during the first decades of the 21st century. The causes for disruption can be traced to the information revolution, which brought such benefits as rapid publication, greater connectivity, and ready access to large databases, along with less desirable practices including image manipulation, plagiarism, and other ethical transgressions. The information revolution has driven the proliferation of journals, expansion of for-profit academic publishing, and empowerment of the open-access movement, each of which has exerted new financial pressures on traditional publishing models. As journals became the focal point for ethical concerns in science, they have adapted by increasing the scope of their duties, which now include archiving of data, enforcement of good practices, establishment of standards for rigor, and training the next generation of reviewers and editors. Here, we consider the seismic changes occurring in scientific publishing and place them into the context of a rapidly changing landscape of scientific and publishing norms.

An evaluation of the preprints produced at the beginning of the 2022 mpox public health emergency

BACKGROUND

Preprints are scientific articles that have not undergone the peer-review process. They allow the latest evidence to be rapidly shared, however it is unclear whether they can be confidently used for decision-making during a public health emergency. This study aimed to compare the data and quality of preprints released during the first four months of the 2022 mpox outbreak to their published versions.

METHODS

Eligible preprints (n = 76) posted between May to August 2022 were identified through an established mpox literature database and followed to July 2024 for changes in publication status. Quality of preprints and published studies was assessed by two independent reviewers to evaluate changes in quality, using validated tools that were available for the study design (n = 33). Tools included the Newcastle-Ottawa Scale; Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2); and JBI Critical Appraisal Checklists. The questions in each tool led to an overall quality assessment of high quality (no concerns with study design, conduct, and/or analysis), moderate quality (minor concerns) or low quality (several concerns). Changes in data (e.g. methods, outcomes, results) for preprint-published pairs (n = 60) were assessed by one reviewer and verified by a second.

RESULTS

Preprints and published versions that could be evaluated for quality (n = 25 pairs) were mostly assessed as low quality. Minimal to no change in quality from preprint to published was identified: all observational studies (10/10), most case series (6/7) and all surveillance data analyses (3/3) had no change in overall quality, while some diagnostic test accuracy studies (3/5) improved or worsened their quality assessment scores. Among all pairs (n = 60), outcomes were often added in the published version (58%) and less commonly removed (18%). Numerical results changed from preprint to published in 53% of studies, however most of these studies (22/32) had changes that were minor and did not impact main conclusions of the study.

CONCLUSIONS

This study suggests the minimal changes in quality, results and main conclusions from preprint to published versions supports the use of preprints, and the use of the same critical evaluation tools on preprints as applied to published studies, in decision-making during a public health emergency.

Describing the Landscape of Medical Education Preprints on MedRxiv: Current Trends and Future Recommendations

PURPOSE

A preprint is a version of a research manuscript posted to a preprint server prior to peer review. Preprints enable authors to quickly and openly share research, afford opportunities for expedient feedback, and enable immediate listing of research on grant and promotion applications. In medical education, most journals welcome preprints, which suggests that preprints play a role in the field's discourse. Yet, little is known about medical education preprints, including author characteristics, preprint use, and ultimate publication status. This study provides an overview of preprints in medical education to better understand their role in the field's discourse.

METHOD

The authors queried medRxiv, a preprint repository, to identify preprints categorized as "medical education" and downloaded related metadata. CrossRef was queried to gather information on preprints later published in journals. Data were analyzed using descriptive statistics.

RESULTS

Between 2019 and 2022, 204 preprints were classified in medRxiv as "medical education," with most deposited in 2021 (n = 76; 37.3%). On average, preprint full-texts were downloaded 1,875.2 times, and all were promoted on social media. Preprints were authored, on average, by 5.9 authors. Corresponding authors were based in 41 countries, with 45.6% in the United States, the United Kingdom, and Canada. Almost half (n = 101; 49.5%) became published articles in predominantly peer-reviewed journals. Preprints appeared in 65 peer-reviewed journals, with BMC Medical Education (n = 9; 8.9%) most represented.

CONCLUSIONS

Medical education research is being deposited as preprints, which are promoted, heavily accessed, and subsequently published in peer-reviewed journals, including medical education journals. Considering the benefits of preprints and the slowness of medical education publishing, it is likely that preprint depositing will increase and preprints will be integrated into the field's discourse. The authors propose next steps to facilitate responsible and effective creation and use of preprints.

Reimagining bioRxiv and preprint servers as platforms for academic learning

A popular preprint server, bioRxiv, is important as a tool for increased visibility for life science research. If used properly, however, bioRxiv can also be an important tool for training, as it may expose trainees (degree-seeking students undertaking research or internships directly related to their field of study) to the peer review process. Here, we offer a comprehensive guide to using bioRxiv as a training tool, as well as offer suggestions for improvements in bioRxiv, including confusion that may be caused by bioRxiv articles appearing on PubMed.

Altmetric Attention Scores and Citations of Published Research With or Without Preprints

This cross-sectional study assesses how frequently research articles published in the clinical journals with high impact factors are preprinted and whether preprinting is associated with changes in media attention and citation counts.

Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials

BACKGROUND

Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials.

METHODS

The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles. Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article.

RESULTS

The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint-article RCTs were included in the analysis. The median (interquartile range) delay between preprint and journal article was 121 (73-187) days, the median sample size was 150 (71-464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as 'some concern' for 80% of RCTs. We found that 81 preprint-article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found.

CONCLUSIONS

Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

Rapid, accurate publication and dissemination of clinical trial results: benefits and challenges

Large-scale clinical trials are essential in cardiology and require rapid, accurate publication, and dissemination. Whereas conference presentations, press releases, and social media disseminate information quickly and often receive considerable coverage by mainstream and healthcare media, they lack detail, may emphasize selected data, and can be open to misinterpretation. Preprint servers speed access to research manuscripts while awaiting acceptance for publication by a journal, but these articles are not formally peer-reviewed and sometimes overstate the findings. Publication of trial results in a major journal is very demanding but the use of existing checklists can help accelerate the process. In case of rejection, procedures such as easing formatting requirements and possibly carrying over peer-review to other journals could speed resubmission. Secondary publications can help maximize benefits from clinical trials; publications of secondary endpoints and subgroup analyses further define treatment effects and the patient populations most likely to benefit. These rely on data access, and although data sharing is becoming more common, many challenges remain. Beyond publication in medical journals, there is a need for wider knowledge dissemination to maximize impact on clinical practice. This might be facilitated through plain language summary publications. Social media, websites, mainstream news outlets, and other publications, although not peer-reviewed, are important sources of medical information for both the public and for clinicians. This underscores the importance of ensuring that the information is understandable, accessible, balanced, and trustworthy. This report is based on discussions held on December 2021, at the 18th Global Cardiovascular Clinical Trialists meeting, involving a panel of editors of some of the top medical journals, as well as members of the lay press, industry, and clinical trialists.

No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study

OBJECTIVES

Preprints became a major source of research communication during the COVID-19 pandemic. We aimed to evaluate whether summary treatment effect estimates differ between preprint and peer-reviewed journal trials.

STUDY DESIGN AND SETTING

A meta-epidemiological study. Data were derived from the COVID-NMA living systematic review (covid-nma.com) up to July 20, 2022. We identified all meta-analyses evaluating pharmacological treatments vs. standard of care or placebo for patients with COVID-19 that included at least one preprint and one peer-reviewed journal article. Difference in effect estimates between preprint and peer-reviewed journal trials were estimated by the ratio of odds ratio (ROR); ROR <1 indicated larger effects in preprint trials.

RESULTS

Thirty-seven meta-analyses including 114 trials (44 preprints and 70 peer-reviewed publications) were selected. The median number of randomized controlled trials (RCTs) per meta-analysis was 2 (interquartile range [IQR], 2-4; maximum, 11), median sample size of RCTs was 199 (IQR, 99-478). Overall, there was no statistically significant difference in summary effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71-1.09; I2 = 17.8%; τ2 = 0.06).

CONCLUSION

We did not find an important difference between summary treatment effects of preprints and summary treatment effects of peer-reviewed publications. Systematic reviewers and guideline developers should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.

The use and acceptability of preprints in health and social care settings: A scoping review

BACKGROUND

Preprints are open and accessible scientific manuscript or report that is shared publicly, through a preprint server, before being submitted to a journal. The value and importance of preprints has grown since its contribution during the public health emergency of the COVID-19 pandemic. Funders and publishers are establishing their position on the use of preprints, in grant applications and publishing models. However, the evidence supporting the use and acceptability of preprints varies across funders, publishers, and researchers. The scoping review explored the current evidence on the use and acceptability of preprints in health and social care settings by publishers, funders, and the research community throughout the research lifecycle.

METHODS

A scoping review was undertaken with no study or language limits. The search strategy was limited to the last five years (2017-2022) to capture changes influenced by COVID-19 (e.g., accelerated use and role of preprints in research). The review included international literature, including grey literature, and two databases were searched: Scopus and Web of Science (24 August 2022).

RESULTS

379 titles and abstracts and 193 full text articles were assessed for eligibility. Ninety-eight articles met eligibility criteria and were included for full extraction. For barriers and challenges, 26 statements were grouped under four main themes (e.g., volume/growth of publications, quality assurance/trustworthiness, risks associated to credibility, and validation). For benefits and value, 34 statements were grouped under six themes (e.g., openness/transparency, increased visibility/credibility, open review process, open research, democratic process/systems, increased productivity/opportunities).

CONCLUSIONS

Preprints provide opportunities for rapid dissemination but there is a need for clear policies and guidance from journals, publishers, and funders. Cautionary measures are needed to maintain the quality and value of preprints, paying particular attention to how findings are translated to the public. More research is needed to address some of the uncertainties addressed in this review.

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We "Undercooking" Our Science?

BACKGROUND

Although biomedical preprint servers have grown rapidly over the past several years, the harm to patient health and safety remains a major concern among several scientific communities. Despite previous studies examining the role of preprints during the Coronavirus-19 pandemic, there is limited information characterizing their impact on scientific communication in orthopaedic surgery.

QUESTIONS/PURPOSES

(1) What are the characteristics (subspecialty, study design, geographic origin, and proportion of publications) of orthopaedic articles on three preprint servers? (2) What are the citation counts, abstract views, tweets, and Altmetric score per preprinted article and per corresponding publication?

METHODS

Three of the largest preprint servers (medRxiv, bioRxiv, and Research Square) with a focus on biomedical topics were queried for all preprinted articles published between July 26, 2014, and September 1, 2021, using the following search terms: "orthopaedic," "orthopedic," "bone," "cartilage," "ligament," "tendon," "fracture," "dislocation," "hand," "wrist," "elbow," "shoulder," "spine," "spinal," "hip," "knee," "ankle," and "foot." Full-text articles in English related to orthopaedic surgery were included, while nonclinical studies, animal studies, duplicate studies, editorials, abstracts from conferences, and commentaries were excluded. A total of 1471 unique preprints were included and further characterized in terms of the orthopaedic subspecialty, study design, date posted, and geographic factors. Citation counts, abstract views, tweets, and Altmetric scores were collected for each preprinted article and the corresponding publication of that preprint in an accepting journal. We ascertained whether a preprinted article was published by searching title keywords and the corresponding author in three peer-reviewed article databases (PubMed, Google Scholar, and Dimensions) and confirming that the study design and research question matched.

RESULTS

The number of orthopaedic preprints increased from four in 2017 to 838 in 2020. The most common orthopaedic subspecialties represented were spine, knee, and hip. From 2017 to 2020, the cumulative counts of preprinted article citations, abstract views, and Altmetric scores increased. A corresponding publication was identified in 52% (762 of 1471) of preprints. As would be expected, because preprinting is a form of redundant publication, published articles that are also preprinted saw greater abstract views, citations, and Altmetric scores on a per-article basis.

CONCLUSION

Although preprints remain an extremely small proportion of all orthopaedic research, our findings suggest that nonpeer-reviewed, preprinted orthopaedic articles are being increasingly disseminated. These preprinted articles have a smaller academic and public footprint than their published counterparts, but they still reach a substantial audience through infrequent and superficial online interactions, which are far from equivalent to the engagement facilitated by peer review. Furthermore, the sequence of preprint posting and journal submission, acceptance, and publication is unclear based on the information available on these preprint servers. Thus, it is difficult to determine whether the metrics of preprinted articles are attributable to preprinting, and studies such as the present analysis will tend to overestimate the apparent impact of preprinting. Despite the potential for preprint servers to function as a venue for thoughtful feedback on research ideas, the available metrics data for these preprinted articles do not demonstrate the meaningful engagement that is achieved by peer review in terms of the frequency or depth of audience feedback.

CLINICAL RELEVANCE

Our findings highlight the need for safeguards to regulate research dissemination through preprint media, which has never been shown to benefit patients and should not be considered as evidence by clinicians. Clinician-scientists and researchers have the most important responsibility of protecting patients from the harm of potentially inaccurate biomedical science and therefore must prioritize patient needs first by uncovering scientific truths through the evidence-based processes of peer review, not preprinting. We recommend all journals publishing clinical research adopt the same policy as Clinical Orthopaedics and Related Research® , The Bone & Joint Journal, The Journal of Bone and Joint Surgery, and the Journal of Orthopaedic Research , removing any papers posted to preprint servers from consideration.

The impact of COVID-19 on clinical research at Australian and New Zealand universities: A qualitative study

Background

The COVID-19 pandemic resulted in the implementation of social distancing measures, travel restrictions, and infection control measures that introduced a myriad of disruptions in the conduct of clinical research worldwide. As a result, many aspects of clinical research were variably impacted.

Aim

To explore the impact of the first 18 months of the COVID-19 pandemic on clinical research across accredited nursing, pharmacy, and medicine program providers in Australian and New Zealand universities.

Methods

Representatives from all program providers across Australian and New Zealand universities, with publicly available contact information, were invited to participate in this qualitative study, whereby semi-structured interviews were completed with participants who held senior research or leadership positions within their institution. Interviews were transcribed verbatim and inductively analysed using thematic content analysis.

Findings

Interviews were conducted with 16 participants between August and October 2021. Two major themes were identified (Immediate Research Impact and Broader Research Impact) with six subthemes: Prioritisation, Continuation, and Dissemination of Research; Modifications to Research; Funding and Changes to Research Focus; Collaboration; Research Workforce; Context-specific Impacts.

Discussion

The impact on clinical research in Australian and New Zealand universities included changes to data collection methods, a perceived decreased quality of research, changes to collaboration, neglect of basic disease research, and loss of the research workforce.

Conclusion

This study highlights the impact of the COVID-19 pandemic on clinical research within the Australian and New Zealand university context. Implications of these impacts should be considered to ensure long-term sustainability of research and preparedness for future disruptions.

ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health

Large Language Models (LLMs) have recently gathered attention with the release of ChatGPT, a user-centered chatbot released by OpenAI. In this perspective article, we retrace the evolution of LLMs to understand the revolution brought by ChatGPT in the artificial intelligence (AI) field. The opportunities offered by LLMs in supporting scientific research are multiple and various models have already been tested in Natural Language Processing (NLP) tasks in this domain. The impact of ChatGPT has been huge for the general public and the research community, with many authors using the chatbot to write part of their articles and some papers even listing ChatGPT as an author. Alarming ethical and practical challenges emerge from the use of LLMs, particularly in the medical field for the potential impact on public health. Infodemic is a trending topic in public health and the ability of LLMs to rapidly produce vast amounts of text could leverage misinformation spread at an unprecedented scale, this could create an "AI-driven infodemic," a novel public health threat. Policies to contrast this phenomenon need to be rapidly elaborated, the inability to accurately detect artificial-intelligence-produced text is an unresolved issue.

Current concerns on journal article with preprint: Anesthesia and Pain Medicine perspectives

Preprints are preliminary research reports that have not yet been peer-reviewed. They have been widely adopted to promote the timely dissemination of research across many scientific fields. In August 1991, Paul Ginsparg launched an electronic bulletin board intended to serve a few hundred colleagues working in a subfield of theoretical high-energy physics, thus launching arXiv, the first and largest preprint platform. Additional preprint servers have since been implemented in different academic fields, such as BioRxiv (2013, Biology; www.biorxiv.org) and medRxiv (2019, Health Science; www.medrxiv.org). While preprint availability has made valuable research resources accessible to the general public, thus bridging the gap between academic and non-academic audiences, it has also facilitated the spread of unsupported conclusions through various media channels. Issues surrounding the preprint policies of a journal must be addressed, ultimately, by editors and include the acceptance of preprint manuscripts, allowing the citation of preprints, maintaining a double-blind peer review process, changes to the preprint's content and authors' list, scoop priorities, commenting on preprints, and preventing the influence of social media. Editors must be able to deal with these issues adequately, to maintain the scientific integrity of their journal. In this review, the history, current status, and strengths and weaknesses of preprints as well as ongoing concerns regarding journal articles with preprints are discussed. An optimal approach to preprints is suggested for editorial board members, authors, and researchers.

PreprintMatch: A tool for preprint to publication detection shows global inequities in scientific publication

Preprints, versions of scientific manuscripts that precede peer review, are growing in popularity. They offer an opportunity to democratize and accelerate research, as they have no publication costs or a lengthy peer review process. Preprints are often later published in peer-reviewed venues, but these publications and the original preprints are frequently not linked in any way. To this end, we developed a tool, PreprintMatch, to find matches between preprints and their corresponding published papers, if they exist. This tool outperforms existing techniques to match preprints and papers, both on matching performance and speed. PreprintMatch was applied to search for matches between preprints (from bioRxiv and medRxiv), and PubMed. The preliminary nature of preprints offers a unique perspective into scientific projects at a relatively early stage, and with better matching between preprint and paper, we explored questions related to research inequity. We found that preprints from low income countries are published as peer-reviewed papers at a lower rate than high income countries (39.6% and 61.1%, respectively), and our data is consistent with previous work that cite a lack of resources, lack of stability, and policy choices to explain this discrepancy. Preprints from low income countries were also found to be published quicker (178 vs 203 days) and with less title, abstract, and author similarity to the published version compared to high income countries. Low income countries add more authors from the preprint to the published version than high income countries (0.42 authors vs 0.32, respectively), a practice that is significantly more frequent in China compared to similar countries. Finally, we find that some publishers publish work with authors from lower income countries more frequently than others.

Preprints in Health Professions Education: Raising Awareness and Shifting Culture

A preprint is a version of a research manuscript posted by its authors to a preprint server before peer review. Preprints are associated with a variety of benefits, including the ability to rapidly communicate research, the opportunity for researchers to receive feedback and raise awareness of their research, and broad and unrestricted access. For early-career researchers, preprints also provide a mechanism for demonstrating research progress and productivity without the lengthy timelines of traditional journal publishing. Despite these benefits, few health professions education (HPE) research articles are deposited as preprints, suggesting that preprinting is not currently integrated into HPE culture. In this article, the authors describe preprints, their benefits and related risks, and the potential barriers that hamper their widespread use within HPE. In particular, the authors propose the barriers of discordant messaging and the lack of formal and informal education on how to deposit, critically appraise, and use preprints. To mitigate these barriers, several recommendations are proposed to facilitate preprints in becoming an accepted and encouraged component of HPE culture, allowing the field to take full advantage of this evolving form of research dissemination.

Comparison of Clinical Study Results Reported in medRxiv Preprints vs Peer-reviewed Journal Articles

IMPORTANCE

Preprints have been widely adopted to enhance the timely dissemination of research across many scientific fields. Concerns remain that early, public access to preliminary medical research has the potential to propagate misleading or faulty research that has been conducted or interpreted in error.

OBJECTIVE

To evaluate the concordance among study characteristics, results, and interpretations described in preprints of clinical studies posted to medRxiv that are subsequently published in peer-reviewed journals (preprint-journal article pairs).

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study assessed all preprints describing clinical studies that were initially posted to medRxiv in September 2020 and subsequently published in a peer-reviewed journal as of September 15, 2022.

MAIN OUTCOMES AND MEASURES

For preprint-journal article pairs describing clinical trials, observational studies, and meta-analyses that measured health-related outcomes, the sample size, primary end points, corresponding results, and overarching conclusions were abstracted and compared. Sample size and results from primary end points were considered concordant if they had exact numerical equivalence.

RESULTS

Among 1399 preprints first posted on medRxiv in September 2020, a total of 1077 (77.0%) had been published as of September 15, 2022, a median of 6 months (IQR, 3-8 months) after preprint posting. Of the 547 preprint-journal article pairs describing clinical trials, observational studies, or meta-analyses, 293 (53.6%) were related to COVID-19. Of the 535 pairs reporting sample sizes in both sources, 462 (86.4%) were concordant; 43 (58.9%) of the 73 pairs with discordant sample sizes had larger samples in the journal publication. There were 534 pairs (97.6%) with concordant and 13 pairs (2.4%) with discordant primary end points. Of the 535 pairs with numerical results for the primary end points, 434 (81.1%) had concordant primary end point results; 66 of the 101 discordant pairs (65.3%) had effect estimates that were in the same direction and were statistically consistent. Overall, 526 pairs (96.2%) had concordant study interpretations, including 82 of the 101 pairs (81.2%) with discordant primary end point results.

CONCLUSIONS AND RELEVANCE

Most clinical studies posted as preprints on medRxiv and subsequently published in peer-reviewed journals had concordant study characteristics, results, and final interpretations. With more than three-fourths of preprints published in journals within 24 months, these results may suggest that many preprints report findings that are consistent with the final peer-reviewed publications.

Why don't we share data and code? Perceived barriers and benefits to public archiving practices

The biological sciences community is increasingly recognizing the value of open, reproducible and transparent research practices for science and society at large. Despite this recognition, many researchers fail to share their data and code publicly. This pattern may arise from knowledge barriers about how to archive data and code, concerns about its reuse, and misaligned career incentives. Here, we define, categorize and discuss barriers to data and code sharing that are relevant to many research fields. We explore how real and perceived barriers might be overcome or reframed in the light of the benefits relative to costs. By elucidating these barriers and the contexts in which they arise, we can take steps to mitigate them and align our actions with the goals of open science, both as individual scientists and as a scientific community.

Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review

Objective

To assess the trustworthiness (ie, complete and consistent reporting of key methods and results between preprint and published trial reports) and impact (ie, effects of preprints on meta-analytic estimates and the certainty of evidence) of preprint trial reports during the covid-19 pandemic.

Design

Retrospective review.

Data sources

World Health Organization covid-19 database and the Living Overview of the Evidence (L-OVE) covid-19 platform by the Epistemonikos Foundation (up to 3 August 2021).

Main outcome measures

Comparison of characteristics of covid-19 trials with and without preprints, estimates of time to publication of covid-19 preprints, and description of differences in reporting of key methods and results between preprints and their later publications. For the effects of eight treatments on mortality and mechanical ventilation, the study comprised meta-analyses including preprints and excluding preprints at one, three, and six months after the first trial addressing the treatment became available either as a preprint or publication (120 meta-analyses in total, 60 of which included preprints and 60 of which excluded preprints) and assessed the certainty of evidence using the GRADE framework.

Results

Of 356 trials included in the study, 101 were only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals. The median time to publication of preprints was about six months. Key methods and results showed few important differences between trial preprints and their subsequent published reports. Apart from two (3.3%) of 60 comparisons, point estimates were consistent between meta-analyses including preprints versus those excluding preprints as to whether they indicated benefit, no appreciable effect, or harm. For nine (15%) of 60 comparisons, the rating of the certainty of evidence was different when preprints were included versus being excluded-the certainty of evidence including preprints was higher in four comparisons and lower in five comparisons.

Conclusion

No compelling evidence indicates that preprints provide results that are inconsistent with published papers. Preprints remain the only source of findings of many trials for several months-an unsuitable length of time in a health emergency that is not conducive to treating patients with timely evidence. The inclusion of preprints could affect the results of meta-analyses and the certainty of evidence. Evidence users should be encouraged to consider data from preprints.

Ensuring an Accurate Scientific Record in an Era of Pre-print Servers

BACKGROUND

Pre-prints have become an increasing part of the biomedical landscape. For example, during the first month of operation, July 2019, medRxiv received 176 submissions, one year later, in June 2020, including the first few months of COVID-19, it received 1866 submissions. The current relevant question is how to ensure an accurate scientific record given that there may be important differences between a pre-print and the peer-reviewed publication.

METHODS

Based upon the experience of the authors, conversations with editors, and a focused selective review of the literature, including the recommendations of some professional groups, a limited number of practical recommendations were formulated.

RESULTS

Peer-reviewed journals should request that authors indicate if the submitted manuscript has been posted on a pre-print server; ensure this is noted in the article if it is published by including the digital object identifier (DOI); and detail any major differences in the conclusions between the pre-print and the article. Pre-print servers should ensure that all content is marked as not peer-reviewed and be prepared to retract any pre-print that is fundamentally flawed within days that could influence clinical or public health recommendations that have therapeutic implications.

CONCLUSION

Authors, those responsible for pre-print servers, and editors of peer- reviewed journals are responsible for ensuring an accurate scientific record.

Forecasting the publication and citation outcomes of COVID-19 preprints

Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.

Assessing Open Science practices in physical activity behaviour change intervention evaluations

Objectives

Concerns on the lack of reproducibility and transparency in science have led to a range of research practice reforms, broadly referred to as 'Open Science'. The extent that physical activity interventions are embedding Open Science practices is currently unknown. In this study, we randomly sampled 100 reports of recent physical activity randomised controlled trial behaviour change interventions to estimate the prevalence of Open Science practices.

Methods

One hundred reports of randomised controlled trial physical activity behaviour change interventions published between 2018 and 2021 were identified, as used within the Human Behaviour-Change Project. Open Science practices were coded in identified reports, including: study pre-registration, protocol sharing, data, materials and analysis scripts sharing, replication of a previous study, open access publication, funding sources and conflict of interest statements. Coding was performed by two independent researchers, with inter-rater reliability calculated using Krippendorff's alpha.

Results

78 of the 100 reports provided details of study pre-registration and 41% provided evidence of a published protocol. 4% provided accessible open data, 8% provided open materials and 1% provided open analysis scripts. 73% of reports were published as open access and no studies were described as replication attempts. 93% of reports declared their sources of funding and 88% provided conflicts of interest statements. A Krippendorff's alpha of 0.73 was obtained across all coding.

Conclusion

Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention reports, whereas funding source and conflict of interest declarations are common. Future physical activity research should increase the reproducibility of their methods and results by incorporating more Open Science practices.

Comparison of COVID-19 and non-COVID-19 papers

OBJECTIVE

The need to generate evidence related to COVID-19, the acceleration of publication and peer-review process and the competition between journals may have influenced the quality of COVID-19 papers. Our objective was to compare the characteristics of COVID-19 papers against those of non-COVID-19 papers and identify the variables in which they differ.

METHOD

We conducted a journal-matched case-control study. Cases were COVID-19 papers and controls were non-COVID-19 papers published between March 2020 and January 2021. Journals belonging to five different Journal Citations Reports categories were selected. Within each selected journal, a COVID-19 paper (where there was one) and another non-COVID-19 paper were selected. Conditional logistic regression models were fitted.

RESULTS

We included 81 COVID-19 and 143 non-COVID-19 papers. Descriptive observational studies and analytical observational studies had, respectively, a 55-fold (odds ratio [OR]: 55.12; 95% confidence interval [95%CI]: 7.41-409.84) and 19-fold (OR: 19.28; 95%CI: 3.09-120.31) higher likelihood of being COVID-19 papers, respectively, and also a higher probability of having a smaller sample size (OR: 7.15; 95%CI: 2.33-21.94). COVID-19 papers had a higher probability of being cited since their publication (OR: 4.97; 95%CI: 1.63-15.10).

CONCLUSIONS

The characteristics of COVID-19 papers differed from those of non-COVID-19 papers published in the first months of the pandemic. In order to ensure the publication of good scientific evidence the quality of COVID-19-papers should be preserved.

Inconsistent and incomplete retraction of published research: A cross-sectional study on Covid-19 retractions and recommendations to mitigate risks for research, policy and practice

BACKGROUND

Retraction of published research can reduce the dissemination of incorrect or misleading information, but concerns have been raised about the clarity and rigor of the retraction process. Failure to clearly and consistently retract research has several risks, for example discredited or erroneous research may inform health research studies (e.g. clinical trials), policies and practices, potentially rendering these unreliable.

OBJECTIVE

To investigate consistency and clarity of research retraction, based on a case study of retracted Covid-19 research.

STUDY DESIGN

A cross-sectional study of retracted Covid-19 articles reporting empirical research findings, based on searches of Medline, Embase and Scopus on 10th July and 19th December 2020.

KEY RESULTS

We included 46 retracted Covid-19 articles. The number eligible for inclusion nearly doubled, from 26 to 46, in five months. Most articles (67%) were retracted from scientific journals and the remainder from preprint servers. Key findings: (1) reasons for retraction were not reported in 33% (15/46) of cases; (2) time from publication to retraction could not be determined in 43% (20/46) of cases; (3) More than half (59%) of retracted Covid-19 articles (27/46) remained available as original unmarked electronic documents after retraction (33% as full text and 26% as an abstract only). Sources of articles post-retraction were preprint servers, ResearchGate and, less commonly, websites including PubMed Central and the World Health Organization. A retracted journal article which controversially claimed a link between 5G technology and Covid-19 remains available in its original full text from at least 60 different websites.

CONCLUSIONS

The retraction process is inconsistent and often ambiguous, with more than half of retracted Covid-19 research articles remaining available, unmarked, from a wide range of online sources. There is an urgent need to improve guidance on the retraction process and to extend this to cover preprint servers. We provide structured recommendations to address these concerns and to reduce the risks that arise when retracted research is inappropriately cited.

MiR-200c-3p expression may be associated with worsening of the clinical course of patients with COVID-19

COVID-19 represents a public health emergency, whose mechanism of which is not fully understood. It is speculated that microRNAs may play a crucial role in host cells after infection by SARS-CoV-2. Thus, our study aimed to analyze the expression of miR-200c-3p in saliva samples from patients with COVID-19. One handred eleven samples from patients with COVID-19 were divided into 4 groups. Group I: 39 patients negative for Covid-19; Group II: 37 positive and symptomatic patients, with no indication of hospitalization; Group III: 21 patients with respiratory disorders (hospitalized); Group IV: 14 patients with severe conditions (oxygen therapy). The expression levels of miR-200c-3p were determined using qPCR. We found greater expression of miR-200c-3p in patients in group IV (p<0.0001), and also verified that patients aged ≥42 years had a higher expression of this miR (p=0.013). Logistic regression analysis revealed that the expression of miR-200c-3p and systemic arterial hypertension are factors independently associated with patients in group IV (p<0.0001). Our results suggest that miR-200c-3p is a predictor of severity independent of COVID-19 risk factors, which could represent a way of screening patients affected by SARS-CoV-2.

Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation

OBJECTIVE

To compare results reporting and the presence of spin in COVID-19 study preprints with their finalised journal publications.

DESIGN

Cross-sectional study.

SETTING

International medical literature.

PARTICIPANTS

Preprints and final journal publications of 67 interventional and observational studies of COVID-19 treatment or prevention from the Cochrane COVID-19 Study Register published between 1 March 2020 and 30 October 2020.

MAIN OUTCOME MEASURES

Study characteristics and discrepancies in (1) results reporting (number of outcomes, outcome descriptor, measure, metric, assessment time point, data reported, reported statistical significance of result, type of statistical analysis, subgroup analyses (if any), whether outcome was identified as primary or secondary) and (2) spin (reporting practices that distort the interpretation of results so they are viewed more favourably).

RESULTS

Of 67 included studies, 23 (34%) had no discrepancies in results reporting between preprints and journal publications. Fifteen (22%) studies had at least one outcome that was included in the journal publication, but not the preprint; eight (12%) had at least one outcome that was reported in the preprint only. For outcomes that were reported in both preprints and journals, common discrepancies were differences in numerical values and statistical significance, additional statistical tests and subgroup analyses and longer follow-up times for outcome assessment in journal publications.At least one instance of spin occurred in both preprints and journals in 23/67 (34%) studies, the preprint only in 5 (7%), and the journal publications only in 2 (3%). Spin was removed between the preprint and journal publication in 5/67 (7%) studies; but added in 1/67 (1%) study.

CONCLUSIONS

The COVID-19 preprints and their subsequent journal publications were largely similar in reporting of study characteristics, outcomes and spin. All COVID-19 studies published as preprints and journal publications should be critically evaluated for discrepancies and spin.

A Review of COVID-19-Related Publications and Lag Times During the First Six Months of the Year 2020

INTRODUCTION

Considering the need for information regarding approaches to prevention and treatment of coronavirus disease 2019 (COVID-19), we sought to determine publication lag times of COVID-19-related original research articles published in top general medicine and emergency medicine (EM) journals. We further sought to characterize the types of COVID-19 publications within these journals.

METHODS

We reviewed 125 top-ranked general medicine journals and 20 top-ranked EM-specific journals for COVID-19-related publications. We abstracted article titles and manuscript details for each COVID-19-related article published between January 1-June 30, 2020, and categorized articles as one of the following: original research; case report; review; or commentary. We abstracted data for preprint publications over the same time period and determined whether articles from the general medicine and EM journals had been previously published as preprint articles. Our primary outcomes were the following: 1) lag time (days) between global cumulative World Health Organization (WHO)-confirmed cases of COVID-19 and publications; 2) lag times between preprint article publication and peer-reviewed journal publication; and 3) lag times between submission and publication in peer-reviewed journals. Our secondary outcome was to characterize COVID-19-related publications.

RESULTS

The first original research publications appeared in a general medicine journal 20 days and in an EM journal 58 days after the first WHO-confirmed case of COVID-19. We found median and mean lag times between preprint publications and journal publications of 32 days (19, 49) and 36 days (22) for general medicine journals, and 26 days (16, 36) and 25 days (13) for EM journals. Median and mean lag times between submission and publication were 30 days (19, 45) and 35 days (13) for general medicine journals, and 23 days (11, 39) and 27 days (19) for EM journals. Of 2530 general medicine journal articles and 351 EM journal articles, 28% and 23.6% were original research. We noted substantial closing of the preprint to peer-reviewed publication (160 days pre-pandemic) and peer-reviewed journal submission to publication (194 days pre-pandemic) lag times for COVID-19 manuscripts.

CONCLUSION

We found a rapid and robust response with shortened publication lag times to meet the need for the publication of original research and other vital medical information related to COVID-19 during the first six months of 2020.

COVID-19 Rapid Review cross-publisher initiative: What we have learned and what we are going to do next

The global crisis sparked collaboration between publishers and service providers to successfully address an immediate problem and demonstrated the possibility for future partnerships.

Encouraging experts to join a reviewer pool and quickly review the preprint and journal submissions, we were able to publish COVID‐19 research more quickly.

The initiative confirmed little author uptake of inter‐publisher journal transfer option.

The collaboration showed wide consensus on open science practices which will ensure faster and more reliable research findings.

Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals

This cross-sectional study examines the concordance between clinical studies posted as preprints and subsequently published in high-impact journals, including key study characteristics, reported results, and study interpretations.

Antimicrobial resistance and COVID-19: Intersections and implications

Before the coronavirus 2019 (COVID-19) pandemic began, antimicrobial resistance (AMR) was among the top priorities for global public health. Already a complex challenge, AMR now needs to be addressed in a changing healthcare landscape. Here, we analyse how changes due to COVID-19 in terms of antimicrobial usage, infection prevention, and health systems affect the emergence, transmission, and burden of AMR. Increased hand hygiene, decreased international travel, and decreased elective hospital procedures may reduce AMR pathogen selection and spread in the short term. However, the opposite effects may be seen if antibiotics are more widely used as standard healthcare pathways break down. Over 6 months into the COVID-19 pandemic, the dynamics of AMR remain uncertain. We call for the AMR community to keep a global perspective while designing finely tuned surveillance and research to continue to improve our preparedness and response to these intersecting public health challenges.