The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape

A new tool for evaluating health equity in academic journals; the Diversity Factor

Current methods to evaluate a journal's impact rely on the downstream citation mapping used to generate the Impact Factor. This approach is a fragile metric prone to being skewed by outlier values and does not speak to a researcher's contribution to furthering health outcomes for all populations. Therefore, we propose the implementation of a Diversity Factor to fulfill this need and supplement the current metrics. It is composed of four key elements: dataset properties, author country, author gender and departmental affiliation. Due to the significance of each individual element, they should be assessed independently of each other as opposed to being combined into a simplified score to be optimized. Herein, we discuss the necessity of such metrics, provide a framework to build upon, evaluate the current landscape through the lens of each key element and publish the findings on a freely available website that enables further evaluation. The OpenAlex database was used to extract the metadata of all papers published from 2000 until August 2022, and Natural language processing was used to identify individual elements. Features were then displayed individually on a static dashboard developed using TableauPublic, which is available at www.equitablescience.com. In total, 130,721 papers were identified from 7,462 journals where significant underrepresentation of LMIC and Female authors was demonstrated. These findings are pervasive and show no positive correlation with the Journal's Impact Factor. The systematic collection of the Diversity Factor concept would allow for more detailed analysis, highlight gaps in knowledge, and reflect confidence in the translation of related research. Conversion of this metric to an active pipeline would account for the fact that how we define those most at risk will change over time and quantify responses to particular initiatives. Therefore, continuous measurement of outcomes across groups and those investigating those outcomes will never lose importance. Moving forward, we encourage further revision and improvement by diverse author groups in order to better refine this concept.

Raising concerns on questionable ethics approvals - a case study of 456 trials from the Institut Hospitalo-Universitaire Méditerranée Infection

BACKGROUND

The practice of clinical research is strictly regulated by law. During submission and review processes, compliance of such research with the laws enforced in the country where it was conducted is not always correctly filled in by the authors or verified by the editors. Here, we report a case of a single institution for which one may find hundreds of publications with seemingly relevant ethical concerns, along with 10 months of follow-up through contacts with the editors of these articles. We thus argue for a stricter control of ethical authorization by scientific editors and we call on publishers to cooperate to this end.

METHODS

We present an investigation of the ethics and legal aspects of 456 studies published by the IHU-MI (Institut Hospitalo-Universitaire Méditerranée Infection) in Marseille, France.

RESULTS

We identified a wide range of issues with the stated research authorization and ethics of the published studies with respect to the Institutional Review Board and the approval presented. Among the studies investigated, 248 were conducted with the same ethics approval number, even though the subjects, samples, and countries of investigation were different. Thirty-nine (39) did not even contain a reference to the ethics approval number while they present research on human beings. We thus contacted the journals that published these articles and provide their responses to our concerns. It should be noted that, since our investigation and reporting to journals, PLOS has issued expressions of concerns for several publications we analyze here.

CONCLUSION

This case presents an investigation of the veracity of ethical approval, and more than 10 months of follow-up by independent researchers. We call for stricter control and cooperation in handling of these cases, including editorial requirement to upload ethical approval documents, guidelines from COPE to address such ethical concerns, and transparent editorial policies and timelines to answer such concerns. All supplementary materials are available.

Science Communication as a Collective Intelligence Endeavor: A Manifesto and Examples for Implementation

Effective science communication is challenging when scientific messages are informed by a continually updating evidence base and must often compete against misinformation. We argue that we need a new program of science communication as collective intelligence-a collaborative approach, supported by technology. This would have four key advantages over the typical model where scientists communicate as individuals: scientific messages would be informed by (a) a wider base of aggregated knowledge, (b) contributions from a diverse scientific community, (c) participatory input from stakeholders, and (d) better responsiveness to ongoing changes in the state of knowledge.

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We "Undercooking" Our Science?

BACKGROUND

Although biomedical preprint servers have grown rapidly over the past several years, the harm to patient health and safety remains a major concern among several scientific communities. Despite previous studies examining the role of preprints during the Coronavirus-19 pandemic, there is limited information characterizing their impact on scientific communication in orthopaedic surgery.

QUESTIONS/PURPOSES

(1) What are the characteristics (subspecialty, study design, geographic origin, and proportion of publications) of orthopaedic articles on three preprint servers? (2) What are the citation counts, abstract views, tweets, and Altmetric score per preprinted article and per corresponding publication?

METHODS

Three of the largest preprint servers (medRxiv, bioRxiv, and Research Square) with a focus on biomedical topics were queried for all preprinted articles published between July 26, 2014, and September 1, 2021, using the following search terms: "orthopaedic," "orthopedic," "bone," "cartilage," "ligament," "tendon," "fracture," "dislocation," "hand," "wrist," "elbow," "shoulder," "spine," "spinal," "hip," "knee," "ankle," and "foot." Full-text articles in English related to orthopaedic surgery were included, while nonclinical studies, animal studies, duplicate studies, editorials, abstracts from conferences, and commentaries were excluded. A total of 1471 unique preprints were included and further characterized in terms of the orthopaedic subspecialty, study design, date posted, and geographic factors. Citation counts, abstract views, tweets, and Altmetric scores were collected for each preprinted article and the corresponding publication of that preprint in an accepting journal. We ascertained whether a preprinted article was published by searching title keywords and the corresponding author in three peer-reviewed article databases (PubMed, Google Scholar, and Dimensions) and confirming that the study design and research question matched.

RESULTS

The number of orthopaedic preprints increased from four in 2017 to 838 in 2020. The most common orthopaedic subspecialties represented were spine, knee, and hip. From 2017 to 2020, the cumulative counts of preprinted article citations, abstract views, and Altmetric scores increased. A corresponding publication was identified in 52% (762 of 1471) of preprints. As would be expected, because preprinting is a form of redundant publication, published articles that are also preprinted saw greater abstract views, citations, and Altmetric scores on a per-article basis.

CONCLUSION

Although preprints remain an extremely small proportion of all orthopaedic research, our findings suggest that nonpeer-reviewed, preprinted orthopaedic articles are being increasingly disseminated. These preprinted articles have a smaller academic and public footprint than their published counterparts, but they still reach a substantial audience through infrequent and superficial online interactions, which are far from equivalent to the engagement facilitated by peer review. Furthermore, the sequence of preprint posting and journal submission, acceptance, and publication is unclear based on the information available on these preprint servers. Thus, it is difficult to determine whether the metrics of preprinted articles are attributable to preprinting, and studies such as the present analysis will tend to overestimate the apparent impact of preprinting. Despite the potential for preprint servers to function as a venue for thoughtful feedback on research ideas, the available metrics data for these preprinted articles do not demonstrate the meaningful engagement that is achieved by peer review in terms of the frequency or depth of audience feedback.

CLINICAL RELEVANCE

Our findings highlight the need for safeguards to regulate research dissemination through preprint media, which has never been shown to benefit patients and should not be considered as evidence by clinicians. Clinician-scientists and researchers have the most important responsibility of protecting patients from the harm of potentially inaccurate biomedical science and therefore must prioritize patient needs first by uncovering scientific truths through the evidence-based processes of peer review, not preprinting. We recommend all journals publishing clinical research adopt the same policy as Clinical Orthopaedics and Related Research® , The Bone & Joint Journal, The Journal of Bone and Joint Surgery, and the Journal of Orthopaedic Research , removing any papers posted to preprint servers from consideration.

The Penn Medicine COVID-19 Therapeutics Committee-Reflections on a Model for Rapid Evidence Review and Dynamic Practice Recommendations During a Public Health Emergency

The Penn Medicine COVID-19 Therapeutics Committee-an interspecialty, clinician-pharmacist, and specialist-front line primary care collaboration-has served as a forum for rapid evidence review and the production of dynamic practice recommendations during the 3-year coronavirus disease 2019 public health emergency. We describe the process by which the committee went about its work and how it navigated specific challenging scenarios. Our target audiences are clinicians, hospital leaders, public health officials, and researchers invested in preparedness for inevitable future threats. Our objectives are to discuss the logistics and challenges of forming an effective committee, undertaking a rapid evidence review process, aligning evidence-based guidelines with operational realities, and iteratively revising recommendations in response to changing pandemic data. We specifically discuss the arc of evidence for corticosteroids; the noble beginnings and dangerous misinformation end of hydroxychloroquine and ivermectin; monoclonal antibodies and emerging viral variants; and patient screening and safety processes for tocilizumab, baricitinib, and nirmatrelvir-ritonavir.

Characterization of Comments About bioRxiv and medRxiv Preprints

Importance

Preprints have been increasingly used in biomedical science, and a key feature of many platforms is public commenting. The content of these comments, however, has not been well studied, and it is unclear whether they resemble those found in journal peer review.

Objective

To describe the content of comments on the bioRxiv and medRxiv preprint platforms.

Design, Setting, and Participants

In this cross-sectional study, preprints posted on the bioRxiv and medRxiv platforms in 2020 were accessed through each platform's application programming interface on March 29, 2021, and a random sample of preprints containing between 1 and 20 comments was evaluated independently by 3 evaluators using an instrument to assess their features and general content.

Main Outcome and Measures

The numbers and percentages of comments from authors or nonauthors were assessed, and the comments from nonauthors were assessed for content. These nonauthor comments were assessed to determine whether they included compliments, criticisms, corrections, suggestions, or questions, as well as their topics (eg, relevance, interpretation, and methods). Nonauthor comments were also analyzed to determine whether they included references, provided a summary of the findings, or questioned the preprint's conclusions.

Results

Of 52 736 preprints, 3850 (7.3%) received at least 1 comment (mean [SD] follow-up, 7.5 [3.6] months), and the 1921 assessed comments (from 1037 preprints) had a median length of 43 words (range, 1-3172 words). The criticisms, corrections, or suggestions present in 694 of 1125 comments (61.7%) were the most prevalent content, followed by compliments (n = 428 [38.0%]) and questions (n = 393 [35.0%]). Criticisms usually regarded interpretation (n = 286), methodological design (n = 267), and data collection (n = 238), while compliments were mainly about relevance (n = 111) and implications (n = 72).

Conclusions and Relevance

In this cross-sectional study of preprint comments, topics commonly associated with journal peer review were frequent. However, only a small percentage of preprints posted on the bioRxiv and medRxiv platforms in 2020 received comments on these platforms. A clearer taxonomy of peer review roles would help to describe whether postpublication peer review fulfills them.

Completeness and consistency of primary outcome reporting in COVID-19 publications in the early pandemic phase: a descriptive study

BACKGROUND

The COVID-19 pandemic saw a steep increase in the number of rapidly published scientific studies, especially early in the pandemic. Some have suggested COVID-19 trial reporting is of lower quality than typical reports, but there is limited evidence for this in terms of primary outcome reporting. The objective of this study was to assess the prevalence of completely defined primary outcomes reported in registry entries, preprints, and journal articles, and to assess consistent primary outcome reporting between these sources.

METHODS

This is a descriptive study of a cohort of registered interventional clinical trials for the treatment and prevention of COVID-19, drawn from the DIssemination of REgistered COVID-19 Clinical Trials (DIRECCT) study dataset. The main outcomes are: 1) Prevalence of complete primary outcome reporting; 2) Prevalence of consistent primary outcome reporting between registry entry and preprint as well as registry entry and journal article pairs.

RESULTS

We analyzed 87 trials with 116 corresponding publications (87 registry entries, 53 preprints and 63 journal articles). All primary outcomes were completely defined in 47/87 (54%) registry entries, 31/53 (58%) preprints and 44/63 (70%) journal articles. All primary outcomes were consistently reported in 13/53 (25%) registry-preprint pairs and 27/63 (43%) registry-journal article pairs. No primary outcome was specified in 13/53 (25%) preprints and 8/63 (13%) journal articles. In this sample, complete primary outcome reporting occurred more frequently in trials with vs. without involvement of pharmaceutical companies (76% vs. 45%), and in RCTs vs. other study designs (68% vs. 49%). The same pattern was observed for consistent primary outcome reporting (with vs. without pharma: 56% vs. 12%, RCT vs. other: 43% vs. 22%).

CONCLUSIONS

In COVID-19 trials in the early phase of the pandemic, all primary outcomes were completely defined in 54%, 58%, and 70% of registry entries, preprints and journal articles, respectively. Only 25% of preprints and 43% of journal articles reported primary outcomes consistent with registry entries.

Habits and perceptions regarding open science by researchers from Spanish institutions

The article describes the results of the online survey on open science (OS) carried out on researchers affiliated with universities and Spanish research centres and focused on open access to scientific publications, the publication process, the management of research data and the review of open articles. The main objective was to identify the perception and habits of researchers with regard to practices closely linked to open science and the scientific value added is that offers an in-depth picture of researchers as one of the main actors to whom this transformation and implementation of open science will fall. It focuses on the different aspects of OS: open access, open data, publication process and open review in order to identify habits and perceptions. This is to make possible an implementation of the OS movement. The survey was carried out among researchers who had published in the years 2020-2021, according to data obtained from WoS. It was emailed to a total of 8,188 researchers and obtained a total of 666 responses, of which 554 were complete, the rest being forms with some questions unanswered. The main results showed that open access still requires the diffusion of practices and services provided by the institution, as well as training (library or equivalent service) and institutional support from the competent authorities (vice rectors or equivalent) in specific aspects such as data management. In the case of data, around 50% of respondents stated they had stored data in a repository, and of all the options, the most frequently given was that of an institutional repository, followed by a discipline repository. Among the main reasons for doing this, we found transparency, visibility of data and the ability to validate results. For those who stated they had never stored data, the most frequent reasons for not having done so were privacy and confidentiality, the lack of a mandated data policy or a lack of knowledge of how to do it. In terms of open peer review, participants mentioned a certain reticence to the opening of evaluations due to potential conflicts of interest that may arise or because lower-quality content might be accepted in order to avoid conflicts. In addition, the hierarchical structure of senior researcher versus junior researcher might affect reviews. The main conclusions indicate a need for persuasion of OA to take place; APCs are an economic barrier rather than the main criterion for journal selection; OPR practices may seem innovative and emerging; scientific and evaluation policies seem to have a clear effect on the behaviour of researchers; researchers state that they share research data more for reasons of persuasion than out of obligation. Researchers do question the pathways or difficulties that may arise on a day-to-day basis and seem aware that we are undergoing change, where academic evaluation or policies related to open science, its implementation and habits among researchers may change. In this sense, more and better support is needed on the part of institutions and faculty support services.

The impact of COVID-19 on open access publishing in radiology and nuclear medicine: an in-depth analysis

In response to the COVID-19 pandemic, numerous initiatives have been implemented to ensure open access availability of COVID-19-related articles to make published articles accessible for anyone. This study aimed to assess the impact of the COVID-19 pandemic on open-access publishing in radiology and nuclear medicine. We conducted a comprehensive analysis of articles and reviews published in these fields during the COVID-19 publishing era using the Web of Science database. We analyzed several indicators between COVID-19 and non-COVID-19 related articles, including the number and percentage of open-access articles, the top ten cited articles, and the number of reviews. In total, 67,100 articles were published in radiology and nuclear medicine between January 2020 and June 2022. Among those, more than half (51.1%) were open-access articles. Among these publications, 2,336 were COVID-19-related, and 64,764 were non-COVID-19-related. However, articles related to COVID-19 had an open access rate of 91.5%, compared to only 49.6% of the non-COVID-19-related articles. Moreover, COVID-19-related articles had a higher percentage of highly cited and hot papers compared to articles not related to COVID-19. Moreover, most highly cited studies were related to chest computerized tomography (CT) scan findings in COVID-19 patients. The findings emphasize the significant proportion of open access COVID-19-related publications in radiology and nuclear medicine, facilitating widespread and timely access to everyone.

Global COVID-19 Policy Engagement With Scientific Research Information: Altmetric Data Study

BACKGROUND

Previous studies on COVID-19 scholarly articles have primarily focused on bibliometric characteristics, neglecting the identification of institutional actors that cite recent scientific contributions related to COVID-19 in the policy domain, and their locations.

OBJECTIVE

The purpose of this study was to assess the online citation network and knowledge structure of COVID-19 research across policy domains over 2 years from January 2020 to January 2022, with a particular emphasis on geographical frequency. Two research questions were addressed. The first question was related to who has been the most active in policy engagement with science and research information sharing during the COVID-19 pandemic, particularly in terms of countries and organization types. The second question was related to whether there are significant differences in the types of coronavirus research shared among countries and continents.

METHODS

The Altmetric database was used to collect policy report citations of scientific articles for 3 topic terms (COVID-19, COVID-19 vaccine, and COVID-19 variants). Altmetric provides the URLs of policy agencies that have cited COVID-19 research. The scientific articles used for Altmetric citations are extracted from journals indexed by PubMed. The numbers of COVID-19, COVID-19 vaccine, and COVID-19 variant research outputs between January 1, 2020, and January 31, 2022, were 216,787, 16,748, and 2777, respectively. The study examined the frequency of citations based on policy institutional domains, such as intergovernmental organizations, national and domestic governmental organizations, and nongovernmental organizations (think tanks and academic institutions).

RESULTS

The World Health Organization (WHO) stood out as the most notable institution citing COVID-19-related research outputs. The WHO actively sought and disseminated information regarding the COVID-19 pandemic. The COVID-19 vaccine citation network exhibited the most extensive connections in terms of degree centrality, 2-local eigenvector centrality, and eigenvector centrality among the 3 key terms. The Netherlands, the United States, the United Kingdom, and Australia were the countries that sought and shared the most information on COVID-19 vaccines, likely due to their high numbers of COVID-19 cases. Developing nations, although gaining quicker access to COVID-19 vaccine information, appeared to be relatively isolated from the enriched COVID-19 pandemic content in the global network.

CONCLUSIONS

The global scientific network ecology during the COVID-19 pandemic revealed distinct types of links primarily centered around the WHO. Western countries demonstrated effective networking practices in constructing these networks. The prominent position of the key term "COVID-19 vaccine" demonstrates that nation-states align with global authority regardless of their national contexts. In summary, the citation networking practices of policy agencies have the potential to uncover the global knowledge distribution structure as a proxy for the networking strategy employed during a pandemic.

The Preprint Club: A blueprint for community-based peer review: A blueprint for community-based peer review

Cross-institutional journal clubs focused on preprints are a new approach to community-based peer review and allow ERCs to gain experience.

Communicating science in the COVID-19 news in the UK during Omicron waves: exploring representations of nature of science with epistemic network analysis

News media plays a vital role in communicating scientific evidence to the public during the COVID-19 pandemic. Such communication is important for convincing the public to follow social distancing guidelines and to respond to health campaigns such as vaccination programmes. However, newspapers were criticised that they focus on the socio-political perspective of science, without explaining the nature of scientific works behind the government's decisions. This paper examines the connections of the nature of science categories in the COVID-19 era by four local newspapers in the United Kingdom between November 2021 to February 2022. Nature of science refers to different aspects of how science works such as aims, values, methods and social institutions of science. Considering the news media may mediate public information and perception of scientific stories, it is relevant to ask how the various British newspapers covered aspects of science during the pandemic. In the period explored, Omicron variant was initially a variant of concern, and an increasing number of scientific evidence showed that the less severity of this variant might move the country from pandemic to endemic. We explored how news articles communicate public health information by addressing how science works during the period when Omicron variants surge. A novel discourse analysis approach, epistemic network analysis is used to characterise the frequency of connections of categories of the nature of science. The connection between political factors and the professional activities of scientists, as well as that with scientific practices are more apparent in left-populated and centralist outlets than in right-populated news outlets. Among four news outlets across the political spectrum, a left-populated newspaper, the Guardian, is not consistent in representing relations of different aspects of the nature of scientific works across different stages of the public health crisis. Inconsistency of addressing aspects of scientific works and a downplay of the cognitive-epistemic nature of scientific works likely lead to failure in trust and consumption of scientific knowledge by the public in the healthcare crisis.

Are COVID-19 systematic reviews up to date and can we tell? A cross-sectional study

BACKGROUND

COVID-19 led to a rapid acceleration in the number of systematic reviews. Readers need to know how up to date evidence is when selecting reviews to inform decisions. This cross-sectional study aimed to evaluate how easily the currency of COVID-19 systematic reviews published early in the pandemic could be determined and how up to date these reviews were at the time of publication.

METHODS

We searched for systematic reviews and meta-analyses relevant to COVID-19 added to PubMed in July 2020 and January 2021, including any that were first published as preprints. We extracted data on the date of search, number of included studies, and date first published online. For the search date, we noted the format of the date and where in the review this was reported. A sample of non-COVID-19 systematic reviews from November 2020 served as a comparator.

RESULTS

We identified 246 systematic reviews on COVID-19. In the abstract of these reviews, just over half (57%) reported the search date (day/month/year or month/year) while 43% failed to report any date. When the full text was considered, the search date was missing from 6% of reviews. The median time from last search to publication online was 91 days (IQR 63-130). Time from search to publication was similar for the subset of 15 rapid or living reviews (92 days) but shorter for the 29 reviews published as preprints (37 days). The median number of studies or publications included per review was 23 (IQR 12-40). In the sample of 290 non-COVID SRs, around two-thirds (65%) reported the search date while a third (34%) did not include any date in the abstract. The median time from search to publication online was 253 days (IQR 153-381) and each review included a median of 12 studies (IQR 8-21).

CONCLUSIONS

Despite the context of the pandemic and the need to easily ascertain the currency of systematic reviews, reporting of the search date information for COVID-19 reviews was inadequate. Adherence to reporting guidelines would improve the transparency and usefulness of systematic reviews to users.

The impact of COVID-19 on clinical research at Australian and New Zealand universities: A qualitative study

Background

The COVID-19 pandemic resulted in the implementation of social distancing measures, travel restrictions, and infection control measures that introduced a myriad of disruptions in the conduct of clinical research worldwide. As a result, many aspects of clinical research were variably impacted.

Aim

To explore the impact of the first 18 months of the COVID-19 pandemic on clinical research across accredited nursing, pharmacy, and medicine program providers in Australian and New Zealand universities.

Methods

Representatives from all program providers across Australian and New Zealand universities, with publicly available contact information, were invited to participate in this qualitative study, whereby semi-structured interviews were completed with participants who held senior research or leadership positions within their institution. Interviews were transcribed verbatim and inductively analysed using thematic content analysis.

Findings

Interviews were conducted with 16 participants between August and October 2021. Two major themes were identified (Immediate Research Impact and Broader Research Impact) with six subthemes: Prioritisation, Continuation, and Dissemination of Research; Modifications to Research; Funding and Changes to Research Focus; Collaboration; Research Workforce; Context-specific Impacts.

Discussion

The impact on clinical research in Australian and New Zealand universities included changes to data collection methods, a perceived decreased quality of research, changes to collaboration, neglect of basic disease research, and loss of the research workforce.

Conclusion

This study highlights the impact of the COVID-19 pandemic on clinical research within the Australian and New Zealand university context. Implications of these impacts should be considered to ensure long-term sustainability of research and preparedness for future disruptions.

Changing word meanings in biomedical literature reveal pandemics and new technologies

While we often think of words as having a fixed meaning that we use to describe a changing world, words are also dynamic and changing. Scientific research can also be remarkably fast-moving, with new concepts or approaches rapidly gaining mind share. We examined scientific writing, both preprint and pre-publication peer-reviewed text, to identify terms that have changed and examine their use. One particular challenge that we faced was that the shift from closed to open access publishing meant that the size of available corpora changed by over an order of magnitude in the last two decades. We developed an approach to evaluate semantic shift by accounting for both intra- and inter-year variability using multiple integrated models. This analysis revealed thousands of change points in both corpora, including for terms such as 'cas9', 'pandemic', and 'sars'. We found that the consistent change-points between pre-publication peer-reviewed and preprinted text are largely related to the COVID-19 pandemic. We also created a web app for exploration that allows users to investigate individual terms ( https://greenelab.github.io/word-lapse/ ). To our knowledge, our research is the first to examine semantic shift in biomedical preprints and pre-publication peer-reviewed text, and provides a foundation for future work to understand how terms acquire new meanings and how peer review affects this process.

ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health

Large Language Models (LLMs) have recently gathered attention with the release of ChatGPT, a user-centered chatbot released by OpenAI. In this perspective article, we retrace the evolution of LLMs to understand the revolution brought by ChatGPT in the artificial intelligence (AI) field. The opportunities offered by LLMs in supporting scientific research are multiple and various models have already been tested in Natural Language Processing (NLP) tasks in this domain. The impact of ChatGPT has been huge for the general public and the research community, with many authors using the chatbot to write part of their articles and some papers even listing ChatGPT as an author. Alarming ethical and practical challenges emerge from the use of LLMs, particularly in the medical field for the potential impact on public health. Infodemic is a trending topic in public health and the ability of LLMs to rapidly produce vast amounts of text could leverage misinformation spread at an unprecedented scale, this could create an "AI-driven infodemic," a novel public health threat. Policies to contrast this phenomenon need to be rapidly elaborated, the inability to accurately detect artificial-intelligence-produced text is an unresolved issue.

Searching for evidence in public health emergencies: a white paper of best practices

Objectives

Information professionals have supported medical providers, administrators and decision-makers, and guideline creators in the COVID-19 response. Searching COVID-19 literature presented new challenges, including the volume and heterogeneity of literature and the proliferation of new information sources, and exposed existing issues in metadata and publishing. An expert panel developed best practices, including recommendations, elaborations, and examples, for searching during public health emergencies.

Methods

Project directors and advisors developed core elements from experience and literature. Experts, identified by affiliation with evidence synthesis groups, COVID-19 search experience, and nomination, responded to an online survey to reach consensus on core elements. Expert participants provided written responses to guiding questions. A synthesis of responses provided the foundation for focus group discussions. A writing group then drafted the best practices into a statement. Experts reviewed the statement prior to dissemination.

Results

Twelve information professionals contributed to best practice recommendations on six elements: core resources, search strategies, publication types, transparency and reproducibility, collaboration, and conducting research. Underlying principles across recommendations include timeliness, openness, balance, preparedness, and responsiveness.

Conclusions

The authors and experts anticipate the recommendations for searching for evidence during public health emergencies will help information specialists, librarians, evidence synthesis groups, researchers, and decision-makers respond to future public health emergencies, including but not limited to disease outbreaks. The recommendations complement existing guidance by addressing concerns specific to emergency response. The statement is intended as a living document. Future revisions should solicit input from a broader community and reflect conclusions of meta-research on COVID-19 and health emergencies.

COVID-19 retracted publications on retraction watch: A systematic survey of their pre-prints and citations

Background

Studies related to the coronavirus disease 2019 (COVID-19) were frequently published as pre-prints prior to undergoing peer-review. However, several publications were later retracted due to ethical concerns or study misconduct. Although these studies have been retracted, the availability of their corresponding pre-prints has never been formally investigated, and may result in the spread of misinformation if they are being used to inform decision-making.

Methods

Our objective was to conduct a systematic survey of retracted COVID-19 publications listed on the Retraction Watch database as of August 15th, 2021. We assessed the availability of corresponding pre-prints for retracted publications, and documented the number of citations and online views.

Results

Our study included 140 retracted COVID-19 publications, and we could not retrieve corresponding pre-prints for 132 retracted publications in our study (94%). Although we were unable to find the majority of pre-prints, they had already been disseminated, with a maximal citation count of 593 and Altmetric score of 558,928.

Conclusion

While it is reassuring that most corresponding pre-prints could not be retrieved, our study highlights the need for online platforms and journals to employ quality assurance methods to prevent the spread of misinformation through citation of retracted papers.

Current concerns on journal article with preprint: Anesthesia and Pain Medicine perspectives

Preprints are preliminary research reports that have not yet been peer-reviewed. They have been widely adopted to promote the timely dissemination of research across many scientific fields. In August 1991, Paul Ginsparg launched an electronic bulletin board intended to serve a few hundred colleagues working in a subfield of theoretical high-energy physics, thus launching arXiv, the first and largest preprint platform. Additional preprint servers have since been implemented in different academic fields, such as BioRxiv (2013, Biology; www.biorxiv.org) and medRxiv (2019, Health Science; www.medrxiv.org). While preprint availability has made valuable research resources accessible to the general public, thus bridging the gap between academic and non-academic audiences, it has also facilitated the spread of unsupported conclusions through various media channels. Issues surrounding the preprint policies of a journal must be addressed, ultimately, by editors and include the acceptance of preprint manuscripts, allowing the citation of preprints, maintaining a double-blind peer review process, changes to the preprint's content and authors' list, scoop priorities, commenting on preprints, and preventing the influence of social media. Editors must be able to deal with these issues adequately, to maintain the scientific integrity of their journal. In this review, the history, current status, and strengths and weaknesses of preprints as well as ongoing concerns regarding journal articles with preprints are discussed. An optimal approach to preprints is suggested for editorial board members, authors, and researchers.

Inter-rater reliability of the Infectious Disease Modeling Reproducibility Checklist (IDMRC) as applied to COVID-19 computational modeling research

Background

Infectious disease computational modeling studies have been widely published during the coronavirus disease 2019 (COVID-19) pandemic, yet they have limited reproducibility. Developed through an iterative testing process with multiple reviewers, the Infectious Disease Modeling Reproducibility Checklist (IDMRC) enumerates the minimal elements necessary to support reproducible infectious disease computational modeling publications. The primary objective of this study was to assess the reliability of the IDMRC and to identify which reproducibility elements were unreported in a sample of COVID-19 computational modeling publications.

Methods

Four reviewers used the IDMRC to assess 46 preprint and peer reviewed COVID-19 modeling studies published between March 13th, 2020, and July 31st, 2020. The inter-rater reliability was evaluated by mean percent agreement and Fleiss' kappa coefficients (κ). Papers were ranked based on the average number of reported reproducibility elements, and average proportion of papers that reported each checklist item were tabulated.

Results

Questions related to the computational environment (mean κ = 0.90, range = 0.90-0.90), analytical software (mean κ = 0.74, range = 0.68-0.82), model description (mean κ = 0.71, range = 0.58-0.84), model implementation (mean κ = 0.68, range = 0.39-0.86), and experimental protocol (mean κ = 0.63, range = 0.58-0.69) had moderate or greater (κ > 0.41) inter-rater reliability. Questions related to data had the lowest values (mean κ = 0.37, range = 0.23-0.59). Reviewers ranked similar papers in the upper and lower quartiles based on the proportion of reproducibility elements each paper reported. While over 70% of the publications provided data used in their models, less than 30% provided the model implementation.

Conclusions

The IDMRC is the first comprehensive, quality-assessed tool for guiding researchers in reporting reproducible infectious disease computational modeling studies. The inter-rater reliability assessment found that most scores were characterized by moderate or greater agreement. These results suggests that the IDMRC might be used to provide reliable assessments of the potential for reproducibility of published infectious disease modeling publications. Results of this evaluation identified opportunities for improvement to the model implementation and data questions that can further improve the reliability of the checklist.

Measuring the Concept of PID Literacy: User Perceptions and Understanding of PIDs in Support of Open Scholarly Infrastructure

The increasing centrality of persistent identifiers (PIDs) to scholarly ecosystems and the contribution they can make to the burgeoning “PID graph” has the potential to transform scholarship. Despite their importance as originators of PID data, little is known about researchers’ awareness and understanding of PIDs, or their efficacy in using them. In this article, we report on the results of an online interactive test designed to elicit exploratory data about researcher awareness and understanding of PIDs. This instrument was designed to explore recognition of PIDs (e.g. Digital Object Identifiers [DOIs], Open Researcher and Contributor IDs [ORCIDs], etc.) and the extent to which researchers correctly apply PIDs within digital scholarly ecosystems, as well as measure researchers’ perceptions of PIDs. Our results reveal irregular patterns of PID understanding and certainty across all participants, though statistically significant disciplinary and academic job role differences were observed in some instances. Uncertainty and confusion were found to exist in relation to dominant schemes such as ORCID and DOIs, even when contextualized within real-world examples. We also show researchers’ perceptions of PIDs to be generally positive but that disciplinary differences can be noted, as well as higher levels of aversion to PIDs in specific use cases and negative perceptions where PIDs are measured on an “activity” semantic dimension. This work therefore contributes to our understanding of scholars’ “PID literacy” and should inform those designing PID-centric scholarly infrastructures that a significant need for training and outreach to active researchers remains necessary.

PreprintMatch: A tool for preprint to publication detection shows global inequities in scientific publication

Preprints, versions of scientific manuscripts that precede peer review, are growing in popularity. They offer an opportunity to democratize and accelerate research, as they have no publication costs or a lengthy peer review process. Preprints are often later published in peer-reviewed venues, but these publications and the original preprints are frequently not linked in any way. To this end, we developed a tool, PreprintMatch, to find matches between preprints and their corresponding published papers, if they exist. This tool outperforms existing techniques to match preprints and papers, both on matching performance and speed. PreprintMatch was applied to search for matches between preprints (from bioRxiv and medRxiv), and PubMed. The preliminary nature of preprints offers a unique perspective into scientific projects at a relatively early stage, and with better matching between preprint and paper, we explored questions related to research inequity. We found that preprints from low income countries are published as peer-reviewed papers at a lower rate than high income countries (39.6% and 61.1%, respectively), and our data is consistent with previous work that cite a lack of resources, lack of stability, and policy choices to explain this discrepancy. Preprints from low income countries were also found to be published quicker (178 vs 203 days) and with less title, abstract, and author similarity to the published version compared to high income countries. Low income countries add more authors from the preprint to the published version than high income countries (0.42 authors vs 0.32, respectively), a practice that is significantly more frequent in China compared to similar countries. Finally, we find that some publishers publish work with authors from lower income countries more frequently than others.

Slow, slow, quick, quick, slow: five altmetric sources observed over a decade show evolving trends, by research age, attention source maturity and open access status

The study of temporal trends in altmetrics is under-developed, and this multi-year observation study addresses some of the deficits in our understanding of altmetric behaviour over time. The attention surrounding research outputs, as partially captured by altmetrics, or alternative metrics, constitutes many varied forms of data. Over the years 2008–2013, a set of 7739 papers were sampled on six occasions. Five altmetric data sources were recorded (Twitter, Mendeley, News, Blogs and Policy) and analysed for temporal trends, with particular attention being paid to their Open Access status and discipline. Twitter attention both starts and ends quickly. Mendeley readers accumulate quickly, and continue to grow over the following years. News and blog attention is quick to start, although news attention persists over a longer timeframe. Citations in policy documents are slow to start, and are observed to be growing over a decade after publication. Over time, growth in Twitter activity is confirmed, alongside an apparent decline in blogging attention. Mendeley usage is observed to grow, but shows signs of recent decline. Policy attention is identified as the slowest form of impact studied by altmetrics, and one that strongly favours the Humanities and Social Sciences. The Open Access Altmetrics Advantage is seen to emerge and evolve over time, with each attention source showing different trends. The existence of late-emergent attention in all attention sources is confirmed.

Weekly updating of guideline recommendations was feasible: the Australian National COVID-19 clinical evidence Taskforce

OBJECTIVES

To investigate how quickly evidence was incorporated into the Australian living guidelines for COVID-19 during the first 12 months of the pandemic.

STUDY DESIGN AND SETTING

For each study concerning drug therapies included in the guideline from April 3, 2020 to April 1, 2021, we extracted the publication date of the study, and the guideline version the study was included in. We analyzed two subgroups of studies as follows: those published in high impact factor journals and those with 100 or more participants.

RESULTS

In the first year, we published 37 major versions of the guidelines, incorporating 129 studies that investigated 48 drug therapies informing 115 recommendations. The median time from first publication of a study to incorporation in the guideline was 27 days (interquartile range [IQR], 16 to 44), ranging from 9 to 234 days. For the 53 studies in the highest impact factor journals, the median was 20 days (IQR 15 to 30), and for the 71 studies with 100 or more participants the median was 22 days (IQR 15 to 36).

CONCLUSION

Developing and sustaining living guidelines where evidence is rapidly incorporated is a resource- and time-intensive undertaking; however, this study demonstrates that it is feasible, even over a long period.

Changes in health communication in the age of COVID-19: A study on the dissemination of preprints to the public

Introduction

Preprints have become an important tool for meeting the challenges of health communication in the context of COVID-19. They allow scientists to disseminate their results more quickly due to the absence of a peer review process. Preprints have been well-received by scientists, however, there have been concerns about the exposure of wider public audiences to preprints due in part to this lack of peer review.

Methods

The aim of this study is to examine the dissemination of preprints on medRxiv and bioRxiv during the COVID-19 pandemic using content analysis and statistical analysis.

Results

Our findings show that preprints have played an unprecedented role in disseminating COVID-19-related science results to the public.

Discussion

While the overall media coverage of preprints is unsatisfactory, digital native news media performed better than legacy media in reporting preprints, which means that we could make the most of digital native media to improve health communication. This study contributes to understanding how science communication has evolved in response to the COVID-19 pandemic and provides some practical recommendations.

Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review

OBJECTIVES

To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process.

STUDY DESIGN AND SETTING

Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts.

RESULTS

Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance.

CONCLUSIONS

The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic.

Getting the most out of maths: How to coordinate mathematical modelling research to support a pandemic, lessons learnt from three initiatives that were part of the COVID-19 response in the UK

In March 2020 mathematics became a key part of the scientific advice to the UK government on the pandemic response to COVID-19. Mathematical and statistical modelling provided critical information on the spread of the virus and the potential impact of different interventions. The unprecedented scale of the challenge led the epidemiological modelling community in the UK to be pushed to its limits. At the same time, mathematical modellers across the country were keen to use their knowledge and skills to support the COVID-19 modelling effort. However, this sudden great interest in epidemiological modelling needed to be coordinated to provide much-needed support, and to limit the burden on epidemiological modellers already very stretched for time. In this paper we describe three initiatives set up in the UK in spring 2020 to coordinate the mathematical sciences research community in supporting mathematical modelling of COVID-19. Each initiative had different primary aims and worked to maximise synergies between the various projects. We reflect on the lessons learnt, highlighting the key roles of pre-existing research collaborations and focal centres of coordination in contributing to the success of these initiatives. We conclude with recommendations about important ways in which the scientific research community could be better prepared for future pandemics. This manuscript was submitted as part of a theme issue on "Modelling COVID-19 and Preparedness for Future Pandemics".

Paper 4: Search methods and approaches for living guidelines

OBJECTIVES

To describe the key features of a continual evidence surveillance process that can be implemented for living guidelines and to outline the considerations and trade-offs in adopting different approaches.

STUDY DESIGN AND SETTING

Members of the Australian Living Evidence Consortium (ALEC), National Institute of Health and Care Excellence (NICE), and the US GRADE Network (USGN) shared their practical experiences of and approaches to establishing surveillance systems for living guidelines. We identified several common components of evidence surveillance and listed the key features and considerations for each component drawn from case studies, highlighting differences with standard guidelines.

RESULTS

We developed guidance that covers the initial information needed to support decisions around suitability for living mode and the practical considerations in setting up continual search surveillance systems (search frequency, sources to search, use of automation, reporting the search, ongoing resources, and evaluation). The case studies draw on our experiences with developing guidelines for COVID-19, as well as for other conditions such as stroke and diabetes, and cover a range of practical approaches, including the use of automation.

CONCLUSION

This paper highlights different approaches to continual evidence surveillance that can be implemented in living guidelines.

Comparison of Preprint Postings of Randomized Clinical Trials on COVID-19 and Corresponding Published Journal Articles: A Systematic Review

IMPORTANCE

Randomized clinical trials (RCTs) on COVID-19 are increasingly being posted as preprints before publication in a scientific, peer-reviewed journal.

OBJECTIVE

To assess time to journal publication for COVID-19 RCT preprints and to compare differences between pairs of preprints and corresponding journal articles.

EVIDENCE REVIEW

This systematic review used a meta-epidemiologic approach to conduct a literature search using the World Health Organization COVID-19 database and Embase to identify preprints published between January 1 and December 31, 2021. This review included RCTs with human participants and research questions regarding the treatment or prevention of COVID-19. For each preprint, a literature search was done to locate the corresponding journal article. Two independent reviewers read the full text, extracted data, and assessed risk of bias using the Cochrane Risk of Bias 2 tool. Time to publication was analyzed using a Cox proportional hazards regression model. Differences between preprint and journal article pairs in terms of outcomes, analyses, results, or conclusions were described. Statistical analysis was performed on October 17, 2022.

FINDINGS

This study included 152 preprints. As of October 1, 2022, 119 of 152 preprints (78.3%) had been published in journals. The median time to publication was 186 days (range, 17-407 days). In a multivariable model, larger sample size and low risk of bias were associated with journal publication. With a sample size of less than 200 as the reference, sample sizes of 201 to 1000 and greater than 1000 had hazard ratios (HRs) of 1.23 (95% CI, 0.80-1.91) and 2.19 (95% CI, 1.36-3.53) for publication, respectively. With high risk of bias as the reference, medium-risk articles with some concerns for bias had an HR of 1.77 (95% CI, 1.02-3.09); those with a low risk of bias had an HR of 3.01 (95% CI, 1.71-5.30). Of the 119 published preprints, there were differences in terms of outcomes, analyses, results, or conclusions in 65 studies (54.6%). The main conclusion in the preprint contradicted the conclusion in the journal article for 2 studies (1.7%).

CONCLUSIONS AND RELEVANCE

These findings suggest that there is a substantial time lag from preprint posting to journal publication. Preprints with smaller sample sizes and high risk of bias were less likely to be published. Finally, although differences in terms of outcomes, analyses, results, or conclusions were observed for preprint and journal article pairs in most studies, the main conclusion remained consistent for the majority of studies.

Science Communication and Biomedical Visualization: Two Sides of the Same Coin

Biomedical visualization has a long history as a tool for education around public health. However, recent advances in our understanding of how to be more effective at communicating complex scientific ideas to a public audience necessitate a re-examination of approaches to biomedical visualization. Scientific knowledge has expanded dramatically in the twenty-first century, as has its availability beyond the scientific arena. This chapter briefly discusses the historical approaches in biomedical visualization from the perspective of Western public health. It also outlines the approach that biomedical visualization should take according to best practices in effective science communication.

Preprints in Health Professions Education: Raising Awareness and Shifting Culture

A preprint is a version of a research manuscript posted by its authors to a preprint server before peer review. Preprints are associated with a variety of benefits, including the ability to rapidly communicate research, the opportunity for researchers to receive feedback and raise awareness of their research, and broad and unrestricted access. For early-career researchers, preprints also provide a mechanism for demonstrating research progress and productivity without the lengthy timelines of traditional journal publishing. Despite these benefits, few health professions education (HPE) research articles are deposited as preprints, suggesting that preprinting is not currently integrated into HPE culture. In this article, the authors describe preprints, their benefits and related risks, and the potential barriers that hamper their widespread use within HPE. In particular, the authors propose the barriers of discordant messaging and the lack of formal and informal education on how to deposit, critically appraise, and use preprints. To mitigate these barriers, several recommendations are proposed to facilitate preprints in becoming an accepted and encouraged component of HPE culture, allowing the field to take full advantage of this evolving form of research dissemination.

COVID-19 and the scientific publishing system: growth, open access and scientific fields

We model the growth of scientific literature related to COVID-19 and forecast the expected growth from 1 June 2021. Considering the significant scientific and financial efforts made by the research community to find solutions to end the COVID-19 pandemic, an unprecedented volume of scientific outputs is being produced. This questions the capacity of scientists, politicians and citizens to maintain infrastructure, digest content and take scientifically informed decisions. A crucial aspect is to make predictions to prepare for such a large corpus of scientific literature. Here we base our predictions on the Autoregressive Integrated Moving Average (ARIMA) and exponential smoothing models using the Dimensions database. This source has the particularity of including in the metadata information on the date in which papers were indexed. We present global predictions, plus predictions in three specific settings: by type of access (Open Access), by domain-specific repository (SSRN and MedRxiv) and by several research fields. We conclude by discussing our findings.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11192-022-04536-x.

Open access and its potential impact on public health - A South African perspective

Traditionally, access to research information has been restricted through journal subscriptions. This means that research entities and individuals who were unable to afford subscription costs did not have access to journal articles. There has however been a progressive shift toward electronic access to journal publications and subsequently growth in the number of journals available globally. In the context of electronic journals, both open access and restricted access options exist. While the latter option is comparable to traditional, subscription-based paper journals, open access journal publications follow an "open science" publishing model allowing scholarly communications and outputs to be publicly available online at no cost to the reader. However, for readers to enjoy open access, publication costs are shifted elsewhere, typically onto academic institutions and authors. SARS-CoV-2, and the resulting COVID-19 pandemic have highlighted the benefits of open science through accelerated research and unprecedented levels of collaboration and data sharing. South Africa is one of the leading open access countries on the African continent. This paper focuses on open access in the South African higher education research context with an emphasis on our Institution and our own experiences. It also addresses the financial implications of open access and provides possible solutions for reducing the cost of publication for researchers and their institutions. Privacy in open access and the role of the Protection of Personal Information Act (POPIA) in medical research and secondary use of data in South Africa will also be discussed.

Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

OBJECTIVES

The aim of this study is to describe (1) registered and (2) published systematic reviews (SRs) on COVID-19 treatments, and to analyze (3) the proportion of publications among registered SRs and (4) the proportion of registrations among published SRs.

STUDY DESIGN AND SETTING

This meta-research study (CRD42021240423) is part of CEOsys (http://www.covid-evidenz.de/). Two reviewers identified protocols in PROSPERO (registered January 2020 to September 2020) and SRs published as preprint or peer-reviewed article in L·OVE (Living OVerview of the Evidence) COVID-19 (by May 2021). SRs of all types assessing COVID-19 treatments in humans were included.

RESULTS

We included 239 PROSPERO protocols and 346 SRs published in L·OVE. In both samples, the affiliation of the corresponding author with an Asian institution, standard SR as review type, and meta-analysis as synthesis method were the most frequent characteristics. Living SRs made up ≤10%. A total of 71 of 239 (29.7%) PROSPERO protocols were published as SR by February 2022, that is, after at least 17 months of follow-up (25 of 71 as preprints, 35.2%). In L·OVE, 261 of 346 (75.4%) SRs published by May 2021 were not registered in PROSPERO.

CONCLUSION

Overall, one-third PROSPERO protocols were published and three-fourth published SRs were not registered. We strongly encourage authors to register and publish their SRs promptly to reduce research waste and to allocate resources efficiently during the pandemic and beyond.

Das Netzwerk Universitätsmedizin: Der Transfer von Erkenntnissen aus der Forschung in die Patient*innenbehandlung am Beispiel von COVID-19

Ein wichtiges Instrument zur Verbreitung neuer Erkenntnisse aus der Forschung in der Patient*innenbehandlung sind Leitlinien. Sie sollen eine möglichst optimale evidenzbasierte Versorgung unterstützen. Für den Umgang mit einer neuen Erkrankung wie COVID-19, für die es initial keine etablierte Versorgungspraxis gab, war die schnelle Erarbeitung aktueller Leitlinien essentiell. Angesichts intensiver weltweiter Forschung entstand in kurzer Zeit sehr viel neues Wissen. Gleichzeitig mussten akut sehr viele Patient*innen behandelt werden. Die Fülle neuer Erkenntnisse musste somit in einer Zeit sehr schnell in die Praxis transferiert werden, in der medizinisches Personal durch den hohen Versorgungsbedarf ohnehin schon sehr stark belastet war. Zudem ist die Erstellung von Leitlinien zeitintensiv und aufwändig. Daher hat das Netzwerk Universitätsmedizin (NUM), ein Forschungsverbund von 36 deutschen Universitätsklinika, im Projekt CEOsys Ansätze entwickelt, um den Transfer neuer Erkenntnisse aus der COVID-19-Forschung in die Patient*innenbehandlung erheblich zu beschleunigen. Die dafür erforderlichen Plattformen müssen nun verstetigt werden, um die Pandemic Preparedness in Deutschland dauerhaft zu verbessern.

Comparison of Clinical Study Results Reported in medRxiv Preprints vs Peer-reviewed Journal Articles

IMPORTANCE

Preprints have been widely adopted to enhance the timely dissemination of research across many scientific fields. Concerns remain that early, public access to preliminary medical research has the potential to propagate misleading or faulty research that has been conducted or interpreted in error.

OBJECTIVE

To evaluate the concordance among study characteristics, results, and interpretations described in preprints of clinical studies posted to medRxiv that are subsequently published in peer-reviewed journals (preprint-journal article pairs).

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study assessed all preprints describing clinical studies that were initially posted to medRxiv in September 2020 and subsequently published in a peer-reviewed journal as of September 15, 2022.

MAIN OUTCOMES AND MEASURES

For preprint-journal article pairs describing clinical trials, observational studies, and meta-analyses that measured health-related outcomes, the sample size, primary end points, corresponding results, and overarching conclusions were abstracted and compared. Sample size and results from primary end points were considered concordant if they had exact numerical equivalence.

RESULTS

Among 1399 preprints first posted on medRxiv in September 2020, a total of 1077 (77.0%) had been published as of September 15, 2022, a median of 6 months (IQR, 3-8 months) after preprint posting. Of the 547 preprint-journal article pairs describing clinical trials, observational studies, or meta-analyses, 293 (53.6%) were related to COVID-19. Of the 535 pairs reporting sample sizes in both sources, 462 (86.4%) were concordant; 43 (58.9%) of the 73 pairs with discordant sample sizes had larger samples in the journal publication. There were 534 pairs (97.6%) with concordant and 13 pairs (2.4%) with discordant primary end points. Of the 535 pairs with numerical results for the primary end points, 434 (81.1%) had concordant primary end point results; 66 of the 101 discordant pairs (65.3%) had effect estimates that were in the same direction and were statistically consistent. Overall, 526 pairs (96.2%) had concordant study interpretations, including 82 of the 101 pairs (81.2%) with discordant primary end point results.

CONCLUSIONS AND RELEVANCE

Most clinical studies posted as preprints on medRxiv and subsequently published in peer-reviewed journals had concordant study characteristics, results, and final interpretations. With more than three-fourths of preprints published in journals within 24 months, these results may suggest that many preprints report findings that are consistent with the final peer-reviewed publications.

Biased, wrong and counterfeited evidences published during the COVID-19 pandemic, a systematic review of retracted COVID-19 papers

In 2020 COVID-19 led to an unprecedented stream of papers being submitted to journals. Scientists and physicians all around the globe were in need for information about this new disease. In this climate, many articles were accepted after extremely fast peer-reviews to provide the scientific community with the latest discoveries and knowledge. Unfortunately, this also led to articles retraction due to authors' misconduct or errors in methodology and/or conclusions. The aim of this study is to investigate the number and characteristics of retracted papers, and to explore the main causes that led to retraction. We conducted a systematic review on retracted articles, using PubMed as data source. Our inclusion criteria were the following: English-language retracted articles that reported original data, results, opinions or hypotheses on COVID-19 and Sars-CoV-2. Twenty-seven retracted articles were identified, mainly reporting observational studies and opinion pieces. Many articles published during the first year of the pandemic have been retracted, mainly due to the authors' scientific misconduct. Duplications, plagiarism, frauds and absence of consent, were the main reasons for retractions. In modern medicine, researchers are required to publish frequently, and, especially during situations like the COVID-19 pandemic, when articles were rapidly published, gaps in peer-reviews system and in the path to scientific publication arose.

Science in motion: A qualitative analysis of journalists' use and perception of preprints

This qualitative study explores how and why journalists use preprints-unreviewed research papers-in their reporting. Through thematic analysis of interviews conducted with 19 health and science journalists in the second year of the COVID-19 pandemic, it applies a theoretical framework that conceptualizes COVID-19 preprint research as a form of post-normal science, characterized by high scientific uncertainty and societal relevance, urgent need for political decision-making, and value-related policy considerations. Findings suggest that journalists approach the decision to cover preprints as a careful calculation, in which the potential public benefits and the ease of access preprints provided were weighed against risks of spreading misinformation. Journalists described viewing unreviewed studies with extra skepticism and relied on diverse strategies to find, vet, and report on them. Some of these strategies represent standard science journalism, while others, such as labeling unreviewed studies as preprints, mark a departure from the norm. However, journalists also reported barriers to covering preprints, as many felt they lacked the expertise or the time required to fully understand or vet the research. The findings suggest that coverage of preprints is likely to continue post-pandemic, with important implications for scientists, journalists, and the publics who read their work.

How often do cancer researchers make their data and code available and what factors are associated with sharing?

BACKGROUND

Various stakeholders are calling for increased availability of data and code from cancer research. However, it is unclear how commonly these products are shared, and what factors are associated with sharing. Our objective was to evaluate how frequently oncology researchers make data and code available and explore factors associated with sharing.

METHODS

A cross-sectional analysis of a random sample of 306 cancer-related articles indexed in PubMed in 2019 which studied research subjects with a cancer diagnosis was performed. All articles were independently screened for eligibility by two authors. Outcomes of interest included the prevalence of affirmative sharing declarations and the rate with which declarations connected to data complying with key FAIR principles (e.g. posted to a recognised repository, assigned an identifier, data license outlined, non-proprietary formatting). We also investigated associations between sharing rates and several journal characteristics (e.g. sharing policies, publication models), study characteristics (e.g. cancer rarity, study design), open science practices (e.g. pre-registration, pre-printing) and subsequent citation rates between 2020 and 2021.

RESULTS

One in five studies declared data were publicly available (59/306, 19%, 95% CI: 15-24%). However, when data availability was investigated this percentage dropped to 16% (49/306, 95% CI: 12-20%), and then to less than 1% (1/306, 95% CI: 0-2%) when data were checked for compliance with key FAIR principles. While only 4% of articles that used inferential statistics reported code to be available (10/274, 95% CI: 2-6%), the odds of reporting code to be available were 5.6 times higher for researchers who shared data. Compliance with mandatory data and code sharing policies was observed in 48% (14/29) and 0% (0/6) of articles, respectively. However, 88% of articles (45/51) included data availability statements when required. Policies that encouraged data sharing did not appear to be any more effective than not having a policy at all. The only factors associated with higher rates of data sharing were studying rare cancers and using publicly available data to complement original research.

CONCLUSIONS

Data and code sharing in oncology occurs infrequently, and at a lower rate than would be expected given the prevalence of mandatory sharing policies. There is also a large gap between those declaring data to be available, and those archiving data in a way that facilitates its reuse. We encourage journals to actively check compliance with sharing policies, and researchers consult community-accepted guidelines when archiving the products of their research.

COVID-19-Related manuscripts: lag from preprint to publication

OBJECTIVE

Preprints have had a prominent role in the swift scientific response to COVID-19. Two years into the pandemic, we investigated how much preprints had contributed to timely data sharing by analyzing the lag time from preprint posting to journal publication.

RESULTS

To estimate the median number of days between the date a manuscript was posted as a preprint and the date of its publication in a scientific journal, we analyzed preprints posted from January 1, 2020, to December 31, 2021 in the NIH iSearch COVID-19 Portfolio database and performed a Kaplan-Meier (KM) survival analysis using a non-mixture parametric cure model. Of the 39,243 preprints in our analysis, 7712 (20%) were published in a journal, after a median lag of 178 days (95% CI: 175-181). Most of the published preprints were posted on the bioRxiv (29%) or medRxiv (65%) servers, which allow authors to choose a subject category when posting. Of the 20,698 preprints posted on these two servers, 7358 (36%) were published, including approximately half of those categorized as biochemistry, biophysics, and genomics, which became published articles within the study interval, compared with 29% categorized as epidemiology and 26% as bioinformatics.

Motivations, concerns and selection biases when posting preprints: A survey of bioRxiv authors

Since 2013, the usage of preprints as a means of sharing research in biology has rapidly grown, in particular via the preprint server bioRxiv. Recent studies have found that journal articles that were previously posted to bioRxiv received a higher number of citations or mentions/shares on other online platforms compared to articles in the same journals that were not posted. However, the exact causal mechanism for this effect has not been established, and may in part be related to authors' biases in the selection of articles that are chosen to be posted as preprints. We aimed to investigate this mechanism by conducting a mixed-methods survey of 1,444 authors of bioRxiv preprints, to investigate the reasons that they post or do not post certain articles as preprints, and to make comparisons between articles they choose to post and not post as preprints. We find that authors are most strongly motivated to post preprints to increase awareness of their work and increase the speed of its dissemination; conversely, the strongest reasons for not posting preprints centre around a lack of awareness of preprints and reluctance to publicly post work that has not undergone a peer review process. We additionally find evidence that authors do not consider quality, novelty or significance when posting or not posting research as preprints, however, authors retain an expectation that articles they post as preprints will receive more citations or be shared more widely online than articles not posted.

Robustness of evidence reported in preprints during peer review

Scientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Expert peer-review scores of preprint quality were not related to eventual publication in a peer-reviewed journal. Uncertainty was reduced during peer review, with CIs reducing by 7% on average. These results support the use of preprints, a component of biomedical research literature, in decision making. These results can also help inform the use of preprints during the ongoing COVID-19 pandemic and future disease outbreaks.

[Informed trust in science: lessons from the COVID 19 pandemic for the conceptualization of science literacy]

Informed trust in science is necessary for the 'interfaces' within the flow of knowledge between citizens' everyday understanding of the pandemic and the dynamically evolving state of knowledge in the sciences. This is the core thesis of this paper. Without science, the COVID-19 pandemic can neither be understood nor controlled, and for this to happen, citizens must engage with science based knowledge. However, such knowledge is dynamic (evolving and intertwined with normative issues). Furthermore, science based knowledge competes with pseudoscientific contributions. As non-experts, laypersons must therefore decide whom to trust. The paper describes the concept of functional scientific literacy as a prerequisite of informed trust. The knowledge bases for judgments of informed trust should be taught in school and judging rationally about the trustworthiness of science-related knowledge claims should be practiced.

Overlay journals: A study of the current landscape

Overlay journals are characterised by their articles being published on open access repositories, often already starting in their initial preprint form as a prerequisite for submission to the journal prior to initiating the peer-review process. In this study we aimed to identify currently active overlay journals and examine their characteristics. We utilised an explorative web search and contacted key service providers for additional information. The final sample consisted of 34 overlay journals. While the results show that new overlay journals have been actively established within recent years, the current presence of overlay journals remains diminutive compared to the overall number of open access journals. Most overlay journals publish articles in natural sciences, mathematics or computer sciences, and are commonly published by groups of academics rather than formal organisations. They may also rank highly within the traditional journal citation metrics. None of the investigated journals required fees from authors, which is likely related to the cost-effective aspects of the overlay publishing model. Both the growth in adoption of open access preprint repositories and researchers’ willingness to publish in overlay journals will determine the model’s wider impact on scholarly publishing.

Overview of the COVID-19 text mining tool interactive demonstration track in BioCreative VII

The coronavirus disease 2019 (COVID-19) pandemic has compelled biomedical researchers to communicate data in real time to establish more effective medical treatments and public health policies. Nontraditional sources such as preprint publications, i.e. articles not yet validated by peer review, have become crucial hubs for the dissemination of scientific results. Natural language processing (NLP) systems have been recently developed to extract and organize COVID-19 data in reasoning systems. Given this scenario, the BioCreative COVID-19 text mining tool interactive demonstration track was created to assess the landscape of the available tools and to gauge user interest, thereby providing a two-way communication channel between NLP system developers and potential end users. The goal was to inform system designers about the performance and usability of their products and to suggest new additional features. Considering the exploratory nature of this track, the call for participation solicited teams to apply for the track, based on their system's ability to perform COVID-19-related tasks and interest in receiving user feedback. We also recruited volunteer users to test systems. Seven teams registered systems for the track, and >30 individuals volunteered as test users; these volunteer users covered a broad range of specialties, including bench scientists, bioinformaticians and biocurators. The users, who had the option to participate anonymously, were provided with written and video documentation to familiarize themselves with the NLP tools and completed a survey to record their evaluation. Additional feedback was also provided by NLP system developers. The track was well received as shown by the overall positive feedback from the participating teams and the users. Database URL: https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vii/track-4/.

Identifying science in the news: An assessment of the precision and recall of Altmetric.com news mention data

The company Altmetric is often used to collect mentions of research in online news stories, yet there have been concerns about the quality of this data. This study investigates these concerns. Using a manual content analysis of 400 news stories as a comparison method, we analyzed the precision and recall with which Altmetric identified mentions of research in 8 news outlets. We also used logistic regression to identify the characteristics of research mentions that influence their likelihood of being successfully identified. We find that, for a predefined set of outlets, Altmetric’s news mention data were relatively accurate (F-score = 0.80), with very high precision (0.95) and acceptable recall (0.70), although recall is below 0.50 for some news outlets. Altmetric is more likely to successfully identify mentions of research that include a hyperlink to the research item, an author name, and/or the title of a publication venue. This data source appears to be less reliable for mentions of research that provide little or no bibliometric information, as well as for identifying mentions of scholarly monographs, conference presentations, dissertations, and non-English research articles. Our findings suggest that, with caveats, scholars can use Altmetric news mention data as a relatively reliable source to identify research mentions across a range of outlets with high precision and acceptable recall, offering scholars the potential to conserve resources during data collection. Our study does not, however, offer an assessment of completeness or accuracy of Altmetric news data overall.