Characterization of Comments About bioRxiv and medRxiv Preprints

Post-publication peer review and the identification of methodological and reporting issues in COVID-19 trials: a qualitative study

OBJECTIVES

We aimed to determine to what extent systematic reviewers and post-preprint and post-publication peer review identified methodological and reporting issues in COVID-19 trials that could be easily resolved by the authors.

DESIGN

Qualitative study.

DATA SOURCES

COVID-NMA living systematic review (covid-nma.com), PubPeer, medRxiv, Research Square, SSRN.

METHODS

We considered randomised controlled trials (RCTs) in COVID-NMA that evaluated pharmacological treatments for COVID-19 and retrieved systematic reviewers' assessments of the risk of bias and outcome reporting bias. We also searched for commentary data on PubPeer and preprint servers up to 6 November 2023. We employed qualitative content analysis to develop themes and domains of methodological and reporting issues identified by commenters.

RESULTS

We identified 500 eligible RCTs. Systematic reviewers identified methodological and reporting issues in 446 (89%) RCTs. In 391 (78%) RCTs, the issues could be easily resolved by the trial authors; issues included incomplete reporting (49%), selection of the reported results (52%) and no access to the pre-specified plan (25%). Alternatively, 74 (15%) RCTs had received at least one comment on PubPeer or preprint servers, totalling 348 comments. In 46 (9%) RCTs, the issues identified by post-preprint and post-publication peer review comments could be easily resolved by the trial authors; the issues were related to incomplete reporting (6%), errors (5%), statistical analysis (3%), inconsistent reporting of methods and analyses (2%), spin (2%), selection of the reported results (1%) and no access to the raw data/pre-specified plan (1%).

CONCLUSIONS

Without changing their process, systematic reviewers identified issues in most RCTs that could be easily resolved by the trial authors; however, the lack of an established author feedback mechanism represents a wasted opportunity for facilitating improvement and enhancing the overall manuscript quality. On the other hand, despite the existing feedback loop to authors present in post-publication peer review, it demonstrated limited effectiveness in identifying methodological and reporting issues.

The present and future of peer review: Ideas, interventions, and evidence

What is wrong with the peer review system? Is peer review sustainable? Useful? What other models exist? These are central yet contentious questions in today's academic discourse. This perspective critically discusses alternative models and revisions to the peer review system. The authors highlight possible changes to the peer review system, with the goal of fostering further dialog among the main stakeholders, including producers and consumers of scientific research. Neither our list of identified issues with the peer review system nor our discussed resolutions are complete. A point of agreement is that fair assessment and efficient change would require more comprehensive and rigorous data on the various aspects of the peer review system.

An international, cross-sectional survey of preprint attitudes among biomedical researchers

Background

Preprints are scientific manuscripts that are made available on open-access servers but are not yet peer-reviewed. Although preprints are becoming more prevalent, uptake is not optimal. Understanding researchers' opinions and attitudes toward preprints is valuable to optimize their use. Understanding knowledge gaps and researchers' attitudes toward preprinting can assist stakeholders, such as journals, funding agencies, and universities, to use preprints more effectively. Here, we aimed to collect perceptions and behaviours regarding preprints across an international sample of biomedical researchers.

Methods

Corresponding authors of articles published in biomedical research journals were identified from a random sample of journals from the MEDLINE database. Their names and email addresses were extracted to invite them to our anonymous, cross-sectional survey, which asked participants questions about their knowledge, attitudes, and opinions regarding preprinting.

Results

The survey was completed by 730 respondents providing a response rate of 3.20% and demonstrated a wide range of attitudes and opinions about preprints with authors from various disciplines and career stages worldwide. Most respondents were familiar with the concept of preprints but most had not previously posted one. The lead author of the project and journal policy had the greatest impact on decisions to post a preprint, whereas employers/research institutes had the least impact. Supporting open science practices was the highest ranked incentive, while increasing authors' visibility was the highest ranked motivation for publishing preprints.

Conclusions

Although many biomedical researchers recognize the benefits of preprints, there is still hesitation among others to engage in this practice. This may be due to the general lack of peer review of preprints and little enthusiasm from external organizations such as journals, funding agencies, and universities. Future work is needed to determine optimal ways to improve researchers' attitudes through modifications to current preprint systems and policies.

An analysis of the effects of sharing research data, code, and preprints on citations

Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains. In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122'000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations. We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% (±.7) on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% (±.8) on average. However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.