Beyond disclaimers: the need for a curation-based model of PubMed

According to its own description, the biomedical meta-database PubMed exists "with the aim of improving health-both globally and personally." Unfortunately, PubMed contains an increasing amount of low-quality research that may detract from this goal. Currently, PubMed warns its users and protects itself from such problems with a disclaimer stating that the presence of any article, book, or document in PubMed does not imply an endorsement of, or concurrence with, its contents by the NLM, the National Institutes of Health (NIH), or the U.S. Federal Government. However, we are critical of a "disclaimer-only" stance and encourage PubMed to take further action against low-quality research being found and indexed in its database, and thus available for use. To address this problem, we offer two lines of reasoning to argue that PubMed should not function merely as a passive index of health-related research. Instead, we first argue that only trustworthy published research is able to further PubMed's goal of health improvement. Secondly, on the basis of surveys, we argue that researchers place a high level of trust in articles that are referenced in this meta-database. We cannot expect any one set of actors to ensure trustworthy content on PubMed, which requires collective responsibility among authors, peer reviewers, editors, and indexers alike. Instead, we propose a curation-based model that incorporates three mechanisms of collaborative content curation: open expert feedback on indexed content, journal auditing, and constant transparent reassessment of indexed entities.

Terms in journal articles associating with high quality: can qualitative research be world-leading?

Purpose

Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of aspects of research that are associated with high or low quality.

Design/methodology/approach

We extracted the words and 2–5-word phrases most strongly associated with different quality scores in each of 34 Units of Assessment (UoAs) in the Research Excellence Framework (REF) 2021. We extracted the terms from 122,331 journal articles 2014–2020 with individual REF2021 quality scores.

Findings

The terms associating with high- or low-quality scores vary between fields but relate to writing styles, methods and topics. We show that the first-person writing style strongly associates with higher quality research in many areas because it is the norm for a set of large prestigious journals. We found methods and topics that associate with both high- and low-quality scores. Worryingly, terms associated with educational and qualitative research attract lower quality scores in multiple areas. REF experts may rarely give high scores to qualitative or educational research because the authors tend to be less competent, because it is harder to do world leading research with these themes, or because they do not value them.

Originality/value

This is the first investigation of journal article terms associating with research quality.

Open peer review: the point of view of scientific journal editors

Academic journals have been incorporating several elements of open science: open access (since 2000), later, the deposit of research data of the articles published, the dissemination of preprints before the publication of the paper and, finally, the open peer review (OPR). While open access is well-established and the inclusion of research data is increasingly widespread, the OPR is just at the beginning of its incorporation as a real alternative to the double-blind model, which is the most widespread and consolidated. The objective of our article is to analyse the opinion of the editors of Spanish scientific journals about the advantages and disadvantages or barriers for the implementation of the OPR. This is a qualitative study that has been carried out from the open answers of a questionnaire sent to the 1875 editors of the Spanish academic journals that appear in the database Dulcinea and that obtained a response of 22.4%. Regarding the limitations, the study is based on the opinions and experience of the editors of Spanish scientific journals, which are mostly published by academic institutions and are in the field of social sciences and humanities. The results focus on delving into the advantages and disadvantages. Among the encouraging factors, the editors point out that to have open reports is very useful for the scientific community, that it recognizes the role of the reviewer, makes it possible to control the arbitrariness of some reviewers, and that it promotes the reviewer-author dialogue. The main barriers discussed are the following: a possible lack of objectivity and rigor, resistance to change a consolidated system (“double-blind”), knowing the author benefits established authors and harms novices, more difficulties for finding reviewers, increases costs and can lengthen the review process.

The Role of Publons in the Context of Open Peer Review

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length

Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative perspective, focusing on reviewer anonymity, review length, and review outcomes. The sample contained 45,385 first round open reviews from published standard journal articles in 288 MDPI journals classified into one or more Scopus disciplinary areas (Health Sciences; Life Sciences; Physical Sciences; Social Sciences). The eight main findings include substantial differences between journals and disciplines in review lengths, reviewer anonymity, review outcomes, and the use of attachments. In particular, Physical Sciences journal reviews tended to be stricter and were more likely to be anonymous. Life Sciences and Social Sciences reviews were the longest overall. Signed reviews tend to be 15% longer (perhaps to be more careful or polite) but gave similar decisions to anonymous reviews. Finally, reviews with major revision outcomes tended to be 68% longer than reviews with for minor revision outcomes, except in a few journals. In conclusion, signing reviews does not seem to threaten the validity of peer review outcomes and authors, editors, and reviewers of multidisciplinary articles should be aware of substantial field differences in what constitutes an appropriate review.