The state of the art in peer review

Predatory publishing in medical education: a rapid scoping review

BACKGROUND

Academic publishing is a cornerstone of scholarly communications, yet is unfortunately open to abuse, having given rise to 'predatory publishers'- groups that employ aggressive marketing tactics, are deficient in methods and ethics, and bypass peer review. Preventing these predatory publishers from infiltrating scholarly activity is of high importance, and students must be trained in this area to increase awareness and reduce use. The scope of this issue in the context of medical students remains unknown, and therefore this sought to examine the breadth of the current literature base.

METHODS

A rapid scoping review was undertaken, adhering to adapted PRISMA guidelines. Six databases (ASSIA, EBSCO, Ovid, PubMed, Scopus, Web of Science) were systematically searched for content related to predatory publishing and medical students. Results were single-screened, facilitated by online reviewing software. Resultant data were narratively described, with common themes identified.

RESULTS

After searching and screening, five studies were included, representing a total of 1338 students. Two predominant themes- understanding, and utilisation- of predatory publishers was identified. These themes revealed that medical students were broadly unaware of the issue of predatory publishing, and that a small number have already, or would consider, using their services.

CONCLUSION

There remains a lack of understanding of the threat that predatory publishers pose amongst medical students. Future research and education in this domain will be required to focus on informing medical students on the issue, and the implication of engaging with predatory publishers.

Editors' perspectives on the selection of reviewers and the quality of reviews

This article describes the outcomes of a survey of 93 editors in chief and associate editors of behavior-analytic journals. We sought information about variables that influence their judgment of the selection of reviewers, selection of review panels, and quality of reviews. When selecting reviewers, participants rated highly expertise on the topic, history of conducting good reviews, and history of writing constructive and respectful reviews. When selecting review panels, participants rated highly stratifying reviewers based on their expertise, avoiding conflicts of interest, and the matching based on the area of expertise between reviewers and authors. When evaluating the quality of a review, participants rated highly considerations related to research design, the science underlying the main idea, and accurate interpretations of the data. Participants did not rate copyediting as important. Overall, the extent to which reviewer selection was influenced by membership in underrepresented groups varied. These findings can inform the development of training programs for teaching peer-review repertoires.

How to improve scientific peer review: Four schools of thought

Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review. Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved. To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school. Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between them. We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.

Ten simple rules for failing successfully in academia

Failure is an integral part of life and by extension academia. At the same time, failure is often ignored, with potentially negative consequences both for the science and the scientists involved. This article provides several strategies for learning from and dealing with failure instead of ignoring it. Hopefully, our recommendations are widely applicable, while still taking into account individual differences between academics. These simple rules allow academics to further develop their own strategies for failing successfully in academia.

Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length

Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative perspective, focusing on reviewer anonymity, review length, and review outcomes. The sample contained 45,385 first round open reviews from published standard journal articles in 288 MDPI journals classified into one or more Scopus disciplinary areas (Health Sciences; Life Sciences; Physical Sciences; Social Sciences). The eight main findings include substantial differences between journals and disciplines in review lengths, reviewer anonymity, review outcomes, and the use of attachments. In particular, Physical Sciences journal reviews tended to be stricter and were more likely to be anonymous. Life Sciences and Social Sciences reviews were the longest overall. Signed reviews tend to be 15% longer (perhaps to be more careful or polite) but gave similar decisions to anonymous reviews. Finally, reviews with major revision outcomes tended to be 68% longer than reviews with for minor revision outcomes, except in a few journals. In conclusion, signing reviews does not seem to threaten the validity of peer review outcomes and authors, editors, and reviewers of multidisciplinary articles should be aware of substantial field differences in what constitutes an appropriate review.

eLife's new model and its impact on science communication

The eLife Early-Career Advisory Group discusses eLife's new peer review and publishing model, and how the whole process of scientific communication could be improved for the benefit of early-career researchers and the entire scientific community.

An overview of innovations in the external peer review of journal manuscripts

Background: There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations. This literature review was part of one such project 'Experiments in peer review' which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in the external peer review of journal manuscripts reported in the scholarly literature and by providing a summary of the different approaches. This did not include interventions in editorial processes. Methods: This review of reviews is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 291 records were screened, with six review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples.   Results: The overview of innovations are drawn from six review articles. The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented. Conclusions: From a simple synthesis of the review authors' conclusions, three key messages are presented: observations on current practice; authors' views on the implications of innovations in peer review; and calls for action in peer review research and practice.

Innovations in peer review in scholarly publishing: a meta-summary

Background: There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations. This literature review was part of one such project 'Experiments in peer review' which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in peer review reported in the scholarly literature and by providing a summary of the different approaches. Methods: This meta-summary is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 247 papers were screened, with 6 review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples.   Results: The summary of innovations are drawn from 6 review articles. The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented. Conclusions: From a simple synthesis of the review authors' conclusions, three key messages are presented: observations on current practice; authors' views on the implications of innovations in peer review; and calls for action in peer review research and practice.

Which peer reviewers voluntarily reveal their identity to authors? Insights into the consequences of open-identities peer review

Identifying reviewers is argued to improve the quality and fairness of peer review, but is generally disfavoured by reviewers. To gain some insight into the factors that influence when reviewers are willing to have their identity revealed, I examined which reviewers voluntarily reveal their identities to authors at the journal Functional Ecology, at which reviewer identities are confidential unless reviewers sign their comments to authors. I found that 5.6% of reviewers signed their comments to authors. This proportion increased slightly over time, from 4.4% in 2003-2005 to 6.7% in 2013-2015. Male reviewers were 1.8 times more likely to sign their comments to authors than were female reviewers, and this difference persisted over time. Few reviewers signed all of their reviews; reviewers were more likely to sign their reviews when their rating of the manuscript was more positive, and papers that had at least one signed review were more likely to be invited for revision. Signed reviews were, on average, longer and recommended more references to authors. My analyses cannot distinguish cause and effect for the patterns observed, but my results suggest that 'open-identities' review, in which reviewers are not permitted to be anonymous, will probably reduce the degree to which reviewers are critical in their assessment of manuscripts and will differentially affect recruitment of male and female reviewers, negatively affecting the diversity of reviewers recruited by journals.

Shared burden is always lighter - Peer-review performance in an ophthalmological journal 2010-2020

PURPOSE

There are concerns in the academic publishing community that it is becoming more difficult to secure reviews for scientific manuscripts. This study examines trends in editorial and peer review processes in an ophthalmological journal over the last decade.

METHODS

A retrospective analysis was performed of editorial data from the journal Acta Ophthalmologica containing all manuscript submissions between 2010 and 2020.

RESULTS

The number of yearly submissions grew between 2010 and 2019 from 1014 to 1623, and in 2020, the number of submissions increased to 2449. In total, the number of submissions increased by 142% between 2010 and 2020. Similarly, the proportion of desk-rejected manuscripts increased from 48% to 67% during the period 2010-2020. The number of invitations needed to obtain one review showed an increase from 1.9 to 2.6 between 2010 and 2019, but remained stable between 2019 and 2020. However, the number of reviewers per reviewed manuscript, reviewed manuscripts per reviewer and time from invitation to completed review assignment remained almost constant between 2010 and 2020. Researchers based in North American were disproportionally often invited to review (18%) compared to their share of published articles (7%), and they also declined review invitation more frequently compared to scholars in other parts of the world.

CONCLUSIONS

The study revealed an increase in submitted manuscripts to an ophthalmological journal over the last decade, with a further increase during the COVID-19 pandemic. The number of reviewer invitations needed to obtain one review grew during the study period but remained constant between 2019 and 2020, despite a vast increase in submitted manuscripts. Hence, the burden for unique reviewers did not increase. Instead, the proportion of desk-rejected manuscripts grew, and the reviewer pool expanded, which allowed the annual average number of reviews by individual reviewers to remain stable.

Asthma in Adult Patients with COVID-19. Prevalence and Risk of Severe Disease

Rationale: Health outcomes of people with coronavirus disease (COVID-19) range from no symptoms to severe illness and death. Asthma, a common chronic lung disease, has been considered likely to increase the severity of COVID-19, although data addressing this hypothesis have been scarce until very recently.Objectives: To review the epidemiologic literature related to asthma's potential role in COVID-19 severity.Methods: Studies were identified through the PubMed (MEDLINE) and medRxiv (preprint) databases using the search terms "asthma," "SARS-CoV-2" (severe acute respiratory syndrome coronavirus 2), and "COVID-19," and by cross-referencing citations in identified studies that were available in print or online before December 22, 2020.Measurements and Main Results: Asthma prevalence data were obtained from studies of people with COVID-19 and regional health statistics. We identified 150 studies worldwide that allowed us to compare the prevalence of asthma in patients with COVID-19 by region, disease severity, and mortality. The results of our analyses do not provide clear evidence of increased risk of COVID-19 diagnosis, hospitalization, severity, or mortality due to asthma.Conclusions: These findings could provide some reassurance to people with asthma regarding its potential to increase their risk of severe morbidity from COVID-19.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

Predatory Publishers/Journals in Medical Sciences: How to Avoid, Stop, and What to Do after Being Scammed by Them?

PURPOSE

The phenomenon of predatory publishers and journals is one of the recent issues to affect the scientific field. It has negative impacts upon scientific research and harmful consequences, including the spread of pseudo-medical sciences. This article discusses general outlines regarding the concept of predatory journals to clarify this issue for junior researchers and academic workers.

METHODS

A brief description of the mechanism of publication within legitimate journals was highlighted. Moreover, the negative impact of this behavior was clarified along with several suggestions that may help authors to avoid predatory journals. Finally, post-predatory publication procedures were discussed, followed by practical steps to eliminate or minimize this phenomenon.

CONCLUSION

The practice of publishing in the predatory journals needs to be addressed; raising this topic in all academic and research institutions may help minimize the impacts. Moreover, this topic must be part of the curriculum for both undergraduate and graduate courses to draw students' attention to this issue. Furthermore, actions should be taken against those who deliberately publish their work in such journals.

Open Accessibility in Education Research: Enhancing the Credibility, Equity, Impact, and Efficiency of Research

Openness is a foundational principle in science. Making the tools and products of scientific research openly accessible advances core aims and values of education researchers, such as the credibility, equity, impact, and efficiency of research. The digital revolution has expanded opportunities for providing greater access to research. In this article, we examine three open-science practices-open data and code, open materials, and open access-that education researchers can use to increase accessibility to the tools and products of research in the field. For each open-science practice, we discuss what the practice is and how it works, its primary benefits, some important limitations and challenges, and two thorny issues.

A Novel Individual Mentored Methodology to Peer Review for Residents/Fellows

Individualized guidance and assistance with constructive criticism as a mentored activity to peer review an article helps instill required rudiments, eliminate bad habits, and is shown to be beneficial to all participants. The Society of Laparoscopic & Robotic Surgeons initiated the R/F article mentoring review opportunity in 2014. The intimacy of actively debated discourse allows exposure to various peer review techniques and debate in tandem with education regarding the merits and faults of an article’s hypothesis and conclusions, and how they are evaluated for publication and responses to authors. The benefits of coaching reassessment of ideas, critical analysis, airing of disparate viewpoints; and the need to update, reinforce, and relearn science is not static and is more robust using this method.

Pandemic publication: correction and erratum in COVID-19 publications

This commentary identifies and characterizes correction and erratum in COVID-19 publications with a scientometric approach by considering their rate of growth, reasons for correction, the time-span between publishing the original and corrected versions, as well as their citation status in four questions. It also suggestions to solve the current issues regarding indexing, retrieving, publishing, and research evaluation.

What is meaningful research and how should we measure it?

We discuss the trend towards using quantitative metrics for evaluating research. We claim that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. After sketching what the future could look like if quantitative metrics are allowed to proliferate, we provide a more detailed discussion on why research is so difficult to evaluate and outline approaches for avoiding such a situation. In particular, we characterize meaningful research as an essentially contested concept and argue that quantitative metrics should always be accompanied by operationalized instructions for their proper use and continuously evaluated via feedback loops. Additionally, we analyze a dataset containing information about computer science publications and their citation history and indicate how quantitative metrics could potentially be calibrated via alternative evaluation methods such as test of time awards. Finally, we argue that, instead of over-relying on indicators, research environments should primarily be based on trust and personal responsibility.

Open up: a survey on open and non-anonymized peer reviewing

BACKGROUND

Our aim is to highlight the benefits and limitations of open and non-anonymized peer review. Our argument is based on the literature and on responses to a survey on the reviewing process of alt.chi, a more or less open review track within the so-called Computer Human Interaction (CHI) conference, the predominant conference in the field of human-computer interaction. This track currently is the only implementation of an open peer review process in the field of human-computer interaction while, with the recent increase in interest in open scientific practices, open review is now being considered and used in other fields.

METHODS

We ran an online survey with 30 responses from alt.chi authors and reviewers, collecting quantitative data using multiple-choice questions and Likert scales. Qualitative data were collected using open questions.

RESULTS

Our main quantitative result is that respondents are more positive to open and non-anonymous reviewing for alt.chi than for other parts of the CHI conference. The qualitative data specifically highlight the benefits of open and transparent academic discussions. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

CONCLUSION

While the benefits are quite clear and the system is generally well-liked by alt.chi participants, they remain reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation.

The limitations to our understanding of peer review

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.

Open Science in the Humanities, or: Open Humanities?

Open science refers to both the practices and norms of more open and transparent communication and research in scientific disciplines and the discourse on these practices and norms. There is no such discourse dedicated to the humanities. Though the humanities appear to be less coherent as a cluster of scholarship than the sciences are, they do share unique characteristics which lead to distinct scholarly communication and research practices. A discourse on making these practices more open and transparent needs to take account of these characteristics. The prevalent scientific perspective in the discourse on more open practices does not do so, which confirms that the discourse’s name, open science, indeed excludes the humanities so that talking about open science in the humanities is incoherent. In this paper, I argue that there needs to be a dedicated discourse for more open research and communication practices in the humanities, one that integrates several elements currently fragmented into smaller, unconnected discourses (such as on open access, preprints, or peer review). I discuss three essential elements of open science—preprints, open peer review practices, and liberal open licences—in the realm of the humanities to demonstrate why a dedicated open humanities discourse is required.

Integrating Molecular Biology and Bioinformatics Education

Combined awareness about the power and limitations of bioinformatics and molecular biology enables advanced research based on high-throughput data. Despite an increasing demand of scientists with a combined background in both fields, the education of dry and wet lab subjects are often still separated. This work describes an example of integrated education with a focus on genomics and transcriptomics. Participants learned computational and molecular biology methods in the same practical course. Peer-review was applied as a teaching method to foster cooperative learning of students with heterogeneous backgrounds. The positive evaluation results indicate that this approach was accepted by the participants and would likely be suitable for wider scale application.

Ten Hot Topics around Scholarly Publishing

The changing world of scholarly communication and the emerging new wave of ‘Open Science’ or ‘Open Research’ has brought to light a number of controversial and hotly debated topics. Evidence-based rational debate is regularly drowned out by misinformed or exaggerated rhetoric, which does not benefit the evolving system of scholarly communication. This article aims to provide a baseline evidence framework for ten of the most contested topics, in order to help frame and move forward discussions, practices, and policies. We address issues around preprints and scooping, the practice of copyright transfer, the function of peer review, predatory publishers, and the legitimacy of ‘global’ databases. These arguments and data will be a powerful tool against misinformation across wider academic research, policy and practice, and will inform changes within the rapidly evolving scholarly publishing system.

Publish-and-Flourish: Using Blockchain Platform to Enable Cooperative Scholarly Communication

Scholarly communication is today immersed in publish-or-perish culture that propels non-cooperative behavior in the sense of strategic games played by researchers. Here we introduce and describe a blockchain based platform for decentralized scholarly communication. The design of the platform rests on community driven publishing reviewing processes and implements cryptoeconomic incentives that promote cooperative user behavior. The key to achieve cooperation in blockchain based scholarly communication is to transform today’s static research paper into a modifiable research paper under continuous peer review process. We introduce and discuss the implementation of a modifiable research paper as a smart contract on the blockchain.

Transparent Attribution of Contributions to Research: Aligning Guidelines to Real-Life Practices

Research studies, especially in the sciences, may benefit from substantial non-author support without which they could not be completed or published. The term “contributorship” was coined in 1997 to recognize all contributions to a research study, but its implementation (mostly in biomedical reports) has been limited to the inclusion of an “Author Contributions” statement that omits other contributions. To standardize the reporting of contributions across disciplines, irrespective of whether a given contribution merits authorship or acknowledgment, the Contributor Roles Taxonomy (CRediT) was launched in 2014. Our assessment, however, shows that in practice, CRediT is a detailed authorship classification that risks denying appropriate credit for persons who contribute as non-authors. To illustrate the shortcomings in CRediT and suggest improvements, in this article we review key concepts of authorship and contributorship and examine the range of non-author contributions that may (or may not) be acknowledged. We then briefly describe different types of editorial support provided by (non-author) translators, authors’ editors and writers, and explain why it is not always acknowledged. Finally, we propose two new CRediT taxa and revisions to three existing taxa regarding both technical and editorial support, as a small but important step to make credit attribution more transparent, accurate and open.

Peer Review Bias: A Critical Review

Various types of bias and confounding have been described in the biomedical literature that can affect a study before, during, or after the intervention has been delivered. The peer review process can also introduce bias. A compelling ethical and moral rationale necessitates improving the peer review process. A double-blind peer review system is supported on equipoise and fair-play principles. Triple- and quadruple-blind systems have also been described but are not commonly used. The open peer review system introduces "Skin in the Game" heuristic principles for both authors and reviewers and has a small favorable effect on the quality of published reports. In this exposition, we present, on the basis of a comprehensive literature search of PubMed from its inception until October 20, 2017, various possible mechanisms by which the peer review process can distort research results, and we discuss the evidence supporting different strategies that may mitigate this bias. It is time to improve the quality, transparency, and accountability of the peer review system.

Guidelines for open peer review implementation

Open peer review (OPR) is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. As more journals move to implement and experiment with the myriad of innovations covered by this term, there is a clear need for best practice guidelines to guide implementation. This brief article aims to address this knowledge gap, reporting work based on an interactive stakeholder workshop to create best-practice guidelines for editors and journals who wish to transition to OPR. Although the advice is aimed mainly at editors and publishers of scientific journals, since this is the area in which OPR is at its most mature, many of the principles may also be applicable for the implementation of OPR in other areas (e.g., books, conference submissions).

Preprints in Scholarly Communication: Re-Imagining Metrics and Infrastructures

Digital scholarship and electronic publishing within scholarly communities change when metrics and open infrastructures take center stage for measuring research impact. In scholarly communication, the growth of preprint repositories as a new model of scholarly publishing over the last three decades has been one of the major developments. As it unfolds, the landscape of scholarly communication is transitioning—with much being privatized as it is made open—and turning towards alternative metrics, such as social media attention, author-level, and article-level metrics. Moreover, the granularity of evaluating research impact through new metrics and social media changes the objective standards of evaluating research performance. Using preprint repositories as a case study, this article situates them in a scholarly web, examining their salient features, benefits, and futures. Moves towards scholarly web development and publishing on the semantic and social web with open infrastructures, citations, and alternative metrics—how preprints advance building the web as data—is discussed. We determine that this will viably demonstrate new metrics and, by enhancing research publishing tools in the scholarly commons, facilitate various communities of practice. However, for preprint repositories to be sustainable, scholarly communities and funding agencies should support continued investment in open knowledge, alternative metrics development, and open infrastructures in scholarly publishing.

The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future

Review, promotion, and tenure (RPT) processes significantly affect how faculty direct their own career and scholarly progression. Although RPT practices vary between and within institutions, and affect various disciplines, ranks, institution types, genders, and ethnicity in different ways, some consistent themes emerge when investigating what faculty would like to change about RPT. For instance, over the last few decades, RPT processes have generally increased the value placed on research, at the expense of teaching and service, which often results in an incongruity between how faculty actually spend their time vs. what is considered in their evaluation. Another issue relates to publication practices: most agree RPT requirements should encourage peer-reviewed works of high quality, but in practice, the value of publications is often assessed using shortcuts such as the prestige of the publication venue, rather than on the quality and rigor of peer review of each individual item. Open access and online publishing have made these issues even murkier due to misconceptions about peer review practices and concerns about predatory online publishers, which leaves traditional publishing formats the most desired despite their restricted circulation. And, efforts to replace journal-level measures such as the impact factor with more precise article-level metrics (e.g., citation counts and altmetrics) have been slow to integrate with the RPT process. Questions remain as to whether, or how, RPT practices should be changed to better reflect faculty work patterns and reduce pressure to publish in only the most prestigious traditional formats. To determine the most useful way to change RPT, we need to assess further the needs and perceptions of faculty and administrators, and gain a better understanding of the level of influence of written RPT guidelines and policy in an often vague process that is meant to allow for flexibility in assessing individuals.