Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution

'Science by consensus' impedes scientific creativity and progress: A simple alternative to funding biomedical research

The very low success rates of grant applications to the National Institutes of Health (NIH) and the National Science Foundation (NSF) are highly detrimental to the progress of science and the careers of scientists. The peer review process that evaluates proposals has been claimed arbitrarily to be the best there is. This consensus system, however, has never been evaluated scientifically against an alternative. Here we delineate the 15 major problems with the peer review process. We challenge the Science Advisor to the President, and the leadership of NIH, NSF, the U.S. National Academy of Sciences and other funding agencies throughout the world to refute each of these criticisms. We call for the implementation of more equitable alternatives that will not constrain the progress of science. We propose a system that will fund at least 80,000 principal investigators, including young scientists, with about half the current NIH budget, seven-times as many as the current number of NIH "research project grants," and that will forego the cumbersome, expensive, and counterproductive "peer" review stage. Further, we propose that the success of the two systems over 5-10 years be compared scientifically.

"It is becoming increasingly difficult to find reviewers"-myths and facts about peer review

A frequent complaint of editors of scientific journals is that it has become increasingly difficult to find reviewers for evaluating submitted manuscripts. Such claims are, most commonly, based on anecdotal evidence. To gain more insight grounded on empirical evidence, editorial data of manuscripts submitted for publication to the Journal of Comparative Physiology A between 2014 and 2021 were analyzed. No evidence was found that more invitations were necessary over time to get manuscripts reviewed; that the reviewer's response time after invitation increased; that the number of reviewers who completed their reports, relative to the number of reviewers who had agreed to review a manuscript, decreased; and that the recommendation behavior of reviewers changed. The only significant trend observed was among reviewers who completed their reports later than agreed. The average number of days that these reviewers submitted their evaluations roughly doubled over the period analyzed. By contrast, neither the proportion of late vs. early reviews, nor the time for completing the reviews among the punctual reviewers, changed. Comparison with editorial data from other journals suggests that journals that serve a smaller community of readers and authors, and whose editors themselves contact potential reviewers, perform better in terms of reviewer recruitment and performance than journals that receive large numbers of submissions and use editorial assistants for sending invitations to potential reviewers.

Peer reviewers' willingness to review, their recommendations and quality of reviews after the Finnish Medical Journal switched from single-blind to double-blind peer review

BACKGROUND

There is a power imbalance between authors and reviewers in single-blind peer review. We explored how switching from single-blind to double-blind peer review affected 1) the willingness of experts to review, 2) their publication recommendations, and 3) the quality of review reports.

METHODS

The Finnish Medical Journal switched from single-blind to double-blind peer review in September 2017. The proportion of review invitations that resulted in a received review report was counted. The reviewers' recommendations of "accept as is", "minor revision", "major revision" or "reject" were explored. The content of the reviews was assessed by two experienced reviewers using the Review Quality Instrument modified to apply to both original research and review manuscripts. The study material comprised reviews submitted from September 2017 to February 2018. The controls were the reviews submitted between September 2015 and February 2016 and between September 2016 and February 2017. The reviewers' recommendations and the scorings of quality assessments were tested with the Chi square test, and the means of quality assessments with the independent-samples t test.

RESULTS

A total of 118 double-blind first-round reviews of 59 manuscripts were compared with 232 single-blind first-round reviews of 116 manuscripts. The proportion of successful review invitations when reviewing single-blinded was 67%, and when reviewing double-blinded, 66%. When reviewing double-blinded, the reviewers recommended accept as is or minor revision less often than during the control period (59% vs. 73%), and major revision or rejection more often (41% vs 27%, P = 0.010). For the quality assessment, 116 reviews from the double-blind period were compared with 104 reviews conducted between September 2016 and February 2017. On a 1-5 scale (1 poor, 5 excellent), double-blind reviews received higher overall proportion of ratings of 4 and 5 than single-blind reviews (56% vs. 49%, P < 0.001). Means for the overall quality of double-blind reviews were 3.38 (IQR, 3.33-3.44) vs. 3.22 (3.17-3.28; P < 0.001) for single-blind reviews.

CONCLUSIONS

The quality of the reviews conducted double-blind was better than of those conducted single-blind. Switching to double-blind review did not alter the reviewers' willingness to review. The reviewers became slightly more critical.

Bridges of perspectives: representation of people with lived experience of spinal cord injury in editorial boards and peer review

BACKGROUND

Diversity among editorial boards and in the peer review process maximizes the likelihood that the dissemination of reported results is both relevant and respectful to readers and end users. Past studies have examined diversity among editorial board members and reviewers for factors such as gender, geographic location, and race, but limited research has explored the representation of people with disabilities. Here, we sought to understand the landscape of inclusivity of people with lived experience of spinal cord injury specifically in journals publishing papers (2012-2022) on their quality of life.

METHODS

An open and closed 12-question adaptive survey was disseminated to 31 journal editors over a one-month period beginning December 2022.

RESULTS

We received 10 fully completed and 5 partially completed survey responses (response rate 48%). Notwithstanding the small sample, over 50% (8/15) of respondents indicated that their journal review practices involve people with lived experience of spinal cord injury, signaling positive even if incomplete inclusivity practices. The most notable reported barriers to achieving this goal related to identifying and recruiting people with lived experience to serve in the review and editorial process.

CONCLUSIONS

In this study we found positive but incomplete trends toward inclusivity in journal practices involving people with lived experience of spinal cord injury. We recommend, therefore, that explicit and genuine efforts are directed toward recruitment through community-based channels. To improve representation even further, we suggest that editors and reviewers be offered the opportunity to self-identify as living with a disability without discrimination or bias.

The peer review process: A survey among scientists in radiology

PURPOSE

To map the experience and view of scientists in radiology on the peer review process.

METHOD

A survey with 12 closed-ended questions and 5 conditional sub-questions was conducted among corresponding authors who published in general radiology journals.

RESULTS

244 corresponding authors participated. In considering a peer review invitation, most respondents found the topic and the availability of time very important (62.1% [144/132] and 57.8% [134/232], respectively), the quality of the abstract, the prestige/impact factor of the journal, and the sense of professional duty important (43.7% [101/231], 42.2% [98/232], and 53.9% [125/232], respectively), and were indifferent about a reward (35.3% [82/232]). However, 61.1% (143/234) believed that a reviewer should be rewarded. Direct financial compensation (27.6% [42/152]), discounted fees for society memberships, conventions, and/or journal subscriptions (24.3% [37/152]), and Continuing Medical Education credits (23.0% [35/152]) were the most frequently desired rewards. 73.4% (179/244) of respondents never received formal peer review training, of whom 31.2% (54/173) would like to, particularly less experienced researchers (Chi-Square P = 0.001). The median reported review time per article was 2.5 h. 75.2% (176/234) of respondents found it acceptable that a manuscript is rejected by an editor without formal peer review. The double-blinded peer review model was preferred by most respondents (42.3% [99/234]). A median of 6 weeks was considered the maximum acceptable time from manuscript submission to initial decision by a journal.

CONCLUSION

Publishers and journal editors may use the experiences and views of authors that were provided in this survey to shape the peer review process.

Tips and guidelines for being a good peer reviewer

Publication is the key means by which science spreads. The purpose of scientific journals is to publish novel and quality articles. The editors of the journals evaluate the content of the manuscripts by submitting them to a process called «peer review», considered today the gold standard to guarantee the adequate publication of scientific articles. A well-crafted and critical peer-review report is a treasure for both authors and editors. In the present manuscript we will examine the key aspects of the peer review process. We will begin by explaining what exactly this process consists of and since when it has existed, and then clarifying why it is so important. Then we will argue why we should want to be reviewers of scientific papers. We will then review what are the fundamental rules to carry out a good review of a manuscript and what aspects of it we should focus on. Later we will see what format a peer review report should have and how to write its different sections, as well as the options for its final resolution. We will pay special attention to commenting on the ethical aspects and the most frequent errors that are made in the evaluation of manuscripts. Finally, we will recognize what the fundamental limitations of peer review are, and we will end by proposing some suggestions for their improvement. Our ultimate goal is to stimulate researchers -and authors- to go one step further and undertake the challenge of being peer reviewers of scientific manuscripts.

Peer Review in a General Medical Research Journal Before and During the COVID-19 Pandemic

Importance

Although peer review is an important component of publication for new research, the viability of this process has been questioned, particularly with the added stressors of the COVID-19 pandemic.

Objective

To characterize rates of peer reviewer acceptance of invitations to review manuscripts, reviewer turnaround times, and editor-assessed quality of reviews before and after the start of the COVID-19 pandemic at a large, open-access general medical journal.

Design, Setting, and Participants

This retrospective, pre-post cohort study examined all research manuscripts submitted to JAMA Network Open between January 1, 2019, and June 29, 2021, either directly or via transfer from other JAMA Network journals, for which at least 1 peer review of manuscript content was solicited. Measures were compared between the period before the World Health Organization declaration of a COVID-19 pandemic on March 11, 2020 (14.3 months), and the period during the pandemic (15.6 months) among all reviewed manuscripts and between pandemic-period manuscripts that did or did not address COVID-19.

Main Outcomes and Measures

For each reviewed manuscript, the number of invitations sent to reviewers, proportions of reviewers accepting invitations, time in days to return reviews, and editor-assessed quality ratings of reviews were determined.

Results

In total, the journal sought review for 5013 manuscripts, including 4295 Original Investigations (85.7%) and 718 Research Letters (14.3%); 1860 manuscripts were submitted during the prepandemic period and 3153 during the pandemic period. Comparing the prepandemic with the pandemic period, the mean (SD) number of reviews rated as high quality (very good or excellent) per manuscript increased slightly from 1.3 (0.7) to 1.5 (0.7) (P < .001), and the mean (SD) time for reviewers to return reviews was modestly shorter (from 15.8 [7.6] days to 14.4 [7.0] days; P < .001), a difference that persisted in linear regression models accounting for manuscript type, study design, and whether the manuscript addressed COVID-19.

Conclusions and Relevance

In this cohort study, the speed and editor-reported quality of peer reviews in an open-access general medical journal improved modestly during the initial year of the pandemic. Additional study will be necessary to understand how the pandemic has affected reviewer burden and fatigue.

The Role of Publons in the Context of Open Peer Review

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

Improving equity, diversity, and inclusion in academia

There are growing bodies of evidence demonstrating the benefits of equity, diversity, and inclusion (EDI) on academic and organizational excellence. In turn, some editors have stated their desire to improve the EDI of their journals and of the wider scientific community. The Royal Society of Chemistry established a minimum set of requirements aimed at improving EDI in scholarly publishing. Additionally, several resources were reported to have the potential to improve EDI, but their effectiveness and feasibility are yet to be determined. In this commentary we suggest six approaches, based on the Royal Society of Chemistry set of requirements, that journals could implement to improve EDI. They are: (1) adopt a journal EDI statement with clear, actionable steps to achieve it; (2) promote the use of inclusive and bias-free language; (3) appoint a journal’s EDI director or lead; (4) establish a EDI mentoring approach; (5) monitor adherence to EDI principles; and (6) publish reports on EDI actions and achievements. We also provide examples of journals that have implemented some of these strategies, and discuss the roles of peer reviewers, authors, researchers, academic institutes, and funders in improving EDI.

Editorial Work and the Peer Review Economy of STS Journals

In this paper, we analyze the role of science and technology studies (STS) journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political-economic and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy.

What works for peer review and decision-making in research funding: a realist synthesis

Introduction

Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.

Methods

Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.

Results

We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.

Conclusions

Reports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-022-00120-2.

Peer-reviewing in Surgical Journals: Revolutionize or Perish?

The gold standard of safe-guarding the quality of published science is peer review. However, this long-standing system has not evolved in today's digital world, where there has been an explosion in the number of publications and surgical journals. A journal's quality depends not only on the quality of papers submitted but is reflected upon the quality of its peer review process. Over the past decade journals are experiencing a rapidly escalating "peer review crisis" with editors struggling in recruiting reliable reviewers who will provide their skilled work for free with ever-diminishing incentives within today's restricted time-constraints. The problem is complex and difficult to solve, but more urgent than ever. Time is valuable and academicians, researchers and clinicians are overburdened and already extremely busy publishing their own research along with their ever growing clinical and administrative duties. Fewer and fewer individuals volunteer to provide their skilled work for free which is expected. The current incentives to review do not have a big impact on one's career and therefore are not realistic effective countermeasures. As the limits of the system are constantly stretched, there will inevitably come a "point of no return" and Surgical Journals will be the ones to first take the hit as there is an overwhelming evidence of burnout in the surgical specialties and the Surgical community is almost 50% smaller than its Medical counterpart. This review identifies the potential causes of the peer-review crisis, outlines the incentives and drawbacks of being a reviewer, summarizes the currently established common practices of rewarding reviewers and the existing and potential solutions to the problem. The magnitude of the problem and unsustainability that will make it perish are discussed along with its current flaws. Finally, recommendations are made to address many of the weaknesses of the system with the hope to revive it.

Converting Access Microbiology to an open research platform: community survey results

Following the Microbiology Society’s successful bid for a Learned Society Curation Award from the Wellcome Trust and Howard Hughes Medical Institute, the Society is converting our sound science, open access journal, Access Microbiology, to an open research platform. As part of this, we conducted a survey of our community to gauge current attitudes towards the platform and here we present some of these results. The majority of respondents (57 %) said they would always or sometimes want to remain anonymous on their peer review report, whilst 75 % of respondents said that as an author they would be happy to make the data underlying their research open. There was a clear desire for a range of research types that are often seen with sound science publications and rigorous research. An encouraging 94 % of respondents stated that the platform is somewhere they would consider publishing, demonstrating the enthusiasm in these respondents for a new publishing platform for their community. Given this data and that from our previous focus group research, the platform will launch as outlined in the original project proposal and adopt a transparent peer review model with an open data policy.

Nonfinancial conflict of interest in peer-review: Some notes for discussion

Strong beliefs can influence the way we deal with emotionally charged topics. Researchers, editors, and reviewers are not an exception. Declaring such nonfinancial conflict of interest when handling or reviewing submitted articles is often obligatory; however, the declaration is not a license to submit a biased review with personal insults or to break the journal's guidelines. This kind of poor practice can be a clear sign of the seriousness of conflict of interest. In this article, I argue that hostile, unethical, and biased behavior of reviewers and editors often arises from a serious nonfinancial conflict of interest, which should not be ignored or undermined.

Shared burden is always lighter - Peer-review performance in an ophthalmological journal 2010-2020

PURPOSE

There are concerns in the academic publishing community that it is becoming more difficult to secure reviews for scientific manuscripts. This study examines trends in editorial and peer review processes in an ophthalmological journal over the last decade.

METHODS

A retrospective analysis was performed of editorial data from the journal Acta Ophthalmologica containing all manuscript submissions between 2010 and 2020.

RESULTS

The number of yearly submissions grew between 2010 and 2019 from 1014 to 1623, and in 2020, the number of submissions increased to 2449. In total, the number of submissions increased by 142% between 2010 and 2020. Similarly, the proportion of desk-rejected manuscripts increased from 48% to 67% during the period 2010-2020. The number of invitations needed to obtain one review showed an increase from 1.9 to 2.6 between 2010 and 2019, but remained stable between 2019 and 2020. However, the number of reviewers per reviewed manuscript, reviewed manuscripts per reviewer and time from invitation to completed review assignment remained almost constant between 2010 and 2020. Researchers based in North American were disproportionally often invited to review (18%) compared to their share of published articles (7%), and they also declined review invitation more frequently compared to scholars in other parts of the world.

CONCLUSIONS

The study revealed an increase in submitted manuscripts to an ophthalmological journal over the last decade, with a further increase during the COVID-19 pandemic. The number of reviewer invitations needed to obtain one review grew during the study period but remained constant between 2019 and 2020, despite a vast increase in submitted manuscripts. Hence, the burden for unique reviewers did not increase. Instead, the proportion of desk-rejected manuscripts grew, and the reviewer pool expanded, which allowed the annual average number of reviews by individual reviewers to remain stable.

Letter to the Editor: publish, publish … cursed!

This paper comments on the phenomenon of “publish or perish” associated with the current information explosion and its awful consequence: the curse that hangs over academia which dooms it to publish incessantly irrelevant and pointless documents. The overabundance of publications is not justified and is not even necessary in many contexts for personal promotion, and even less for the advancement of science. Therefore, the current role of scientific journals is highly questionable that its aim could be misleading. Huge numbers of articles are published, but they are not read because the aim is principally “publish for publish,” or publication for its own sake. The standard corrective tool for improving scientific communication—peer review—cannot function adequately, and biases and perversions are introduced which undermine society’s confidence in the scientific enterprise. A dark landscape unfurls itself across the world of scientific information, forcing us to question and improve its current state. Methodologically this paper goes halfway between the essay and the review trying to provoke engaged and useful controversy.

What is meaningful research and how should we measure it?

We discuss the trend towards using quantitative metrics for evaluating research. We claim that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. After sketching what the future could look like if quantitative metrics are allowed to proliferate, we provide a more detailed discussion on why research is so difficult to evaluate and outline approaches for avoiding such a situation. In particular, we characterize meaningful research as an essentially contested concept and argue that quantitative metrics should always be accompanied by operationalized instructions for their proper use and continuously evaluated via feedback loops. Additionally, we analyze a dataset containing information about computer science publications and their citation history and indicate how quantitative metrics could potentially be calibrated via alternative evaluation methods such as test of time awards. Finally, we argue that, instead of over-relying on indicators, research environments should primarily be based on trust and personal responsibility.

The limitations to our understanding of peer review

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.

How Many Papers Should Scientists Be Reviewing? An Analysis Using Verified Peer Review Reports

The current peer review system is under stress from ever increasing numbers of publications, the proliferation of open-access journals and an apparent difficulty in obtaining high-quality reviews in due time. At its core, this issue may be caused by scientists insufficiently prioritising reviewing. Perhaps this low prioritisation is due to a lack of understanding on how many reviews need to be conducted by researchers to balance the peer review process. I obtained verified peer review data from 142 journals across 12 research fields, for a total of over 300,000 reviews and over 100,000 publications, to determine an estimate of the numbers of reviews required per publication per field. I then used this value in relation to the mean numbers of authors per publication per field to highlight a ‘review ratio’: the expected minimum number of publications an author in their field should review to balance their input (publications) into the peer review process. On average, 3.49 ± 1.45 (SD) reviews were required for each scientific publication, and the estimated review ratio across all fields was 0.74 ± 0.46 (SD) reviews per paper published per author. Since these are conservative estimates, I recommend scientists aim to conduct at least one review per publication they produce. This should ensure that the peer review system continues to function as intended.

Librarians as methodological peer reviewers for systematic reviews: results of an online survey

BACKGROUND

Developing a comprehensive, reproducible literature search is the basis for a high-quality systematic review (SR). Librarians and information professionals, as expert searchers, can improve the quality of systematic review searches, methodology, and reporting. Likewise, journal editors and authors often seek to improve the quality of published SRs and other evidence syntheses through peer review. Health sciences librarians contribute to systematic review production but little is known about their involvement in peer reviewing SR manuscripts.

METHODS

This survey aimed to assess how frequently librarians are asked to peer review systematic review manuscripts and to determine characteristics associated with those invited to review. The survey was distributed to a purposive sample through three health sciences information professional listservs.

RESULTS

There were 291 complete survey responses. Results indicated that 22% (n = 63) of respondents had been asked by journal editors to peer review systematic review or meta-analysis manuscripts. Of the 78% (n = 228) of respondents who had not already been asked, 54% (n = 122) would peer review, and 41% (n = 93) might peer review. Only 4% (n = 9) would not review a manuscript. Respondents had peer reviewed manuscripts for 38 unique journals and believed they were asked because of their professional expertise. Of respondents who had declined to peer review (32%, n = 20), the most common explanation was "not enough time" (60%, n = 12) followed by "lack of expertise" (50%, n = 10).The vast majority of respondents (95%, n = 40) had "rejected or recommended a revision of a manuscript| after peer review. They based their decision on the "search methodology" (57%, n = 36), "search write-up" (46%, n = 29), or "entire article" (54%, n = 34). Those who selected "other" (37%, n = 23) listed a variety of reasons for rejection, including problems or errors in the PRISMA flow diagram; tables of included, excluded, and ongoing studies; data extraction; reporting; and pooling methods.

CONCLUSIONS

Despite being experts in conducting literature searches and supporting SR teams through the review process, few librarians have been asked to review SR manuscripts, or even just search strategies; yet many are willing to provide this service. Editors should involve experienced librarians with peer review and we suggest some strategies to consider.

Gender diversity of editorial boards and gender differences in the peer review process at six journals of ecology and evolution

Despite substantial progress for women in science, women remain underrepresented in many aspects of the scholarly publication process. We examined how the gender diversity of editors and reviewers changed over time for six journals in ecology and evolution (2003–2015 for four journals, 2007–2015 or 2009–2015 for the other two), and how several aspects of the peer review process differed between female and male editors and reviewers. We found that for five of the six journals, women were either absent or very poorly represented as handling editors at the beginning of our dataset. The representation of women increased gradually and consistently, with women making up 29% of the handling editors (averaged across journals) in 2015, similar to the representation of women as last authors on ecology papers (23% in 2015) but lower than the proportion of women among all authors (31%) and among members of the societies that own the journals (37%–40%). The proportion of women among reviewers has also gradually but consistently increased over time, reaching 27% by 2015. Female editors invited more female reviewers than did male editors, and this difference increased with age of the editor. Men and women who were invited to review did not differ in whether they responded to the review invitation, but, of those that responded, women were slightly more likely to agree to review. In contrast, women were less likely than men to accept invitations to serve on journal editorial boards. Our analyses indicate that there has been progress in the representation of women as reviewers and editors in ecology and evolutionary biology, but women are still underrepresented among the gatekeepers of scholarly publishing relative to their representation among researchers.

The state of the art in peer review

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.

The effectiveness of journals as arbiters of scientific impact

Academic publishers purport to be arbiters of knowledge, aiming to publish studies that advance the frontiers of their research domain. Yet the effectiveness of journal editors at identifying novel and important research is generally unknown, in part because of the confidential nature of the editorial and peer review process. Using questionnaires, we evaluated the degree to which journals are effective arbiters of scientific impact on the domain of Ecology, quantified by three key criteria. First, journals discriminated against low-impact manuscripts: The probability of rejection increased as the number of citations gained by the published paper decreased. Second, journals were more likely to publish high-impact manuscripts (those that obtained citations in 90th percentile for their journal) than run-of-the-mill manuscripts; editors were only 23% and 41% as likely to reject an eventual high-impact paper (pre- versus postreview rejection) compared to a run-of-the-mill paper. Third, editors did occasionally reject papers that went on to be highly cited. Error rates were low, however: Only 3.8% of rejected papers gained more citations than the median article in the journal that rejected them, and only 9.2% of rejected manuscripts went on to be high-impact papers in the (generally lower impact factor) publishing journal. The effectiveness of scientific arbitration increased with journal prominence, although some highly prominent journals were no more effective than much less prominent ones. We conclude that the academic publishing system, founded on peer review, appropriately recognizes the significance of research contained in manuscripts, as measured by the number of citations that manuscripts obtain after publication, even though some errors are made. We therefore recommend that authors reduce publication delays by choosing journals appropriate to the significance of their research.

Research Integrity and Peer Review-past highlights and future directions

In May 2016, we launched Research Integrity and Peer Review, an international, open access journal with fully open peer review (reviewers are identified on their reports and named reports are published alongside the article) to provide a home for research on research and publication ethics, research reporting, and research on peer review. As the journal enters its third year, we reflect on recent events and highlights for the journal and explore how the journal is faring in terms of gender and diversity in peer review. We also share the particular interests of our Editors-in-Chief regarding models of peer review, reporting quality, common research integrity issues that arise during the publishing process, and how people interact with the published literature. We continue to encourage further research into peer review, research and publication ethics and research reporting, as we believe that all new initiatives should be evidence-based. We also remain open to constructive discussions of the developments in the field that offer new solutions.

The ClarivateTM Analytics acquisition of Publons – an evolution or commodification of peer review?

Without peer reviewers, the entire scholarly publishing system as we currently know it would collapse. However, as it currently stands, publishing is an extremely exploitative system, relative to other business models, in which trained and specialized labor is exploited, in the form of editors and peer reviewers, primarily by for-profit publishers, in return for a pat on the back, and a public nod of thanks. This is the “standardized” and “accepted” form for deriving mainstream peer reviewed literature. However, except for open peer review, where reports are open and identities are known, traditional peer review is closed, and the content of peer reports is known only to the authors and editors involved. Publons launched in 2012 as a platform that would offer recognition to peer reviewers for their work. In 2016, Publons rewarded the most productive reviewers with a “Sentinels of Science” award, accompanied by a dismal monetary reward (38 US cents/review) for their efforts. A site aimed at registering pre- and post-publication peer efforts, Publons was perceived as a positive step towards a more transparent peer review system. However, the continued presence of fake peer reviews and a spike in retractions, even among publishers that were Publons sponsors, suggests that perhaps peers may be exploiting Publons to get recognition for superficial or poor peer review. Since all reviews are not public, their content and quality cannot be verified. On 1 June 2017, ClarivateTM Analytics, which owns the journal impact factor—most likely the most gamed non-academic factor in academic publishing—which is a measure of the number of citations of papers in journals, many of which are published by the for-profit publishers—including Publons sponsors—that “employ” free peer reviewers to quality check the literature they then sell for profit, purchased Publons. Touting the purchase as a way to increase transparency, and stamp out fake peer review, some who had supported Publons felt betrayed, even cancelling their Publons accounts immediately when learning of this purchase. Their concerns included the possible “gaming” of peer review as had taken place with the journal impact factor. This commentary examines possible positive and negative aspects of this business transaction, and what it might mean to academics and publishers.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of Web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform current models while avoiding as many of the biases of existing systems as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that, at least partially, resolves many of the technical and social issues associated with peer review, and can potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Collister L, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Greshake Tzovaras B, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Nørgaard Kjær J, O'Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.