The limitations to our understanding of peer review

Identifying and Minimizing Incentives for Competing Interests in Sports Medicine Publications

Academics in sports medicine as well as other medical fields are generally expected to publish research and opinions in peer-reviewed journals. The peer-review process is intended to protect against the publication of flawed research and unsubstantiated claims. However, both financial and non-financial competing interests may result in sub-optimal results by affecting investigators, editors, peer reviewers, academic institutions, and publishers. In this article, we focus on the non-financial competing interests created in our current academic system. Because these competing interests are embedded in our current scholastic framework, the potential biases are difficult to quantify. To minimize the effect of these competing interests, we review and highlight some underlying incentives for each stakeholder and some potential solutions to mitigate their effects.

Better nanoscience through open, collaborative, and critical discussions

We aim to foster a discussion of science correction and of how individual researchers can improve the quality and control of scientific production. This is crucial because although the maintenance of rigorous standards and the scrupulous control of research findings and methods are sometimes taken for granted, in practice, we are routinely confronted with articles that contain errors.

The Peer Review System: A Journal Editor's 30-Year Perspective

The peer review system has become the standard by which scientific articles are refereed. Unfortunately, even from its beginnings in the mid-1800s it has been fraught with difficulties. Potential reviewers are volunteers who may be inundated with requests to review yet these reviews take considerable time and effort. There is little motivation to complete a review causing significant delays in the publication process. There may be biases unintentionally built into the system between reviewers, authors, editors, and journals. Attempts to overcome these biases by various blinding schemes have been met with limited success. Finally, the recent advent of Artificial Intelligence has the potential to completely upend the system, for good or bad.

Comments about comments: peer review and the amazing editorial board of Brain Communications

Our editor discusses our editorial board members, who come from eight countries on four continents, and wider issues of the peer review system.

Editors' perspectives on the selection of reviewers and the quality of reviews

This article describes the outcomes of a survey of 93 editors in chief and associate editors of behavior-analytic journals. We sought information about variables that influence their judgment of the selection of reviewers, selection of review panels, and quality of reviews. When selecting reviewers, participants rated highly expertise on the topic, history of conducting good reviews, and history of writing constructive and respectful reviews. When selecting review panels, participants rated highly stratifying reviewers based on their expertise, avoiding conflicts of interest, and the matching based on the area of expertise between reviewers and authors. When evaluating the quality of a review, participants rated highly considerations related to research design, the science underlying the main idea, and accurate interpretations of the data. Participants did not rate copyediting as important. Overall, the extent to which reviewer selection was influenced by membership in underrepresented groups varied. These findings can inform the development of training programs for teaching peer-review repertoires.

ChatGPT identifies gender disparities in scientific peer review

The peer review process is a critical step in ensuring the quality of scientific research. However, its subjectivity has raised concerns. To investigate this issue, I examined over 500 publicly available peer review reports from 200 published neuroscience papers in 2022-2023. OpenAI's generative artificial intelligence ChatGPT was used to analyze language use in these reports, which demonstrated superior performance compared to traditional lexicon- and rule-based language models. As expected, most reviews for these published papers were seen as favorable by ChatGPT (89.8% of reviews), and language use was mostly polite (99.8% of reviews). However, this analysis also demonstrated high levels of variability in how each reviewer scored the same paper, indicating the presence of subjectivity in the peer review process. The results further revealed that female first authors received less polite reviews than their male peers, indicating a gender bias in reviewing. In addition, published papers with a female senior author received more favorable reviews than papers with a male senior author, for which I discuss potential causes. Together, this study highlights the potential of generative artificial intelligence in performing natural language processing of specialized scientific texts. As a proof of concept, I show that ChatGPT can identify areas of concern in scientific peer review, underscoring the importance of transparent peer review in studying equitability in scientific publishing.

A health sciences researcher's experience of manuscript review comments, 2020-2022

BACKGROUND

 Peer review frequently improves a manuscript, but authors may consider some reviewer feedback negative, inappropriate or unclear. This study aims to summarise and analyse review comments received by authors.

METHODS

 This longitudinal study included all submissions of which the researcher was an author, reviewed by any journal during 2020-2022. First-round reviews were retrieved from emails and documents received by the authors or the faculty's medical editors or the journal platforms. A confidential datasheet with review items compiled from literature and the researcher's experience as author and reviewer was completed for each submission. Review comments were noted verbatim for subjective items such as rude or vague statements.

RESULTS

 The 65 submissions received 118 reviews from 36 journals, mainly in the form of unstructured narrative reports (59%). The majority of first-round reviews (58%), including those for rejected submissions, contained some positive comments. Reviewers frequently (75% of reviews, 88% of submissions) required some expansion of information. Vague and incorrect statements occurred in 15% and 18% of reviews, respectively. Only two reviews contained statements that could be considered rude. The types of comments made were associated with the review format.

CONCLUSION

 The majority of reviews contained some positive comments and rude comments were extremely rare. Reviewers frequently requested the expansion of information provided.Contribution: This study gives insight to authors, reviewers and editors regarding the type and tone of review comments. This could guide authors during manuscript preparation and authors, reviewers and editors during the review process.

The PRISMATIC project: protocol for a research programme on novel methods to improve reporting and peer review of systematic reviews of health evidence

BACKGROUND

Incomplete reporting about what systematic reviewers did and what they found prevents users of the report from being able to fully interpret the findings and understand the limitations of the underlying evidence. Reporting guidelines such as the PRISMA statement and its extensions are designed to improve reporting. However, there are important inconsistencies across the various PRISMA reporting guidelines, which causes confusion and misinterpretation. Coupled with this, users might need to consult multiple guidelines to gain a full understanding of the guidance. Furthermore, the current passive strategy of implementing PRISMA has not fully brought about needed improvements in the completeness of systematic review reporting.

METHODS

The PRISMATIC ('PRISMA, Technology, and Implementation to enhance reporting Completeness') project aims to use novel methods to enable more efficient and effective translation of PRISMA reporting guidelines into practice. We will establish a working group who will develop a unified PRISMA statement that harmonises content across the main PRISMA guideline and several of its extensions. We will then develop a web application that generates a reporting template and checklist customised to the characteristics and methods of a systematic review ('PRISMA-Web app') and conduct a randomised trial to evaluate its impact on authors' reporting. We will also develop a web application that helps peer reviewers appraise systematic review manuscripts ('PRISMA-Peer app') and conduct a diagnostic accuracy study to evaluate its impact on peer reviewers' detection of incomplete reporting.

DISCUSSION

We anticipate the novel guidance and web-based apps developed throughout the project will substantively enhance the completeness of reporting of systematic reviews of health evidence, ultimately benefiting users who rely on systematic reviews to inform health care decision-making.

A critical scientific evaluation of a purportedly negative data report - response to Seneviratne et al. 2022

A core principle in the pursuit of scientific knowledge is that science is self-correcting and that important results should be replicable. Hypotheses need to be reinforced, adjusted, or rejected when novel results are obtained. Replication of results confirms hypotheses and enhances their integration into scientific practice. In contrast, publication of substantiated and replicated negative findings (i.e., non-significant or opposite findings) can be the basis to reject erroneous hypotheses or develop alternative strategies for investigation. Replication is a problem in all research fields. The Psychology Reproductivity Project reported that only 36% of 'highly influential' published research in highly ranked journals were reproduced. Similar to positive data, negative data can be flawed. Errors in a negative data set can be based on methodology, statistics, conceptual defects, and flawed peer review. The peer review process has received progressive scrutiny. A large-scale review of the peer review process of manuscripts submitted to the British Medical Journal group indicated that the process could be characterized as inconsistent, inaccurate, and biased. Further analysis indicated that the peer process is easily manipulated, indicative of a failed system, is a major factor behind the lack of replication in science (acceptance of flawed manuscripts), suppresses opposing scientific evidence and views, and causes gaps in and lack of growth of science. Complicating the integrity of scientific publication is the role of Editors/Researchers. Ethical guidelines exist for major publishing houses about editorial ethics, behavior, and practice.

Analysis of Retracted Manuscripts in Chemistry: Errors vs Misconduct

In this Viewpoint we discuss Chemistry manuscripts retracted during the 2001-2021 period (a total of 1292 journal articles retrieved from the Retraction Watch database). We showed that 58.5% of Chemistry manuscripts were retracted due to misconduct; of them, 40.5% of retractions were due to self-plagiarism and 36% due to fraud. Errors and concerns unrelated to misconduct constituted 26% of all retractions. Retracted manuscripts had a median retraction time of 1.7 years and peer-review time of 71 days (but only 43 days for fraudulent manuscripts).

Scope and financial impact of unpublished data and unused samples among U.S. academic and government researchers

Unpublished data and unused samples are common byproducts of research activity, but little is known about the scope and economic impact of their disuse. To fill this knowledge gap, we collected self-reported anonymous survey responses from 301 academic and government scientists from randomly selected institutions. Respondents estimated that they published ∼60% of their data and 95% had unpublished data. Of those collecting specimens, 60% stored unused samples. Systemic and logistical issues were identified as major contributory factors. The median cumulative self-reported estimated value of unused resources per researcher was $28,857, with life science ($36k) and government ($109k) researchers reporting the costliest assets. Using NSF headcounts, we estimated that the current cumulative value of unused resources at universities is approximately $6.2 billion, about 7% of the current annual R&D budget. These findings provide actionable information that can be used by decision makers to reduce obstacles that undermine scientific progress and productivity.

The rise of preprints in earth sciences

The rate of science information's spread has accelerated in recent years. In this context, it appears that many scientific disciplines are beginning to recognize the value and possibility of sharing open access (OA) online manuscripts in their preprint form. Preprints are academic papers that are published but have not yet been evaluated by peers. They have existed in research at least since the 1960s and the creation of ArXiv in physics and mathematics. Since then, preprint platforms-which can be publisher- or community-driven, profit or not for profit, and based on proprietary or free and open source software-have gained popularity in many fields (for example, bioRxiv for the biological sciences). Today, there are many platforms that are either disciplinary-specific or cross-domain, with exponential development over the past ten years. Preprints as a whole still make up a very small portion of scholarly publishing, but a large group of early adopters are testing out these value-adding tools across a much wider range of disciplines than in the past. In this opinion article, we provide perspective on the three main options available for earth scientists, namely EarthArXiv, ESSOAr/ESS Open Archive and EGUsphere.

How to improve scientific peer review: Four schools of thought

Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review. Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved. To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school. Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between them. We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.

Scientists' perception of pilot study quality was influenced by statistical significance and study design

OBJECTIVES

Preliminary studies play a key role in developing large-scale interventions but may be held to higher or lower scientific standards during the peer review process because of their preliminary study status.

STUDY DESIGN AND SETTING

Abstracts from 5 published obesity prevention preliminary studies were systematically modified to generate 16 variations of each abstract. Variations differed by 4 factors: sample size (n = 20 vs. n = 150), statistical significance (P < 0.05 vs. P > 0.05), study design (single group vs. randomized 2 groups), and preliminary study status (presence/absence of pilot language). Using an online survey, behavioral scientists were provided with a randomly selected variation of each of the 5 abstracts and blinded to the existence of other variations. Respondents rated each abstract on aspects of study quality.

RESULTS

Behavioral scientists (n = 271, 79.7% female, median age 34 years) completed 1,355 abstract ratings. Preliminary study status was not associated with perceived study quality. Statistically significant effects were rated as more scientifically significant, rigorous, innovative, clearly written, warranted further testing, and had more meaningful results. Randomized designs were rated as more rigorous, innovative, and meaningful.

CONCLUSION

Findings suggest reviewers place a greater value on statistically significant findings and randomized control design and may overlook other important study characteristics.

The peer review process: A survey among scientists in radiology

PURPOSE

To map the experience and view of scientists in radiology on the peer review process.

METHOD

A survey with 12 closed-ended questions and 5 conditional sub-questions was conducted among corresponding authors who published in general radiology journals.

RESULTS

244 corresponding authors participated. In considering a peer review invitation, most respondents found the topic and the availability of time very important (62.1% [144/132] and 57.8% [134/232], respectively), the quality of the abstract, the prestige/impact factor of the journal, and the sense of professional duty important (43.7% [101/231], 42.2% [98/232], and 53.9% [125/232], respectively), and were indifferent about a reward (35.3% [82/232]). However, 61.1% (143/234) believed that a reviewer should be rewarded. Direct financial compensation (27.6% [42/152]), discounted fees for society memberships, conventions, and/or journal subscriptions (24.3% [37/152]), and Continuing Medical Education credits (23.0% [35/152]) were the most frequently desired rewards. 73.4% (179/244) of respondents never received formal peer review training, of whom 31.2% (54/173) would like to, particularly less experienced researchers (Chi-Square P = 0.001). The median reported review time per article was 2.5 h. 75.2% (176/234) of respondents found it acceptable that a manuscript is rejected by an editor without formal peer review. The double-blinded peer review model was preferred by most respondents (42.3% [99/234]). A median of 6 weeks was considered the maximum acceptable time from manuscript submission to initial decision by a journal.

CONCLUSION

Publishers and journal editors may use the experiences and views of authors that were provided in this survey to shape the peer review process.

The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research

Artificial Intelligence (AI) has revolutionized various domains, including education and research. Natural language processing (NLP) techniques and large language models (LLMs) such as GPT-4 and BARD have significantly advanced our comprehension and application of AI in these fields. This paper provides an in-depth introduction to AI, NLP, and LLMs, discussing their potential impact on education and research. By exploring the advantages, challenges, and innovative applications of these technologies, this review gives educators, researchers, students, and readers a comprehensive view of how AI could shape educational and research practices in the future, ultimately leading to improved outcomes. Key applications discussed in the field of research include text generation, data analysis and interpretation, literature review, formatting and editing, and peer review. AI applications in academics and education include educational support and constructive feedback, assessment, grading, tailored curricula, personalized career guidance, and mental health support. Addressing the challenges associated with these technologies, such as ethical concerns and algorithmic biases, is essential for maximizing their potential to improve education and research outcomes. Ultimately, the paper aims to contribute to the ongoing discussion about the role of AI in education and research and highlight its potential to lead to better outcomes for students, educators, and researchers.

Predicting and reasoning about replicability using structured groups

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ('investigate', 'discuss', 'estimate' and 'aggregate'). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as 'effect size' and 'reputation' (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

Racism and censorship in the editorial and peer review process

Psychology aims to capture the diversity of our human experience, yet racial inequity ensures only specific experiences are studied, peer-reviewed, and eventually published. Despite recent publications on racial bias in research topics, study samples, academic teams, and publication trends, bias in the peer review process remains largely unexamined. Drawing on compelling case study examples from APA and other leading international journals, this article proposes key mechanisms underlying racial bias and censorship in the editorial and peer review process, including bias in reviewer selection, devaluing racialized expertise, censorship of critical perspectives, minimal consideration of harm to racialized people, and the publication of unscientific and racist studies. The field of psychology needs more diverse researchers, perspectives, and topics to reach its full potential and meet the mental health needs of communities of colour. Several recommendations are called for to ensure the APA can centre racial equity throughout the editorial and review process.

Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review

BACKGROUND

The emergence of systems based on large language models (LLMs) such as OpenAI's ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks.

METHODS

To investigate the potential impact of using LLMs on the peer review process, we used five core themes within discussions about peer review suggested by Tennant and Ross-Hellauer. These include 1) reviewers' role, 2) editors' role, 3) functions and quality of peer reviews, 4) reproducibility, and 5) the social and epistemic functions of peer reviews. We provide a small-scale exploration of ChatGPT's performance regarding identified issues.

RESULTS

LLMs have the potential to substantially alter the role of both peer reviewers and editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs' training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia. Regarding performance, we identified major enhancements in a short period and expect LLMs to continue developing.

CONCLUSIONS

We believe that LLMs are likely to have a profound impact on academia and scholarly communication. While potentially beneficial to the scholarly communication system, many uncertainties remain and their use is not without risks. In particular, concerns about the amplification of existing biases and inequalities in access to appropriate infrastructure warrant further attention. For the moment, we recommend that if LLMs are used to write scholarly reviews and decision letters, reviewers and editors should disclose their use and accept full responsibility for data security and confidentiality, and their reports' accuracy, tone, reasoning and originality.

Meta-analysis of (single-cell method) benchmarks reveals the need for extensibility and interoperability

Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and ultimately guide users on what tools to use. Benchmarking can also be important for community building and advancing methods in a principled way. We conducted a meta-analysis of recent single-cell benchmarks to summarize the scope, extensibility, and neutrality, as well as technical features and whether best practices in open data and reproducible research were followed. The results highlight that while benchmarks often make code available and are in principle reproducible, they remain difficult to extend, for example, as new methods and new ways to assess methods emerge. In addition, embracing containerization and workflow systems would enhance reusability of intermediate benchmarking results, thus also driving wider adoption.

The power of peer networking for improving STEM faculty job applications: a successful pilot programme

To attain a faculty position, postdoctoral fellows submit job applications that require considerable time and effort to produce. Although mentors and colleagues review these applications, postdocs rarely receive iterative feedback from reviewers with the breadth of expertise typically found on an academic search committee. To address this gap, we describe an international peer-reviewing programme for postdocs across disciplines to receive reciprocal, iterative feedback on faculty applications. A participant survey revealed that nearly all participants would recommend the programme to others. Furthermore, our programme was more likely to attract postdocs who struggled to find mentoring, possibly because of their identity as a woman or member of an underrepresented population in STEM or because they changed fields. Between 2018 and 2021, our programme provided nearly 150 early career academics with a diverse and supportive community of peer mentors during the difficult search for a faculty position and continues to do so today. As the transition from postdoc to faculty represents the largest 'leak' in the academic pipeline, implementation of similar programmes by universities or professional societies would provide psycho-social support necessary to prevent attrition of individuals from underrepresented populations as well as increase the chances of success for early career academics in their search for independence.

Towards peer review as a group engagement

I discuss from an economic perspective two of the most recent suggestions to reform the peer review system: (a) payment to referees; (b) ex post peer review. I show that strong economic arguments militate against these ideas. With respect to payment to referees I use results from the economic analysis of prosocial behavior and the private production of public goods, which show that the supply of monetary incentives has the paradoxical effect of reducing the willingness of agents to collaborate, insofar as they substitute intrincic motivation with extrinsic motivation. With respect to ex post peer review, I show that it fails to offer sufficient incentives to researchers, since it is anonymous, depersonalized, and weak in its marginal impact on publishing decisions. I take this argument to criticize the lack of theorizing, in the side of radical proponents of Open access, about the conditions for transition from the subscription model to the Open model. It is this lack of critical attention to economic arguments that has led to the unintended but dramatic outcome of a net increase in the cost of scientific publishing, as documented in very recent papers. Finally, I advance a proposal for admitting payments to referees, but not as individuals but as groups of researchers. I offer this idea to open discussion.

Research Methods: How to Perform an Effective Peer Review

Scientific peer review has existed for centuries and is a cornerstone of the scientific publication process. Because the number of scientific publications has rapidly increased over the past decades, so has the number of peer reviews and peer reviewers. In this paper, drawing on the relevant medical literature and our collective experience as peer reviewers, we provide a user guide to the peer review process, including discussion of the purpose and limitations of peer review, the qualities of a good peer reviewer, and a step-by-step process of how to conduct an effective peer review.

Investing in the Academic Writing: Training Future Reviewers and Sustaining Efficient and Quality Peer Review

Peer reviewers are considered gatekeepers in academic writing who play a pivotal and essential role during the publication process. Excellent manuscripts invariably need excellent reviewers. Producing peer reviewers with such caliber is time-consuming albeit necessary for the progress and continuity of academia. Despite the popular belief that an experienced author invariably makes a good reviewer, the reality is far-fetched. This suggests the need for peer reviewer training, which should be effective, logistically affordable, and demonstrate long-lasting positive impacts. Open review, co-review, and several reviewer training programs are already in place for this purpose with varying efficiencies. This narrative review discusses the current modalities available to a junior reviewer to improve his/her review skills and proposes a reviewer residency concept that could be adopted as a part of peer reviewer training.

The Role of Publons in the Context of Open Peer Review

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

Inequity in Peer Review in Communication Sciences and Disorders

PURPOSE

The American Speech-Language-Hearing Association (ASHA) has committed to advancing diversity, equity, and inclusion (DEI) by retaining and advancing Black, Indigenous, and people of color (BIPOC) individuals in the discipline of communication sciences and disorders (CSD), amid critical shortages of faculty to train the next generation of practitioners and researchers. Publishing research is central to the recruitment, retention, and advancement of faculty. However, inequity in peer review may systematically target BIPOC scholars, adding yet another barrier to their success as faculty. This viewpoint article addresses the challenge of inequity in peer review and provides some practical strategies for developing equitable peer-review practices. First, we describe the demographics of ASHA constituents, including those holding research doctorates, who would typically be involved in peer review. Next, we explore the peer-review process, describing how inequity in peer review may adversely impact BIPOC authors or research with BIPOC communities. Finally, we offer real-world examples of and a framework for equitable peer review.

CONCLUSIONS

Inequity at the individual and systemic levels in peer review can harm BIPOC CSD authors. Such inequity has effects not limited to peer review itself and exerts long-term adverse effects on the recruitment, retention, and advancement of BIPOC faculty in CSD. To uphold ASHA's commitment to DEI and to move the discipline of CSD forward, it is imperative to build equity into the editorial structure for publishing, the composition of editorial boards, and journals content. While we focus on inequity in CSD, these issues are relevant to other disciplines.

Editorial Work and the Peer Review Economy of STS Journals

In this paper, we analyze the role of science and technology studies (STS) journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political-economic and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy.

Accelerated Peer Review and Paper Processing Models in Academic Publishing

Some journals and publishers offer a free or paid rapid peer review service. In the latter case, such a service is offered at a premium, i.e., for an additional fee, and authors receive, in return, a privileged service, namely faster peer review. In the cut-throat world of survival in academia, the difference of a few weeks or months in terms of speed of peer review and publication may bring untold benefits to authors that manage to benefit from accelerated peer review. We examine the deontological aspects behind this two-tier peer review system, including some positive, but mainly negative, aspects. Some paid accelerated peer review services thrive. We examine the paid accelerated peer review services by Taylor & Francis, Future Medicine Ltd., Elsevier, and two stand-alone journals that are OASPA members. This suggests that there is a demand, and thus market, for faster peer review. However, this privilege risks creating a two-tiered system that may divide academics between those who can pay versus those who cannot. We recommend that those papers that have benefited from accelerated peer review clearly indicate this in the published papers, as either a disclaimer or within the acknowledgements, for maximum transparency of the peer review and publication process.

Reimagining peer review as an expert elicitation process

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the ‘best’ experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated.

Covid-19 refereeing duration and impact in major medical journals

Two partly conflicting academic pressures from the seriousness of the Covid-19 pandemic are the need for faster peer review of Covid-19 health-related research and greater scrutiny of its findings. This paper investigates whether decreases in peer review durations for Covid-19 articles were universal across 97 major medical journals, Nature, Science, and Cell. The results suggest that on average, Covid-19 articles submitted during 2020 were reviewed 1.7–2.1 times faster than non-Covid-19 articles submitted during 2017–2020. Nevertheless, whilst the review speed of Covid-19 research was particularly fast during the first five months (1.9–3.4 times faster) of the pandemic (January–May 2020), this speed advantage was no longer evident for articles submitted November–December 2020. Faster peer review also associates with higher citation impact for Covid-19 articles in the same journals, suggesting it did not usually compromise the scholarly impact of important Covid-19 research. Overall, then, it seems that core medical and general journals responded quickly but carefully to the pandemic, although the situation returned closer to normal within a year.

Peer Review

https://publons.com/publon/10.1162/qss_a_00176