CREDIBILITY, PEER REVIEW, AND NATURE, 1945-1990

Biomedical publishing: Past historic, present continuous, future conditional

Academic journals have been publishing the results of biomedical research for more than 350 years. Reviewing their history reveals that the ways in which journals vet submissions have changed over time, culminating in the relatively recent appearance of the current peer-review process. Journal brand and Impact Factor have meanwhile become quality proxies that are widely used to filter articles and evaluate scientists in a hypercompetitive prestige economy. The Web created the potential for a more decoupled publishing system in which articles are initially disseminated by preprint servers and then undergo evaluation elsewhere. To build this future, we must first understand the roles journals currently play and consider what types of content screening and review are necessary and for which papers. A new, open ecosystem involving preprint servers, journals, independent content-vetting initiatives, and curation services could provide more multidimensional signals for papers and avoid the current conflation of trust, quality, and impact. Academia should strive to avoid the alternative scenario, however, in which stratified publisher silos lock in submissions and simply perpetuate this conflation.

Publication speed in pharmacy practice journals: A comparative analysis

Background

Scholarly publishing system relies on external peer review. However, the duration of publication process is a major concern for authors and funding bodies.

Objective

To evaluate the duration of the publication process in pharmacy practice journals compared with other biomedical journals indexed in PubMed.

Methods

All the articles published from 2009 to 2018 by the 33 pharmacy practice journals identified in Mendes et al. study and indexed in PubMed were gathered as study group. A comparison group was created through a random selection of 3000 PubMed PMIDs for each year of study period. Articles with publication dates outside the study period were excluded. Metadata of both groups of articles were imported from PubMed. The duration of editorial process was calculated with three periods: acceptance lag (days between ‘submission date’ and ‘acceptance date’), lead lag (days between ‘acceptance date’ and ‘online publication date’), and indexing lag (days between ‘online publication date’ and ‘Entry date’). Null hypothesis significance tests and effect size measures were used to compare these periods between both groups.

Results

The 33 pharmacy practice journals published 26,256 articles between 2009 and 2018. Comparison group random selection process resulted in a pool of 23,803 articles published in 5,622 different journals. Acceptance lag was 105 days (IQR 57–173) for pharmacy practice journals and 97 days (IQR 56–155) for the comparison group with a null effect difference (Cohen’s d 0.081). Lead lag was 13 (IQR 6–35) and 23 days (IQR 9–45) for pharmacy practice and comparison journals, respectively, which resulted in a small effect. Indexing lag was 5 days (IQR 2–46) and 4 days (IQR 2–12) for pharmacy practice and control journals, which also resulted in a small effect. Slight positive time trend was found in pharmacy practice acceptance lag, while slight negative trends were found for lead and indexing lags for both groups.

Conclusions

Publication process duration of pharmacy practice journals is similar to a general random sample of articles from all disciplines.

Peer review: the case for neutral language

Peer review is an integral part of scientific life, determining success in publishing, grant applications, and professional appointments. We argue for the importance of neutral language in peer review and provide examples of nonneutral linguistic and stylistic devices that emphasize a reviewer's personal response to the manuscript rather than their objective assessment.

The malate-aspartate shuttle (Borst cycle): How it started and developed into a major metabolic pathway

This article presents a personal and critical review of the history of the malate–aspartate shuttle (MAS), starting in 1962 and ending in 2020. The MAS was initially proposed as a route for the oxidation of cytosolic NADH by the mitochondria in Ehrlich ascites cell tumor lacking other routes, and to explain the need for a mitochondrial aspartate aminotransferase (glutamate oxaloacetate transaminase 2 [GOT2]). The MAS was soon adopted in the field as a major pathway for NADH oxidation in mammalian tissues, such as liver and heart, even though the energetics of the MAS remained a mystery. Only in the 1970s, LaNoue and coworkers discovered that the efflux of aspartate from mitochondria, an essential step in the MAS, is dependent on the proton‐motive force generated by the respiratory chain: for every aspartate effluxed, mitochondria take up one glutamate and one proton. This makes the MAS in practice uni‐directional toward oxidation of cytosolic NADH, and explains why the free NADH/NAD ratio is much higher in the mitochondria than in the cytosol. The MAS is still a very active field of research. Most recently, the focus has been on the role of the MAS in tumors, on cells with defects in mitochondria and on inborn errors in the MAS. The year 2019 saw the discovery of two new inborn errors in the MAS, deficiencies in malate dehydrogenase 1 and in aspartate transaminase 2 (GOT2). This illustrates the vitality of ongoing MAS research.

What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis

Background

Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers’ comments.

Methods

A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January – 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports.

Results

A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes “writing criteria” (dimension II) is the top ranking theme, followed by comments in relation to the “methods” (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts.

Conclusions

This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called “Degree of data transformation” to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.

The limitations to our understanding of peer review

Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.

The state of the art in peer review

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and communications infrastructure and suggest why uptake of new models of peer review appears to have been so low compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture.

The ability of different peer review procedures to flag problematic publications

There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review’s ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

The changing forms and expectations of peer review

The quality and integrity of the scientific literature have recently become the subject of heated debate. Due to an apparent increase in cases of scientific fraud and irreproducible research, some have claimed science to be in a state of crisis. A key concern in this debate has been the extent to which science is capable of self-regulation. Among various mechanisms, the peer review system in particular is considered an essential gatekeeper of both quality and sometimes even integrity in science. However, the allocation of responsibility for integrity to the peer review system is fairly recent and remains controversial. In addition, peer review currently comes in a wide variety of forms, developed in the expectation they can address specific problems and concerns in science publishing. At present, there is a clear need for a systematic analysis of peer review forms and the concerns underpinning them, especially considering a wave of experimentation fuelled by internet technologies and their promise to improve research integrity and reporting. We describe the emergence of current peer review forms by reviewing the scientific literature on peer review and by adding recent developments based on information from editors and publishers. We analyse the rationale for developing new review forms and discuss how they have been implemented in the current system. Finally, we give a systematisation of the range of discussed peer review forms. We pay detailed attention to the emergence of the expectation that peer review can maintain 'the integrity of science's published record', demonstrating that this leads to tensions in the academic debate about the responsibilities and abilities of the peer review system.

A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of Web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform current models while avoiding as many of the biases of existing systems as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that, at least partially, resolves many of the technical and social issues associated with peer review, and can potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.