Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Evaluation and testing of the proportional hazards assumption in analysis of time-to-event data in subgroup analysis of randomised controlled trials: a meta-epidemiological study

BACKGROUND

When Cox regression models are used to analyse time-to-event data, the proportional hazard assumption (PHA) must be reassured to obtain valid results. Transparent reporting of the statistics used is therefore essential to interpret research. This study aimed to assess the quality of statistical reporting and testing of the PHA in subgroup analysis of surgical randomised controlled trials (RCTs).

METHODS

All published articles (see appendix 1) in the top quartile (25%) of surgical journals from 2019 to 2021 were screened in a literature review according to the ClarivateTM journal citation report impact factor. Subgroup analyses of surgical RCT data that used Cox models were identified. Statistical reporting was rated using a previously established 12-item PHA Reporting Score as our primary endpoint. For original surgical publications, the PHA was formally tested on reconstructed time-to-event data from Kaplan-Meier estimators. Methodological reporting quality was rated according to the CONSORT statement. Digitalisation was only possible in studies where a Kaplan-Meier estimator including numbers at risk per time interval was published. All results from the subgroup analyses were compared to primary surgical RCT reports and benchmark RCTs using Cox models published in the New England Journal of Medicine and The Lancet.

RESULTS

Thirty-two studies reporting secondary subgroup analyses on surgical RCT data using Cox models were identified. Statistical reporting of surgical subgroup publications was significantly inferior compared to original benchmark publications: median PHA Reporting Score 50% (interquartile range [IQR]: 39 to 58) vs 58% (IQR: 42 to 67), p <0.001. The subgroups did not differ in comparison to primary surgical RCTs: median PHA Reporting Score 50% (IQR: 39 to 58) vs 42% (IQR: 33 to 58), p = 0.286. Adherence to the CONSORT reporting standards did significantly differ between subgroup studies and benchmark publications (p <0.001) as well as between subgroup studies and primary surgical RCT reports: 13 (12.5 to 14) vs 13 (IQR: 11 to 13), p = 0.042.

CONCLUSION

Statistical methodological reporting of secondary subgroup analyses from surgical RCTs was inferior to benchmark publications but not worse than primary surgical RCT reports. A comprehensive statistical review process and statistical reporting guidelines might help improve the reporting quality.

The Most Cited Publications in Abdominal Wall Reconstruction-A Bibliometric Analysis

BACKGROUND

Abdominal wall reconstruction (AWR) is a treatment option for structural defects of the abdominal wall. The most frequently cited publications related to AWR have not been quantitatively or qualitatively assessed. This bibliometric analysis characterizes and assesses the most frequently cited AWR publications, to identify trends, gaps, and guide future efforts for the international research community.

METHODS

The 100 most cited publications in AWR were identified on Web of Science, across all available journal years (from May 1964 to December 2023). Study details, including the citation count, main content focus, and outcome measures, were extracted and tabulated from each publication. Oxford Centre for Evidence-Based Medicine levels of evidence (LOE) of each study were also assessed.

RESULTS

The 100 most cited publications in AWR were cited by a total of 9674 publications. Citations per publication ranged from 43 to 414 (mean 96.7 ± 52.48). Most publications were LOE 3 (n = 60), representative of the large number of retrospective cohort studies. The number of publications for LOE 5, 4, 3, 2, and 1 was 21, 2, 60, 2, and 12, respectively. The main content focus was surgical technique in 44 publications followed by outcomes in 38 publications. Patient-reported outcome measures were used in 3 publications, and no publications reported validated esthetic outcome measures.

CONCLUSIONS

Overall, 3 was the LOE for most frequently cited AWR publications, with more publications below LOE 3 than above LOE 3. Validated outcome measures and patient-reported outcome measures were infrequently incorporated in the studies evaluated.

A health sciences researcher's experience of manuscript review comments, 2020-2022

BACKGROUND

 Peer review frequently improves a manuscript, but authors may consider some reviewer feedback negative, inappropriate or unclear. This study aims to summarise and analyse review comments received by authors.

METHODS

 This longitudinal study included all submissions of which the researcher was an author, reviewed by any journal during 2020-2022. First-round reviews were retrieved from emails and documents received by the authors or the faculty's medical editors or the journal platforms. A confidential datasheet with review items compiled from literature and the researcher's experience as author and reviewer was completed for each submission. Review comments were noted verbatim for subjective items such as rude or vague statements.

RESULTS

 The 65 submissions received 118 reviews from 36 journals, mainly in the form of unstructured narrative reports (59%). The majority of first-round reviews (58%), including those for rejected submissions, contained some positive comments. Reviewers frequently (75% of reviews, 88% of submissions) required some expansion of information. Vague and incorrect statements occurred in 15% and 18% of reviews, respectively. Only two reviews contained statements that could be considered rude. The types of comments made were associated with the review format.

CONCLUSION

 The majority of reviews contained some positive comments and rude comments were extremely rare. Reviewers frequently requested the expansion of information provided.Contribution: This study gives insight to authors, reviewers and editors regarding the type and tone of review comments. This could guide authors during manuscript preparation and authors, reviewers and editors during the review process.

A scoping review on biomedical journal peer review guides for reviewers

BACKGROUND

Peer review is widely used in academic fields to assess a manuscript's significance and to improve its quality for publication. This scoping review will assess existing peer review guidelines and/or checklists intended for reviewers of biomedical journals and provide an overview on the review guidelines.

METHODS

PubMed, Embase, and Allied and Complementary Medicine (AMED) databases were searched for review guidelines from the date of inception until February 19, 2021. There was no date restriction nor article type restriction. In addition to the database search, websites of journal publishers and non-publishers were additionally hand-searched.

RESULTS

Of 14,633 database publication records and 24 website records, 65 publications and 14 websites met inclusion criteria for the review (78 records in total). From the included records, a total of 1,811 checklist items were identified. The items related to Methods, Results, and Discussion were found to be the highly discussed in reviewer guidelines.

CONCLUSION

This review identified existing literature on peer review guidelines and provided an overview of the current state of peer review guides. Review guidelines were varying by journals and publishers. This calls for more research to determine the need to use uniform review standards for transparent and standardized peer review.

PROTOCOL REGISTRATION

The protocol for this study has been registered at Research Registry (www.researchregistry.com): reviewregistry881.

How to Review a Surgical Scientific Paper: A Guide for Critical Appraisal

It is important for surgeons to participate in the peer-review process of scientific literature. As the number of published manuscripts continues to increase, there is a great need for volunteerism in this arena. However, there is little formal or informal training, which can help surgeons provide unbiased and meaningful reviews. Therefore, it is critical to provide more resources and guidelines to aid surgeons during the review process. The purpose of this paper is to provide a structured guide for a quality review of a surgical paper. This review represents the work of the Association of Women Surgeons Publications Committee.

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

The global Covid-19 pandemic has had a considerable impact on the scientific enterprise, including scholarly publication and peer-review practices. Several studies have assessed these impacts, showing among others that medical journals have strongly accelerated their review processes for Covid-19-related content. This has raised questions and concerns regarding the quality of the review process and the standards to which manuscripts are held for publication. To address these questions, this study sets out to assess qualitative differences in review reports and editorial decision letters for Covid-19 related, articles not related to Covid-19 published during the 2020 pandemic, and articles published before the pandemic. It employs the open peer-review model at the British Medical Journal and eLife to study the content of review reports, editorial decisions, author responses, and open reader comments. It finds no clear differences between the review processes of articles not related to Covid-19 published during or before the pandemic. However, it does find notable diversity between Covid-19 and non-Covid-19-related articles, including fewer requests for additional experiments, more cooperative comments, and different suggestions to address too strong claims. In general, the findings suggest that both reviewers and journal editors implicitly and explicitly use different quality criteria to assess Covid-19-related manuscripts, hence transforming science’s main evaluation mechanism for their underlying studies and potentially affecting their public dissemination.

Journal editors' perspectives on the communication practices in biomedical journals: a qualitative study

OBJECTIVE

To generate an understanding of the communication practices that might influence the peer-review process in biomedical journals.

METHOD

Recruitment was based on purposive maximum variation sampling. We conducted semistructured interviews. Data were analysed using thematic analysis method.

PARTICIPANTS

56 journal editors from general medicine (n=13) and specialty (n=43) biomedical journals. Most were editor-in-chiefs (n=39), men (n=40) and worked part time (n=50).

RESULTS

Our analysis generated four themes (1) providing minimal guidance to peer reviewers-two subthemes described the way journal editors rationalised their behaviour: (a) peer reviewers should know without guidelines how to review and (b) detailed guidance and structure might have a negative effect; (2) communication strategies of engagement with peer reviewers-two opposing strategies that journal editors employed to handle peer reviewers: (a) use of direct and personal communication to motivate peer reviewers and (b) use of indirect communication to avoid conflict; (3) concerns about impact of review model on communication-maintenance of anonymity as a means of facilitating critical and unburdened communication and minimising biases and (4) different practices in the moderation of communication between authors and peer reviewers-some journal editors actively interjected themselves into the communication chain to guide authors through peer reviewers' comments, others remained at a distance, leaving it to the authors to work through peer reviewers' comments.

CONCLUSIONS

These journal editors' descriptions reveal several communication practices that might have a significant impact on the peer-review process. Editorial strategies to manage miscommunication are discussed. Further research on these proposed strategies and on communication practices from the point of view of authors and peer reviewers is warranted.

What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis

Background

Peer review is at the heart of the scientific process. With the advent of digitisation, journals started to offer electronic articles or publishing online only. A new philosophy regarding the peer review process found its way into academia: the open peer review. Open peer review as practiced by BioMed Central (BMC) is a type of peer review where the names of authors and reviewers are disclosed and reviewer comments are published alongside the article. A number of articles have been published to assess peer reviews using quantitative research. However, no studies exist that used qualitative methods to analyse the content of reviewers’ comments.

Methods

A focused mapping review and synthesis (FMRS) was undertaken of manuscripts reporting qualitative research submitted to BMC open access journals from 1 January – 31 March 2018. Free-text reviewer comments were extracted from peer review reports using a 77-item classification system organised according to three key dimensions that represented common themes and sub-themes. A two stage analysis process was employed. First, frequency counts were undertaken that allowed revealing patterns across themes/sub-themes. Second, thematic analysis was conducted on selected themes of the narrative portion of reviewer reports.

Results

A total of 107 manuscripts submitted to nine open-access journals were included in the FMRS. The frequency analysis revealed that among the 30 most frequently employed themes “writing criteria” (dimension II) is the top ranking theme, followed by comments in relation to the “methods” (dimension I). Besides that, some results suggest an underlying quantitative mindset of reviewers. Results are compared and contrasted in relation to established reporting guidelines for qualitative research to inform reviewers and authors of frequent feedback offered to enhance the quality of manuscripts.

Conclusions

This FMRS has highlighted some important issues that hold lessons for authors, reviewers and editors. We suggest modifying the current reporting guidelines by including a further item called “Degree of data transformation” to prompt authors and reviewers to make a judgment about the appropriateness of the degree of data transformation in relation to the chosen analysis method. Besides, we suggest that completion of a reporting checklist on submission becomes a requirement.

A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals

BACKGROUND

Although peer reviewers play a key role in the manuscript review process, their roles and tasks are poorly defined. Clarity around this issue is important as it may influence the quality of peer reviewer reports. This scoping review explored the roles and tasks of peer reviewers of biomedical journals.

METHODS

Comprehensive literature searches were conducted in Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, EMBASE, MEDLINE, PsycINFO, Scopus and Web of Science from inception up to May 2017. There were no date and language restrictions. We also searched for grey literature. Studies with statements mentioning roles, tasks and competencies pertaining to the role of peer reviewers in biomedical journals were eligible for inclusion. Two reviewers independently performed study screening and selection. Relevant statements were extracted, collated and classified into themes.

RESULTS

After screening 2763 citations and 600 full-text papers, 209 articles and 13 grey literature sources were included. A total of 1426 statements related to roles were extracted, resulting in 76 unique statements. These were grouped into 13 emergent themes: proficient experts in their field (3 items), dutiful/altruistic towards scientific community (7 items), familiar with journal (2 items), unbiased and ethical professionals (18 items), self-critical professionals (4 items), reliable professionals (7 items), skilled critics (15 items), respectful communicators (6 items), gatekeepers (2 items), educators (2 items), advocates for author/editor/reader (3 items) and advisors to editors (2 items). Roles that do not fall within the remit of peer reviewers were also identified (5 items). We also extracted 2026 statements related to peer reviewers' tasks, resulting in 73 unique statements. These were grouped under six themes: organisation and approach to reviewing (10 items), make general comments (10 items), assess and address content for each section of the manuscript (36 items), address ethical aspects (5 items), assess manuscript presentation (8 items) and provide recommendations (4 items).

CONCLUSIONS

Peer reviewers are expected to perform a large number of roles and tasks for biomedical journals. These warrant further discussion and clarification in order not to overburden these key actors.