Manuscript peer review at the AJR: facts, figures, and quality assessment

Tools used to assess the quality of peer review reports: a methodological systematic review

Background

A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research.

Methods

We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis.

Results

We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of ‘quality’. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer’s comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18).

Conclusion

Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0688-x) contains supplementary material, which is available to authorized users.

Added value of double reading in diagnostic radiology,a systematic review

Objectives

Double reading in diagnostic radiology can find discrepancies in the original report, but a systematic program of double reading is resource consuming. There are conflicting opinions on the value of double reading. The purpose of the current study was to perform a systematic review on the value of double reading.

Methods

A systematic review was performed to find studies calculating the rate of misses and overcalls with the aim of establishing the added value of double reading by human observers.

Results

The literature search resulted in 1610 hits. After abstract and full-text reading, 46 articles were selected for analysis. The rate of discrepancy varied from 0.4 to 22% depending on study setting. Double reading by a sub-specialist, in general, led to high rates of changed reports.

Conclusions

The systematic review found rather low discrepancy rates. The benefit of double reading must be balanced by the considerable number of working hours a systematic double-reading scheme requires. A more profitable scheme might be to use systematic double reading for selected, high-risk examination types. A second conclusion is that there seems to be a value of sub-specialisation for increased report quality. A consequent implementation of this would have far-reaching organisational effects.

Key Points

• In double reading, two or more radiologists read the same images.

• A systematic literature review was performed.

• The discrepancy rates varied from 0.4 to 22% in various studies.

• Double reading by sub-specialists found high discrepancy rates.

Electronic supplementary material

The online version of this article (10.1007/s13244-018-0599-0) contains supplementary material, which is available to authorised users.

Reviewing the Reviewers: Comparison of Review Quality and Reviewer Characteristics at theAmerican Journal of Roentgenology

OBJECTIVE

The purpose of our study was to determine which manuscript reviewer characteristics are most strongly associated with reviewer performance as judged by editors of the American Journal of Roentgenology (AJR).

MATERIALS AND METHODS

At the AJR, manuscript reviews are rated by the journal editors on a subjective scale from 1 (lowest) to 4, on the basis of the value, thoroughness, and punctuality of the critique. We obtained all scores for AJR reviewers and determined the average score for each reviewer. We also sent a questionnaire to 989 reviewers requesting specific information regarding the age, sex, radiology subspecialty, number of years serving as a reviewer, academic rank, and practice type of the reviewer. The demographic profiles were correlated with the average quality score for each reviewer. Statistical analysis included correlation analysis and analysis of variance modeling. Reviewer quality scores were also correlated with the scoring of individual reviews and ultimate disposition of 196 manuscripts sent to the AJR during the same period.

RESULTS

Responses to the questionnaire were obtained from 821 reviewers (83.0%), for whom quality scores were available for 714 (87.0%). Correlation analysis shows that the quality score of reviewers strongly correlated with younger age (p = 0.001). A statistically significant correlation between quality score and practice type was seen (p = 0.008), with reviewers from academic institutions receiving higher scores. No significant correlation was found between quality score and sex (p = 0.72), years of reviewing (p = 0.26), academic rank (p = 0.10), or the ultimate disposition of the manuscript (p = 0.40). The quality score of the reviewers showed no variation by subspecialty (p = 0.99).

CONCLUSION

The highest-rated AJR reviewers tended to be young and from academic institutions. The quality of peer review did not correlate with the sex, academic rank, or subspecialty of the reviewer.

Peer Review at theAmerican Journal of Roentgenology:How Reviewer and Manuscript Characteristics Affected Editorial Decisions on 196 Major Papers

OBJECTIVE

The objective of this study was to examine the relative influence of manuscript characteristics and peer-reviewer attributes in the assessment of manuscripts.

MATERIALS AND METHODS

Over a 6-month period, all major papers submitted to the American Journal of Roentgenology (AJR) were entered into a database that recorded manuscript characteristics, demographic profiles of reviewers, and the disposition of the manuscript. Manuscript characteristics included reviewer ratings on five scales (rhetoric, structure, science, import, and overall recommendation); the subspecialty class of the paper; the primary imaging technique; and the country of origin. Demographic profiles of the reviewers included age, sex, subspecialty, years of reviewing, academic rank, and practice type. Statistical analysis included correlation analysis, ordinal logistic regression, and analysis of variance.

RESULTS

A total of 445 reviews of 196 manuscripts were the work of 335 reviewers. Of the 196 submitted manuscripts, 20 (10.2%) were accepted, 106 (54.1%) were rejected, and 70 (35.7%) were rejected with the opportunity to resubmit. Regarding manuscript characteristics, we found that the country of origin, score on the science scale, and score on the import scale were statistically significant variables for predicting the final disposition of a manuscript. Of the reviewer attributes, we found a statistically significant association between greater reviewer age and also higher academic rank with lower scores on the import scale. Reviewer concordance was higher for structure, science, and overall scores than on the rhetoric and import scores. Greater variability in the overall scoring of papers could be attributed to the reviewer than the manuscript, but both factors combined explain only 23% of the total variability.

CONCLUSION

At the AJR, manuscript acceptance was most strongly associated with reviewer scoring of the science and import of a major paper and also with the country of origin. Reviewers who were older and of higher academic rank tended to discount the importance of manuscripts.

The peer-review process of the Journal of Neurosurgery

Peer review is the process by which scientific articles are evaluated and selected for publication. To clarify this procedure for readers and writers, the authors present a detailed description of peer review at the Journal of Neurosurgery (JNS) in the context of other journals. They discuss the unique characteristics of JNS's peer-review process and how it contributes to the quality of the JNS.

How to "peer review" a medical journal manuscript

BACKGROUND

The accuracy, validity, and appropriateness of what gets published in the medical literature depends upon a pool of competent peer referees. However, the requisite skills needed to become such a peer are not taught at the residency training program level or subsequently by the journals themselves, who offer no training and little written instruction.

OBJECTIVE

To outline a logical, orderly approach to the task of reviewing a "raw" medical manuscript. By breaking the overall endeavor down into smaller, step-by-step components, the novice reviewer should attain the direction and skills to complete a review with confidence.

METHODS

Explore the role of the reviewer, discuss the philosophy of review, identify the key elements to look for when reading and rereading the manuscript, and outline a orderly approach to the decision making process.

CONCLUSIONS

A "peer" is a physician with expertise on the subject under scrutiny who spends sufficient time and thought to fulfill two main obligations: to render an honest, unbiased decision on whether or not the manuscript should be published and, if it is acceptable, to help make it better. It is possible to perform the review process in a relatively simple, organized, and logical manner.