The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey

Nurses’ Motivations, Barriers, and Facilitators to Engage in a Peer Review Process: A Qualitative Study Protocol

Peer review supports the integrity and quality of scientific publishing. However, although it is a fundamental part of the publishing process, peer review can also be challenging for reviewers, editors, and other stakeholders. The present study aims to explore the nurses’ motivations, barriers, and facilitators in engaging in a peer review process. This qualitative, descriptive exploratory study will be developed in partnerships with three research centers. Researchers followed the consolidated criteria for reporting qualitative research (COREQ) checklist to ensure the quality of this study protocol. According to the selection criteria, the purposive sampling will be used to recruit nurse researchers that act as peer reviewers for several scientific journals in various fields of knowledge. Interviews will be conducted until data have been sufficiently consistent with meeting the initial objectives. Researchers will develop a guide comprising a set of open-ended questions to collect participants’ characteristics, descriptive review behavior, and perceptions regarding their motivations, barriers, and facilitators. Researchers will analyze data using an inductive process of content analysis with the help of the QDA Miner Lite database. Findings from this study will generate knowledge that may help stakeholders identify facilitating factors and barriers and guide the development of strategies to remove or minimize these barriers.

Scientists from Minority-Serving Institutions and Their Participation in Grant Peer Review

Funding disparities currently exist across racial groups for grant applicants and between minority-serving institutions (MSIs) and traditionally White institutions (TWIs), with implicit bias in grant review a likely source of these differences. A survey examining grant review participation was disseminated to thousands of MSI-based scientists, and their responses were compared with those in a previous study of predominantly TWI-based scientists. Despite similar levels of grant submission, only 45% of the MSI-based scientists had recently participated in grant review, which is much lower than the rate reported in the previous study. MSI-based scientists indicated not being invited and a general lack of time (largely because of teaching and service responsibilities) as barriers. Nevertheless, the majority of the MSI-based respondents were interested in reviewing and in training in grant reviewing. These results serve as a call to action for research funders to increase both invitations to review and training opportunities for MSI-based scientists.

Grant Review Feedback: Appropriateness and Usefulness

The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56-60% of applicants determined the feedback to be appropriate (fair, well-written, and well-informed), although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents' feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals.

Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion

Background

Funding agencies have long used panel discussion in the peer review of research grant proposals as a way to utilize a set of expertise and perspectives in making funding decisions. Little research has examined the quality of panel discussions and how effectively they are facilitated.

Methods

Here, we present a mixed-method analysis of data from a survey of reviewers focused on their perceptions of the quality, effectiveness, and influence of panel discussion from their last peer review experience.

Results

Reviewers indicated that panel discussions were viewed favorably in terms of participation, clarifying differing opinions, informing unassigned reviewers, and chair facilitation. However, some reviewers mentioned issues with panel discussions, including an uneven focus, limited participation from unassigned reviewers, and short discussion times. Most reviewers felt the discussions affected the review outcome, helped in choosing the best science, and were generally fair and balanced. However, those who felt the discussion did not affect the outcome were also more likely to evaluate panel communication negatively, and several reviewers mentioned potential sources of bias related to the discussion. While respondents strongly acknowledged the importance of the chair in ensuring appropriate facilitation of the discussion to influence scoring and to limit the influence of potential sources of bias from the discussion on scoring, nearly a third of respondents did not find the chair of their most recent panel to have performed these roles effectively.

Conclusions

It is likely that improving chair training in the management of discussion as well as creating review procedures that are informed by the science of leadership and team communication would improve review processes and proposal review reliability.

A CTSA-based consultation service to advance research on special and underserved populations

In this report, we describe the implementation and short-term outcomes of a Special Populations Consultation Service within the University of California, Los Angeles (UCLA) Clinical and Translational Science Institute (CTSI). With the goal of increasing the quality and quantity of special population (SP) research, the UCLA CTSI Integrating Special Populations program designed a consultation service to support faculty and trainees conducting research involving one of three CTSI “special populations:” children, older adults, and/or minority; underserved; or health disparity populations. The Special Populations Consultation Service offers three types of activities: grant proposal studios, career consultations, and project reviews. UCLA CTSI faculty with appropriate content expertise serve as consultants. We evaluated this consultation model using satisfaction surveys and by quantifying funded grants and reported changes in career goals in SP research. Between 2016 and 2019, the Special Populations Consultation Service provided 59 consultations including 42 grant studios and was used by researchers at all levels from all four UCLA CTSI institutions. Recipients rated the consultations very highly. Funding success rates were 57% following K-level grant studios and 28% following R-level grant studios. Users of project and career consultations commonly attributed career accomplishments in part to their consultation experiences. The SP Consultation Service is feasible and acceptable and appears to enhance careers of investigators studying special populations.