Community partners' responses to items assessing stakeholder engagement: Cognitive response testing in measure development

Community Engagement in Research: An Updated Systematic Review of Quantitative Engagement Measurement Scales for Health Studies

Though the interest in community engagement in research (CEnR) protocols has increased, studies reporting on the findings of tested CEnR engagement measurement scales for health studies are sparse. A systematic review was conducted from January 1 to March 1, 2023, to identify validated, quantitative CEnR engagement measurement tools for health studies. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology was employed. The rigor of scale development, testing, and implementation was explored, and a `best practices evaluation conducted. Themes on the readiness of scales for implementation in health research studies were narratively compiled. Nineteen studies met the search inclusion criteria-reporting on the development, testing, and implementation of seven CEnR engagement measurement scales for health studies. Scale implementation studies precipitated only two of the studies. None of the scales followed the rigorous process dictated in best practices; however, at this time, three scales have gone through the most robust testing processes. Advancement of the science of engagement measurement requires consensus on terminology, application of best practices for scale development and testing protocols, and consistency of reporting findings.

Construct validation of the Research Engagement Survey Tool (REST)

BACKGROUND

The Research Engagement Survey Tool (REST) was developed to examine the level of partner (e.g., patients, caregivers, advocates, clinicians, community members) engagement in research studies. The REST is aligned with eight engagement principles based on the literature and consensus reached through a five round Delphi process. Each of the engagement principles has three-five corresponding items that are assessed on two Likert type scales quantity (how often: never, rarely, sometimes, often, always, not applicable) and quality (how well: poor, fair, good, very good, excellent, not applicable). We conducted a comprehensive validation of the REST. Despite the importance of partner engagement in research, currently no gold standard measure exists.

METHODS

Multiple strategies were employed to validate the REST. Here, we examine the internal consistency of items for each of the eight engagement principles. In addition, we examine the convergent validity of the comprehensive (32-item) REST with other measures (e.g., medical mistrust, Community Engagement in Research Index, Partnership Self-Assessment Tool, Wilder collaboration inventory, Partnership Assessment In community-based Research). We propose two scoring approaches for the REST; one aligned with the engagement principles and the other aligned with levels of community engagement: (1) outreach and education, (2) consultation, (3) cooperation, (4) collaboration, and (5) partnership.

RESULTS

The REST has strong internal consistency (Cronbach's alpha > 0.75) for each of the eight engagement principals measured on both scales (quality and quantity). The REST had negligible (e.g., medical mistrust, community engagement in research index), low (e.g., Partnership Assessment In community-based Research, Partnership Self-Assessment Tool- benefits scale), and moderate (e.g., Wilder collaboration inventory, Partnership Self-Assessment Tool- synergy scale) statistically significant correlations with other measures based on the Spearman rank correlation coefficient. These results suggest the REST is measuring something similar and correlated to the existing measures, but it captures a different construct (perceived research engagement).

CONCLUSIONS

The REST is a valid and reliable tool to assess research engagement of community health stakeholders in the research process. Valid tools to assess research engagement are necessary to examine the impact of engagement on the scientific process and scientific discovery and move the field of stakeholder engagement from best practices and lessons learned to evidence-based approaches based on empirical data.

A Study Examining the Usefulness of a New Measure of Research Engagement

INTRODUCTION

Engagement of relevant stakeholders' ideas, opinions, and concerns is critical to the success of modern research projects. We have developed a tool to measure stakeholder engagement, called the Research Engagement Survey Tool (REST). The purpose of this paper is to present the implementation and uptake of the stakeholder engagement measure REST among research teams, including the assessment of barriers and facilitating factors for use of the new research engagement measure in practice.

METHODS

In this implementation study, project team members participated in baseline and follow-up web-based surveys. Web-based interviews were conducted with a subset of project teams that implemented the REST. On the baseline survey, project teams were asked to provide details about up to three ongoing or recently completed projects, were asked if they agreed with compensation for REST completion, and were asked if they would like to send the survey to stakeholders or would prefer our project team to email their project stakeholders. Follow-up surveys contained questions on reactions to implementing REST and results of REST.

RESULTS

Project team members/researchers who completed the baseline survey (n=86) were mostly female (79%) and Non-Hispanic/Latino(a) White (76%). Those who implemented REST were also mostly female (86%) and Non-Hispanic/Latino(a) White (71%), with an average of 11 years in academic research. About 98% of all participants completing the baseline survey had the capacity to survey partners, while 100% of all teams who implemented REST did. A small portion of respondents indicated the time commitment of REST would be a barrier (29% of baseline survey respondents, 10% of those who implemented REST) and indicated workload would be a barrier (31% of baseline survey respondents, 14% of those who implemented REST).

DISCUSSION

The data presented here indicate that REST implementation is feasible in a volunteer group of ongoing research projects.

Development and Co-design of NeuroOrb: A Novel “Serious Gaming” System Targeting Cognitive Impairment in Parkinson’s Disease

Whilst Parkinson’s disease (PD) is typically thought of as a motor disease, a significant number of individuals also experience cognitive impairment (CI), ranging from mild-CI to dementia. One technique that may prove effective in delaying the onset of CI in PD is cognitive training (CT); however, evidence to date is variable. This may be due to the implementation of CT in this population, with the motor impairments of PD potentially hampering the ability to use standard equipment, such as pen-and-paper or a computer mouse. This may, in turn, promote negative attitudes toward the CT paradigm, which may correlate with poorer outcomes. Consequently, optimizing a system for the delivery of CT in the PD population may improve the accessibility of and engagement with the CT paradigm, subsequently leading to better outcomes. To achieve this, the NeuroOrb Gaming System was designed, coupling a novel accessible controller, specifically developed for use with people with motor impairments, with a “Serious Games” software suite, custom-designed to target the cognitive domains typically affected in PD. The aim of the current study was to evaluate the usability of the NeuroOrb through a reiterative co-design process, in order to optimize the system for future use in clinical trials of CT in individuals with PD. Individuals with PD (n = 13; mean age = 68.15 years; mean disease duration = 8 years) were recruited from the community and participated in three co-design loops. After implementation of key stakeholder feedback to make significant modifications to the system, system usability was improved and participant attitudes toward the NeuroOrb were very positive. Taken together, this provides rationale for moving forward with a future clinical trial investigating the utility of the NeuroOrb as a tool to deliver CT in PD.

Development and Validation of a Brief Version of the Research Engagement Survey Tool

The Research Engagement Survey Tool (REST) examines the level of partner engagement in research studies. This study used mixed methods, including web-based surveys (N = 336), a modified Delphi process (N = 18), and cognitive response interviews (N = 16), with convenience sampling to develop and validate a short version of the REST. We conducted factor analysis and calculated internal consistency for the condensed REST. We validated the condensed REST against the comprehensive REST. All analyses were carried out on two scales (quality and quantity) based on Likert-type response options. We examined convergent validity with other measures theoretically associated with the REST (e.g., the Community Engagement Research Index and the Partnership Self-Assessment Tool). This study produced a 9-item condensed version of the REST. The condensed REST loads on 1 factor, has high internal consistency (Cronbach’s alpha = 0.92 for the quantity scale; 0.94 for the quality scale), is significantly correlated (ρ = 0.97; p < 0.001 for both scales) with the comprehensive (32-item) REST, and has negligible, low, and moderate correlation with other measures (e.g., the Partnership Assessment In community-based Research, trust in medical researchers, and the Coalition Self-Assessment Survey). Use of the condensed REST will reduce participant burden and time to complete. This standardized and validated quantitative measure is useful to compare engagement across projects or within a project over time.