Systematic Review of Quantitative Measures of Stakeholder Engagement

Patient engagement in designing, conducting, and disseminating clinical pain research: IMMPACT recommended considerations

ABSTRACT

In the traditional clinical research model, patients are typically involved only as participants. However, there has been a shift in recent years highlighting the value and contributions that patients bring as members of the research team, across the clinical research lifecycle. It is becoming increasingly evident that to develop research that is both meaningful to people who have the targeted condition and is feasible, there are important benefits of involving patients in the planning, conduct, and dissemination of research from its earliest stages. In fact, research funders and regulatory agencies are now explicitly encouraging, and sometimes requiring, that patients are engaged as partners in research. Although this approach has become commonplace in some fields of clinical research, it remains the exception in clinical pain research. As such, the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials convened a meeting with patient partners and international representatives from academia, patient advocacy groups, government regulatory agencies, research funding organizations, academic journals, and the biopharmaceutical industry to develop consensus recommendations for advancing patient engagement in all stages of clinical pain research in an effective and purposeful manner. This article summarizes the results of this meeting and offers considerations for meaningful and authentic engagement of patient partners in clinical pain research, including recommendations for representation, timing, continuous engagement, measurement, reporting, and research dissemination.

Stakeholder Engagement in Type 1 Diabetes Research, Quality Improvement, and Clinical Care

The integration of stakeholder engagement (SE) in research, quality improvement (QI), and clinical care has gained significant traction. Type 1 diabetes is a chronic disease that requires complex daily management and care from a multidisciplinary team across the lifespan. Inclusion of key stakeholder voices, including patients, caregivers, health care providers and community advocates, in the research process and implementation of clinical care is critical to ensure representation of perspectives that match the values and goals of the patient population. This review describes the current framework for SE and its application to research, QI, and clinical care across the lifespan.

A scoping review of the globally available tools for assessing health research partnership outcomes and impacts

BACKGROUND

Health research partnership approaches have grown in popularity over the past decade, but the systematic evaluation of their outcomes and impacts has not kept equal pace. Identifying partnership assessment tools and key partnership characteristics is needed to advance partnerships, partnership measurement, and the assessment of their outcomes and impacts through systematic study.

OBJECTIVE

To locate and identify globally available tools for assessing the outcomes and impacts of health research partnerships.

METHODS

We searched four electronic databases (Ovid MEDLINE, Embase, CINAHL + , PsychINFO) with an a priori strategy from inception to June 2021, without limits. We screened studies independently and in duplicate, keeping only those involving a health research partnership and the development, use and/or assessment of tools to evaluate partnership outcomes and impacts. Reviewer disagreements were resolved by consensus. Study, tool and partnership characteristics, and emerging research questions, gaps and key recommendations were synthesized using descriptive statistics and thematic analysis.

RESULTS

We screened 36 027 de-duplicated citations, reviewed 2784 papers in full text, and kept 166 studies and three companion reports. Most studies originated in North America and were published in English after 2015. Most of the 205 tools we identified were questionnaires and surveys targeting researchers, patients and public/community members. While tools were comprehensive and usable, most were designed for single use and lacked validity or reliability evidence. Challenges associated with the interchange and definition of terms (i.e., outcomes, impacts, tool type) were common and may obscure partnership measurement and comparison. Very few of the tools identified in this study overlapped with tools identified by other, similar reviews. Partnership tool development, refinement and evaluation, including tool measurement and optimization, are key areas for future tools-related research.

CONCLUSION

This large scoping review identified numerous, single-use tools that require further development and testing to improve their psychometric and scientific qualities. The review also confirmed that the health partnership research domain and its measurement tools are still nascent and actively evolving. Dedicated efforts and resources are required to better understand health research partnerships, partnership optimization and partnership measurement and evaluation using valid, reliable and practical tools that meet partners' needs.

Community Engagement in Research: An Updated Systematic Review of Quantitative Engagement Measurement Scales for Health Studies

Though the interest in community engagement in research (CEnR) protocols has increased, studies reporting on the findings of tested CEnR engagement measurement scales for health studies are sparse. A systematic review was conducted from January 1 to March 1, 2023, to identify validated, quantitative CEnR engagement measurement tools for health studies. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology was employed. The rigor of scale development, testing, and implementation was explored, and a `best practices evaluation conducted. Themes on the readiness of scales for implementation in health research studies were narratively compiled. Nineteen studies met the search inclusion criteria-reporting on the development, testing, and implementation of seven CEnR engagement measurement scales for health studies. Scale implementation studies precipitated only two of the studies. None of the scales followed the rigorous process dictated in best practices; however, at this time, three scales have gone through the most robust testing processes. Advancement of the science of engagement measurement requires consensus on terminology, application of best practices for scale development and testing protocols, and consistency of reporting findings.

Partnering with patients and caregivers to enrich research and care in kidney disease: values and strategies

Patient and caregiver involvement broadens the scope of new knowledge generated from research and can enhance the relevance, quality and impact of research on clinical practice and health outcomes. Incorporating the perspectives of people with lived experience of chronic kidney disease (CKD) affords new insights into the design of interventions, study methodology, data analysis and implementation and has value for patients, healthcare professionals and researchers alike. However, patient involvement in CKD research has been limited and data on which to inform best practice is scarce. A number of frameworks have been developed for involving patients and caregivers in research in CKD and in health research more broadly. These frameworks provide an overall conceptual structure to guide the planning and implementation of research partnerships and describe values that are essential and strategies considered best practice when working with diverse stakeholder groups. This article aims to provide a summary of the strategies most widely used to support multistakeholder partnerships, the different ways patients and caregivers can be involved in research and the methods used to amalgamate diverse and at times conflicting points of view.

Using ethnographic approaches to document, evaluate, and facilitate virtual community-engaged implementation research

BACKGROUND

Community Advisory Boards (CABs) have been frequently used to engage diverse partners to inform research projects. Yet, evaluating the quality of engagement has not been routine. We describe a multi-method ethnographic approach documenting and assessing partner engagement in two "virtual" CABs, for which we conducted all meetings remotely.

METHODS

Two research projects for increasing equitable COVID-19 testing, vaccination, and clinical trial participation for underserved communities involved remote CAB meetings. Thirty-three partners representing 17 community groups participated in 15 sessions across the two CABs facilitated by a social change organization. We developed ethnographic documentation forms to assess multiple aspects of CAB member engagement (e.g., time spent speaking, modality used, types of interactions). Documenters were trained to observe CAB sub-groups via virtual sessions. Debriefing with the documentation team after CAB meetings supported quality assurance and process refinement. CAB members completed a brief validated survey after each meeting to assess the quality and frequency of engagement. Content and rapid thematic analysis were used to analyze documentation data. Quantitative data were summarized as frequencies and means. Qualitative and quantitative findings were triangulated.

RESULTS

A total of 4,540 interactions were identified across 15 meetings. The most frequent interaction was providing information (44%), followed by responding (37-38%). The quality and frequency of stakeholder engagement were rated favorably (average 4.7 of 5). Most CAB members (96%) reported good/excellent engagement. Specific comments included appreciation for the diversity of perspectives represented by the CAB members and suggestions for improved live interpretation. Debriefing sessions led to several methodological refinements for the documentation process and forms.

CONCLUSION

We highlight key strategies for documenting and assessing community engagement. Our methods allowed for rich ethnographic data collection that refined our work with community partners. We recommend ongoing trainings, including debriefing sessions and routinely reviewed assessment of data to strengthen meaningful community engagement.

Tools for assessing health research partnership outcomes and impacts: a systematic review

OBJECTIVE

To identify and assess the globally available valid, reliable and acceptable tools for assessing health research partnership outcomes and impacts.

METHODS

We searched Ovid MEDLINE, Embase, CINAHL Plus and PsycINFO from origin to 2 June 2021, without limits, using an a priori strategy and registered protocol. We screened citations independently and in duplicate, resolving discrepancies by consensus and retaining studies involving health research partnerships, the development, use and/or assessment of tools to evaluate partnership outcomes and impacts, and reporting empirical psychometric evidence. Study, tool, psychometric and pragmatic characteristics were abstracted using a hybrid approach, then synthesized using descriptive statistics and thematic analysis. Study quality was assessed using the quality of survey studies in psychology (Q-SSP) checklist.

RESULTS

From 56 123 total citations, we screened 36 027 citations, assessed 2784 full-text papers, abstracted data from 48 studies and one companion report, and identified 58 tools. Most tools comprised surveys, questionnaires and scales. Studies used cross-sectional or mixed-method/embedded survey designs and employed quantitative and mixed methods. Both studies and tools were conceptually well grounded, focusing mainly on outcomes, then process, and less frequently on impact measurement. Multiple forms of empirical validity and reliability evidence was present for most tools; however, psychometric characteristics were inconsistently assessed and reported. We identified a subset of studies (22) and accompanying tools distinguished by their empirical psychometric, pragmatic and study quality characteristics. While our review demonstrated psychometric and pragmatic improvements over previous reviews, challenges related to health research partnership assessment and the nascency of partnership science persist.

CONCLUSION

This systematic review identified multiple tools demonstrating empirical psychometric evidence, pragmatic strength and moderate study quality. Increased attention to psychometric and pragmatic requirements in tool development, testing and reporting is key to advancing health research partnership assessment and partnership science. PROSPERO CRD42021137932.

How are health research partnerships assessed? A systematic review of outcomes, impacts, terminology and the use of theories, models and frameworks

Background

Accurate, consistent assessment of outcomes and impacts is challenging in the health research partnerships domain. Increased focus on tool quality, including conceptual, psychometric and pragmatic characteristics, could improve the quantification, measurement and reporting partnership outcomes and impacts. This cascading review was undertaken as part of a coordinated, multicentre effort to identify, synthesize and assess a vast body of health research partnership literature.

Objective

To systematically assess the outcomes and impacts of health research partnerships, relevant terminology and the type/use of theories, models and frameworks (TMF) arising from studies using partnership assessment tools with known conceptual, psychometric and pragmatic characteristics.

Methods

Four electronic databases were searched (MEDLINE, Embase, CINAHL Plus and PsycINFO) from inception to 2 June 2021. We retained studies containing partnership evaluation tools with (1) conceptual foundations (reference to TMF), (2) empirical, quantitative psychometric evidence (evidence of validity and reliability, at minimum) and (3) one or more pragmatic characteristics. Outcomes, impacts, terminology, definitions and TMF type/use were abstracted verbatim from eligible studies using a hybrid (independent abstraction–validation) approach and synthesized using summary statistics (quantitative), inductive thematic analysis and deductive categories (qualitative). Methodological quality was assessed using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

Results

Application of inclusion criteria yielded 37 eligible studies. Study quality scores were high (mean 80%, standard deviation 0.11%) but revealed needed improvements (i.e. methodological, reporting, user involvement in research design). Only 14 (38%) studies reported 48 partnership outcomes and 55 impacts; most were positive effects (43, 90% and 47, 89%, respectively). Most outcomes were positive personal, functional, structural and contextual effects; most impacts were personal, functional and contextual in nature. Most terms described outcomes (39, 89%), and 30 of 44 outcomes/impacts terms were unique, but few were explicitly defined (9, 20%). Terms were complex and mixed on one or more dimensions (e.g. type, temporality, stage, perspective). Most studies made explicit use of study-related TMF (34, 92%). There were 138 unique TMF sources, and these informed tool construct type/choice and hypothesis testing in almost all cases (36, 97%).

Conclusion

This study synthesized partnership outcomes and impacts, deconstructed term complexities and evolved our understanding of TMF use in tool development, testing and refinement studies. Renewed attention to basic concepts is necessary to advance partnership measurement and research innovation in the field.

Systematic review protocol registration: PROSPERO protocol registration: CRD42021137932 https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=137932.

Contributors to negative biopsychosocial health or performance outcomes in rugby players (CoNBO): a systematic review and Delphi study protocol

The importance of contributors that can result in negative player outcomes in sport and the feasibility and barriers to modifying these to optimise player health and well-being have yet to be established. Within rugby codes (rugby league, rugby union and rugby sevens), within male and female cohorts across playing levels (full-time senior, part-time senior, age grade), this project aims to develop a consensus on contributors to negative biopsychosocial outcomes in rugby players (known as the CoNBO study) and establish stakeholder perceived importance of the identified contributors and barriers to their management. This project will consist of three parts; part 1: a systematic review, part 2: a three-round expert Delphi study and part 3: stakeholder rating of feasibility and barriers to management. Within part 1, systematic searches of electronic databases (PubMed, Scopus, MEDLINE, SPORTDiscus, CINAHL) will be performed. The systematic review protocol is registered with PROSPERO. Studies will be searched to identify physical, psychological and/or social factors resulting in negative player outcomes in rugby. Part 2 will consist of a three-round expert Delphi consensus study to establish additional physical, psychological and/or social factors that result in negative player outcomes in rugby and their importance. In part 3, stakeholders (eg, coaches, chief executive officers and players) will provide perceptions of the feasibility and barriers to modifying the identified factors within their setting. On completion, several manuscripts will be submitted for publication in peer-reviewed journals. The findings of this project have worldwide relevance for stakeholders in the rugby codes. PROSPERO registration number CRD42022346751.

Construct validation of the Research Engagement Survey Tool (REST)

BACKGROUND

The Research Engagement Survey Tool (REST) was developed to examine the level of partner (e.g., patients, caregivers, advocates, clinicians, community members) engagement in research studies. The REST is aligned with eight engagement principles based on the literature and consensus reached through a five round Delphi process. Each of the engagement principles has three-five corresponding items that are assessed on two Likert type scales quantity (how often: never, rarely, sometimes, often, always, not applicable) and quality (how well: poor, fair, good, very good, excellent, not applicable). We conducted a comprehensive validation of the REST. Despite the importance of partner engagement in research, currently no gold standard measure exists.

METHODS

Multiple strategies were employed to validate the REST. Here, we examine the internal consistency of items for each of the eight engagement principles. In addition, we examine the convergent validity of the comprehensive (32-item) REST with other measures (e.g., medical mistrust, Community Engagement in Research Index, Partnership Self-Assessment Tool, Wilder collaboration inventory, Partnership Assessment In community-based Research). We propose two scoring approaches for the REST; one aligned with the engagement principles and the other aligned with levels of community engagement: (1) outreach and education, (2) consultation, (3) cooperation, (4) collaboration, and (5) partnership.

RESULTS

The REST has strong internal consistency (Cronbach's alpha > 0.75) for each of the eight engagement principals measured on both scales (quality and quantity). The REST had negligible (e.g., medical mistrust, community engagement in research index), low (e.g., Partnership Assessment In community-based Research, Partnership Self-Assessment Tool- benefits scale), and moderate (e.g., Wilder collaboration inventory, Partnership Self-Assessment Tool- synergy scale) statistically significant correlations with other measures based on the Spearman rank correlation coefficient. These results suggest the REST is measuring something similar and correlated to the existing measures, but it captures a different construct (perceived research engagement).

CONCLUSIONS

The REST is a valid and reliable tool to assess research engagement of community health stakeholders in the research process. Valid tools to assess research engagement are necessary to examine the impact of engagement on the scientific process and scientific discovery and move the field of stakeholder engagement from best practices and lessons learned to evidence-based approaches based on empirical data.

Veteran Engagement in Health Services Research: a Conceptual Model

With 20 million living veterans and millions more immediate family members, and approximately 9 million veterans enrolled in the nationally networked VA healthcare system, representing the interests and needs of veterans in this complex community is a substantial endeavor. Based on the importance of engaging Veterans in research, the VA Health Services Research and Development (HSR&D) Service convened a Working Group of VA researchers and Veterans to conduct a review of patient engagement models and develop recommendations for an approach to engage Veterans in health research that would incorporate their unique lived experiences and interests, and their perspectives on research priorities. The Working Group considered the specific context for Veteran engagement in research that includes other VA stakeholders from the operational and clinical leadership of the VA Health Administration (VHA). The resulting model identifies the range of potential stakeholders and three domains of relevant constructs-processes expected to facilitate Veteran engagement in research with other stakeholders, individual stakeholder and external factors, and outcomes. The expectation is that Veteran engagement will benefit research to policy and practice translation, including increasing the transparency of research and producing knowledge that is readily accepted and implemented in healthcare.

Navigating ethical challenges of conducting randomized clinical trials on COVID-19

BACKGROUND

The contemporary frameworks for clinical research require informed consent for research participation that includes disclosure of material information, comprehension of disclosed information and voluntary consent to research participation. There is thus an urgent need to test, and an ethical imperative, to test, modify or refine medications or healthcare plans that could reduce patient morbidity, lower healthcare costs or strengthen healthcare systems.

METHODS

Conceptual review.

DISCUSSION

Although some allocation principles seem better than others, no single moral principle allocates interventions justly, necessitating combining the moral principles into multiprinciple allocation systems. The urgency notwithstanding, navigating ethical challenges related to conducting corona virus disease (COVID-19) clinical trials is mandatory, in order to safeguard the safety and welfare of research participants, ensure autonomy of participants, reduce possibilities for exploitation and ensure opportunities for research participation. The ethical challenges to can be categorized as challenges in allocation of resources for research; challenges of clinical equipoise in relation to the research questions; challenges of understanding disclosed information in potential participants; and challenges in obtaining informed consent.

CONCLUSION

To navigate these challenges, stakeholders need a delicate balance of moral principles during allocation of resources for research. Investigators need to apply information processing theories to aid decision-making about research participation or employ acceptable modifications to improve the informed consent process. Research and ethics committees should strengthen research review and oversight to ensure rigor, responsiveness and transparency.

Psychometric Properties of Heavy Work Investment Measures: A Systematic Review

In recent years, the study of heavy work investment (HWI) has been diversifying greatly in the various fields of application in the organizational field, for example, occupational health, human resources, quality at work among others. However, to date, no systematic review has been carried out to examine the methodological quality of the instruments designed to measure HWI. Therefore, the present systematic review examines the psychometric properties of three main measures of HWI: Workaholism Battery (WorkBAT), Work Addiction Risk Test (WART), and Dutch Work Addiction Scale (DUWAS). Five electronic databases were systematically searched, selecting psychometric articles. Of the 2621 articles identified, 35 articles met all inclusion criteria published between 1992 and 2019. The findings indicated that most of the articles were focused on reviewing psychometric properties, analyses were conducted from classical test theory, collected validity evidence based on internal structure and relationship with other variables, and reliability of scores was obtained through the internal consistency method. Of the instruments reviewed, the DUWAS is the one with the highest methodological quality. Recommendations are made for future research to address the psychometric study of these instruments based on recent advances in the field of organizational measurement.

A Stakeholder-Engaged Approach to Development of an Animal-Assisted Intervention for Obesity Prevention Among Youth With Autism Spectrum Disorder and Their Pet Dogs

Stakeholder involvement in research has been demonstrated to increase the effectiveness, validity, and quality of a study. This paper describes the engagement of a stakeholder panel in the development and implementation of an animal-assisted intervention (AAI) assessment and program for children diagnosed with Autism Spectrum Disorder (ASD). Canines for Autism Activity and Nutrition (CAAN) aims to promote physical activity and wellness among children diagnosed with ASD by integrating activities with their pet dog during the child's ongoing Applied Behavioral Analysis (ABA) in-home therapy sessions. Feedback from stakeholders guided program development at each stage of the research process, including this publication. Utilizing a stakeholder-informed approach was essential for the development of assessment tools, program materials, and program design. Methods that may assist others to effectively partner with stakeholders to implement an AAI among children diagnosed with ASD or related disorders are described.

Storytelling training to promote stakeholder engagement in research dissemination

Storytelling is increasingly recognized as a culturally relevant, human-centered strategy and has been linked to improvements in health knowledge, behavior, and outcomes. The Community Engagement Program of the Johns Hopkins Institute for Clinical and Translational Research designed and implemented a storytelling training program as a potentially versatile approach to promote stakeholder engagement. Data collected from multiple sources, including participant ratings, responses to open-ended questions, and field notes, consistently pointed to high-level satisfaction and acceptability of the program. As a next step, the storytelling training process and its impact need to be further investigated.

Rethinking innovation and the role of stakeholder engagement in sport and exercise medicine

In sport and exercise medicine, increasing pressure to improve athlete health outcomes and performance with limited resources has prompted an emphasis on innovation. A key component to innovation is stakeholder participation and engagement, that is, the involvement of those affected by the outcomes, such as end users and actors (the person(s) performing the required actions/behaviour change), of the research process. Several research frameworks in sport and exercise medicine highly recommend stakeholder engagement as part of the research process. There are, however, different levels to how engaged a stakeholder can be in a research project, and this level of engagement may be dependent on the researchers' goals. Stakeholder engagement can be organised on a continuum based on the stakeholder's relationship to the research and how involved they are in the project's decision-making process. This continuum can be used as a rating scale to evaluate and monitor the degree of perceived stakeholder participation in a research project. There are different paths to innovation in research, which are interlinked, and ideas and knowledge flow between them. Considering the continuum of stakeholder engagement and paths to innovation, this article highlights how different research types require different degrees of stakeholder engagement.

From intent to implementation: Factors affecting public involvement in life science research

Public involvement is key to closing the gap between research production and research use, and the only way to achieving ultimate transparency in science. The majority of life science research is not public-facing, but is funded by the public and impacts communities. We undertook an exploratory survey of researchers within the life sciences to better understand their views and perceived challenges to involving the public in their research. As survey response rate could not be determined, interpretation of the results must be cautious. We had a valid response cohort of n = 110 researchers, of whom 90% were primarily laboratory based. Using a mixed methods approach, we demonstrate that a top-down approach is key to motivate progression of life scientists from feeling positive towards public involvement to actually engaging in it. Researchers who viewed public involvement as beneficial to their research were more likely to have direct experience of doing it. We demonstrate that the systemic flaws in the way life sciences research enterprise is organised, including the promotion system, hyper-competition, and time pressures are major barriers to involving the public in the scientific process. Scientists are also apprehensive of being involuntarily involved in the current politicized climate; misinformation and publicity hype surrounding science nowadays makes them hesitant to share their early and in-progress research. The time required to deliberate study design and relevance, plan and build relationships for sustained involvement, provide and undertake training, and improve communication in the current research environment is often considered nonpragmatic, particularly for early career researchers. In conclusion, a top-down approach involving institutional incentives and infrastructure appears most effective at transitioning researchers from feeling positive towards public involvement to actually implementing it.

A Stakeholder-Engaged Process for Adapting an Evidence-Based Intervention for Posttraumatic Stress Disorder for Peer Delivery

PURPOSE

Peer providers represent a growing sector of the U.S. workforce, yet guidance is needed on best practices for adapting behavioral health interventions for peer delivery.

METHODS

We utilized the Framework for Reporting Adaptations and Modifications to Evidence-based interventions (FRAME; Wiltsey Stirman et al. 2013, 2019) to describe how we systematically adapted Skills Training in Affective and Interpersonal Regulation (STAIR) for posttraumatic stress disorder (PTSD) for peer delivery. Our process was iterative and relied on engagement of multiple stakeholders, including a work group of organizational leaders (N = 5), peer interventionists (N = 4), intervention experts (N = 2), and trial participants (N = 18). The FRAME was used to guide rapid coding across multiple data sources, including researcher field notes, meeting minutes, and intervention manual documents, and content analysis of semi-structured interviews with peer interventionists and trial participants.

RESULTS

Phase 1 (pre-trial) focused on modifications for fit with the local context and peer model. Key modifications focused on improving intervention design and packaging, removing clinical and stigmatizing language, and addressing peer interventionist training gaps. We used a hybrid approach to delivery, whereby we integrated peer model principles (self-disclosure, mutuality) into a directive approach. Phase 2 (trial) included reactive fidelity-consistent adaptations, such as additional educational resources. Phase 3 (post-trial) focused on adaptations to support roll-out of the intervention at the peer organization (e.g., group format).

CONCLUSION

Our stakeholder-engaged process may serve as a helpful exemplar to others tailoring interventions for peer delivery. Future research is needed to understand the role of stakeholder engagement and adaptation in implementation success.

Exploring community engaged research experiences and preferences: a multi-level qualitative investigation

BACKGROUND

Community engagement may make research more relevant, translatable, and sustainable, hence improving the possibility of reducing health disparities. The purpose of this study was to explore strategies for community engagement adopted by research teams and identify areas for enhancing engagement in future community engaged research.

METHODS

The Community Engagement Program of the Johns Hopkins Institute for Clinical and Translational Research hosted a forum to engage researchers and community partners in group discussion to reflect on their diverse past and current experiences in planning, implementing, and evaluating community engagement in health research. A total of 50 researchers, research staff, and community partners participated in five concurrent semi-structured group interviews and a whole group wrap-up session. Group interviews were audiotaped, transcribed verbatim, and analyzed using content analysis.

RESULTS

Four themes with eight subthemes were identified. Main themes included: Community engagement is an ongoing and iterative process; Community partner roles must be well-defined and clearly communicated; Mutual trust and transparency are central to community engagement; and Measuring community outcomes is an evolving area. Relevant subthemes were: engaging community partners in various stages of research; mission-driven vs. "checking the box"; breadth and depth of engagement; roles of community partner; recruitment and selection of community partners; building trust; clear communication for transparency; and conflict in community engaged research.

CONCLUSION

The findings highlight the benefits and challenges of community engaged research. Enhanced capacity building for community engagement, including training and communication tools for both community and researcher partners, are needed.

Community partners' responses to items assessing stakeholder engagement: Cognitive response testing in measure development

Background

Despite recognition of the importance of stakeholder input into research, there is a lack of validated measures to assess how well constituencies are engaged and their input integrated into research design. Measurement theory suggests that a community engagement measure should use clear and simple language and capture important components of underlying constructs, resulting in a valid measure that is accessible to a broad audience.

Objective

The primary objective of this study was to evaluate how community members understood and responded to a measure of community engagement developed to be reliable, valid, easily administered, and broadly usable.

Method

Cognitive response interviews were completed, during which participants described their reactions to items and how they processed them. Participants were asked to interpret item meaning, paraphrase items, and identify difficult or problematic terms and phrases, as well as provide any concerns with response options while responding to 16 of 32 survey items.

Results

The results of the cognitive response interviews of participants (N = 16) suggest concerns about plain language and literacy, clarity of question focus, and the lack of context clues to facilitate processing in response to items querying research experience. Minimal concerns were related to response options. Participants suggested changes in words and terms, as well as item structure.

Conclusion

Qualitative research can improve the validity and accessibility of measures that assess stakeholder experience of community-engaged research. The findings suggest wording and sentence structure changes that improve ability to assess implementation of community engagement and its impact on research outcomes.

Protocol for assessing stakeholder engagement in the development and evaluation of the Informed Health Choices resources teaching secondary school students to think critically about health claims and choices

Background

As part of a five year plan (2019–2023), the Informed Health Choices Project, is developing and evaluating resources for helping secondary school students learn to think critically about health claims and choices. We will bring together key stakeholders; such as secondary school teachers and students, our main target for the IHC secondary school resources, school administrators, policy makers, curriculum development specialists and parents, to enable us gain insight about the context.

Objectives

To ensure that stakeholders are effectively and appropriately engaged in the design, evaluation and dissemination of the learning resources.

To evaluate the extent to which stakeholders were successfully engaged.

Methods

Using a multi-stage stratified sampling method, we will identify a representative sample of secondary schools with varied characteristics that might modify the effects of the learning resources such as, the school location (rural, semi-urban or urban), ownership (private, public) and ICT facilities (under resourced, highly resourced). A sample of schools will be randomly selected from the schools in each stratum. We will aim to recruit a diverse sample of students and secondary school teachers from those schools. Other stakeholders will be purposively selected to ensure a diverse range of experience and expertise.

Results

Together with the teacher and student networks and the advisory panels, we will establish measurable success criteria that reflect the objectives of engaging stakeholders at the start of the project and evaluate the extent to which those criteria were met at the end of the project.

Conclusion

We aim for an increase in research uptake, improve quality and appropriateness of research results, accountability and social justice.

Engaging patients and stakeholders in preresearch: findings from the Pipeline to Proposal Awards Initiative

Aim: The Patient-Centered Outcomes Research Institute Pipeline to Proposal (P2P) Awards Initiative funded 177 awardees to engage patients and stakeholder partners in preresearch. Based on P2P, we described engagement strategies; outcomes; facilitators; and challenges to inform research funders and stakeholders participating in preresearch. Materials & methods: We used a qualitative approach based on content analysis of program data and interviews with P2P awardees and partners. Results: Awardees developed partnership infrastructure by recruiting patients and stakeholders, establishing clear roles, and providing training. Building trust was key to engaging patients and stakeholders in preresearch. Awardees reported partners were more likely to engage in PCOR in the future. Conclusion: P2P awardees increased capacity of patient and stakeholder partnerships to conduct PCOR.

Utilizing patient advocates in Parkinson's disease: A proposed framework for patient engagement and the modern metrics that can determine its success

The wide application of patient engagement and its associated benefits has increased across government, academic and pharmaceutical research. However, neither an identified standard practice for the process of engagement, nor utilization of common metrics to assess associated outcomes, exists. Parkinson's Foundation developed a patient engagement framework and metrics to assess engagement within the academic research and drug development sectors. This approach was developed over the course of several years through assessing the literature, acquiring feedback from researchers and people with Parkinson's disease and adapting practices to be relevant and generalizable across patient engagement projects. This framework includes the: 1) creation of a scope of work, 2) establishment of guiding principles, 3) selection and training of participants, 4) co‐determination of project metrics, 5) execution of the project and 6) dissemination of project findings. Parkinson's Foundation has also worked with academic, government and pharmaceutical stakeholders to identify metrics that assess both the quality of patient engagement and outcomes associated with patient engagement on projects. By improving patient engagement project methodologies and metrics, global clinical trials can have access to evidence‐based patient engagement practices to more efficiently capture the needs of, and potentially benefit, the patient community.

Partnerships, Processes, and Outcomes: A Health Equity-Focused Scoping Meta-Review of Community-Engaged Scholarship

In recent decades, there has been remarkable growth in scholarship examining the usefulness of community-engaged research (CEnR) and community-based participatory research (CBPR) for eliminating health inequities.This article seeks to synthesize the extant literature of systematic reviews, scoping reviews, and other related reviews regarding the context, processes, and research designs and interventions underlying CEnR that optimize its effectiveness. Through a scoping review, we have utilized an empirically derived framework of CBPR to map this literature and identify key findings and priorities for future research. Our study found 100 reviews of CEnR that largely support the CBPR conceptual framework.

Reaching Consensus on Principles of Stakeholder Engagement in Research

BACKGROUND

Stakeholder-engaged research is an umbrella term for the types of research that have community, patient, and/or stakeholder engagement, feedback, and bidirectional communication as approaches used in the research process. The level of stakeholder engagement across studies can vary greatly, from minimal engagement to fully collaborative partnerships.

OBJECTIVES

To present the process of reaching consensus among stakeholder and academic experts on the stakeholder engagement principles (EPs) and to identify definitions for each principle.

METHODS

We convened 19 national experts, 18 of whom remained engaged in a five-round Delphi process. The Delphi panel consisted of a broad range of stakeholders (e.g., patients, caregivers, advocacy groups, clinicians, researchers). We used web-based surveys for most rounds (1-3 and 5) and an in-person meeting for round 4. Panelists evaluated EP titles and definitions with a goal of reaching consensus (>80% agreement). Panelists' comments guided modifications, with greater weight given to non-academic stakeholder input.

CONCLUSIONS

EP titles and definitions were modified over five Delphi rounds. The panel reached consensus on eight EPs (dropping four, modifying four, and adding one) and corresponding definitions. The Delphi process allowed for a stakeholder-engaged approach to methodological research. Stakeholder engagement in research is time consuming and requires greater effort but may yield a better, more relevant outcome than more traditional scientist-only processes. This stakeholder-engaged process of reaching consensus on EPs and definitions provides a key initial step for the content validation of a survey tool to examine the level of stakeholder engagement in research studies.

Stakeholder engagement in research: a scoping review of current evaluation methods

Aim: Evaluating stakeholder engagement can capture what meaningful engagement in research entails, how it develops, and how it is experienced by all collaborators. We conducted a scoping review of recent approaches for evaluating engagement in research and present a descriptive overview of our findings. Methods: We searched peer-reviewed journal articles published worldwide in English between January 2013 and June 2018. Results: Our final sample consisted of 17 articles. Various approaches for evaluating stakeholder engagement were identified including qualitative approaches, surveys and engagement logs. Discussion & conclusion: We identified evaluation approaches that varied in quality, detail and methods. Valid, systematic and inclusive approaches that are developed with research partners and are inclusive of diverse perspectives are an important area for future research.

Content validation of a quantitative stakeholder engagement measure

AIM

Using a stakeholder-engaged approach, this study conducted content validation and item reduction of a quantitative measure of research engagement.

METHODS

A five-round modified Delphi process was used to reach consensus on items. Rounds 1-3 and 5 were conducted using web-based surveys. Round 4 consisted of a 2-day, in-person meeting. Delphi panelists received individualized reports outlining individual and aggregate group responses after rounds 1-3.

RESULTS

Over the five-round process, items were added, dropped, modified, and moved from one engagement principle to another. The number of items was reduced from 48 to 32, with three to five items corresponding to eight engagement principles.

CONCLUSIONS

Research that develops standardized, reliable, and accurate measures to assess stakeholder engagement is essential to understanding the impact of engagement on scientific discovery and the scientific process. Valid quantitative measures to assess stakeholder engagement in research are necessary to assess associations between engagement and research outcomes.

Parenting Programs for Underserved Populations in Low- and Middle-Income Countries: Issues of Scientific Integrity and Social Justice

Research suggests that parenting programs are effective for preventing behavioral and emotional difficulties in children, but a lot more attention needs to be paid to issues of context and culture during the development, testing and implementation of these interventions. The views and needs of underserved and disenfranchised communities in the US and the Global South are often not taken into account for the development and testing of interventions. The successful implementation of evidence-based interventions for vulnerable children and families in underserved and marginalized communities requires careful consideration of how existing paradigms of prevention, evaluation, and implementation science impact issues of social justice and equity. This paper will describe how a team of parenting program researchers has been collaborating with their partners globally in generating local knowledge by balancing the need for rigorous scientific methods with issues of power. Authors from the U.S., Latin America, Africa and Southeast Asia draw on their experiences regarding challenges and successes with issues regarding study design and measurement, the transferability and adaptation of interventions, and the dissemination and implementation of different parenting interventions while placing communities at the center of their efforts through participatory methods. We describe innovative approaches that span the continuum of intervention development, adaptation, optimization, evaluation, implementation, and scale up of different parenting programs for vulnerable children and families across the world. We conclude by offering specific and pragmatic recommendations to increase access of culturally relevant and effective parenting programs in these communities.

Engaging knowledge users in a systematic review on the comparative effectiveness of geriatrician-led models of care is possible: A cross-sectional survey using the Patient Engagement Evaluation Tool

BACKGROUND

A systematic review (SR) was conducted to evaluate the comparative effectiveness of geriatrician-led models of care, and an integrated knowledge translation (iKT) approach facilitated SR relevance. Activities to engage knowledge users (KUs) in the SR were evaluated for perceived level of engagement.

STUDY DESIGN AND SETTING

KUs included patients, caregivers, geriatricians, and policymakers from three Canadian provinces. Activities included 1) modified Delphi to select outcomes; 2) cross-sectional survey to select outcome measures, and 3) in-person meeting to discuss SR findings. KU engagement was assessed using the Patient Engagement Evaluation Tool (PEET) after the second and third activities. KUs rated the extent of successful engagement using a 7-point Likert scale ranging from "no extent" to "very large extent."

RESULTS

In total, 15 KUs completed the PEET: eight geriatricians, four policymakers, two patients, and one caregiver. Median engagement scores across all activities (median range: 6.00-6.50) indicated that KUs felt engaged. Differences were observed for activity type; perceived engagement at in-person meeting resulted in higher meta-criteria scores for trust (P = 0.005), legitimacy (P = 0.003), fairness (P = 0.013), and competency (P = 0.035) compared with online activities.

CONCLUSIONS

KUs can be engaged meaningfully in SR processes. Their perceived engagement was higher for in-person than for online activities.

From Craft to Reflective Art and Science Comment on "Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review"

Patient engagement practices are increasingly incorporated in health research, governance, and care. More recently, a large number of evaluation tools and metrics have been developed to support engagement evaluation. This growing interest in evaluation reflects a maturation of the patient engagement field, moving from a "craft" to a reflective "art and science," with more explicit expected benefits and risks, better understood conditions for success and failure, and increasingly rigorous evaluation instruments to improve engagement theories and interventions. It also supports a more critical view of engagement science, moving beyond reductionist views of engagement as a "black box technology" to a more subtle view of this broad category of complex interventions. Structured evaluation can advance patient engagement by supporting more reflective partnerships between patients, clinicians, health system leaders and citizens. This can help clarify mutual (and potentially contradictory) expectations toward engagement, provide a reality check toward claims of benefits and harms, and increase health systems’ capacity to implement effective engagement practices over time. To do so, closer collaborations are required between engagement scientists and practitioners to align the theories, practice and evaluation of patient and community engagement.

Improving Community Advisory Board Engagement in Precision Medicine Research to Reduce Health Disparities

Community Advisory Boards (CABs) are used in efforts to reduce health disparities; however, there is little documentation in the literature regarding their use in precision medicine research. In this case study, an academic-CAB partnership developed a questionnaire and patient educational materials for two precision smoking cessation interventions that involved use of genetic or genetically-informed information. The community-engaged research (CEnR) literature provided a framework for enhancing benefits to CAB members involved in developing research documents for use with a low-income, ethnically diverse population of smokers.The academic partners integrated three CEnR strategies: 1) in-meeting statements acknowledging their desire to learn from community partners, 2) in-meeting written feedback to and from community partners, and 3) a survey to obtain CAB member feedback post-meetings. Strategies 1 and 2 yielded modifications to pertinent study materials, as well as suggestions for improving meeting operations that were then adopted, as appropriate, by the academic partners. The survey indicated that CAB members valued the meeting procedure changes which appeared to have contributed to improvements in attendance and satisfaction with the meetings. Further operationalization of relevant partnership constructs and development of tools for measuring these aspects of community-academic partnerships is warranted to support community engagement in precision medicine research studies.

What do people think about genetics? A systematic review

Genetics is increasingly becoming a part of modern medical practice. How people think about genetics' use in medicine and their daily lives is therefore essential. Earlier studies indicated mixed attitudes about genetics. However, this might be changing. Using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) as a guideline, we initially reviewed 442 articles that looked at awareness, attitudes, knowledge, and perception of risks among the general and targeted recruitment populations. After fitting our criteria (from the last 5 years, conducted in the USA, non-provider populations, quantitative results reported, and assessed participants 18 years and older), finally 51 eligible articles were thematically coded and presented in this paper. Awareness is reported as relatively high in the studies reviewed. Attitudes are mixed but with higher proportions reporting positive attitudes towards genetic testing and counseling. Self-reported knowledge is reasonably high, specifically with the effects of specific programs developed to raise knowledge levels of the general and targeted recruited populations. Perception of risk is somewhat aligned with actual risk. With the reasonable positive reports of genetic awareness and knowledge, there is similar positive attitude and perception of risk, supporting the need for continued dissemination of such knowledge. Given interest in incorporating community participation in genomic educational strategies, we provide this review as a baseline from which to launch community-specific educational supports and tools.

Stakeholder engagement to inform HIV clinical trials: a systematic review of the evidence

INTRODUCTION

Stakeholder engagement is an essential component of HIV clinical trials. We define stakeholder engagement as an input by individuals or groups with an interest in HIV clinical trials to inform the design or conduct of said trials. Despite its value, stakeholder engagement to inform HIV clinical trials has not been rigorously examined. The purpose of our systematic review is to examine stakeholder engagement for HIV clinical trials and compare it to the recommendations of the UNAIDS/AVAC Good Participatory Practice (GPP) guidelines.

METHODS

We used the PRISMA checklist and identified English language studies describing stakeholder engagement to inform HIV clinical trials. Four databases (PubMed, Ovid, CINAHL and Web of Science) and six journals were searched, with additional studies identified using handsearching and expert input. Two independent reviewers examined citations, abstracts and full texts. Data were extracted on country, engagement methods, stakeholder types and purpose of stakeholder engagement. Based on the GPP guidelines, we examined how frequently stakeholder engagement was conducted to inform clinical trial research question development, protocol development, recruitment, enrolment, follow-up, results and dissemination.

RESULTS AND DISCUSSION

Of the 917 citations identified, 108 studies were included in the analysis. Forty-eight studies (44.4%) described stakeholder engagement in high-income countries, thirty (27.8%) in middle-income countries and nine (8.3%) in low-income countries. Fourteen methods for stakeholder engagement were identified, including individual (e.g. interviews) and group (e.g. community advisory boards) strategies. Thirty-five types of stakeholders were engaged, with approximately half of the studies (60; 55.6%) engaging HIV-affected community stakeholders (e.g. people living with HIV, at-risk or related populations of interest). We observed greater frequency of stakeholder engagement to inform protocol development (49 studies; 45.4%) and trial recruitment (47 studies; 43.5%). Fewer studies described stakeholder engagement to inform post-trial processes related to trial results (3; 2.8%) and dissemination (11; 10.2%).

CONCLUSIONS

Our findings identify important directions for future stakeholder engagement research and suggestions for policy. Most notably, we found that stakeholder engagement was more frequently conducted to inform early stages of HIV clinical trials compared to later stages. In order to meet recommendations established in the GPP guidelines, greater stakeholder engagement across all clinical trial stages is needed.

Patient and public engagement in research and health system decision making: A systematic review of evaluation tools

BACKGROUND

Patient and public engagement is growing, but evaluative efforts remain limited. Reviews looking at evaluation tools for patient engagement in individual decision making do exist, but no similar articles in research and health systems have been published.

OBJECTIVE

Systematically review and appraise evaluation tools for patient and public engagement in research and health system decision making.

METHODS

We searched literature published between January 1980 and February 2016. Electronic databases (Ovid MEDLINE, Embase, Cochrane Database of Systematic Reviews, CINAHL and PsycINFO) were consulted, as well as grey literature obtained through Google, subject-matter experts, social media and engagement organization websites. Two independent reviewers appraised the evaluation tools based on 4 assessment criteria: scientific rigour, patient and public perspective, comprehensiveness and usability.

RESULTS

In total, 10 663 unique references were identified, 27 were included. Most of these tools were developed in the last decade and were designed to support improvement of engagement activities. Only 11% of tools were explicitly based on a literature review, and just 7% were tested for reliability. Patients and members of the public were involved in designing 56% of the tools, mainly in the piloting stage, and 18.5% of tools were designed to report evaluation results to patients and the public.

CONCLUSION

A growing number of evaluation tools are available to support patient and public engagement in research and health system decision making. However, the scientific rigour with which such evaluation tools are developed could be improved, as well as the level of patient and public engagement in their design and reporting.

Rules of engagement: perspectives on stakeholder engagement for genomic biobanking research in South Africa

BACKGROUND

Genomic biobanking research is undergoing exponential growth in Africa raising a host of legal, ethical and social issues. Given the scientific complexity associated with genomics, there is a growing recognition globally of the importance of science translation and community engagement (CE) for this type of research, as it creates the potential to build relationships, increase trust, improve consent processes and empower local communities. Despite this level of recognition, there is a lack of empirical evidence of the practise and processes for effective CE in genomic biobanking in Africa.

METHODS

To begin to address this vacuum, 17 in-depth face to face interviews were conducted with South African experts in genomic biobanking research and CE to provide insight into the process, benefits and challenges of CE in South Africa. Emerging themes were analysed using a contextualised thematic approach.

RESULTS

Several themes emerged concerning the conduct of CE in genomic biobanking research in Africa. Although the literature tends to focus on the local community in CE, respondents in this study described three different layers of stakeholder engagement: community level, peer level and high level. Community level engagement includes potential participants, community advisory boards (CAB) and field workers; peer level engagement includes researchers, biobankers and scientists, while high level engagement includes government officials, funders and policy makers. Although education of each stakeholder layer is important, education of the community layer can be most challenging, due to the complexity of the research and educational levels of stakeholders in this layer.

CONCLUSION

CE is time-consuming and often requires an interdisciplinary research team approach. However careful planning of the engagement strategy, including an understanding of the differing layers of stakeholder engagement, and the specific educational needs at each layer, can help in the development of a relationship based on trust between the research team and various stakeholder groups. Since the community layer often comprises vulnerable populations in low and middle income countries (LMICs), co-development of innovative educational tools on genomic biobanking is essential. CE is clearly a component of a broader process best described as stakeholder engagement.