Informing the development of the SUCCEED reporting guideline for studies on the scaling of health interventions: A systematic review

BACKGROUND

Quality reporting contributes to effective translation of health research in practice and policy. As an initial step in the development of a reporting guideline for scaling, the Standards for reporting stUdies of sCaling evidenCEd-informED interventions (SUCCEED), we performed a systematic review to identify relevant guidelines and compile a list of potential items.

METHODS

We conducted a systematic review according to Cochrane method guidelines. We searched the following databases: MEDLINE, Embase, PsycINFO, Cochrane Library, CINAHL, Web of Science, from their respective inceptions. We also searched websites of relevant organizations and Google. We included any document that provided instructions or recommendations, e.g., reporting guideline, checklist, guidance, framework, standard; could inform the design or reporting of scaling interventions; and related to the health sector. We extracted characteristics of the included guidelines and assessed their methodological quality using a 3-item internal validity assessment tool. We extracted all items from the guidelines and classified them according to the main sections of reporting guidelines (title, abstract, introduction, methods, results, discussion and other information). We performed a narrative synthesis based on descriptive statistics.

RESULTS

Of 7704 records screened (published between 1999 and 2019), we included 39 guidelines, from which data were extracted from 57 reports. Of the 39 guidelines, 17 were for designing scaling interventions and 22 for reporting implementation interventions. At least one female author was listed in 31 guidelines, and 21 first authors were female. None of the authors belonged to the patient stakeholder group. Only one guideline clearly identified a patient as having participated in the consensus process. More than half the guidelines (56%) had been developed using an evidence-based process. In total, 750 items were extracted from the 39 guidelines and distributed into the 7 main sections.

CONCLUSION

Relevant items identified could inform the development of a reporting guideline for scaling studies of evidence-based health interventions. This and our assessment of guidelines could contribute to better reporting in the science and practice of scaling.

Spontaneous Scaling of a Primary Care Innovation in Real-Life Conditions: Protocol for a Case Study

BACKGROUND

Scaling effective primary care innovations to benefit more people is of interest to decision makers. However, we know little about how promising innovations are being scaled "spontaneously," that is, without deliberate guidance.

OBJECTIVE

We aim to observe, document, and analyze how, in real-life conditions, 1 primary care innovation spontaneously scales up across Quebec, Canada.

METHODS

We will conduct a participative study using a descriptive single-case study. It will be guided by the McLean and Gargani principles for scaling and reported according to the COREQ (Consolidated Criteria for Reporting Qualitative Research) guidelines. Informed by an integrated knowledge translation approach, our steering committee will include patient users throughout the project. Inspired by the Quebec College of Family Physician's "Dragons' Den" primary care program, we will identify a promising primary care innovation that is being or will be scaled spontaneously. We will interview the innovation team about their scaling experiences every month for 1 year. We will conduct interviews and focus groups with decision makers, health care providers, and end users in the innovation team and the target site about their experience of both scaling and receiving the scaled innovation and document meetings as nonparticipant observers. Interview transcripts and documentary data will be analyzed to (1) compare the spontaneous scaling plan and implementation with the McLean and Gargani principles for scaling and (2) determine how it was consistent with or diverged from the 4 McLean and Gargani guiding principles: justification, optimal scale, coordination, and dynamic evaluation.

RESULTS

This study was funded in March 2020 by the Canadian Institutes of Health Research. Recruitment began in November 2023 and data collection began in December 2023. Results are expected to be published in the first quarter of 2024.

CONCLUSIONS

Our study will advance the science of scaling by providing practical evidence-based material about scaling health and social care innovations in real-world settings using the 4 guiding principles of McLean and Gargani.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54855.

Evaluating the quality of research co-production: Research Quality Plus for Co-Production (RQ + 4 Co-Pro)

BACKGROUND

Co-production is an umbrella term used to describe the process of generating knowledge through partnerships between researchers and those who will use or benefit from research. Multiple advantages of research co-production have been hypothesized, and in some cases documented, in both the academic and practice record. However, there are significant gaps in understanding how to evaluate the quality of co-production. This gap in rigorous evaluation undermines the potential of both co-production and co-producers.

METHODS

This research tests the relevance and utility of a novel evaluation framework: Research Quality Plus for Co-Production (RQ + 4 Co-Pro). Following a co-production approach ourselves, our team collaborated to develop study objectives, questions, analysis, and results sharing strategies. We used a dyadic field-test design to execute RQ + 4 Co-Pro evaluations amongst 18 independently recruited subject matter experts. We used standardized reporting templates and qualitative interviews to collect data from field-test participants, and thematic assessment and deliberative dialogue for analysis. Main limitations include that field-test participation included only health research projects and health researchers and this will limit perspective included in the study, and, that our own co-production team does not include all potential perspectives that may add value to this work.

RESULTS

The field test surfaced strong support for the relevance and utility of RQ + 4 Co-Pro as an evaluation approach and framework. Research participants shared opportunities for fine-tuning language and criteria within the prototype version, but also, for alternative uses and users of RQ + 4 Co-Pro. All research participants suggested RQ + 4 Co-Pro offered an opportunity for improving how co-production is evaluated and advanced. This facilitated our revision and publication herein of a field-tested RQ + 4 Co-Pro Framework and Assessment Instrument.

CONCLUSION

Evaluation is necessary for understanding and improving co-production, and, for ensuring co-production delivers on its promise of better health.. RQ + 4 Co-Pro provides a practical evaluation approach and framework that we invite co-producers and stewards of co-production-including the funders, publishers, and universities who increasingly encourage socially relevant research-to study, adapt, and apply.

Decisions and Decisional Needs of Canadians From all Provinces and Territories During the COVID-19 Pandemic: Population-Based Cross-sectional Surveys

BACKGROUND

Never before COVID-19 had Canadians faced making health-related decisions in a context of significant uncertainty. However, little is known about which type of decisions and the types of difficulties that they faced.

OBJECTIVE

We sought to identify the health-related decisions and decisional needs of Canadians.

METHODS

Our study was codesigned by researchers and knowledge users (eg, patients, clinicians). Informed by the CHERRIES (the Checklist for Reporting Results of Internet E-Surveys) reporting guideline, we conducted 2 online surveys of random samples drawn from the Leger consumer panel of 400,000 Canadians. Eligible participants were adults (≥18 years) who received or were receiving any health services in the past 12 months for themselves (adults) or for their child (parent) or senior with cognitive impairment (caregiver). We assessed decisions and decisional needs using questions informed by the Ottawa Decision Support Framework, including decisional conflict and decision regret using the Decision Conflict Scale (DCS) and the Decision Regret Scale (DRS), respectively. Descriptive statistics were conducted for adults who had decided for themselves or on behalf of someone else. Significant decisional conflict (SDC) was defined as a total DCS score of >37.5 out of 100, and significant decision regret was defined as a total DRS score of >25 out of 100.

RESULTS

From May 18 to June 4, 2021, 14,459 adults and 6542 parents/caregivers were invited to participate. The invitation view rate was 15.5% (2236/14,459) and 28.3% (1850/6542); participation rate, 69.3% (1549/2236) and 28.7% (531/1850); and completion rate, 97.3% (1507/1549) and 95.1% (505/531), respectively. The survey was completed by 1454 (97.3%) adults and 438 (95.1%) parents/caregivers in English (1598/1892, 84.5%) or French (294/1892, 15.5%). Respondents from all 10 Canadian provinces and the northern territories represented a range of ages, education levels, civil statuses, ethnicities, and annual household income. Of 1892 respondents, 541 (28.6%) self-identified as members of marginalized groups. The most frequent decisions were (adults vs parents/caregivers) as follows: COVID-19 vaccination (490/1454, 33.7%, vs 87/438, 19.9%), managing a health condition (253/1454, 17.4%, vs 47/438, 10.7%), other COVID-19 decisions (158/1454, 10.9%, vs 85/438, 19.4%), mental health care (128/1454, 8.8%, vs 27/438, 6.2%), and medication treatments (115/1454, 7.9%, vs 23/438, 5.3%). Caregivers also reported decisions about moving family members to/from nursing or retirement homes (48/438, 11.0%). Adults (323/1454, 22.2%) and parents/caregivers (95/438, 21.7%) had SDC. Factors making decisions difficult were worrying about choosing the wrong option (557/1454, 38.3%, vs 184/438, 42.0%), worrying about getting COVID-19 (506/1454, 34.8%, vs 173/438, 39.5%), public health restrictions (427/1454, 29.4%, vs 158/438, 36.1%), information overload (300/1454, 20.6%, vs 77/438, 17.6%), difficulty separating misinformation from scientific evidence (297/1454, 20.4%, vs 77/438, 17.6%), and difficulty discussing decisions with clinicians (224/1454, 15.4%, vs 51/438, 11.6%). For 1318 (90.6%) adults and 366 (83.6%) parents/caregivers who had decided, 353 (26.8%) and 125 (34.2%) had significant decision regret, respectively. In addition, 1028 (50%) respondents made their decision alone without considering the opinions of clinicians.

CONCLUSIONS

During COVID-19, Canadians who responded to the survey faced several new health-related decisions. Many reported unmet decision-making needs, resulting in SDC and decision regret. Interventions can be designed to address their decisional needs and support patients facing new health-related decisions.

Evaluating research co-production: protocol for the Research Quality Plus for Co-Production (RQ+ 4 Co-Pro) framework

Background

Research co-production is an umbrella term used to describe research users and researchers working together to generate knowledge. Research co-production is used to create knowledge that is relevant to current challenges and to increase uptake of that knowledge into practice, programs, products, and/or policy. Yet, rigorous theories and methods to assess the quality of co-production are limited. Here we describe a framework for assessing the quality of research co-production—Research Quality Plus for Co-Production (RQ+ 4 Co-Pro)—and outline our field test of this approach.

Methods

Using a co-production approach, we aim to field test the relevance and utility of the RQ+ 4 Co-Pro framework. To do so, we will recruit participants who have led research co-production projects from the international Integrated Knowledge Translation Research Network. We aim to sample 16 to 20 co-production project leads, assign these participants to dyadic groups (8 to 10 dyads), train each participant in the RQ+ 4 Co-Pro framework using deliberative workshops and oversee a simulation assessment exercise using RQ+ 4 Co-Pro within dyadic groups. To study this experience, we use a qualitative design to collect participant demographic information and project demographic information and will use in-depth semi-structured interviews to collect data related to the experience each participant has using the RQ+ 4 Co-Pro framework.

Discussion

This study will yield knowledge about a new way to assess research co-production. Specifically, it will address the relevance and utility of using RQ+ 4 Co-Pro, a framework that includes context as an inseparable component of research, identifies dimensions of quality matched to the aims of co-production, and applies a systematic and transferable evaluative method for reaching conclusions. This is a needed area of innovation for research co-production to reach its full potential. The findings may benefit co-producers interested in understanding the quality of their work, but also other stewards of research co-production. Accordingly, we undertake this study as a co-production team representing multiple perspectives from across the research enterprise, such as funders, journal editors, university administrators, and government and health organization leaders.

Evidence on scaling in health and social care: protocol for a living umbrella review

BACKGROUND

There is a growing interest in scaling effective health innovations to promote equitable access to high-quality health services worldwide. However, multiple challenges persist in scaling innovations. In this study, we aim to summarize the scaling evidence in the health and social care literature and identify current knowledge gaps.

METHODS

We will conduct a living umbrella review according to the Joanna Briggs Institute Reviewers' Manual. We will consider all knowledge syntheses addressing scaling in health or social care (e.g., any setting, any clinical area) and conducted in a systematic way. We will search the following electronic databases: MEDLINE (Ovid), Embase, PsychINFO (Ovid), CINAHL (EBSCO), Web of Science, The Cochrane Library, Sociological Abstract (Proquest), Academic Search Premier (EBSCO), and Proquest Dissertations & Theses Global, from inception. Furthermore, we will conduct searches of the grey literature. No restriction regarding date or language will be applied. Each phase of the review will be processed by two independent reviewers. We will develop a data extraction form on Covidence. We will assess the methodological quality of the included reviews using AMSTAR2 and the risk of bias using ROBIS. Results will be presented in tabular form and accompanied by a narrative synthesis covering the traditional themes of scaling science that emerge from the analysis, such as coverage, range, and sustainability, as well as themes less covered in the literature, including reporting guidance, models, tools, barriers, and/or facilitators to scaling innovations, evidence regarding application in high-income or low-income countries, and end-user engagement. We will disseminate the findings via publications and through relevant networks.

DISCUSSION

The findings of the umbrella review will facilitate access to scaling evidence in the literature and help strengthen the science of scaling for researchers, policy makers, and program managers. Finally, this work will highlight important knowledge gaps and help prioritize future research questions.

SYSTEMATIC REVIEW REGISTRATION

This protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on November 11, 2020 (registration number: CRD42020183774 ).

Strategies for involving patients and the public in scaling-up initiatives in health and social services: protocol for a scoping review and Delphi survey

BACKGROUND

The scale-up of evidence-based innovations is required to reduce waste and inequities in health and social services (HSS). However, it often tends to be a top-down process initiated by policy makers, and the values of the intended beneficiaries are forgotten. Involving multiple stakeholders including patients and the public in the scaling-up process is thus essential but highly complex. We propose to identify relevant strategies for meaningfully and equitably involving patients and the public in the science and practice of scaling up in HSS.

METHODS

We will adapt our overall method from the RAND/UCLA Appropriateness Method. Following this, we will perform a two-prong study design (knowledge synthesis and Delphi study) grounded in an integrated knowledge translation approach. This approach involves extensive participation of a network of stakeholders interested in patient and public involvement (PPI) in scaling up and a multidisciplinary steering committee. We will conduct a systematic scoping review following the methodology recommended in the Joanna Briggs Institute Reviewers Manual. We will use the following eligibility criteria: (1) participants-any stakeholder involved in creating or testing a strategy for PPI; (2) intervention-any PPI strategy proposed for scaling-up initiatives; (3) comparator-no restriction; (4) outcomes: any process or outcome metrics related to PPI; and (5) setting-HSS. We will search electronic databases (e.g., Medline, Web of Science, Sociological Abstract) from inception onwards, hand search relevant websites, screen the reference lists of included records, and consult experts in the field. Two reviewers will independently select and extract eligible studies. We will summarize data quantitatively and qualitatively and report results using the PRISMA extension for Scoping Reviews (PRISMA-ScR) checklist. We will conduct an online Delphi survey to achieve consensus on the relevant strategies for PPI in scaling-up initiatives in HSS. Participants will include stakeholders from low-, middle-, and high-income countries. We anticipate that three rounds will allow an acceptable degree of agreement on research priorities.

DISCUSSION

Our findings will advance understanding of how to meaningfully and equitably involve patients and the public in scaling-up initiatives for sustainable HSS.

SYSTEMATIC REVIEW REGISTRATION

We registered this protocol with the Open Science Framework on August 19, 2020 ( https://osf.io/zqpx7/ ).

Essential items for reporting of scaling studies of health interventions (SUCCEED): protocol for a systematic review and Delphi process

BACKGROUND

The lack of a reporting guideline for scaling of evidence-based practices (EBPs) studies has prompted the registration of the Standards for reporting studies assessing the impact of scaling strategies of EBPs (SUCCEED) with EQUATOR Network. The development of SUCCEED will be guided by the following main steps recommended for developing health research reporting guidelines.

METHODS

Executive Committee. We established a committee composed of members of the core research team and of an advisory group. Systematic review. The protocol was registered with the Open Science Framework on 29 November 2019 (https://osf.io/vcwfx/). We will include reporting guidelines or other reports that may include items relevant to studies assessing the impact of scaling strategies. We will search the following electronic databases: EMBASE, PsycINFO, Cochrane Library, CINAHL, Web of Science, from inception. In addition, we will systematically search websites of EQUATOR and other relevant organizations. Experts in the field of reporting guidelines will also be contacted. Study selection and data extraction will be conducted independently by two reviewers. A narrative analysis will be conducted to compile a list of items for the Delphi exercise.

CONSENSUS PROCESS

We will invite panelists with expertise in: development of relevant reporting guidelines, methodologists, content experts, patient/member of the public, implementers, journal editors, and funders. We anticipated that three rounds of web-based Delphi consensus will be needed for an acceptable degree of agreement. We will use a 9-point scale (1 = extremely irrelevant to 9 = extremely relevant). Participants' response will be categorized as irrelevant (1-3), equivocal (4-6) and relevant (7-9). For each item, the consensus is reached if at least 80% of the participants' votes fall within the same category. The list of items from the final round will be discussed at face-to-face consensus meeting. Guideline validation. Participants will be authors of scaling studies. We will collect quantitative (questionnaire) and qualitative (semi-structured interview) data. Descriptive analyses will be conducted on quantitative data and constant comparative techniques on qualitative data.

DISCUSSION

Essential items for reporting scaling studies will contribute to better reporting of scaling studies and facilitate the transparency and scaling of evidence-based health interventions.

Translating research into action: an international study of the role of research funders

Background

It is widely accepted that research can lead to improved health outcomes. However, translating research into meaningful impacts in peoples’ lives requires actions that stretch well beyond those traditionally associated with knowledge creation. The research reported in this manuscript provides an international review of health research funders’ efforts to encourage this process of research uptake, application and scaling, often referred to as knowledge translation.

Methods

We conducted web-site review, document review and key informant interviews to investigate knowledge translation at 26 research funding agencies. The sample comprises the regions of Australia, Europe and North America, and a diverse range of funder types, including biomedical, clinical, multi-health domain, philanthropic, public and private organisations. The data builds on a 2008 study by the authors with the same international sample, which permitted longitudinal trend analysis.

Results

Knowledge translation is an objective of growing significance for funders across each region studied. However, there is no clear international consensus or standard on how funders might support knowledge translation. We found that approaches and mechanisms vary across region and funder type. Strategically tailored funding opportunities (grants) are the most prevalent modality of support. The most common funder-driven strategy for knowledge translation within these grants is the linking of researchers to research users. Funders could not to provide empirical evidence to support the majority of the knowledge translation activities they encourage or undertake.

Conclusions

Knowledge translation at a research funder relies on context. Accordingly, we suggest that the diversity of approaches uncovered in our research is fitting. We argue that evaluation of funding agency efforts to promote and/or support knowledge translation should be prioritised and actioned. It is paradoxical that funders’ efforts to get evidence into practice are not themselves evidence based.

Survey of Argentine health researchers on the use of evidence in policymaking

OBJECTIVE

In this study, Argentine health researchers were surveyed regarding their perceptions of facilitators and barriers to evidence-based policymaking in Argentina, as well as their publication activities, and research environment satisfaction.

METHODS

A self-administered online survey was sent to health researchers in Argentina. The survey questions were based on a preceding qualitative study of Argentine health researchers, as well as the scientific literature.

RESULTS

Of the 647 researchers that were reached, 226 accessed the survey, for a response rate of 34.9%. Over 80% of researchers surveyed had never been involved in or contributed to decision-making, while over 90% of researchers indicated they would like to be involved in the decision-making process. Decision-maker self-interest was perceived to be the driving factor in the development of health and healthcare policies. Research conducted by a research leader was seen to be the most influential factor in influencing health policy, followed by policy relevance of the research. With respect to their occupational environment, researchers rated highest and most favourably the opportunities available to present, discuss and publish research results and their ability to further their education and training. Argentine researchers surveyed demonstrated a strong interest and willingness to contribute their work and expertise to inform Argentine health policy development.

CONCLUSION

Despite Argentina's long scientific tradition, there are relatively few institutionalized linkages between health research results and health policymaking. Based on the results of this study, the disconnect between political decision-making and the health research system, coupled with fewer opportunities for formalized or informal researcher/decision-maker interaction, contribute to the challenges in evidence informing health policymaking in Argentina. Improving personal contact and the building of relationships between researchers and policymakers in Argentina will require taking into account researcher perceptions of policymakers, as highlighted in this study.

Understanding the performance and impact of public knowledge translation funding interventions: protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs

Background

The Canadian Institutes of Health Research (CIHR) has defined knowledge translation (KT) as a dynamic and iterative process that includes the synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products, and strengthen the healthcare system. CIHR, the national health research funding agency in Canada, has undertaken to advance this concept through direct research funding opportunities in KT. Because CIHR is recognized within Canada and internationally for leading and funding the advancement of KT science and practice, it is essential and timely to evaluate this intervention, and specifically, these funding opportunities.

Design

The study will employ a novel method of participatory, utilization-focused evaluation inspired by the principles of integrated KT. It will use a mixed methods approach, drawing on both quantitative and qualitative data, and will elicit participation from CIHR funded researchers, knowledge users, KT experts, as well as other health research funding agencies. Lines of inquiry will include an international environmental scan, document/data reviews, in-depth interviews, targeted surveys, case studies, and an expert review panel. The study will investigate how efficiently and effectively the CIHR model of KT funding programs operates, what immediate outcomes these funding mechanisms have produced, and what impact these programs have had on the broader state of health research, health research uptake, and health improvement.

Discussion

The protocol and results of this evaluation will be of interest to those engaged in the theory, practice, and evaluation of KT. The dissemination of the study protocol and results to both practitioners and theorists will help to fill a gap in knowledge in three areas: the role of a public research funding agency in facilitating KT, the outcomes and impacts KT funding interventions, and how KT can best be evaluated.