A systems approach for optimizing implementation to impact: meeting report and proceedings of the 2019 In the Trenches: Implementation to Impact International Summit

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

Policy options for strengthening evidence-informed health policy-making in Iran: overall SASHA project findings

Background

The institutionalization of evidence-informed health policy-making (EIHP) is complex and complicated. It is complex because it has many players and is complicated because its institutionalization will require many changes that will be challenging to make. Like many other issues, strengthening EIHP needs a road map, which should consider challenges and address them through effective, harmonized and contextualized strategies. This study aims to develop a road map for enhancing EIHP in Iran based on steps of planning.

Methods

This study consisted of three phases: (1) identifying barriers to EIHP, (2) recognizing interventions and (3) measuring the use of evidence in Iran's health policy-making. A set of activities was established for conducting these, including foresight, systematic review and policy dialogue, to identify the current and potential barriers for the first phase. For the second phase, an evidence synthesis was performed through a scoping review, by searching the websites of benchmark institutions which had good examples of EIHP practices in order to extract and identify interventions, and through eight policy dialogues and two broad opinion polls to contextualize the list of interventions. Simultaneously, two qualitative-quantitative studies were conducted to design and use a tool for assessing EIHP in the third phase.

Results

We identified 97 barriers to EIHP and categorized them into three groups, including 35 barriers on the “generation of evidence” (push side), 41 on the “use of evidence” (pull side) and 21 on the “interaction between these two” (exchange side). The list of 41 interventions identified through evidence synthesis and eight policy dialogues was reduced to 32 interventions after two expert opinion polling rounds. These interventions were classified into four main strategies for strengthening (1) the education and training system (6 interventions), (2) the incentives programmes (7 interventions), (3) the structure of policy support organizations (4 interventions) and (4) the enabling processes to support EIHP (15 interventions).

Conclusion

The policy options developed in the study provide a comprehensive framework to chart a path for strengthening the country’s EIHP considering both global practices and the context of Iran. It is recommended that operational plans be prepared for road map interventions, and the necessary resources provided for their implementation. The implementation of the road map will require attention to the principles of good governance, with a focus on transparency and accountability.

Systematic review and policy dialogue to determine challenges in evidence-informed health policy-making: findings of the SASHA study

Background

Various interventions have been undertaken in Iran to promote evidence-informed health policy-making (EIHP). Identifying the challenges in EIHP is the first step toward strengthening EIHP in each country through the design of tailored interventions. Therefore, the current study was conducted to synthesize the results of earlier studies and to finalize the list of barriers to EIHP in Iran.

Methods

To identify the barriers to EIHP in Iran, two steps were taken: a systematic review and policy dialogue. To conduct the systematic review, three Iranian databases and PubMed, Health Systems Evidence (HSE), Embase, and Scopus were searched. The reference lists of included papers and documentation from some local organizations were hand-searched. Upon conducting the systematic review, given the significance of stakeholders in clarifying the problem of EIHP, policy dialogue was used to complete the list previously extracted and to do advocacy. Selection criteria for the stakeholders included influential and informed individuals from knowledge-producing, knowledge-utilizing, and knowledge-brokering organizations. Semi-structured interviews were held with three important absent stakeholders.

Results

Challenges specific to Iran that were identified included the lack of integration of the health ministry and the medical universities, lack of ties between health knowledge utilization organizations, failure to establish long-term research plans, neglect of national research needs at the time of recruiting human resources in knowledge-producing organizations, and duplication and lack of coordination in routine data obtained from surveillance systems, disease registration systems, and censuses. It seems that some challenges are common across countries, including neglecting the importance of inter- and intra-disciplinary studies, the capacity of policy-makers and managers to utilize evidence, the criteria for evaluating the performance of policy-makers, managers, and academic members, the absence of long-term programmes in knowledge-utilizing organizations, the rapid replacement of policy-makers and managers, and lack of use of evaluation studies.

Conclusions

In this study, we tried to identify the challenges regarding EIHP in Iran using a systematic review and policy dialogue approach. This is the first step toward determining the best interventions to improve evidence-informed policy-making in each country, because these challenges are contextual and need to be investigated contextually.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00717-x.