Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali's essential medicines

Bridging the gap between research evidence and its implementation in public health practice: case studies of embedded research model

AIM

To investigate the potential of embedded research in bridging the gap between research evidence and its implementation in public health practice.

METHODS

Using a case study methodology, semi-structured interviews were conducted with 4 embedded researchers, 9 public health practitioners, and 4 other stakeholders (2 teachers and 2 students) across four case study sites. Sites and individuals were purposively selected. Sites included two local authorities, one secondary school, and one sports organisation. Thematic data analysis was adopted to analyse the qualitative data.

RESULTS

Four themes were identified: (1) building and maintaining relationships, (2) working with stakeholders, (3) informing practice, and (4) critical reflection.

CONCLUSIONS

Embedded researchers build and maintain relationships with practitioners and other stakeholders to produce research. Evidence from the co-produced research informs future practice and research to improve service and delivery rendered to the public. Thus, embedded researchers use their role to bridge the research evidence - implementation gap in public health practice.

Factors influencing the utilization of doctoral research findings at a university in KwaZulu-Natal, South Africa: Views of academic leaders

BACKGROUND

Translation of health research findings into policy remains a challenge in sub-Saharan Africa. Factors influencing health research utilization are poorly described in literature. Therefore, identifying factors that influence the utilization of research findings for policy formulation is essential to facilitate implementation of evidence-based interventions. The purpose of this study was to explore the views of academic leaders as to why doctoral research is not adequately used in policymaking.

METHODS

In-depth interviews were held with purposively selected key informants from the College of Health Sciences. An open-ended interview guide aimed at exploring college leadership views on factors influencing utilization of PhD generated knowledge into policy was used. Data was analysed thematically using NVivo 12 software. Thematic analysis was used to generate themes around the factors influencing utilization of doctoral research into policy.

RESULTS

Factors such as inaccessibility of research results, lack of funding, poor quality of research, lack of continuity in translating research into policy, lack of timeliness of research results and lack of collaboration between researchers and policymakers hindered the utilization of PhD generated knowledge. Participants recommended engagement with the Department of Health/policymakers, collaboration with Department of Health/policymakers, increasing enrolment of South African citizens into PhD program, making final research products available to Department of Health/policymakers, and provision of funding for dissemination of research results.

CONCLUSION

The study demonstrated that final doctoral research results are mainly disseminated through journal articles and theses. Participants cited inaccessibility of research findings, lack of funding and poor-quality research as the most common factors hindering utilization of doctoral research findings. The study also recommended availing adequate funding for dissemination of research results, collaboration between researchers and policymakers, facilitation of policymaker-researcher engagement to find best ways of using research findings to influence policy and making final research products accessible to policymakers. Further research to gain the perspective of policymakers as to why doctoral research is not adequately used in policy formulation is recommended.

Implementing a Canadian shared-care ADHD program in Beijing: Barriers and facilitators to consider prior to start-up

BACKGROUND

The shared care pathway for ADHD is a program developed in Canada with two main strategies: (a) implement a shared care pathway between general practitioners (GPs) and specialists, and (b) step up or down care so that the patient is treated at the most appropriate level of care, depending on the complexity or outcome of their illness. The current study aims to identify the challenges and facilitators of implementing this program in a Chinese mental health service setting.

METHODS

Two focus groups were conducted using semi-structured interviews with a total of 7 health care providers in Beijing. An adapted grounded theory methodology using open-ended, axial and selective coding was used for data analysis.

RESULTS

We identified three main levels related to barriers and facilitators: (1) a sociocultural level of patients' and health care providers' perspectives; (2) a structural level related to internal and external organizational environments; (3) and the level of the intervention itself with its characteristics. The project is generally aligned with the mandates and goals of the health system, but two of the main obstacles are the varying qualifications of physicians in hospitals of different levels, implying different needs and flexible and adapted training programs, and the lack of appropriate patient referral systems between the different hospital levels.

CONCLUSION

Our study highlights the importance of consultation to obtain a "lay of the land" for deciding on the implementation steps of an a priori well accepted model of care.

National adaptation and implementation of WHO Model List of Essential Medicines: A qualitative evidence synthesis

BACKGROUND

The World Health Organization Model List of Essential Medicines (WHO EML) has played a critical role in guiding the country-level selection and financing of medicines for more than 4 decades. It continues to be a relevant evidence-based policy that can support universal health coverage (UHC) and access to essential medicines. The objective of this review was to identify factors affecting adaptation and implementation of WHO EML at the national level.

METHODS AND FINDINGS

We conducted a qualitative evidence synthesis by searching 10 databases (including CINAHL, Embase, Ovid MEDLINE, Scopus, and Web of Science) through October 2021. Primary qualitative studies focused on country-level implementation of WHO EML were included. The qualitative findings were populated in the Supporting the Use of Research Evidence (SURE) framework, and key themes were identified through an iterative process. We appraised the papers using the Critical Appraisal Skills Programme (CASP) tool and assessed our confidence in the findings using the Grading of Recommendations Assessment, Development and Evaluation working group-Confidence in Evidence from Reviews of Qualitative research (GRADE-CERQual). We screened 1,567 unique citations, reviewed 183 full texts, and included 23 studies, from 30 settings. Non-English studies and experiences and perceptions of stakeholders published in gray literature were not collected. Our findings centered around 3 main ideas pertaining to national adaptation and implementation of WHO EML: (1) the importance of designing institutions, governance, and leadership for national medicines lists (NMLs), particularly the consideration of transparency, coordination capacity, legislative mechanisms, managing regional differences, and clinical guidance; (2) the capacity to manage evidence to inform NML updates, including processes for contextualizing global evidence, utilizing local data and expert knowledge, and assessing budget impact, to which locally relevant cost-effectiveness information plays an important role; and (3) the influence of NML on purchasing and prescribing by altering provider incentives, through linkages to systems for financing and procurement and donor influence.

CONCLUSIONS

This qualitative evidence synthesis underscores the complexity and interdependencies inherent to implementation of WHO EML. To maximize the value of NMLs, greater investments should be made in processes and institutions that are needed to support various stages of the implementation pathway from global norms to adjusting prescribed behavior. Moreover, further research on linkages between NMLs, procurement, and the availability of medicines will provide additional insight into optimal NML implementation.

PROTOCOL REGISTRY

PROSPERO CRD42018104112.

The role of government agencies and other actors in influencing access to medicines in three East African countries

The WHO Model List of Essential Medicines (MLEM) has since 1977 helped prioritize and ensure availability of medicines especially in low- and middle-income countries. The MLEM consists mainly of generic medicines, though recent trends point towards listing expensive on-patent medicines and increasing global support for medicines against non-communicable diseases. However, the implications of such changes for national essential medicines list (NEML) updates for access to essential medicines has received relatively little attention. This study examined how government agencies and other actors in Kenya, Uganda and Tanzania participate in and influence the NEML update process and subsequent availability of prioritized medicines; and the alignment of these processes to WHO guidance. A mixed study design was used, with qualitative documentary review, key informant interviews and thematic data analysis. Results show that NEML updating processes were similar amongst the three countries and aligned to WHO guidelines, albeit conducted irregularly, with tendency to reprioritization during procurement stages, and were not always accompanied by revision of clinical guidelines. Variations were noted in the inclusion of medicines against cancer and hepatitis C, and the utilization of health technology assessment (HTA). For medicines against diseases with high global engagement, such as HIV/AIDS and TB, national stakeholders had more limited inputs in prioritization and funding. Furthermore, national actors were not influenced by the pharmaceutical industry during the NEML update process, nor were any conflicting agendas identified between health, trade and industrial policies. Hence, the study suggests that more attention should be paid to the combination of HTAs and NEMLs, particularly as countries work towards universal health coverage, in addition to heightened awareness of how global disease-specific initiatives may confound national implementation of the NEML. The study concludes with a call to strengthen country-level policy and procedural coherence around the process of prioritizing and ensuring availability of essential medicines.

Health technology assessment in sub-Saharan Africa: a descriptive analysis and narrative synthesis

Background

Countries in Sub-Saharan Africa (SSA) are moving towards universal health coverage. The process of Health Technology Assessment (HTA) can support decisions relating to benefit package design and service coverage. HTA involves institutional cooperation with agreed methods and procedural standards. We systematically reviewed the literature on policies and capacity building to support HTA institutionalisation in SSA.

Methods

We systematically reviewed the literature by searching major databases (PubMed, Embase, etc.) until June 2019 using terms considering three aspects: HTA; health policy, decision making; and SSA. We quantitatively extracted and descriptively analysed content and conducted a narrative synthesis eliciting themes from the selected literature, which varied in study type and apporach.

Results

Half of the 49 papers identified were primary research studies and mostly qualitative. Five countries were represented in six of ten studies; South Africa, Ghana, Uganda, Cameroon, and Ethiopia. Half of first authors were from SSA. Most informants were policy makers. Five themes emerged: (1) use of HTA; (2) decision-making in HTA; (3) values and criteria for setting priority areas in HTA; (4) involving stakeholders in HTA; and (5) specific examples of progress in HTA in SSA. The first one was the main theme where there was little use of evidence and research in making policy. The awareness of HTA and economic evaluation was low, with inadequate expertise and a lack of local data and tools.

Conclusions

Despite growing interest in HTA in SSA countries, awareness remains low and HTA-related activities are uncoordinated and often disconnected from policy. Further training and skills development are needed, firmly linked to a strategy focusing on strengthening within-country partnerships, particularly among researchers and policy makers. The international community has an important role here by supporting policy- relevant technical assistance, highlighting that sustainable financing demands evidence-based processes for effective resource allocation, and catalysing knowledge-sharing opportunities among countries facing similar challenges.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12962-021-00293-5.

Bugs in the Bed: Addressing the Contradictions of Embedded Science with Agile Implementation Research

Implementation research often fails to have its intended impact on what programs actually do. Embedding research within target organizational systems represents an effective response to this problem. However, contradictions associated with the approach often prevent its application. We present case studies of the application of embedded implementation research in Bangladesh, Ghana, and Tanzania where initiatives to strengthen community-based health systems were conducted using the embedded science model. In 2 of the cases, implementation research standards that are typically embraced without question were abandoned to ensure pursuit of embedded science. In the third example, statistical rigor was sustained, but this feature of the design was inconsistent with embedded science. In general, rigorous statistical designs employ units of observation that are inconsistent with organizational units that managers can control. Structural contradictions impede host institution ownership of research processes and utilization of results. Moreover, principles of scientific protocol leadership are inconsistent with managerial leadership. These and other embedded implementation science attributes are reviewed together with contradictions that challenged their pursuit in each case. Based on strategies that were effectively applied to offsetting challenges, a process of merging research with management is proposed that is derived from computer science. Known as "agile science," this paradigm combines scientific rigor with management decision making. This agile embedded research approach is designed to sustain scientific rigor while optimizing the integration of learning into managerial decision making.

What do we need to know? Data sources to support evidence-based decisions using health technology assessment in Ghana

BACKGROUND

Evidence-based decision-making for prioritising health is assisted by health technology assessment (HTA) to integrate data on effectiveness, costs and equity to support transparent decisions. Ghana is moving towards universal health coverage, facilitated mainly by the National Health Insurance Scheme (NHIS) established in 2003. The Government of Ghana is committed to institutionalising HTA for priority-setting. We aimed to identify and describe the sources of accessible data to support HTA in Ghana.

METHODS

We identified and described data sources encompassing six main domains using an existing framework. The domains were epidemiology, clinical efficacy, costs, health service use and consumption, quality of life, and equity. We used existing knowledge, views of stakeholders, and searches of the literature and internet.

RESULTS

The data sources for each of the six domains vary in extent and quality. Ghana has several large data sources to support HTA (e.g. Demographic Health Surveys) that have rigorous quality assurance processes. Few accessible data sources were available for costs and resource utilisation. The NHIS is a potentially rich source of data on resource use and costs but there are some limits on access. There are some data on equity but data on quality of life are limited.

CONCLUSIONS

A small number of quality data sources are available in Ghana but there are some gaps with respect to HTA based on greater use of local and contextualised information. Although more data are becoming available for monitoring, challenges remain in terms of their usefulness for HTA, and some information may not be available in disaggregated form to enable specific analyses. We support recent initiatives for the routine collection of comprehensive and reliable data that is easily accessible for HTA users. A commitment to HTA will require concerted efforts to leverage existing data sources, for example, from the NHIS, and develop and maintain new data (e.g. local health utility estimates). It will be critical that an overarching strategic and mandatory approach to the collection and use of health information is developed for Ghana in parallel to, and informed by, the development of HTA approaches to support resource allocation decisions. The key to HTA is to use the best available data while being open about its limitations and the impact on uncertainty.

Health equity monitoring is essential in public health: lessons from Mozambique

Background

Countries must be able to describe and monitor their populations health and well-being needs in an attempt to understand and address them. The Sustainable Development Goals (SDGs) have re-emphasized the need to invest in comprehensive health information systems to monitor progress towards health equity; however, knowledge on the capacity of health information systems to be able do this, particularly in low-income countries, remains very limited. As a case study, we aimed to evaluate the current capacity of the national health information systems in Mozambique, and the available indicators to monitor health inequalities, in line with SDG 3 (Good Health and Well Being for All at All Ages).

Methods

A data source mapping of the health information system in Mozambique was conducted. We followed the World Health Organization’s methodology of assessing data sources to evaluate the information available for every equity stratifier using a three-point scale: 1 - information is available, 2 - need for more information, and 3 - an information gap. Also, for each indicator we estimated the national average inequality score.

Results

Eight data sources contain health information to measure and monitor progress towards health equity in line with the 27 SDG3 indicators. Seven indicators bear information with nationally funded data sources, ten with data sources externally funded, and ten indicators either lack information or it does not applicable for the matter of the study. None of the 27 indicators associated with SDG3 can be fully disaggregated by equity stratifiers; they either lack some information (15 indicators) or do not have information at all (nine indicators). The indicators that contain more information are related to maternal and child health.

Conclusions

There are important information gaps in Mozambique’s current national health information system which prevents it from being able to comprehensively measure and monitor health equity. Comprehensive national health information systems are an essential public health need. Significant policy and political challenges must also be addressed to ensure effective interventions and action towards health equity in the country.

Organisational factors that facilitate research use in public health policy-making: a scoping review

Background

Although important syntheses and theoretical works exist in relation to understanding the organisational factors that facilitate research use, these contributions differ in their scope and object of study as well as their theoretical underpinnings. Therefore, from an exploratory angle, it may be useful to map out the current literature on organisational factors of research use in public health policy-making when revisiting existing theories and frameworks to gain further theoretical insights.

Methods

Herein, a scoping review technique and thematic content analysis were used to bring together findings from both synthesised and empirical studies of different types to map out the organisational factors that facilitate research use in public health policy-making.

Results

A total of 14 reviews and 40 empirical studies were included in the analysis. These were thematically coded and the intra-organisational factors reported as enabling research use were examined. Five main categories of organisational factors that advance research use in policy organisations – (1) individual factors, (2) the management of research integration, (3) organisational systems and infrastructures of research use, (4) institutional structures and rules for policy-making, and (5) organisational characteristics – were derived as well as 18 subcategories and a total of 64 specific factors, where 27 factors were well supported by research.

Conclusions

Using a scoping review methodology, the intra-organisational factors influencing research use in policy-making (including individual factors) were systematically mapped and the theories applied in this area of research were assessed. The review findings confirm the importance of an intra-organisational perspective when exploring research use, showing that many organisational factors are critical facilitators of research use but also that many factors and mechanisms are understudied. The synthesis shows a lack of studies on politicians and the need for more theoretically founded research. Despite increased efforts to update the existing evidential and theoretical basis of research use, we still need frameworks that combine different approaches and theories to help us grasp the complex organisational mechanisms that facilitate research use in policy settings.

Fostering access to and use of contextualised knowledge to support health policy-making: lessons from the Policy Information Platform in Nigeria

Background

Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Creating a one-stop shop for evidence is an approach that can facilitate timely access to the best evidence to inform policy decisions. We report outcomes after implementation of the Policy Information Platform (PIP), a pilot one-stop evidence repository in Nigeria designed to alleviate barriers to accessing policy-relevant knowledge.

Methods

This cross-sectional study involved five phases, namely (1) consultation with Nigerian policy-makers to identify priority policy issues, areas of health policy information needs, and challenges and capacity constraints in accessing evidence for policy-making; (2) a stakeholder engagement workshop to formally launch the PIP; (3) extraction of data and other information from scientific articles, policy briefs, evaluation reports, grey literature and health policy documents relevant to policy-making in Nigeria (identified by Google and PubMed searches and by examination of websites of relevant Nigerian government ministries, agencies and parastatals), for use in developing the PIP website; (4) promotion of the PIP in national and state health policy meetings; and (5) evaluation of the PIP using a stakeholder survey questionnaire distributed via email and critical appraisal of the grey literature included in the PIP using the authority, accuracy, coverage, objectivity, date and significance (AACODS) checklist.

Results

Priority policy areas identified by policy-makers were disease control and prevention, population health issues and health administration. Challenges identified by policy-makers were a lack of adequate capacity to access policy-relevant evidence and transform the evidence into policy. Policy-makers suggested using systematic reviews, policy briefs and rapid response mechanisms and involving policy-makers in research as ways of increasing evidence uptake for policy. A total of 126 policy-relevant, peer-reviewed scientific articles, 85 health policy documents and 201 policy-relevant grey literature documents were selected for inclusion in the PIP. Of the 195 individuals contacted via email to evaluate the PIP, 31 (15.9%) provided a response. Respondents noted that the PIP facilitated access to information based on local evidence and context-sensitive data. Barriers identified included lack of knowledge about the PIP and limited capacity of end-users to use the data compiled in the platform.

Conclusion

An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ access to evidence for use in policy-making and practice.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0431-4) contains supplementary material, which is available to authorized users.

Challenges to the implementation of malaria policies in Malawi

BACKGROUND

Despite malaria prevention initiatives, malaria remains a major health problem in Malawi, especially for pregnant mothers and children under the age of five. To reduce the malaria burden, Malawi established its first National Malaria Control Programme in 1984. Implementation of evidence-based policies contributed to malaria prevalence dropping from 43% in 2010 to 22% in 2017. In this study, we explored challenges to implementing malaria policies in Malawi from the perspective of key stakeholders in the country.

METHODS

In this qualitative study, we conducted in-depth interviews with 27 key informants from April to July 2015. We stopped sampling new participants when themes became saturated. Purposive and snowballing sampling techniques were used to identify key informants including malaria researchers that were policy advisors, policy makers, programme managers, and other key stakeholders. Interviews were conducted in English, recorded and transcribed, and imported into QSR Nvivo 11 for coding and analysis. Data were analysed using the qualitative content analysis approach.

RESULTS

Participants identified three main categories of challenges to the implementation of malaria policies. First structural challenges include inadequate resources, unavailability of trained staff, poor supervision and mentorship of staff, and personnel turnover in government. The second challenge is unilateral implementation of policies. The third category is the inadequately informed policy development and includes lack of platforms to engage with communities, top-down approach in policy formulation and lack of understanding of socio-cultural factors affecting policy uptake by communities.

CONCLUSIONS

Policy makers should recognize that inadequate support of policy objectives leads to an implementation gap. Therefore, policy development and implementation should not be viewed as distinct, but rather as interactive processes shaping each other. Support for health policy and systems research should be mobilized to strengthen the health system. Detailed assessment of implementation challenges to specific malaria policies should also be conducted to address these challenges and support the shift from the paradigm of malaria prevention and control to elimination in Malawi.

Evidence map of knowledge translation strategies, outcomes, facilitators and barriers in African health systems

Background

The need for research-based knowledge to inform health policy formulation and implementation is a chronic global concern impacting health systems functioning and impeding the provision of quality healthcare for all. This paper provides a systematic overview of the literature on knowledge translation (KT) strategies employed by health system researchers and policy-makers in African countries.

Methods

Evidence mapping methodology was adapted from the social and health sciences literature and used to generate a schema of KT strategies, outcomes, facilitators and barriers. Four reference databases were searched using defined criteria. Studies were screened and a searchable database containing 62 eligible studies was compiled using Microsoft Access. Frequency and thematic analysis were used to report study characteristics and to establish the final evidence map. Focus was placed on KT in policy formulation processes in order to better manage the diversity of available literature.

Results

The KT literature in African countries is widely distributed, problematically diverse and growing. Significant disparities exist between reports on KT in different countries, and there are many settings without published evidence of local KT characteristics. Commonly reported KT strategies include policy briefs, capacity-building workshops and policy dialogues. Barriers affecting researchers and policy-makers include insufficient skills and capacity to conduct KT activities, time constraints and a lack of resources. Availability of quality locally relevant research was the most reported facilitator. Limited KT outcomes reflect persisting difficulties in outcome identification and reporting.

Conclusion

This study has identified substantial geographical gaps in knowledge and evidenced the need to boost local research capacities on KT practices in low- and middle-income countries. Evidence mapping is also shown to be a useful approach that can assist local decision-making to enhance KT in policy and practice.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0419-0) contains supplementary material, which is available to authorized users.

Knowledge translation for public health in low- and middle- income countries: a critical interpretive synthesis

Background

Effective knowledge translation allows the optimisation of access to and utilisation of research knowledge in order to inform and enhance public health policy and practice. In low- and middle- income countries, there are substantial complexities that affect the way in which research can be utilised for public health action. This review attempts to draw out concepts in the literature that contribute to defining some of the complexities and contextual factors that influence knowledge translation for public health in low- and middle- income countries.

Methods

A Critical Interpretive Synthesis was undertaken, a method of analysis which allows a critical review of a wide range of heterogeneous evidence, through incorporating systematic review methods with qualitative enquiry techniques. A search for peer-reviewed articles published between 2000 and 2016 on the topic of knowledge translation for public health in low- and middle – income countries was carried out, and 85 articles were reviewed and analysed using this method.

Results

Four main concepts were identified: 1) tension between ‘global’ and ‘local’ health research, 2) complexities in creating and accessing evidence, 3) contextualising knowledge translation strategies for low- and middle- income countries, and 4) the unique role of non-government organisations in the knowledge translation process.

Conclusion

This method of review has enabled the identification of key concepts that may inform practice or further research in the field of knowledge translation in low- and middle- income countries.

Electronic supplementary material

The online version of this article (10.1186/s41256-018-0084-9) contains supplementary material, which is available to authorized users.

A review of the process of knowledge transfer and use of evidence in reproductive and child health in Ghana

Background

The paper carries out a situational analysis to examine the production, dissemination and utilisation of reproductive and child health-related evidence to inform policy formulation in Ghana’s health sector.

Methods

The study used Wald’s model of knowledge production, transfer and utilisation as a conceptual model to collect relevant data via interviews and administration of questionnaire to a network of persons who either previously or currently hold policy-relevant positions in Ghana’s health sector. Additional data was also gathered through a scoping review of the knowledge transfer and research utilisation literature, existing reproductive and child health policies, protocols and guidelines and information available on the websites of relevant institutions in Ghana’s health sector.

Results

The findings of the study suggest that the health sector in Ghana has major strengths (strong knowledge production capacity, a positive environment for the promotion of evidence-informed policy) and opportunities (access to major donors who have the resources to fund good quality research and access to both local and international networks for collaborative research). What remains a challenge, however, is the absence of a robust institutional-wide mechanism for collating research needs and communicating these to researchers, communicating research findings in forms that are friendlier to policy-makers and the inability to incorporate funding for research into the budget of the health sector.

Conclusion

The study concludes, admonishing the Ministry of Health and its agencies to leverage on the existing strengths and opportunities to address the identified challenges.

Organisational capacity and its relationship to research use in six Australian health policy agencies

There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation's culture towards research use, must also be considered.

Designing evaluation studies to optimally inform policy: what factors do policy-makers in China consider when making resource allocation decisions on healthcare worker training programmes?

BACKGROUND

In light of the gap in evidence to inform future resource allocation decisions about healthcare provider (HCP) training in low- and middle-income countries (LMICs), and the considerable donor investments being made towards training interventions, evaluation studies that are optimally designed to inform local policy-makers are needed. The aim of our study is to understand what features of HCP training evaluation studies are important for decision-making by policy-makers in LMICs. We investigate the extent to which evaluations based on the widely used Kirkpatrick model - focusing on direct outcomes of training, namely reaction of trainees, learning, behaviour change and improvements in programmatic health indicators - align with policy-makers' evidence needs for resource allocation decisions. We use China as a case study where resource allocation decisions about potential scale-up (using domestic funding) are being made about an externally funded pilot HCP training programme.

METHODS

Qualitative data were collected from high-level officials involved in resource allocation at the national and provincial level in China through ten face-to-face, in-depth interviews and two focus group discussions consisting of ten participants each. Data were analysed manually using an interpretive thematic analysis approach.

RESULTS

Our study indicates that Chinese officials not only consider information about the direct outcomes of a training programme, as captured in the Kirkpatrick model, but also need information on the resources required to implement the training, the wider or indirect impacts of training, and the sustainability and scalability to other settings within the country. In addition to considering findings presented in evaluation studies, we found that Chinese policy-makers pay close attention to whether the evaluations were robust and to the composition of the evaluation team.

CONCLUSIONS

Our qualitative study indicates that training programme evaluations that focus narrowly on direct training outcomes may not provide sufficient information for policy-makers to make decisions on future training programmes. Based on our findings, we have developed an evidence-based framework, which incorporates but expands beyond the Kirkpatrick model, to provide conceptual and practical guidance that aids in the design of training programme evaluations better suited to meet the information needs of policy-makers and to inform policy decisions.

Promoting evidence-informed policymaking through capacity enhancement in implementation research for health researchers and policymakers in Nigeria: A cross-sectional study

BACKGROUND

Capacity constraints on implementation research among policymakers and researchers are a major challenge to the evidence to policy link. This study was designed to bring together senior policymakers and researchers in Nigeria to consider issues around research-to-policy interface and enhance their capacity on implementation research.

METHODS

The design was a cross-sectional study. A 3-day joint implementation research workshop was held for policymakers and researchers using World Health Organization/TDR Implementation Research Toolkit. Assessment of participants' capacity for evidence-informed policymaking and knowledge on implementation research was done using a 5-point Likert scale questionnaire. A postworkshop key informant interview was also conducted.

RESULTS

A total of 20 researchers and 15 policymakers participated in the study. The interaction/partnership between policymakers and researchers was generally rare in terms of priority-setting process, involvement as coinvestigators, and executing strategies to support policymakers' use of research findings. The mean ratings (MNRs) recorded mostly ranged from 1.80 to 1.89 on the 5-point scale. Researchers were rarely involved in the generation of policy-relevant research that satisfies policymakers' needs with MNR very low at 1.74. The MNRs for capacity to acquire, assess, and adapt research were generally considerably higher among researchers (3.16-3.82) than policymakers (2.27-3.20). There was a general consensus that the training tremendously improved participants' understanding and use of implementation research.

CONCLUSION

Policymakers and researchers are increasingly recognizing their need to work with each other in the interest of the health systems. There is a need to create more capacity enhancement platforms that will facilitate the interface between them.

Factors Associated With Unhealthy Snacks Consumption Among Adolescents in Iran's Schools

Background: Well-informed interventions are needed if school-based health promotion is to be effective. Among other aims, the Iranian Health Promoting School (IHPS) program that was launched in 2011, has an important aim of promoting dietary behaviors of adolescents. The present study, therefore, aimed to investigate the factors affecting unhealthy snacking of adolescents and provide evidence for a more effective IHPS program.

Methods: In a cross-sectional study design, 1320 students from 40 schools in Kerman city were selected using a proportional stratified random sampling method. A modified qualitative Food Frequency Questionnaire (FFQ) was used to gather data about unhealthy snacking behavior. Data about intrapersonal and environmental factors were obtained using a validated and reliable questionnaire. A mixed-effects negative-binomial regression model was used to analyze the data.

Results: Taste and sensory perception (prevalence rate ratio [PRR]=1.18; 95% CI: 1.09-1.27), being a male (PRR=1.20; 95% CI: 1.05-1.38) and lower nutritional knowledge (PRR=0.96; 95% CI: 0.91-0.99) were associated with higher weekly unhealthy snaking. Perceived self-efficacy (PRR=0.95; 95% CI: 0.91-1.00) negatively influenced the frequency of unhealthy snaking, with this approaching significance (P<.06). In case of environmental factors, high socio-economic status (SES) level (PRR=1.45; 95% CI: 1.26-1.67), single-parent family (PRR=1.14; 95% CI: 1.01-1.30), more social norms pressure (PRR=1.08; 95% CI: 1.01-1.17), pocket money allowance (PRR=1.21; 95% CI: 1.09-1.34), easy accessibility (PRR=1.06; 95% CI:1.01-1.11), and less perceived parental control (PRR=0.96; 95% CI: 0.92-0.99) all had a role in higher consumption of unhealthy snacks. Interestingly, larger school size was associated with less unhealthy snacking (PRR=0.79; 95% CI: 0.68-0.92).

Conclusion: Unhealthy snacking behavior is influenced by individual, socio-cultural and physical-environmental influences, namely by factors relating to poor parenting practices, high SES level, family characteristics, improper social norms pressure, and less knowledge and self-efficacy of students. This evidence can be used to inform a more evidencebased IHPS program through focusing on supportive strategies at the home, school, and local community levels.

A Qualitative Assessment of the Evidence Utilization for Health Policy-Making on the Basis of SUPPORT Tools in a Developing Country

Background: SUPPORT tools consist of 18 articles addressing the health policy-makers so that they can learn how to make evidence-informed health policies. These tools have been particularly recommended for developing countries. The present study tries to explain the process of evidence utilization for developing policy documents in the Iranian Ministry of Health and Medical Education (MoHME) and to compare the findings with those of SUPPORT tools.

Methods: A qualitative research was conducted, using the framework analysis approach. Participants consisted of senior managers and technicians in MoHME. Purposeful sampling was done, with a maximum variety, for the selection of research participants: individuals having at least 5 years of experience in preparing evidence-based policy documents. Face-to-face interviews were conducted for data collection. As a guideline for the interviews, ‘the Utilization of Evidence in Policy-Making Organizations’ procedure was used. The data were analyzed through the analysis of the framework method using MAXQDA 10 software.

Results: The participants acquired the research evidence in a topic-based form, and they were less likely to search on the basis of the evidence pyramid. To assess the quality of evidence, they did not use standard critical tools; to adapt the evidence and interventions with the local setting, they did not use the ideas and experiences of all stakeholders, and in preparing the evidence-based policy documents, they did not take into consideration the window of opportunity, did not refrain from using highly technical terms, did not write user-friendly summaries, and did not present alternative policy options. In order to develop health policies, however, they used the following innovations: attention to the financial burden of policy issues on the agenda, sensitivity analysis of the preferred policy option on the basis of technical, sociopolitical, and economic feasibility, advocacy from other scholars, using the multi-criteria decision-making models for the prioritization of policy options, implementation of policy based on the degree of readiness of policy-implementing units, and the classification of policy documents on the basis of different conditions of policy-making (urgent, short-term, and long-term).

Conclusion: Findings showed that the process of evidence utilization in IR-MoH enjoys some innovations for the support of health policy development. The present study provides IR-MoH with considerable opportunities for the improvement of evidence-informed health policy-making. Moreover, the SUPPORT process and tools are recommended to be used in developing countries.

The West African experience in establishing steering committees for better collaboration between researchers and decision-makers to increase the use of health research findings

Background

Aware of the advantages of a project steering committee (SC) in terms of influencing the development of evidence-based health policies, the West African Health Organisation (WAHO) encouraged and supported the creation of such SCs around four research projects in four countries (Burkina Faso, Nigeria, Senegal and Sierra Leone). This study was conducted to describe the process that was used to establish these committees and its findings aim to assist other stakeholders in initiating this type of process.

Methods

This is a cross-sectional, qualitative study of the initiative’s four projects. In addition to a literature review and a review of the project documents, an interview guide was used to collect data from 14 members of the SCs, research teams, WAHO and the International Development Research Center. The respondents were selected with a view to reaching data saturation. The technique of thematic analysis by simple categorisation was used.

Results

To set up the SCs, a research team in each country worked with health authorities to identify potential members, organise meetings with these members and sought the authorities’ approval to formalise the SCs. The SCs’ mission was to provide technical assistance to the researchers during the implementation phase and to facilitate the transfer and use of the findings. The ‘doing by learning’ approach used by each research team, combined with WAHO’s catalytic role with each country’s Ministry of Health, helped each SC manage its contextual difficulties and function effectively.

Conclusion

The involvement of technical and financial partners motivated the researchers and ministries of health, who, in turn, motivated other actors to volunteer on the SCs. The ‘doing by learning’ approach made it possible to develop strategies adapted to each context to create, facilitate and operate each SC and manage its difficulties. To reproduce such an experience, a strong understanding of the local context and the involvement of strong partners are required.

Improving maternal and child health policymaking processes in Nigeria: an assessment of policymakers' needs, barriers and facilitators of evidence-informed policymaking

BACKGROUND

In Nigeria, interest in the evidence-to-policy process is gaining momentum among policymakers involved in maternal, newborn and child health (MNCH). However, numerous gaps exist among policymakers on use of research evidence in policymaking. The objective of this study was to assess the perception of MNCH policymakers regarding their needs and the barriers and facilitators to use of research evidence in policymaking in Nigeria.

METHODS

The study design was a cross-sectional assessment of perceptions undertaken during a national MNCH stakeholders' engagement event convened in Abuja, Nigeria. A questionnaire designed to assess participants' perceptions was administered in person. Group consultations were also held, which centred on policymakers' evidence-to-policy needs to enhance the use of evidence in policymaking.

RESULTS

A total of 40 participants completed the questionnaire and participated in the group consultations. According to the respondents, the main barriers to evidence use in MNCH policymaking include inadequate capacity of organisations to conduct policy-relevant research; inadequate budgetary allocation for policy-relevant research; policymakers' indifference to research evidence; poor dissemination of research evidence to policymakers; and lack of interaction fora between researchers and policymakers. The main facilitators of use of research evidence for policymaking in MNCH, as perceived by the respondents, include capacity building for policymakers on use of research evidence in policy formulation; appropriate dissemination of research findings to relevant stakeholders; involving policymakers in research design and execution; and allowing policymakers' needs to drive research. The main ways identified to promote policymakers' use of evidence for policymaking included improving policymakers' skills in information and communication technology, data use, analysis, communication and advocacy.

CONCLUSION

To improve the use of research evidence in policymaking in Nigeria, there is a need to establish mechanisms that will facilitate the movement from evidence to policy and address the needs identified by policymakers. It is also imperative to improve organisational initiatives that facilitate use of research evidence for policymaking.

Research dissemination workshops: observations and implications based on an experience in Burkina Faso

Background

In Burkina Faso, malaria remains the primary cause of healthcare use, morbidity and child mortality. Therefore, efforts are needed to support the knowledge transfer and application of the results of numerous studies to better formulate and implement programs in the fight against the malaria pandemic. To this end, a 2-day dissemination workshop was held to share the most recent results produced by a multidisciplinary research team. The objective of the present study was to evaluate the workshop and the policy briefs distributed there, the effects these produced on research results use and the processes that facilitated, or not, the application of the knowledge transmitted.

Methods

A mixed-methods design was used. The data were drawn from a quantitative evaluation questionnaire completed after the workshop (n = 25/31) and qualitative interviews conducted with the researchers and various actors who attended the workshop (n = 11) and with participants in working groups (n = 40) that later analysed the policy briefs distributed at the workshop.

Results

The participants recognised the quality of the research results presented, but felt that more needed to be done to adapt the researchers’ language and improve the functioning of the workshop. The potential effects of the workshop were rather limited. Effects were mainly at two levels: individual (e.g. acquisition of new knowledge, personal awareness raising) and local (e.g. change of practice in a local non-governmental organisation). Most participants perceived the utility of the research results, but several reported that their narrow decisional power limited their ability to apply this knowledge.

Conclusions

This study showed the importance of workshops to inform key actors of research results and the need to undertake several different activities to increase the chances that the knowledge will be applied. Several recommendations are proposed to improve knowledge translation approaches in the West African context, including organising working and discussion groups, developing an action plan at the end of the workshop and offering support to participants after the workshop, among others.

Harmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue?

BACKGROUND

Health sector and programme performance assessments provide a rich source of contextual data directly linked to implementation of programmes and can inform health policy dialogue, planning and resource allocation. In seeking to maximise this opportunity, there are challenges to overcome. A meeting convened by the World Health Organization African Region discussed the strengths, weaknesses and challenges to harmonising and standardising health sector and programme performance assessments, as well as use of evidence from such processes in decision making. This article synthesises the deliberations which emerged from the meeting. Discussing these in light of other literature we propose practical options to standardising health sector and programme performance assessment and improve realisation of using evidence in decision making.

DISCUSSION

Use of evidence generated from health sector and programme performance assessments into regular country processes of sectoral monitoring, dialogue and policy modification is crucial. However, this process faces several challenges. Identified challenges were categorised under several themes, namely the weak institutional capacities for monitoring and evaluation in reference to weak health information systems, a lack of tools and skills, and weak accountability mechanisms; desynchronised planning timeframes between programme and overall health sector strategies; inadequate time to undertake comprehensive and good quality performance assessment; weak mechanisms for following up on implementation of recommendations; lack of effective stakeholder participation; and divergent political aspirations.

CONCLUSION

The question of what performance assessment is for in a country must be asked and answered clearly if the utility of these processes is to be realised. Standardising programme and sector reviews offers numerable opportunities that need to be maximised. Identified challenges need to be overcome through strengthened Ministry of Health leadership, effective stakeholder engagement and institutionalising follow-up mechanisms for agreed recommendations. In addition, health sector performance assessments need to be institutionalised as part of the accountability mechanism, and they must be planned for and funding secured within annual budget and medium term expenditure frameworks.

Harmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue?

BACKGROUND

Health sector and programme performance assessments provide a rich source of contextual data directly linked to implementation of programmes and can inform health policy dialogue, planning and resource allocation. In seeking to maximise this opportunity, there are challenges to overcome. A meeting convened by the World Health Organization African Region discussed the strengths, weaknesses and challenges to harmonising and standardising health sector and programme performance assessments, as well as use of evidence from such processes in decision making. This article synthesises the deliberations which emerged from the meeting. Discussing these in light of other literature we propose practical options to standardising health sector and programme performance assessment and improve realisation of using evidence in decision making.

DISCUSSION

Use of evidence generated from health sector and programme performance assessments into regular country processes of sectoral monitoring, dialogue and policy modification is crucial. However, this process faces several challenges. Identified challenges were categorised under several themes, namely the weak institutional capacities for monitoring and evaluation in reference to weak health information systems, a lack of tools and skills, and weak accountability mechanisms; desynchronised planning timeframes between programme and overall health sector strategies; inadequate time to undertake comprehensive and good quality performance assessment; weak mechanisms for following up on implementation of recommendations; lack of effective stakeholder participation; and divergent political aspirations.

CONCLUSION

The question of what performance assessment is for in a country must be asked and answered clearly if the utility of these processes is to be realised. Standardising programme and sector reviews offers numerable opportunities that need to be maximised. Identified challenges need to be overcome through strengthened Ministry of Health leadership, effective stakeholder engagement and institutionalising follow-up mechanisms for agreed recommendations. In addition, health sector performance assessments need to be institutionalised as part of the accountability mechanism, and they must be planned for and funding secured within annual budget and medium term expenditure frameworks.

A Knowledge Brokering Program in Burkina Faso (West Africa): Reflections from Our Experience

Abstract-In Burkina Faso, inadequate interaction among researchers, decision makers, and practitioners, together with low use of research results, impedes the development of health policies and interventions to improve equity. A knowledge translation strategy was implemented as part of a research program. The broker and his team promoted links between actors (health agents, nongovernmental organizations, public administration, policy makers, researchers), provided them with research results related to their needs, and supported them in applying this knowledge in their practices. The strategy was first implemented in Kaya District, Burkina Faso. To increase impact on population health, the strategy included widening the sphere of action through collaboration with the Ministry of Health. The broker was affiliated with a public health consulting firm in the capital, Ouagadougou, and supported by Canadian experts and a senior Burkinabè broker. Evaluation shows that research use increased at the local level among health mutuals, regional nongovernmental organizations, and health professionals in Kaya, but the objective of reaching Ministry of Health decision makers was not achieved. Results highlight the need for better training in knowledge transfer for both local and international researchers and proper identification of the gateways to reach high level decision makers. This ambitious strategy encountered several obstacles: difficult access to decision makers, poor team communication, and broker's nonconducive working environment. Future brokering strategies should analyze the political situation in depth to determine when and how to approach national and regional decision makers; invest time and effort in developing different actors' (including researchers') knowledge transfer skills; and ensure sufficient and good quality communications and resources within the team.

Malaria research and its influence on anti-malarial drug policy in Malawi: a case study

Background

In 1993, Malawi changed its first-line anti-malarial treatment for uncomplicated malaria from chloroquine to sulfadoxine-pyrimethamine (SP), and in 2007, it changed from SP to lumefantrine-artemether. The change in 1993 raised concerns about whether it had occurred timely and whether it had potentially led to early development of Plasmodium falciparum resistance to SP. This case study examined evidence from Malawi in order to assess if the policy changes were justifiable and supported by evidence.

Methods

A systematic review of documents and published evidence between 1984 and 1993, when chloroquine was the first-line drug, and 1994 and 2007, when SP was the first-line drug, was conducted herein. The review was accompanied with key informant interviews.

Results

A total of 1287 publications related to malaria drug policy changes in sub-Saharan Africa were identified. Using the inclusion criteria, four articles from 1984 to 1993 and eight articles from 1994 to 2007 were reviewed. Between 1984 and 1993, three studies reported on chloroquine poor efficacy prompting policy change according to WHO’s recommendation. From 1994 to 2007, four studies conducted in the early years of policy change reported a high SP efficacy of above 80%, retaining it as a first-line drug. Unpublished sentinel site studies between 2005 and 2007 showed a reduced efficacy of SP, influencing policy change to lumefantrine-artemether. The views of key informants indicate that the switch from chloroquine to SP was justified based on local evidence despite unavailability of WHO’s policy recommendations, while the switch to lumefantrine-artemether was uncomplicated as the country was following the recommendations from WHO.

Conclusion

Ample evidence from Malawi influenced and justified the policy changes. Therefore, locally generated evidence is vital for decision making during policy change.

A Cross-sectional Study Assessing Predictors of Essential Medicines Prescribing Behavior Based on Information-motivation-behavioral Skills Model among County Hospitals in Anhui, China

Background:

The self-consciousness and practicality of preferentially prescribed essential medicines (EMs) are not high enough in county hospitals. The purposes of this study were to use the information-motivation-behavioral skills (IMB) model to identify the predictors of essential medicines prescribing behavior (EMPB) among doctors and to examine the association between demographic variables, IMB, and EMPB.

Methods:

A cross-sectional study was carried out to assess predictive relationships among demographic variables and IMB model variables using an anonymous questionnaire administered in nine county hospitals of Anhui province. A structural equation model was constructed for the IMB model to test the instruments using analysis of moment structures 17.0.

Results:

A total of 732 participants completed the survey. The average age of the participants was 37.7 ± 8.9 years old (range: 22–67 years old). The correct rate of information was 90.64%. The average scores of the motivation and behavioral skills were 45.46 ± 7.34 (hundred mark system: 75.77) and 19.92 ± 3.44 (hundred mark system: 79.68), respectively. Approximately half (50.8%) of respondents reported that the proportion of EM prescription was below 60%. The final revised model indicated a good fit to the data (χ²/df = 4.146, goodness of fit index = 0.948, comparative fit index = 0.938, root mean square error of approximation = 0.066). More work experience (β = 0.153, P < 0.001) and behavioral skills (β = 0.449, P < 0.001) predicted more EMPB. Higher income predicted less information (β = −0.197, P < 0.001) and motivation (β = −0.204, P < 0.001). Behavioral skills were positively predicted by information (β = 0.135, P < 0.001) and motivation (β = 0.742, P < 0.001).

Conclusion:

The present study predicted some factors of EMPB, and specified the relationships among the model variables. The utilization rate of EM was not high enough. Motivation and behavior skills were crucial factors affecting EMPB. The influence of demographic variables, such as income and work experience, on EMPB should be fully appreciated. Comprehensive intervention measures should be implemented from multiple perspectives.

The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy

Background

Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).

Method

We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.

Results

The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).

Conclusion

We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0069-9) contains supplementary material, which is available to authorized users.

Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review

BACKGROUND

We completed a scoping review on the barriers and facilitators to use of systematic reviews by health care managers and policy makers, including consideration of format and content, to develop recommendations for systematic review authors and to inform research efforts to develop and test formats for systematic reviews that may optimise their uptake.

METHODS

We used the Arksey and O'Malley approach for our scoping review. Electronic databases (e.g., MEDLINE, EMBASE, PsycInfo) were searched from inception until September 2014. Any study that identified barriers or facilitators (including format and content features) to uptake of systematic reviews by health care managers and policy makers/analysts was eligible for inclusion. Two reviewers independently screened the literature results and abstracted data from the relevant studies. The identified barriers and facilitators were charted using a barriers and facilitators taxonomy for implementing clinical practice guidelines by clinicians.

RESULTS

We identified useful information for authors of systematic reviews to inform their preparation of reviews including providing one-page summaries with key messages, tailored to the relevant audience. Moreover, partnerships between researchers and policy makers/managers to facilitate the conduct and use of systematic reviews should be considered to enhance relevance of reviews and thereby influence uptake.

CONCLUSIONS

Systematic review authors can consider our results when publishing their systematic reviews. These strategies should be rigorously evaluated to determine impact on use of reviews in decision-making.

Utilization of research findings for health policy making and practice: evidence from three case studies in Bangladesh

Background

In striving to contribute towards improved health outcomes, health research institutions generate and accumulate huge volumes of relevant but often underutilized data. This study explores activities undertaken by researchers from the International Centre for Diarrhoeal Disease Research, Bangladesh (icddr,b), an international research institution that promotes the utilization of their findings in the policymaking processes in Bangladesh.

Methods

The study used an exploratory case study design and employed qualitative methods to explore activities implemented to promote research utilization and the extent to which researchers felt that their findings contributed to the policymaking process. Data were collected between September and December 2011 through key informant interviews, focus group discussions with study investigators, and database and document reviews. We reviewed findings from 19 reproductive health studies conducted and completed by icddr,b researchers between 2001 and 2011. We interviewed 21 key informants, including 13 researchers, two policy makers, and six programme implementers. Data were entered into Microsoft Word and analyzed manually following a thematic framework approach. Following the World Health Organization/Turning Research into Practice (WHO/TRIP) framework, three case studies of how research findings were utilized in the policymaking processes in Bangladesh were documented.

Results

Activities implemented to promote research utilization included conducting dissemination workshops, publishing scientific papers, developing policy briefs, providing technical assistance to policymakers and programme implementers, holding one-on-one meetings, and joining advocacy networks. The majority of the researchers (12 of 13) reported that their study findings were utilized to influence policymaking processes at different levels. However, some researchers reported being unaware of whether and how their findings were utilized. As regards actual utilization of research findings, the evidence from the three case studies indicate that research findings can be utilized instrumentally, conceptually and symbolically, and at different stages within the policymaking process, including agenda setting and policy formulation and implementation.

Conclusions

The results show that research findings from icddr,b were promoted and utilized in health policymaking processes in Bangladesh using a variety of utilization approaches. These results suggest a need for using multiple approaches to promote utilization of research findings in health policymaking processes.

Survey of Argentine health researchers on the use of evidence in policymaking

OBJECTIVE

In this study, Argentine health researchers were surveyed regarding their perceptions of facilitators and barriers to evidence-based policymaking in Argentina, as well as their publication activities, and research environment satisfaction.

METHODS

A self-administered online survey was sent to health researchers in Argentina. The survey questions were based on a preceding qualitative study of Argentine health researchers, as well as the scientific literature.

RESULTS

Of the 647 researchers that were reached, 226 accessed the survey, for a response rate of 34.9%. Over 80% of researchers surveyed had never been involved in or contributed to decision-making, while over 90% of researchers indicated they would like to be involved in the decision-making process. Decision-maker self-interest was perceived to be the driving factor in the development of health and healthcare policies. Research conducted by a research leader was seen to be the most influential factor in influencing health policy, followed by policy relevance of the research. With respect to their occupational environment, researchers rated highest and most favourably the opportunities available to present, discuss and publish research results and their ability to further their education and training. Argentine researchers surveyed demonstrated a strong interest and willingness to contribute their work and expertise to inform Argentine health policy development.

CONCLUSION

Despite Argentina's long scientific tradition, there are relatively few institutionalized linkages between health research results and health policymaking. Based on the results of this study, the disconnect between political decision-making and the health research system, coupled with fewer opportunities for formalized or informal researcher/decision-maker interaction, contribute to the challenges in evidence informing health policymaking in Argentina. Improving personal contact and the building of relationships between researchers and policymakers in Argentina will require taking into account researcher perceptions of policymakers, as highlighted in this study.

Using conjoint analysis to develop a system to score research engagement actions by health decision makers

Background

Effective use of research to inform policymaking can be strengthened by policymakers undertaking various research engagement actions (e.g., accessing, appraising, and applying research). Consequently, we developed a thorough measurement and scoring tool to assess whether and how policymakers undertook research engagement actions in the development of a policy document. This scoring tool breaks down each research engagement action into its key ‘subactions’ like a checklist. The primary aim was to develop the scoring tool further so that it assigned appropriate scores to each subaction based on its effectiveness for achieving evidence-informed policymaking. To establish the relative effectiveness of these subactions, we conducted a conjoint analysis, which was used to elicit the opinions and preferences of knowledge translation experts.

Method

Fifty-four knowledge translation experts were recruited to undertake six choice surveys. Respondents were exposed to combinations of research engagement subactions called ‘profiles’, and rated on a 1–9 scale whether each profile represented a limited (1–3), moderate (4–6), or extensive (7–9) example of each research engagement action. Generalised estimating equations were used to analyse respondents’ choice data, where a utility coefficient was calculated for each subaction. A large utility coefficient indicates that a subaction was influential in guiding experts’ ratings of extensive engagement with research.

Results

The calculated utilities were used as the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of engaging with research: searching academic literature databases, obtaining systematic reviews and peer-reviewed research, appraising relevance by verifying its applicability to the policy context, appraising quality by evaluating the validity of the method and conclusions, engaging in thorough collaborations with researchers, and undertaking formal research projects to inform the policy in question.

Conclusions

We have generated an empirically-derived and context-sensitive method of measuring and scoring the extent to which policymakers engaged with research to inform policy development. The scoring system can be used by organisations to quantify staff research engagement actions and thus provide them with insights into what types of training, systems, and tools might improve their staff’s research use capacity.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-015-0013-z) contains supplementary material, which is available to authorized users.

Implementation of a health policy advisory committee as a knowledge translation platform: the Nigeria experience

BACKGROUND

In recent times, there has been a growing demand internationally for health policies to be based on reliable research evidence. Consequently, there is a need to strengthen institutions and mechanisms that can promote interactions among researchers, policy-makers and other stakeholders who can influence the uptake of research findings. The Health Policy Advisory Committee (HPAC) is one of such mechanisms that can serve as an excellent forum for the interaction of policy-makers and researchers. Therefore, the need to have a long term mechanism that allows for periodic interactions between researchers and policy-makers within the existing government system necessitated our implementation of a newly established HPAC in Ebonyi State Nigeria, as a Knowledge Translation (KT) platform. The key study objective was to enhance the capacity of the HPAC and equip its members with the skills/competence required for the committee to effectively promote evidence informed policy-making and function as a KT platform.

METHODS

A series of capacity building programmes and KT activities were undertaken including: i) Capacity building of the HPAC using Evidence-to-Policy Network (EVIPNet) SUPPORT tools; ii) Capacity enhancement mentorship programme of the HPAC through a three-month executive training programme on health policy/health systems and KT in Ebonyi State University Abakaliki; iii) Production of a policy brief on strategies to improve the performance of the Government's Free Maternal and Child Health Care Programme in Ebonyi State Nigeria; and iv) Hosting of a multi-stakeholders policy dialogue based on the produced policy brief on the Government's Free Maternal and Child Health Care Programme.

RESULTS

The study findings indicated a noteworthy improvement in knowledge of evidence-to-policy link among the HPAC members; the elimination of mutual mistrust between policy-makers and researchers; and an increase in the awareness of importance of HPAC in the Ministry of Health (MoH).

CONCLUSION

Findings from this study suggest that a HPAC can function as a KT platform and can introduce a new dimension towards facilitating evidence-to-policy link into the operation of the MoH, and can serve as an excellent platform to bridge the gap between research and policy.

"Hearing from all sides" How legislative testimony influences state level policy-makers in the United States

BACKGROUND

This paper investigates whether state legislators find testimony influential, to what extent testimony influences policy-makers' decisions, and defines the features of testimony important in affecting policy-makers' decisions.

METHODS

We used a mixed method approach to analyze responses from 862 state-level legislators in the United States (U.S.). Data were collected via a phone survey from January-October, 2012. Qualitative data were analyzed using a general inductive approach and codes were designed to capture the most prevalent themes. Descriptive statistics and cross tabulations were also completed on thematic and demographic data to identify additional themes.

RESULTS

Most legislators, regardless of political party and other common demographics, find testimony influential, albeit with various definitions of influence. While legislators reported that testimony influenced their awareness or encouraged them to take action like conducting additional research, only 6% reported that testimony changes their vote. Among those legislators who found testimony influential, characteristics of the presenter (e.g., credibility, knowledge of the subject) were the most important aspects of testimony. Legislators also noted several characteristics of testimony content as important, including use of credible, unbiased information and data.

CONCLUSION

Findings from this study can be used by health advocates, researchers, and individuals to fine tune the delivery of materials and messages to influence policy-makers during legislative testimony. Increasing the likelihood that information from scholars will be used by policy-makers may lead to the adoption of more health policies that are informed by scientific and practice-based evidence.

National health policy-makers' views on the clarity and utility of Countdown to 2015 country profiles and reports: findings from two exploratory qualitative studies

BACKGROUND

The use of sets of indicators to assess progress has become commonplace in the global health arena. Exploratory research has suggested that indicators used for global monitoring purposes can play a role in national policy-making, however, the mechanisms through which this occurs are poorly understood. This article reports findings from two qualitative studies that aimed to explore national policy-makers' interpretation and use of indicators from country profiles and reports developed by Countdown to 2015.

METHODS

An initial study aimed at exploring comprehension of Countdown data was conducted at the 2010 joint Women Deliver/Countdown conference. A second study was conducted at the 64th World Health Assembly in 2011, specifically targeting national policy-makers. Semi-structured interviews were carried out with 29 and 22 participants, respectively, at each event. Participants were asked about their understanding of specific graphs and indicators used or proposed for use in Countdown country profiles, and their perception of how such data can inform national policy-making. Responses were categorised using a framework analysis.

RESULTS

Respondents in both studies acknowledged the importance of the profiles for tracking progress on key health indicators in and across countries, noting that they could be used to highlight changes in coverage, possible directions for future policy, for lobbying finance ministers to increase resources for health, and to stimulate competition between neighbouring or socioeconomically similar countries. However, some respondents raised questions about discrepancies between global estimates and data produced by national governments, and some struggled to understand the profile graphs shown in the absence of explanatory text. Some respondents reported that use of Countdown data in national policy-making was constrained by limited awareness of the initiative, insufficient detail in the country profiles to inform policy, and the absence of indicators felt to be more appropriate to their own country contexts.

CONCLUSIONS

The two studies emphasise the need for country consultations to ensure that national policy-makers understand how to interpret and use tools like the Countdown profile for planning purposes. They make clear the value of qualitative research for refining tools used to promote accountability, and the need for country level Countdown-like processes.

Development of evidence-based health policy documents in developing countries: a case of Iran

Background:

Evidence-based policy documents that are well developed by senior civil servants and are timely available can reduce the barriers to evidence utilization by health policy makers. This study examined the barriers and facilitators in developing evidence-based health policy documents from the perspective of their producers in a developing country.

Methods:

In a qualitative study with a framework analysis approach, we conducted semi-structured interviews using purposive and snowball sampling. A qualitative analysis software (MAXQDA-10) was used to apply the codes and manage the data. This study was theory-based and the results were compared to exploratory studies about the factors influencing evidence-based health policymaking.

Results:

18 codes and three main themes of behavioral, normative, and control beliefs were identified. Factors that influence the development of evidence-based policy documents were identified by the participants: behavioral beliefs included quality of policy documents, use of resources, knowledge and innovation, being time-consuming and contextualization; normative beliefs included policy authorities, policymakers, policy administrators, and co-workers; and control beliefs included recruitment policy, performance management, empowerment, management stability, physical environment, access to evidence, policy making process, and effect of other factors.

Conclusion:

Most of the cited barriers to the development of evidence-based policy were related to control beliefs, i.e. barriers at the organizational and health system levels. This study identified the factors that influence the development of evidence-based policy documents based on the components of the theory of planned behavior. But in exploratory studies on evidence utilization by health policymakers, the identified factors were only related to control behaviors. This suggests that the theoretical approach may be preferable to the exploratory approach in identifying the barriers and facilitators of a behavior.

A systematic review of barriers to and facilitators of the use of evidence by policymakers

Background

The gap between research and practice or policy is often described as a problem. To identify new barriers of and facilitators to the use of evidence by policymakers, and assess the state of research in this area, we updated a systematic review.

Methods

Systematic review. We searched online databases including Medline, Embase, SocSci Abstracts, CDS, DARE, Psychlit, Cochrane Library, NHSEED, HTA, PAIS, IBSS (Search dates: July 2000 - September 2012). Studies were included if they were primary research or systematic reviews about factors affecting the use of evidence in policy. Studies were coded to extract data on methods, topic, focus, results and population.

Results

145 new studies were identified, of which over half were published after 2010. Thirteen systematic reviews were included. Compared with the original review, a much wider range of policy topics was found. Although still primarily in the health field, studies were also drawn from criminal justice, traffic policy, drug policy, and partnership working. The most frequently reported barriers to evidence uptake were poor access to good quality relevant research, and lack of timely research output. The most frequently reported facilitators were collaboration between researchers and policymakers, and improved relationships and skills. There is an increasing amount of research into new models of knowledge transfer, and evaluations of interventions such as knowledge brokerage.

Conclusions

Timely access to good quality and relevant research evidence, collaborations with policymakers and relationship- and skills-building with policymakers are reported to be the most important factors in influencing the use of evidence. Although investigations into the use of evidence have spread beyond the health field and into more countries, the main barriers and facilitators remained the same as in the earlier review. Few studies provide clear definitions of policy, evidence or policymaker. Nor are empirical data about policy processes or implementation of policy widely available. It is therefore difficult to describe the role of evidence and other factors influencing policy. Future research and policy priorities should aim to illuminate these concepts and processes, target the factors identified in this review, and consider new methods of overcoming the barriers described.

A systematic review of barriers to and facilitators of the use of evidence by policymakers

BACKGROUND

The gap between research and practice or policy is often described as a problem. To identify new barriers of and facilitators to the use of evidence by policymakers, and assess the state of research in this area, we updated a systematic review.

METHODS

Systematic review. We searched online databases including Medline, Embase, SocSci Abstracts, CDS, DARE, Psychlit, Cochrane Library, NHSEED, HTA, PAIS, IBSS (Search dates: July 2000 - September 2012). Studies were included if they were primary research or systematic reviews about factors affecting the use of evidence in policy. Studies were coded to extract data on methods, topic, focus, results and population.

RESULTS

145 new studies were identified, of which over half were published after 2010. Thirteen systematic reviews were included. Compared with the original review, a much wider range of policy topics was found. Although still primarily in the health field, studies were also drawn from criminal justice, traffic policy, drug policy, and partnership working. The most frequently reported barriers to evidence uptake were poor access to good quality relevant research, and lack of timely research output. The most frequently reported facilitators were collaboration between researchers and policymakers, and improved relationships and skills. There is an increasing amount of research into new models of knowledge transfer, and evaluations of interventions such as knowledge brokerage.

CONCLUSIONS

Timely access to good quality and relevant research evidence, collaborations with policymakers and relationship- and skills-building with policymakers are reported to be the most important factors in influencing the use of evidence. Although investigations into the use of evidence have spread beyond the health field and into more countries, the main barriers and facilitators remained the same as in the earlier review. Few studies provide clear definitions of policy, evidence or policymaker. Nor are empirical data about policy processes or implementation of policy widely available. It is therefore difficult to describe the role of evidence and other factors influencing policy. Future research and policy priorities should aim to illuminate these concepts and processes, target the factors identified in this review, and consider new methods of overcoming the barriers described.

Embedding health policy and systems research into decision-making processes in low- and middle-income countries

Attention is increasingly directed to bridging the gap between the production of knowledge and its use for health decision-making in low- and middle-income countries (LMICs). An important and underdeveloped area of health policy and systems research (HPSR) is the organization of this process. Drawing from an interdisciplinary conception of embeddedness, a literature review was conducted to identify examples of embedded HPSR used to inform decision-making in LMICs. The results of the literature review were organized according to the World Health Organization’s Building Blocks Framework. Next, a conceptual model was created to illustrate the arrangement of organizations that produce embedded HPSR and the characteristics that facilitate its uptake into the arena of decision-making. We found that multiple forces converge to create context-specific pathways through which evidence enters into decision-making. Depending on the decision under consideration, the literature indicates that decision-makers may call upon an intricate combination of actors for sourcing HPSR. While proximity to decision-making does have advantages, it is not the position of the organization within the network, but rather the qualities the organization possesses, that enable it to be embedded. Our findings suggest that four qualities influence embeddedness: reputation, capacity, quality of connections to decision-makers, and quantity of connections to decision-makers and others. In addition to this, the policy environment (e.g. the presence of legislation governing the use of HPSR, presence of strong civil society, etc.) strongly influences uptake. Through this conceptual model, we can understand which conditions are likely to enhance uptake of HPSR in LMIC health systems. This raises several important considerations for decision-makers and researchers about the arrangement and interaction of evidence-generating organizations in health systems.

Assessment of organizational capacity for evidence-based health systems operations in Nigeria

In Nigeria, health outcomes are unacceptably low largely due to the inability of the health system to function optimally. As part of a strategy to strengthen the health system, an assessment of institutional capacity for use of evidence for health system operations was conducted. The health system operations in terms of stewardship, health administration, service delivery, and access to essential medical products/technologies were fairly adequate. In terms of generation/strategic use of information, health financing, and health workforce, the operations were generally inadequate. There is need to evolve strategies that will guarantee equitable and sustained improvements across health services and health outcomes.

Use of health systems evidence by policymakers in eastern Mediterranean countries: views, practices, and contextual influences

Background

Health systems evidence can enhance policymaking and strengthen national health systems. In the Middle East, limited research exists on the use of evidence in the policymaking process. This multi-country study explored policymakers’ views and practices regarding the use of health systems evidence in health policymaking in 10 eastern Mediterranean countries, including factors that influence health policymaking and barriers and facilitators to the use of evidence.

Methods

This study utilized a survey adapted and customized from a similar tool developed in Canada. Health policymakers from 10 countries (Algeria, Bahrain, Jordan, Lebanon Oman, Pakistan, Palestine, Sudan, Tunisia, and Yemen) were surveyed. Descriptive and bi-variate analyses were performed for quantitative questions and thematic analysis was done for qualitative questions.

Results

A total of 237 policymakers completed the survey (56.3% response rate). Governing parties, limited funding for the health sector and donor organizations exerted a strong influence on policymaking processes. Most (88.5%) policymakers reported requesting evidence and 43.1% reported collaborating with researchers. Overall, 40.1% reported that research evidence is not delivered at the right time. Lack of an explicit budget for evidence-informed health policymaking (55.3%), lack of an administrative structure for supporting evidence-informed health policymaking processes (52.6%), and limited value given to research (35.9%) all limited the use of research evidence. Barriers to the use of evidence included lack of research targeting health policy, lack of funding and investments, and political forces. Facilitators included availability of health research and research institutions, qualified researchers, research funding, and easy access to information.

Conclusions

Health policymakers in several countries recognize the importance of using health systems evidence. Study findings are important in light of changes unfolding in some Arab countries and can help undertake an analysis of underlying transformations and their respective health policy implications including the way evidence will be used in policy decisions.

Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers

In Nigeria, the lack of adequate understanding of the complex nature of translating research into policy and the incompatibility existing between researchers and policymakers constitute a great challenge to evidence-informed policymaking. To address these challenges a one-day evidence-to-policy training workshop was organised for policymakers, researchers and other major stakeholders in the health sector in southeastern Nigeria. Of the 104 individuals invited to the workshop 87 (83.6%) attended. The workshop training sessions focused on capacity development for evidence-informed health policy-making and building effective linkages/partnerships. The post-workshop assessment indicated significant improvements in participants' knowledge, their understanding of the health policymaking process and the use of evidence compared to their pre-workshop status. Using a focus group discussion, major strategies identified by participants that can bridge the gap between health policymakers and researchers included: involving both parties in planning and execution of health research and health programmes; promoting dialogue between researchers and policymakers; institutionalising research grants and commissioning research in health ministries; and ensuring that researchers are made to focus on the core needs of policymakers. There is need for further discussion and debate on the researchers and policymakers partnership concept in low income settings.

The emergence and current performance of a health research system: lessons from Guinea Bissau

BACKGROUND

Little is known about how health research systems (HRS) in low-income countries emerge and evolve over time, and how this process relates to their performance. Understanding how HRSs emerge is important for the development of well functioning National Health Research Systems (NHRS). The aim of this study was to assess how the HRS in Guinea Bissau has emerged and evolved over time and how the present system functions.

METHODS

We used a qualitative case-study methodology to explore the emergence and current performance of the HRS, using the NHRS framework. We reviewed documents and carried out 39 in-depth interviews, ranging from health research to policy and practice stakeholders. Using an iterative approach, we undertook a thematic analysis of the data.

RESULTS

The research practices in Guinea Bissau led to the emergence of a HRS with both local and international links and strong dependencies on international partners and donors. The post-colonial, volatile and resource-dependent context, changes in donor policies, training of local researchers and nature of the research findings influenced how the HRS evolved. Research priorities have mostly been set by 'expatriate' researchers and focused on understanding and reducing child mortality. Research funding is almost exclusively provided by foreign donors and international agencies. The training of Guinean researchers started in the mid-nineties and has since reinforced the links with the health system, broadened the research agenda and enhanced local use of research. While some studies have made an important contribution to global health, the use of research within Guinea Bissau has been constrained by the weak and donor dependent health system, volatile government, top-down policies of international agencies, and the controversial nature of some of the research findings.

CONCLUSIONS

In Guinea Bissau a de facto 'system' of research has emerged through research practices and co-evolving national and international research and development dynamics. If the aim of research is to contribute to local decision making, it is essential to modulate the emerged system by setting national research priorities, aligning funding, building national research capacity and linking research to decision making processes. Donors and international agencies can contribute to this process by coordinating their efforts and aligning to national priorities.

Identifying barriers to the availability and use of Magnesium Sulphate Injection in resource poor countries: a case study in Zambia

Background

Pre-eclampsia and eclampsia are serious complications of pregnancy and major causes of maternal mortality and morbidity worldwide. According to systematic reviews and WHO guidelines magnesium sulphate injection (MgSO4) should be the first -line treatment for severe pre-eclampsia and eclampsia. Studies have shown that this safe and effective medicine is unavailable and underutilized in many resource poor countries. The objective of this study was to identify barriers to the availability and use of MgSO4 in the Zambian Public Health System.

Methods

A 'fishbone' (Ishikawa) diagram listing probable facilitators to the availability and use of MgSO4 identified from the literature was used to develop an assessment tool. Barriers to availability and use of MgSO4 were assessed at the regulatory/government, supply, procurement, distribution, health facility and health professional levels. The assessment was completed during August 2008 using archival data, and observations at a pragmatic sample of health facilities providing obstetric services in Lusaka District, Zambia.

Results

The major barrier to the availability of MgSO4 within the public health system in Zambia was lack of procurement by the Ministry of Health. Other barriers identified included a lack of demand by health professionals at the health centre level and a lack of in-service training in the use of MgSO4. Where there was demand by obstetricians, magnesium sulphate injection was being procured from the private sector by the hospital pharmacy despite not being registered and licensed for use for the treatment of severe pre-eclampsia and eclampsia by the national Pharmaceutical Regulatory Authority.

Conclusions

The case study in Zambia highlights the complexities that underlie making essential medicines available and used appropriately. The fishbone diagram is a useful theoretical framework for illustrating the complexity of translating research findings into clinical practice. A better understanding of the supply system and of the pattern of demand for MgSO4 in Zambia should enable policy makers and stakeholders to develop and implement appropriate interventions to improve the availability and use of MgSO4.

Iraqi health system in kurdistan region: medical professionals' perspectives on challenges and priorities for improvement

BACKGROUND

The views of medical professionals on efficiency of health system and needs for any changes are very critical and constitute a cornerstone for any health system improvement. This is particularly relevant to Iraqi Kurdistan case as the events of the last few decades have significantly devastated the national Iraqi health system while the necessity for adopting a new health care system is increasingly recognized since 2004. This study aims to examine the regional health system in Iraqi Kurdistan from medical professionals' perspectives and try to define its problems and priorities for improvement.

METHODS

A survey questionnaire was developed and administered to a convenience sample of 250 medical professionals in Erbil governorate. The questionnaire included four items; rating of the quality of services and availability of resources in the health institutions, view on different aspects of the health system, the perceived priority needs for health system improvement and gender and professional characteristics of the respondents.

RESULTS

The response rate to the survey was 83.6%. A high proportion of respondents rated the different aspects of services and resources in the health institutions as weak or very weak including the availability of the required quantity and quality of medicines (68.7%), the availability of sufficient medical equipment and investigation tools (68.7%), and the quality of offered services (65.3%). Around 72% of respondents had a rather negative view on the overall health system. The weak role of medical research, the weak role of professional associations in controlling the system and the inefficient health education were identified as important problems in the current health system (87.9%, 87.1% and 84.9%, respectively). The priority needs of health system improvement included adoption of social insurance for medical care of the poor (82%), enhancing the role of family medicine (77.2%), adopting health insurance system (76.1%) and periodic scientific evaluation of physicians and other health staff (69.8%).

CONCLUSION

Medical professionals were generally unsatisfied with the different aspects of the health system in Iraqi Kurdistan region. A number of problems and different priority needs for health system improvement have been recognized that require to be studied in more details.