Developing and refining the methods for a 'one-stop shop' for research evidence about health systems

Models of service delivery in adult cochlear implantation and evaluation of outcomes: A scoping review of delivery arrangements

BACKGROUND

This study aimed to describe available evidence of cochlear implantation delivery arrangements in adults and the outcomes by which these service models are measured.

METHODS

Scoping review of English language, primary studies conducted on adults (≥18 years) with ten or more subjects, published between January 2000 and June 2022, which assessed the effects of delivery arrangements of cochlear implantation were included. MEDLINE, EMBASE, CINAHL Plus, AMED, PsycINFO, LILACS, KoreaMed, IndMed, Cochrane CRCT, ISRCTN registry, WHO ICTRP and Web of Science were systematically searched. Included studies had to have a method section explicitly measure at least one of the Cochrane Effective Practice and Organization of Care (EPOC) outcome category. Criteria for systematic reviews and delivery arrangement category based on EPOC taxonomy was included in data extraction. Data was narratively synthesized based on EPOC categories.

RESULTS

A total of 8135 abstracts were screened after exclusion of duplicates, of these 357 studies fulfilled the inclusion criteria. Around 40% of the studies investigated how care is delivered, focusing on quality and safety systems. New care pathways to coordinate care and the use of information and communication technology were emerging areas. There was little evidence on continuity, coordination and integration of care, how the workforce is managed, where care is provided and changes in the healthcare environment. The main outcome measure for various delivery arrangements were the health status and performance in a test.

CONCLUSION

A substantial body of evidence exists about safety and efficacy of cochlear implantation in adults, predominantly focused on surgical aspects and this area is rapidly growing. There is a lack of evidence on aspects of care delivery that may have more impact on patients' experience such as continuity, coordination and integration of care and should be a focus of future research. This would lead to a better understanding of how patient's view CI experience, associated costs and the value of different care models.

Using and improving the PHISICC paper-based tools in the health facility laboratories: Examples of Human Centered Design taking systems thinking into practice, in Côte d'Ivoire and Nigeria

Background

Health workers in low- and middle-income countries are increasingly demanded to collect more and more data to report them to higher levels of the health information system (HIS), in detriment of useful data for clinical and public health decision-making, potentially compromising the quality of their health care provison. In order to support health workers' decision-making, we engaged with partners in Côte d'Ivoire, Mozambique and Nigeria in a research project to conceive, design, produce, implement and test paper-based health information tools: the PHISICC tools. Our aim was to understand the use of PHISICC tools by health workers and to improve them based on their feedback.

Methods

The design Health Facility Laboratories (HF Labs) in Côte d'Ivoire and in Nigeria were set up after months of use of PHISICC tools. Activities were structured in three phases or 'sprints' of co-creative research. We used a transdisciplinary approach, including anthropology and Human Centered Design (HCD), observations, shadowing, structured interviews and co-creation.

Results

Health workers appreciated the standardization of the tools across different health care areas, with a common visual language that optimized use. Several design issues were raised, in terms of formats and contents. They strongly appreciated how the PHISICC registers guided their clinical decision-making and how it facilitated tallying and counting for monthly reporting. However, adherence to new procedures was not universal. The co-creation sessions resulted in modifications to the PHISICC tools of out-patient care and postnatal care.

Discussion

Although health systems and systemic thinking allowed the teams to embrace complexity, it was the HCD approach that actually produced a shift in researchers' mind-set: from HIS as data management tools to HIS as quality of care instruments. HCD allowed navigating the complexity of health systems interventions due to its capacity to operate change: it not only allowed us to understand how the PHISICC tools were used but also how to further improve them. In the absence of (or even with) an analytical health systems framework, HCD approaches can work in real-life situations for the ideation, testing and implementation of interventions to improve health systems and health status outcomes.

Leveling up evidence syntheses: filling conceptual gaps of the role of midwifery in health systems through a network analysis

OBJECTIVE

In the research note, our main objective is to explore the value of combining an evidence synthesis with a network analysis. The discussion is based on a critical interpretive synthesis, which combines systematic review methodology with qualitive inquiry, and 'research concept' network analysis focused on understanding the roles of midwives in health systems. The interpretative analytic approach of a critical interpretive synthesis has a high explanatory value by allowing for the review of a diverse body of literature and is well-suited to delving into areas that are not well understood, such as midwifery.

RESULTS

Network analyses use graphs to represent relationships between concepts and brought to light important additional insights into the literature that were not present in the evidence synthesis alone. Given the lack of theoretical development in the area of midwifery in health systems, the critical interpretive synthesis allowed for the generation of concepts used to inform a theoretical framework, while the novel application of an exploratory network analysis deepened understanding of conceptual areas of saturation within the field, as well as identifying critical gaps in the literature.

Health equity and health system strengthening - Time for a WHO re-think

The pursuit of health equity is foundational to the global health enterprise. But while moral concerns over health inequities can galvanise political commitment, how such concerns can or should translate into practice remains less clear. This paper reviews evolving ways that equity goals have featured in key World Health Organization (WHO)-related policy documents, before discussing the heuristic value and empirical traction that the concept of equity can bring to the health system strengthening (HSS) agenda. We argue that while health equity is often presented as the overarching goal of HSS, in practice this is typically circumscribed to the provision of healthcare services. Although healthcare equity is important, we suggest that this narrow focus risks losing sight of the structural political, social and economic drivers of health and health inequities, as well as the broader contexts of care and complex socio-political mechanisms through which health systems are strengthened. Drawing on new lines of empirical inquiry, we propose that broadening the equity lens for HSS -offers exciting opportunities to put health systems at the heart of a more ambitious equity agenda in global health.

Community health workers at the dawn of a new era: 3. Programme governance

BACKGROUND

Community health workers (CHWs) can play a critical role in primary healthcare and are seen widely as important to achieving the health-related Sustainable Development Goals (SDGs). The COVID-19 pandemic has emphasized the key role of CHWs. Improving how CHW programmes are governed is increasingly recognized as important for achieving universal access to healthcare and other health-related goals. This paper, the third in a series on "Community Health Workers at the Dawn of a New Era", aims to raise critical questions that decision-makers need to consider for governing CHW programmes, illustrate the options for governance using examples of national CHW programmes, and set out a research agenda for understanding how CHW programmes are governed and how this can be improved.

METHODS

We draw from a review of the literature as well as from the knowledge and experience of those involved in the planning and management of CHW programmes.

RESULTS

Governing comprises the processes and structures through which individuals, groups, programmes, and organizations exercise rights, resolve differences, and express interests. Because CHW programmes are located between the formal health system and communities, and because they involve a wide range of stakeholders, their governance is complex. In addition, these programmes frequently fall outside of the governance structures of the formal health system or are poorly integrated with it, making governing these programmes more challenging. We discuss the following important questions that decision-makers need to consider in relation to governing CHW programmes: (1) How and where within political structures are policies made for CHW programmes? (2) Who implements decisions regarding CHW programmes and at what levels of government? (3) What laws and regulations are needed to support the programme? (4) How should the programme be adapted across different settings or groups within the country or region?

CONCLUSION

The most appropriate and acceptable models for governing CHW programmes depend on communities, on local health systems, and on the political system in which the programme is located. Stakeholders in each setting need to consider what systems are currently in place and how they might be adapted to local needs and systems.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Publication and related bias in quantitative health services and delivery research: a multimethod study

Background

Bias in the publication and reporting of research findings (referred to as publication and related bias here) poses a major threat in evidence synthesis and evidence-based decision-making. Although this bias has been well documented in clinical research, little is known about its occurrence and magnitude in health services and delivery research.

Objectives

To obtain empirical evidence on publication and related bias in quantitative health services and delivery research; to examine current practice in detecting/mitigating this bias in health services and delivery research systematic reviews; and to explore stakeholders’ perception and experiences concerning such bias.

Methods

The project included five distinct but interrelated work packages. Work package 1 was a systematic review of empirical and methodological studies. Work package 2 involved a survey (meta-epidemiological study) of randomly selected systematic reviews of health services and delivery research topics (n = 200) to evaluate current practice in the assessment of publication and outcome reporting bias during evidence synthesis. Work package 3 included four case studies to explore the applicability of statistical methods for detecting such bias in health services and delivery research. In work package 4 we followed up four cohorts of health services and delivery research studies (total n = 300) to ascertain their publication status, and examined whether publication status was associated with statistical significance or perceived ‘positivity’ of study findings. Work package 5 involved key informant interviews with diverse health services and delivery research stakeholders (n = 24), and a focus group discussion with patient and service user representatives (n = 8).

Results

We identified only four studies that set out to investigate publication and related bias in health services and delivery research in work package 1. Three of these studies focused on health informatics research and one concerned health economics. All four studies reported evidence of the existence of this bias, but had methodological weaknesses. We also identified three health services and delivery research systematic reviews in which findings were compared between published and grey/unpublished literature. These reviews found that the quality and volume of evidence and effect estimates sometimes differed significantly between published and unpublished literature. Work package 2 showed low prevalence of considering/assessing publication (43%) and outcome reporting (17%) bias in health services and delivery research systematic reviews. The prevalence was lower among reviews of associations than among reviews of interventions. The case studies in work package 3 highlighted limitations in current methods for detecting these biases due to heterogeneity and potential confounders. Follow-up of health services and delivery research cohorts in work package 4 showed positive association between publication status and having statistically significant or positive findings. Diverse views concerning publication and related bias and insights into how features of health services and delivery research might influence its occurrence were uncovered through the interviews with health services and delivery research stakeholders and focus group discussion conducted in work package 5.

Conclusions

This study provided prima facie evidence on publication and related bias in quantitative health services and delivery research. This bias does appear to exist, but its prevalence and impact may vary depending on study characteristics, such as study design, and motivation for conducting the evaluation. Emphasis on methodological novelty and focus beyond summative assessments may mitigate/lessen the risk of such bias in health services and delivery research. Methodological and epistemological diversity in health services and delivery research and changing landscape in research publication need to be considered when interpreting the evidence. Collection of further empirical evidence and exploration of optimal health services and delivery research practice are required.

Study registration

This study is registered as PROSPERO CRD42016052333 and CRD42016052366.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 33. See the NIHR Journals Library website for further project information.

Reporting of conflicts of interest by authors of primary studies on health policy and systems research: a cross-sectional survey

OBJECTIVES

The objective of this study was to assess the frequency and types of conflict of interest (COI) disclosed by authors of primary studies of health policy and systems research (HPSR).

DESIGN

We conducted a cross-sectional survey using standard systematic review methodology for study selection and data extraction. We conducted descriptive analyses.

SETTING

We collected data from papers published in 2016 in 'health policy and service journals' category in Web of Science database.

PARTICIPANTS

We included primary studies (eg, randomised controlled trials, cohort studies, qualitative studies) of HPSR published in English in 2016 peer-reviewed health policy and services journals.

OUTCOME MEASURES

Reported COI disclosures including whether authors reported COI or not, form in which COI disclosures were provided, number of authors per paper who report any type of COI, number of authors per paper who report specific types and subtypes of COI.

RESULTS

We included 200 eligible primary studies of which 132 (66%) included COI disclosure statements of authors. Of the 132 studies, 19 (14%) had at least one author reporting at least one type of COI and the most frequently reported type was individual financial COI (n=15, 11%). None of the authors reported individual intellectual COIs or personal COIs. Financial and individual COIs were reported more frequently compared with non-financial and institutional COIs.

CONCLUSION

A low percentage of HPSR primary studies included authors reporting COI. Non-financial or institutional COIs were the least reported types of COI.

A critical interpretive synthesis of the roles of midwives in health systems

BACKGROUND

Midwives' roles in sexual and reproductive health and rights continues to evolve. Understanding the profession's role and how midwives can be integrated into health systems is essential in creating evidence-informed policies. Our objective was to develop a theoretical framework of how political system factors and health systems arrangements influence the roles of midwives within the health system.

METHODS

A critical interpretive synthesis was used to develop the theoretical framework. A range of electronic bibliographic databases (CINAHL, EMBASE, Global Health database, HealthSTAR, Health Systems Evidence, MEDLINE and Web of Science) was searched through to 14 May 2020 as were policy and health systems-related and midwifery organisation websites. A coding structure was created to guide the data extraction.

RESULTS

A total of 4533 unique documents were retrieved through electronic searches, of which 4132 were excluded using explicit criteria, leaving 401 potentially relevant records, in addition to the 29 records that were purposively sampled through grey literature. A total of 100 documents were included in the critical interpretive synthesis. The resulting theoretical framework identified the range of political and health system components that can work together to facilitate the integration of midwifery into health systems or act as barriers that restrict the roles of the profession.

CONCLUSIONS

Any changes to the roles of midwives in health systems need to take into account the political system where decisions about their integration will be made as well as the nature of the health system in which they are being integrated. The theoretical framework, which can be thought of as a heuristic, identifies the core contextual factors that governments can use to best leverage their position when working to improve sexual and reproductive health and rights.

Policymaker and stakeholder perspectives on access to assistive technologies in Canada: challenges and proposed solutions for enhancing equitable access

Purpose: Unmet needs for assistive technologies (ATs) exist and the need for ATs is growing owing to demographic changes worldwide. Little comprehensive research has examined equity of access to ATs in Canada. Our study elucidates perspectives of policymakers and stakeholders on challenges and solutions for enhancing equitable access to ATs to advance policy discussions.Methods: We conducted a qualitative interview study with a purposive sample of policymakers and stakeholders. Stakeholders were from non-profit organisations; private insurance companies; ageing or technology industries; and advocacy, consumer, and support groups. We used thematic analysis to develop themes that summarised and facilitated data interpretation.Results: We conducted 24 interviews involving 32 participants. We present three themes: (1) User experiences, detailing challenges experienced by AT system users; (2) System characteristics: Challenges and solutions, outlining governance, financial, and delivery arrangements that create challenges for accessing AT, as well as participants' proposed solutions; and (3) Shifts in models and principles, for approaches that may foster equitable access to ATs. We consolidate results into a set of valued qualities of a system that can enhance equitable AT access, and relate results to relevant national and international activities.Conclusions: This is the most comprehensive study of Canadian policymaker and stakeholder views on AT access to date. Identified challenges and solutions point to opportunities for policy action and to support work to create a national vision for AT access that strengthens the potential for ATs to enable daily activity participation, independence, and societal inclusion of seniors and people with disabilities.IMPLICATIONS FOR REHABILITATIONAT use supports daily activity participation, independence, and societal inclusion of seniors and people with disabilities.There is an urgent need to ensure that those who need ATs have access to them, considering the benefits of their use, current unmet needs for ATs, and the anticipated demand for ATs because of the ageing population and increased prevalence of chronic disease and disability.A comprehensive understanding of policymakers' and stakeholders' perspectives on challenges and potential solutions for enhancing equitable access to ATs is critical to support development of evidence- and values-informed policies.Understanding challenges and solutions identified by diverse policymakers and stakeholders can lead to national and local opportunities for policy action and support work to create a national vision for enhancing equitable access to AT.

Knowledge translation strategies for dissemination with a focus on healthcare recipients: an overview of systematic reviews

BACKGROUND

While there is an ample literature on the evaluation of knowledge translation interventions aimed at healthcare providers, managers, and policy-makers, there has been less focus on patients and their informal caregivers. Further, no overview of the literature on dissemination strategies aimed at healthcare users and their caregivers has been conducted. The overview has two specific research questions: (1) to determine the most effective strategies that have been used to disseminate knowledge to healthcare recipients, and (2) to determine the barriers (and facilitators) to dissemination of knowledge to this group.

METHODS

This overview used systematic review methods and was conducted according to a pre-defined protocol. A comprehensive search of ten databases and five websites was conducted. Both published and unpublished reviews in English, Spanish, or Portuguese were included. A methodological quality assessment was conducted; low-quality reviews were excluded. A narrative synthesis was undertaken, informed by a matrix of strategy by outcome measure. The Health System Evidence taxonomy for "consumer targeted strategies" was used to separate strategies into one of six categories.

RESULTS

We identified 44 systematic reviews that describe the effective strategies to disseminate health knowledge to the public, patients, and caregivers. Some of these reviews also describe the most important barriers to the uptake of these effective strategies. When analyzing those strategies with the greatest potential to achieve behavioral changes, the majority of strategies with sufficient evidence of effectiveness were combined, frequent, and/or intense over time. Further, strategies focused on the patient, with tailored interventions, and those that seek to acquire skills and competencies were more effective in achieving these changes. In relation to barriers and facilitators, while the lack of health literacy or e-literacy could increase inequities, the benefits of social media were also emphasized, for example by widening access to health information for ethnic minorities and lower socioeconomic groups.

CONCLUSIONS

Those interventions that have been shown to be effective in improving knowledge uptake or health behaviors should be implemented in practice, programs, and policies-if not already implemented. When implementing strategies, decision-makers should consider the barriers and facilitators identified by this overview to ensure maximum effectiveness.

PROTOCOL REGISTRATION

PROSPERO: CRD42018093245.

Supporting the use of research evidence in decision-making in crisis zones in low- and middle-income countries: a critical interpretive synthesis

BACKGROUND

Decision-makers in crisis zones are faced with the challenge of having to make health-related decisions under limited time and resource constraints and in light of the many factors that can influence their decisions, of which research evidence is just one. To address a key gap in the research literature about how best to support the use of research evidence in such situations, we conducted a critical interpretive synthesis approach to develop a conceptual framework that outlines the strategies that leverage the facilitators and address the barriers to evidence use in crisis zones.

METHODS

We systematically reviewed both empirical and non-empirical literature and used an interpretive analytic approach to synthesise the results and develop the conceptual framework. We used a 'compass' question to create a detailed search strategy and conducted electronic searches in CINAHL, EMBASE, MEDLINE, SSCI and Web of Science. A second reviewer was assigned to a representative sample of articles. We purposively sampled additional papers to fill in conceptual gaps.

RESULTS

We identified 21 eligible papers to be analysed and purposively sampled an additional 6 to fill conceptual gaps. The synthesis resulted in a conceptual framework that focuses on evidence use in crisis zones examined through the lens of four systems - political, health, international humanitarian aid and health research. Within each of the four systems, the framework identifies the most actionable strategies that leverage the facilitators and address the barriers to evidence use.

CONCLUSIONS

This study presents a new conceptual framework that outlines strategies that leverage the facilitators and address the barriers to evidence use in crisis zones within different systems. This study expands on the literature pertaining to evidence-informed decision-making.

Assessment of publication bias and outcome reporting bias in systematic reviews of health services and delivery research: A meta-epidemiological study

Strategies to identify and mitigate publication bias and outcome reporting bias are frequently adopted in systematic reviews of clinical interventions but it is not clear how often these are applied in systematic reviews relating to quantitative health services and delivery research (HSDR). We examined whether these biases are mentioned and/or otherwise assessed in HSDR systematic reviews, and evaluated associating factors to inform future practice. We randomly selected 200 quantitative HSDR systematic reviews published in the English language from 2007-2017 from the Health Systems Evidence database (www.healthsystemsevidence.org). We extracted data on factors that may influence whether or not authors mention and/or assess publication bias or outcome reporting bias. We found that 43% (n = 85) of the reviews mentioned publication bias and 10% (n = 19) formally assessed it. Outcome reporting bias was mentioned and assessed in 17% (n = 34) of all the systematic reviews. Insufficient number of studies, heterogeneity and lack of pre-registered protocols were the most commonly reported impediments to assessing the biases. In multivariable logistic regression models, both mentioning and formal assessment of publication bias were associated with: inclusion of a meta-analysis; being a review of intervention rather than association studies; higher journal impact factor, and; reporting the use of systematic review guidelines. Assessment of outcome reporting bias was associated with: being an intervention review; authors reporting the use of Grading of Recommendations, Assessment, Development and Evaluations (GRADE), and; inclusion of only controlled trials. Publication bias and outcome reporting bias are infrequently assessed in HSDR systematic reviews. This may reflect the inherent heterogeneity of HSDR evidence and different methodological approaches to synthesising the evidence, lack of awareness of such biases, limits of current tools and lack of pre-registered study protocols for assessing such biases. Strategies to help raise awareness of the biases, and methods to minimise their occurrence and mitigate their impacts on HSDR systematic reviews, are needed.

Stakeholders' experiences with the evidence aid website to support 'real-time' use of research evidence to inform decision-making in crisis zones: a user testing study

Background

Humanitarian action in crisis zones is fraught with many challenges, including lack of timely and accessible research evidence to inform decision-making about humanitarian interventions. Evidence websites have the potential to address this challenge. Evidence Aid is the only evidence website designed for crisis zones that focuses on providing research evidence in the form of systematic reviews. The objective of this study is to explore stakeholders’ views of Evidence Aid, contributing further to our understanding of the use of research evidence in decision-making in crisis zones.

Methods

We designed a qualitative user-testing study to collect interview data from stakeholders about their impressions of Evidence Aid. Eligible stakeholders included those with and without previous experience of Evidence Aid. All participants were either currently working or have worked within the last year in a crisis zone. Participants were asked to perform the same user experience-related tasks and answer questions about this experience and their knowledge needs. Data were analysed using a deductive framework analysis approach drawing on Morville’s seven facets of the user experience — findability, usability, usefulness, desirability, accessibility, credibility and value.

Results

A total of 31 interviews were completed with senior decision-makers (n = 8), advisors (n = 7), field managers (n = 7), analysts/researchers (n = 5) and healthcare providers (n = 4). Participant self-reported knowledge needs varied depending on their role. Overall, participants did not identify any ‘major’ problems (highest order) and identified only two ‘big’ problems (second highest order) with using the Evidence Aid website, namely the lack of a search engine on the home page and that some full-text articles linked to/from the site require a payment. Participants identified seven specific suggestions about how to improve Evidence Aid, many of which can also be applied to other evidence websites.

Conclusions

Stakeholders in crisis zones found Evidence Aid to be useful, accessible and credible. However, they experienced some problems with the lack of a search engine on the home page and the requirement for payment for some full-text articles linked to/from the site.

Fostering access to and use of contextualised knowledge to support health policy-making: lessons from the Policy Information Platform in Nigeria

Background

Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Creating a one-stop shop for evidence is an approach that can facilitate timely access to the best evidence to inform policy decisions. We report outcomes after implementation of the Policy Information Platform (PIP), a pilot one-stop evidence repository in Nigeria designed to alleviate barriers to accessing policy-relevant knowledge.

Methods

This cross-sectional study involved five phases, namely (1) consultation with Nigerian policy-makers to identify priority policy issues, areas of health policy information needs, and challenges and capacity constraints in accessing evidence for policy-making; (2) a stakeholder engagement workshop to formally launch the PIP; (3) extraction of data and other information from scientific articles, policy briefs, evaluation reports, grey literature and health policy documents relevant to policy-making in Nigeria (identified by Google and PubMed searches and by examination of websites of relevant Nigerian government ministries, agencies and parastatals), for use in developing the PIP website; (4) promotion of the PIP in national and state health policy meetings; and (5) evaluation of the PIP using a stakeholder survey questionnaire distributed via email and critical appraisal of the grey literature included in the PIP using the authority, accuracy, coverage, objectivity, date and significance (AACODS) checklist.

Results

Priority policy areas identified by policy-makers were disease control and prevention, population health issues and health administration. Challenges identified by policy-makers were a lack of adequate capacity to access policy-relevant evidence and transform the evidence into policy. Policy-makers suggested using systematic reviews, policy briefs and rapid response mechanisms and involving policy-makers in research as ways of increasing evidence uptake for policy. A total of 126 policy-relevant, peer-reviewed scientific articles, 85 health policy documents and 201 policy-relevant grey literature documents were selected for inclusion in the PIP. Of the 195 individuals contacted via email to evaluate the PIP, 31 (15.9%) provided a response. Respondents noted that the PIP facilitated access to information based on local evidence and context-sensitive data. Barriers identified included lack of knowledge about the PIP and limited capacity of end-users to use the data compiled in the platform.

Conclusion

An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ access to evidence for use in policy-making and practice.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0431-4) contains supplementary material, which is available to authorized users.

Are we entering a new era for qualitative research? Using qualitative evidence to support guidance and guideline development by the World Health Organization

Qualitative approaches are one of several methodologies utilised within the social sciences. New developments within qualitative methods are widening the opportunities for using qualitative evidence to inform health policy and systems decisions. In this commentary, we discuss how, in our work with the World Health Organization (WHO), we have explored ways of broadening the types of evidence used to develop evidence-informed guidance for health systems.Health systems decisions are commonly informed by evidence on the effectiveness of health system interventions. However, decision makers and other stakeholders also typically have additional questions, including how different stakeholders value different outcomes, the acceptability and feasibility of different interventions and the impacts of these interventions on equity and human rights. Evidence from qualitative research can help address these questions, and a number of WHO guidelines are now using qualitative evidence in this way. This growing use of qualitative evidence to inform decision making has been facilitated by recent methodological developments, including robust methods for qualitative evidence syntheses and approaches for assessing how much confidence to place in findings from such syntheses. For research evidence to contribute optimally to improving and sustaining the performance of health systems, it needs to be transferred easily between different elements of what has been termed the 'evidence ecosystem'. This ecosystem includes primary and secondary evidence producers, guidance developers and those implementing and evaluating interventions to strengthen health systems. We argue that most of the elements of an ecosystem for qualitative evidence are now in place - an important milestone that suggests that we are entering a new era for qualitative research. However, a number of challenges and constraints remain. These include how to build stronger links between the communities involved in the different parts of the qualitative evidence ecosystem and the need to strengthen capacity, particularly in low and middle income countries, to produce and utilise qualitative evidence and decision products informed by such evidence. We invite others who want to support the wider use of qualitative evidence in decision processes to look for opportunities in their settings to put this into practice.

The reporting of funding in health policy and systems research: a cross-sectional study

BACKGROUND

Major research-reporting statements, such as PRISMA and CONSORT, require authors to provide information about funding. The objectives of this study were (1) to assess the reporting of funding in health policy and systems research (HPSR) papers and (2) to assess the funding reporting policies of journals publishing on HPSR.

METHODS

We conducted two cross-sectional surveys for papers published in 2016 addressing HPSR (both primary studies and systematic reviews) and for journals publishing on HPSR (both journals under the 'Health Policy and Services' (HPS) category in the Web of Science, and non-HPS journals that published on HPSR). Teams of two reviewers selected studies and abstracted data in duplicate and independently. We conducted descriptive analyses and a regression analysis to investigate the association between reporting of funding by papers and the journal's characteristics.

RESULTS

We included 400 studies (200 systematic reviews and 200 primary studies) that were published in 198 journals. Approximately one-third (31%) of HPSR papers did not report on funding. Of those that did, only 11% reported on the role of funders (15% of systematic reviews and 7% of primary studies). Of the 198 journals publishing on HPSR, 89% required reporting of the source of funding. Of those that did, about one-third (34%) required reporting of the role of funders. Journals classified under the HPS category (n = 72) were less likely than non-HPS journals that published HPSR studies (n = 142) to require information on the role of funders (15% vs. 32%). We did not find any of the journals' characteristics to be associated with the reporting of funding by papers.

CONCLUSIONS

Despite the majority of journals publishing on HPSR requiring the reporting of funding, approximately one-third of HPSR papers did not report on the funding source. Moreover, few journals publishing on HPSR required the reporting of the role of funders, and few HPSR papers reported on that role.

Applying a logical theory of change for strengthening research uptake in policy: a case study of the Evidence Informed Decision Making Network of the Caribbean

OBJECTIVE

Health policymakers in the Caribbean face challenges with research use in decision-making. Although copious approaches to strengthen evidence-informed policy can be found in the literature, these strategies should be applied and evaluated in specific settings. We developed a theory of change for strengthening research uptake in health policy, and the interventions were implemented as the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe). We assessed the model's logic and evaluated whether the expected outcome was achieved.

METHODS

The model was mapped in three stages: problem identification; goal determination; and backward linking of interventions. Beneficiaries were surveyed to assess the design logic and to evaluate the main outcome.

RESULTS

A total of 137 respondents completed evaluation questionnaires. The inclusion of evidence briefs, stakeholder dialogues, a research database, and training programs for policymakers in the model was validated. Respondents also reported their intention to act on research evidence to which they were exposed. After respondents had participated in stakeholder dialogues, the mean intention-to-use score was 6.4 on a scale of 1 (strongly disagree) to 7 (strongly agree), and 6.3 on the same scale, after exposure to training.

CONCLUSIONS

This work provides initial validation of EvIDeNCe as a consolidated strategy to strengthen the application of research in policy in the Caribbean. To our knowledge, it is the first study to develop and apply a comprehensive model of this type to the Caribbean. The findings support results from similar initiatives in other countries, but additional work is needed to evaluate the overall impact of the initiative.

Reporting of Financial and Non-financial Conflicts of Interest in Systematic Reviews on Health Policy and Systems Research: A Cross Sectional Survey

Background: Systematic reviews are increasingly used to inform health policy-making. The conflicts of interest (COI) of the authors of systematic reviews may bias their results and influence their conclusions. This may in turn lead to misguided public policies and systems level decisions. In order to mitigate the adverse impact of COI, scientific journals require authors to disclose their COIs. The objective of this study was to assess the frequency and different types of COI that authors of systematic reviews on health policy and systems research (HSPR) report.

Methods: We conducted a cross sectional survey. We searched the Health Systems Evidence (HSE) database of McMaster Health Forum for systematic reviews published in 2015. We extracted information regarding the characteristics of the systematic reviews and the associated COI disclosures. We conducted descriptive analyses.

Results: Eighty percent of systematic reviews included authors’ COI disclosures. Of the 160 systematic reviews that included COI disclosures, 15% had at least one author reporting at least one type of COI. The two most frequently reported types of COI were individual financial COI and individual scholarly COI (11% and 4% respectively). Institutional COIs were less commonly reported than individual COIs (3% and 15% respectively) and non-financial COIs were less commonly reported than financial COIs (6% and 14% respectively). Only one systematic review reported the COI disclosure by editors, and none reported disclosure by peer reviewers. All COI disclosures were in the form of a narrative statement in the main document and none in an online document.

Conclusion: A fifth of systematic reviews in HPSR do not include a COI disclosure statement, highlighting the need for journals to strengthen and/or better implement their COI disclosure policies. While only 15% of identified disclosure statements report any COI, it is not clear whether this indicates a low frequency of COI versus an underreporting of COI, or both.

Addressing overuse of health services in health systems: a critical interpretive synthesis

Background

Health systems are increasingly focusing on the issue of ‘overuse’ of health services and how to address it. We developed a framework focused on (1) the rationale and context for health systems prioritising addressing overuse, (2) elements of a comprehensive process and approach to reduce overuse and (3) implementation considerations for addressing overuse.

Methods

We conducted a critical interpretive synthesis informed by a stakeholder-engagement process. The synthesis identified relevant empirical and non-empirical articles about system-level overuse. Two reviewers independently screened records, assessed for inclusion and conceptually mapped included articles. From these, we selected a purposive sample, created structured summaries of key findings and thematically synthesised the results.

Results

Our search identified 3545 references, from which we included 251. Most articles (76%; n = 192) were published within 5 years of conducting the review and addressed processes for addressing overuse (63%; n = 158) or political and health system context (60%; n = 151). Besides negative outcomes at the patient, system and global level, there were various contextual factors to addressing service overuse that seem to be key issue drivers. Processes for addressing overuse can be grouped into three elements comprising a comprehensive approach, including (1) approaches to identify overused health services, (2) stakeholder- or patient-led approaches and (3) government-led initiatives. Key implementation considerations include the need to develop ‘buy in’ from stakeholders and citizens.

Conclusions

Health systems want to ensure the use of high-value services to keep citizens healthy and avoid harm. Our synthesis can be used by policy-makers, stakeholders and researchers to understand how the issue has been prioritised, what approaches have been used to address it and implementation considerations.

Systematic review registration

PROSPERO CRD42014013204.

Development of measurable indicators to enhance public health evidence-informed policy-making

BACKGROUND

Ensuring health policies are informed by evidence still remains a challenge despite efforts devoted to this aim. Several tools and approaches aimed at fostering evidence-informed policy-making (EIPM) have been developed, yet there is a lack of availability of indicators specifically devoted to assess and support EIPM. The present study aims to overcome this by building a set of measurable indicators for EIPM intended to infer if and to what extent health-related policies are, or are expected to be, evidence-informed for the purposes of policy planning as well as formative and summative evaluations.

METHODS

The indicators for EIPM were developed and validated at international level by means of a two-round internet-based Delphi study conducted within the European project 'REsearch into POlicy to enhance Physical Activity' (REPOPA). A total of 82 researchers and policy-makers from the six European countries (Denmark, Finland, Italy, the Netherlands, Romania, the United Kingdom) involved in the project and international organisations were asked to evaluate the relevance and feasibility of an initial set of 23 indicators developed by REPOPA researchers on the basis of literature and knowledge gathered from the previous phases of the project, and to propose new indicators.

RESULTS

The first Delphi round led to the validation of 14 initial indicators and to the development of 8 additional indicators based on panellists' suggestions; the second round led to the validation of a further 11 indicators, including 6 proposed by panellists, and to the rejection of 6 indicators. A total of 25 indicators were validated, covering EIPM issues related to human resources, documentation, participation and monitoring, and stressing different levels of knowledge exchange and involvement of researchers and other stakeholders in policy development and evaluation.

CONCLUSION

The study overcame the lack of availability of indicators to assess if and to what extent policies are realised in an evidence-informed manner thanks to the active contribution of researchers and policy-makers. These indicators are intended to become a shared resource usable by policy-makers, researchers and other stakeholders, with a crucial impact on fostering the development of policies informed by evidence.

A comparative evaluation of PDQ-Evidence

Background

A strategy for minimising the time and obstacles to accessing systematic reviews of health system evidence is to collect them in a freely available database and make them easy to find through a simple ‘Google-style’ search interface. PDQ-Evidence was developed in this way. The objective of this study was to compare PDQ-Evidence to six other databases, namely Cochrane Library, EVIPNet VHL, Google Scholar, Health Systems Evidence, PubMed and Trip.

Methods

We recruited healthcare policy-makers, managers and health researchers in low-, middle- and high-income countries. Participants selected one of six pre-determined questions. They searched for a systematic review that addressed the chosen question and one question of their own in PDQ-Evidence and in two of the other six databases which they would normally have searched. We randomly allocated participants to search PDQ-Evidence first or to search the two other databases first. The primary outcomes were whether a systematic review was found and the time taken to find it. Secondary outcomes were perceived ease of use and perceived time spent searching. We asked open-ended questions about PDQ-Evidence, including likes, dislikes, challenges and suggestions for improvements.

Results

A total of 89 people from 21 countries completed the study; 83 were included in the primary analyses and 6 were excluded because of data errors that could not be corrected. Most participants chose PubMed and Cochrane Library as the other two databases. Participants were more likely to find a systematic review using PDQ-Evidence than using Cochrane Library or PubMed for the pre-defined questions. For their own questions, this difference was not found. Overall, it took slightly less time to find a systematic review using PDQ-Evidence. Participants perceived that it took less time, and most participants perceived PDQ-Evidence to be slightly easier to use than the two other databases. However, there were conflicting views about the design of PDQ-Evidence.

Conclusions

PDQ-Evidence is at least as efficient as other databases for finding health system evidence. However, using PDQ-Evidence is not intuitive for some people.

Trial registration

The trial was prospectively registered in the ISRCTN registry 17 April 2015. Registration number: ISRCTN12742235.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0299-8) contains supplementary material, which is available to authorized users.

Lost in Translation: Piloting a Novel Framework to Assess the Challenges in Translating Scientific Uncertainty From Empirical Findings to WHO Policy Statements

BACKGROUND

Calls for evidence-informed public health policy, with implicit promises of greater program effectiveness, have intensified recently. The methods to produce such policies are not self-evident, requiring a conciliation of values and norms between policy-makers and evidence producers. In particular, the translation of uncertainty from empirical research findings, particularly issues of statistical variability and generalizability, is a persistent challenge because of the incremental nature of research and the iterative cycle of advancing knowledge and implementation. This paper aims to assess how the concept of uncertainty is considered and acknowledged in World Health Organization (WHO) policy recommendations and guidelines.

METHODS

We selected four WHO policy statements published between 2008-2013 regarding maternal and child nutrient supplementation, infant feeding, heat action plans, and malaria control to represent topics with a spectrum of available evidence bases. Each of these four statements was analyzed using a novel framework to assess the treatment of statistical variability and generalizability.

RESULTS

WHO currently provides substantial guidance on addressing statistical variability through GRADE (Grading of Recommendations Assessment, Development, and Evaluation) ratings for precision and consistency in their guideline documents. Accordingly, our analysis showed that policy-informing questions were addressed by systematic reviews and representations of statistical variability (eg, with numeric confidence intervals). In contrast, the presentation of contextual or "background" evidence regarding etiology or disease burden showed little consideration for this variability. Moreover, generalizability or "indirectness" was uniformly neglected, with little explicit consideration of study settings or subgroups.

CONCLUSION

In this paper, we found that non-uniform treatment of statistical variability and generalizability factors that may contribute to uncertainty regarding recommendations were neglected, including the state of evidence informing background questions (prevalence, mechanisms, or burden or distributions of health problems) and little assessment of generalizability, alternate interventions, and additional outcomes not captured by systematic review. These other factors often form a basis for providing policy recommendations, particularly in the absence of a strong evidence base for intervention effects. Consequently, they should also be subject to stringent and systematic evaluation criteria. We suggest that more effort is needed to systematically acknowledge (1) when evidence is missing, conflicting, or equivocal, (2) what normative considerations were also employed, and (3) how additional evidence may be accrued.

Delivery arrangements for health systems in low-income countries: an overview of systematic reviews

BACKGROUND

Delivery arrangements include changes in who receives care and when, who provides care, the working conditions of those who provide care, coordination of care amongst different providers, where care is provided, the use of information and communication technology to deliver care, and quality and safety systems. How services are delivered can have impacts on the effectiveness, efficiency and equity of health systems. This broad overview of the findings of systematic reviews can help policymakers and other stakeholders identify strategies for addressing problems and improve the delivery of services.

OBJECTIVES

To provide an overview of the available evidence from up-to-date systematic reviews about the effects of delivery arrangements for health systems in low-income countries. Secondary objectives include identifying needs and priorities for future evaluations and systematic reviews on delivery arrangements and informing refinements of the framework for delivery arrangements outlined in the review.

METHODS

We searched Health Systems Evidence in November 2010 and PDQ-Evidence up to 17 December 2016 for systematic reviews. We did not apply any date, language or publication status limitations in the searches. We included well-conducted systematic reviews of studies that assessed the effects of delivery arrangements on patient outcomes (health and health behaviours), the quality or utilisation of healthcare services, resource use, healthcare provider outcomes (such as sick leave), or social outcomes (such as poverty or employment) and that were published after April 2005. We excluded reviews with limitations important enough to compromise the reliability of the findings. Two overview authors independently screened reviews, extracted data, and assessed the certainty of evidence using GRADE. We prepared SUPPORT Summaries for eligible reviews, including key messages, 'Summary of findings' tables (using GRADE to assess the certainty of the evidence), and assessments of the relevance of findings to low-income countries.

MAIN RESULTS

We identified 7272 systematic reviews and included 51 of them in this overview. We judged 6 of the 51 reviews to have important methodological limitations and the other 45 to have only minor limitations. We grouped delivery arrangements into eight categories. Some reviews provided more than one comparison and were in more than one category. Across these categories, the following intervention were effective; that is, they have desirable effects on at least one outcome with moderate- or high-certainty evidence and no moderate- or high-certainty evidence of undesirable effects. Who receives care and when: queuing strategies and antenatal care to groups of mothers. Who provides care: lay health workers for caring for people with hypertension, lay health workers to deliver care for mothers and children or infectious diseases, lay health workers to deliver community-based neonatal care packages, midlevel health professionals for abortion care, social support to pregnant women at risk, midwife-led care for childbearing women, non-specialist providers in mental health and neurology, and physician-nurse substitution. Coordination of care: hospital clinical pathways, case management for people living with HIV and AIDS, interactive communication between primary care doctors and specialists, hospital discharge planning, adding a service to an existing service and integrating delivery models, referral from primary to secondary care, physician-led versus nurse-led triage in emergency departments, and team midwifery. Where care is provided: high-volume institutions, home-based care (with or without multidisciplinary team) for people living with HIV and AIDS, home-based management of malaria, home care for children with acute physical conditions, community-based interventions for childhood diarrhoea and pneumonia, out-of-facility HIV and reproductive health services for youth, and decentralised HIV care. Information and communication technology: mobile phone messaging for patients with long-term illnesses, mobile phone messaging reminders for attendance at healthcare appointments, mobile phone messaging to promote adherence to antiretroviral therapy, women carrying their own case notes in pregnancy, interventions to improve childhood vaccination. Quality and safety systems: decision support with clinical information systems for people living with HIV/AIDS. Complex interventions (cutting across delivery categories and other health system arrangements): emergency obstetric referral interventions.

AUTHORS' CONCLUSIONS

A wide range of strategies have been evaluated for improving delivery arrangements in low-income countries, using sound systematic review methods in both Cochrane and non-Cochrane reviews. These reviews have assessed a range of outcomes. Most of the available evidence focuses on who provides care, where care is provided and coordination of care. For all the main categories of delivery arrangements, we identified gaps in primary research related to uncertainty about the applicability of the evidence to low-income countries, low- or very low-certainty evidence or a lack of studies.

Implementation strategies for health systems in low-income countries: an overview of systematic reviews

BACKGROUND

A key function of health systems is implementing interventions to improve health, but coverage of essential health interventions remains low in low-income countries. Implementing interventions can be challenging, particularly if it entails complex changes in clinical routines; in collaborative patterns among different healthcare providers and disciplines; in the behaviour of providers, patients or other stakeholders; or in the organisation of care. Decision-makers may use a range of strategies to implement health interventions, and these choices should be based on evidence of the strategies' effectiveness.

OBJECTIVES

To provide an overview of the available evidence from up-to-date systematic reviews about the effects of implementation strategies for health systems in low-income countries. Secondary objectives include identifying needs and priorities for future evaluations and systematic reviews on alternative implementation strategies and informing refinements of the framework for implementation strategies presented in the overview.

METHODS

We searched Health Systems Evidence in November 2010 and PDQ-Evidence up to December 2016 for systematic reviews. We did not apply any date, language or publication status limitations in the searches. We included well-conducted systematic reviews of studies that assessed the effects of implementation strategies on professional practice and patient outcomes and that were published after April 2005. We excluded reviews with limitations important enough to compromise the reliability of the review findings. Two overview authors independently screened reviews, extracted data and assessed the certainty of evidence using GRADE. We prepared SUPPORT Summaries for eligible reviews, including key messages, 'Summary of findings' tables (using GRADE to assess the certainty of the evidence) and assessments of the relevance of findings to low-income countries.

MAIN RESULTS

We identified 7272 systematic reviews and included 39 of them in this overview. An additional four reviews provided supplementary information. Of the 39 reviews, 32 had only minor limitations and 7 had important methodological limitations. Most studies in the reviews were from high-income countries. There were no studies from low-income countries in eight reviews.Implementation strategies addressed in the reviews were grouped into four categories - strategies targeting:1. healthcare organisations (e.g. strategies to change organisational culture; 1 review);2. healthcare workers by type of intervention (e.g. printed educational materials; 14 reviews);3. healthcare workers to address a specific problem (e.g. unnecessary antibiotic prescription; 9 reviews);4. healthcare recipients (e.g. medication adherence; 15 reviews).Overall, we found the following interventions to have desirable effects on at least one outcome with moderate- or high-certainty evidence and no moderate- or high-certainty evidence of undesirable effects.1.Strategies targeted at healthcare workers: educational meetings, nutrition training of health workers, educational outreach, practice facilitation, local opinion leaders, audit and feedback, and tailored interventions.2.Strategies targeted at healthcare workers for specific types of problems: training healthcare workers to be more patient-centred in clinical consultations, use of birth kits, strategies such as clinician education and patient education to reduce antibiotic prescribing in ambulatory care settings, and in-service neonatal emergency care training.3. Strategies targeted at healthcare recipients: mass media interventions to increase uptake of HIV testing; intensive self-management and adherence, intensive disease management programmes to improve health literacy; behavioural interventions and mobile phone text messages for adherence to antiretroviral therapy; a one time incentive to start or continue tuberculosis prophylaxis; default reminders for patients being treated for active tuberculosis; use of sectioned polythene bags for adherence to malaria medication; community-based health education, and reminders and recall strategies to increase vaccination uptake; interventions to increase uptake of cervical screening (invitations, education, counselling, access to health promotion nurse and intensive recruitment); health insurance information and application support.

AUTHORS' CONCLUSIONS

Reliable systematic reviews have evaluated a wide range of strategies for implementing evidence-based interventions in low-income countries. Most of the available evidence is focused on strategies targeted at healthcare workers and healthcare recipients and relates to process-based outcomes. Evidence of the effects of strategies targeting healthcare organisations is scarce.

Governance arrangements for health systems in low-income countries: an overview of systematic reviews

BACKGROUND

Governance arrangements include changes in rules or processes that determine authority and accountability for health policies, organisations, commercial products and health professionals, as well as the involvement of stakeholders in decision-making. Changes in governance arrangements can affect health and related goals in numerous ways, generally through changes in authority, accountability, openness, participation and coherence. A broad overview of the findings of systematic reviews can help policymakers, their technical support staff and other stakeholders to identify strategies for addressing problems and improving the governance of their health systems.

OBJECTIVES

To provide an overview of the available evidence from up-to-date systematic reviews about the effects of governance arrangements for health systems in low-income countries. Secondary objectives include identifying needs and priorities for future evaluations and systematic reviews on governance arrangements and informing refinements of the framework for governance arrangements outlined in the overview.

METHODS

We searched Health Systems Evidence in November 2010 and PDQ Evidence up to 17 December 2016 for systematic reviews. We did not apply any date, language or publication status limitations in the searches. We included well-conducted systematic reviews of studies that assessed the effects of governance arrangements on patient outcomes (health and health behaviours), the quality or utilisation of healthcare services, resource use (health expenditures, healthcare provider costs, out-of-pocket payments, cost-effectiveness), healthcare provider outcomes (such as sick leave), or social outcomes (such as poverty, employment) and that were published after April 2005. We excluded reviews with limitations that were important enough to compromise the reliability of the findings of the review. Two overview authors independently screened reviews, extracted data and assessed the certainty of evidence using GRADE. We prepared SUPPORT Summaries for eligible reviews, including key messages, 'Summary of findings' tables (using GRADE to assess the certainty of the evidence) and assessments of the relevance of findings to low-income countries.

MAIN RESULTS

We identified 7272 systematic reviews and included 21 of them in this overview (19 primary reviews and 2 supplementary reviews). We focus here on the results of the 19 primary reviews, one of which had important methodological limitations. The other 18 were reliable (with only minor limitations).We grouped the governance arrangements addressed in the reviews into five categories: authority and accountability for health policies (three reviews); authority and accountability for organisations (two reviews); authority and accountability for commercial products (three reviews); authority and accountability for health professionals (seven reviews); and stakeholder involvement (four reviews).Overall, we found desirable effects for the following interventions on at least one outcome, with moderate- or high-certainty evidence and no moderate- or high-certainty evidence of undesirable effects. Decision-making about what is covered by health insurance- Placing restrictions on the medicines reimbursed by health insurance systems probably decreases the use of and spending on these medicines (moderate-certainty evidence). Stakeholder participation in policy and organisational decisions- Participatory learning and action groups for women probably improve newborn survival (moderate-certainty evidence).- Consumer involvement in preparing patient information probably improves the quality of the information and patient knowledge (moderate-certainty evidence). Disclosing performance information to patients and the public- Disclosing performance data on hospital quality to the public probably encourages hospitals to implement quality improvement activities (moderate-certainty evidence).- Disclosing performance data on individual healthcare providers to the public probably leads people to select providers that have better quality ratings (moderate-certainty evidence).

AUTHORS' CONCLUSIONS

Investigators have evaluated a wide range of governance arrangements that are relevant for low-income countries using sound systematic review methods. These strategies have been targeted at different levels in health systems, and studies have assessed a range of outcomes. Moderate-certainty evidence shows desirable effects (with no undesirable effects) for some interventions. However, there are important gaps in the availability of systematic reviews and primary studies for the all of the main categories of governance arrangements.

Financial arrangements for health systems in low-income countries: an overview of systematic reviews

BACKGROUND

One target of the Sustainable Development Goals is to achieve "universal health coverage, including financial risk protection, access to quality essential health-care services and access to safe, effective, quality and affordable essential medicines and vaccines for all". A fundamental concern of governments in striving for this goal is how to finance such a health system. This concern is very relevant for low-income countries.

OBJECTIVES

To provide an overview of the evidence from up-to-date systematic reviews about the effects of financial arrangements for health systems in low-income countries. Secondary objectives include identifying needs and priorities for future evaluations and systematic reviews on financial arrangements, and informing refinements in the framework for financial arrangements presented in the overview.

METHODS

We searched Health Systems Evidence in November 2010 and PDQ-Evidence up to 17 December 2016 for systematic reviews. We did not apply any date, language, or publication status limitations in the searches. We included well-conducted systematic reviews of studies that assessed the effects of financial arrangements on patient outcomes (health and health behaviours), the quality or utilisation of healthcare services, resource use, healthcare provider outcomes (such as sick leave), or social outcomes (such as poverty, employment, or financial burden of patients, e.g. out-of-pocket payment, catastrophic disease expenditure) and that were published after April 2005. We excluded reviews with limitations important enough to compromise the reliability of the findings. Two overview authors independently screened reviews, extracted data, and assessed the certainty of evidence using GRADE. We prepared SUPPORT Summaries for eligible reviews, including key messages, 'Summary of findings' tables (using GRADE to assess the certainty of the evidence), and assessments of the relevance of findings to low-income countries.

MAIN RESULTS

We identified 7272 reviews and included 15 in this overview, on: collection of funds (2 reviews), insurance schemes (1 review), purchasing of services (1 review), recipient incentives (6 reviews), and provider incentives (5 reviews). The reviews were published between 2008 and 2015; focused on 13 subcategories; and reported results from 276 studies: 115 (42%) randomised trials, 11 (4%) non-randomised trials, 23 (8%) controlled before-after studies, 51 (19%) interrupted time series, 9 (3%) repeated measures, and 67 (24%) other non-randomised studies. Forty-three per cent (119/276) of the studies included in the reviews took place in low- and middle-income countries. Collection of funds: the effects of changes in user fees on utilisation and equity are uncertain (very low-certainty evidence). It is also uncertain whether aid delivered under the Paris Principles (ownership, alignment, harmonisation, managing for results, and mutual accountability) improves health outcomes compared to aid delivered without conforming to those principles (very low-certainty evidence). Insurance schemes: community-based health insurance may increase service utilisation (low-certainty evidence), but the effects on health outcomes are uncertain (very low-certainty evidence). It is uncertain whether social health insurance improves utilisation of health services or health outcomes (very low-certainty evidence). Purchasing of services: it is uncertain whether increasing salaries of public sector healthcare workers improves the quantity or quality of their work (very low-certainty evidence). Recipient incentives: recipient incentives may improve adherence to long-term treatments (low-certainty evidence), but it is uncertain whether they improve patient outcomes. One-time recipient incentives probably improve patient return for start or continuation of treatment (moderate-certainty evidence) and may improve return for tuberculosis test readings (low-certainty evidence). However, incentives may not improve completion of tuberculosis prophylaxis, and it is uncertain whether they improve completion of treatment for active tuberculosis. Conditional cash transfer programmes probably lead to an increase in service utilisation (moderate-certainty evidence), but their effects on health outcomes are uncertain. Vouchers may improve health service utilisation (low-certainty evidence), but the effects on health outcomes are uncertain (very low-certainty evidence). Introducing a restrictive cap may decrease use of medicines for symptomatic conditions and overall use of medicines, may decrease insurers' expenditures on medicines (low-certainty evidence), and has uncertain effects on emergency department use, hospitalisations, and use of outpatient care (very low-certainty evidence). Reference pricing, maximum pricing, and index pricing for drugs have mixed effects on drug expenditures by patients and insurers as well as the use of brand and generic drugs. Provider incentives: the effects of provider incentives are uncertain (very low-certainty evidence), including: the effects of provider incentives on the quality of care provided by primary care physicians or outpatient referrals from primary to secondary care, incentives for recruiting and retaining health professionals to serve in remote areas, and the effects of pay-for-performance on provider performance, the utilisation of services, patient outcomes, or resource use in low-income countries.

AUTHORS' CONCLUSIONS

Research based on sound systematic review methods has evaluated numerous financial arrangements relevant to low-income countries, targeting different levels of the health systems and assessing diverse outcomes. However, included reviews rarely reported social outcomes, resource use, equity impacts, or undesirable effects. We also identified gaps in primary research because of uncertainty about applicability of the evidence to low-income countries. Financial arrangements for which the effects are uncertain include external funding (aid), caps and co-payments, pay-for-performance, and provider incentives. Further studies evaluating the effects of these arrangements are needed in low-income countries. Systematic reviews should include all outcomes that are relevant to decision-makers and to people affected by changes in financial arrangements.

McMaster Optimal Aging Portal: an evidence-based database for geriatrics-focused health professionals

Objective

The objective of this work was to provide easy access to reliable health information based on good quality research that will help health care professionals to learn what works best for seniors to stay as healthy as possible, manage health conditions and build supportive health systems. This will help meet the demands of our aging population that clinicians provide high quality care for older adults, that public health professionals deliver disease prevention and health promotion strategies across the life span, and that policymakers address the economic and social need to create a robust health system and a healthy society for all ages.

Results

The McMaster Optimal Aging Portal’s (Portal) professional bibliographic database contains high quality scientific evidence about optimal aging specifically targeted to clinicians, public health professionals and policymakers. The database content comes from three information services: McMaster Premium LiteratUre Service (MacPLUS™), Health Evidence™ and Health Systems Evidence. The Portal is continually updated, freely accessible online, easily searchable, and provides email-based alerts when new records are added. The database is being continually assessed for value, usability and use. A number of improvements are planned, including French language translation of content, increased linkages between related records within the Portal database, and inclusion of additional types of content. While this article focuses on the professional database, the Portal also houses resources for patients, caregivers and the general public, which may also be of interest to geriatric practitioners and researchers.

The Evidence-Informed Policy Network (EVIPNet) in Chile: lessons learned from a year of coordinated efforts

Informing the health policymaking process with the best available scientific evidence has become relevant to health systems globally. Knowledge Translation Platforms (KTP), such as the World Health Organization's Evidence Informed Policy Networks (EVIPNet), are a recognized strategy for linking research to action. This report describes the experience of implementing EVIPNet in Chile, from its objectives, organizational structure, strategy, activities, and main outputs, to its evolution over the course of its first year. Lessons learned are also covered. Of the activities initiated by EVIPNet-Chile, the Rapid Response Service proved to be a good starting point for engaging policymakers. Capacity building workshops and policy dialogues with relevant stakeholders were also successful. Additionally, EVIPNet-Chile developed a model for engaging academic institutions in policymaking through a network focused on preparing evidence briefs. A number of challenges, such as changing methods for producing rapid evidence syntheses, were also identified. This KTP implementation model located in a Ministry of Health could contribute to the development of similar initiatives in other health systems.

Identifying and characterising health policy and system-relevant documents in Uganda: a scoping review to develop a framework for the development of a one-stop shop

BACKGROUND

Health policymakers in low- and middle-income countries continue to face difficulties in accessing and using research evidence for decision-making. This study aimed to identify and provide a refined categorisation of the policy documents necessary for building the content of a one-stop shop for documents relevant to health policy and systems in Uganda. The on-line resource is to facilitate timely access to well-packaged evidence for decision-making.

METHODS

We conducted a scoping review of Uganda-specific, health policy, and systems-relevant documents produced between 2000 and 2014. Our methods borrowed heavily from the 2005 Arksey and O'Malley approach for scoping reviews and involved five steps, which that include identification of the research question; identification of relevant documents; screening and selection of the documents; charting of the data; and collating, summarising and reporting results. We searched for the documents from websites of relevant government institutions, non-governmental organisations, health professional councils and associations, religious medical bureaus and research networks. We presented the review findings as numerical analyses of the volume and nature of documents and trends over time in the form of tables and charts.

RESULTS

We identified a total of 265 documents including policies, strategies, plans, guidelines, rapid response summaries, evidence briefs for policy, and dialogue reports. The top three clusters of national priority areas addressed in the documents were governance, coordination, monitoring and evaluation (28%); disease prevention, mitigation, and control (23%); and health education, promotion, environmental health and nutrition (15%). The least addressed were curative, palliative care, rehabilitative services and health infrastructure, each addressed in three documents (1%), and early childhood development in one document. The volume of documents increased over the past 15 years; however, the distribution of the different document types over time has not been uniform.

CONCLUSION

The review findings are necessary for mobilising and packaging the local policy-relevant documents in Uganda in a one-stop shop; where policymakers could easily access them to address pressing questions about the health system and interventions. The different types of available documents and the national priority areas covered provide a good basis for building and organising the content in a meaningful way for the resource.

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research

Background

Capacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers’ capacity to engage with and use research.

Methods

We used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales).

Results

Response rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal.

Conclusions

Initial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0162-8) contains supplementary material, which is available to authorized users.

User Experiences of the McMaster Optimal Aging Portal's Evidence Summaries and Blog Posts: Usability Study

BACKGROUND

Evidence summaries and blogs can support evidence-informed healthy aging, by presenting high-quality health research evidence in plain language for a nonprofessional (citizen) audience.

OBJECTIVE

Our objective was to explore citizens' perceptions about the usability of evidence summaries and blog posts on the Web-based McMaster Optimal Aging Portal.

METHODS

Twenty-two citizens (aged 50 years and older) and informal caregivers participated in a qualitative study using a think-aloud method and semistructured interviews. Eleven interviews were conducted in person, 7 over the telephone, and 4 by Skype.

RESULTS

We identified themes that fell under 4 user-experience categories: (1) desirability: personal relevance, (2) understandability: language comprehension, grasping the message, dealing with uncertainty, (3) usability: volume of information, use of numbers, and (4) usefulness: intention to use, facility for sharing.

CONCLUSIONS

Participants recognized that high-quality evidence on aging was valuable. Their intended use of the information was influenced by how much it applied to their own health circumstances or those of a loved one. Some specific formatting features that were preferred included consistent layout, content organized by subheadings, catchy titles, numerical information summarized in a table, and inclusion of a glossary.

Which health research gets used and why? An empirical analysis of 30 cases

BACKGROUND

While health research is considered essential for improving health worldwide, it remains unclear how it is best organized to contribute to health. This study examined research that was part of a Ghanaian-Dutch research program that aimed to increase the likelihood that results would be used by funding research that focused on national research priorities and was led by local researchers. The aim of this study was to map the contribution of this research to action and examine which features of research and translation processes were associated with the use of the results.

METHODS

Using Contribution Mapping, we systematically examined how 30 studies evolved and how results were used to contribute to action. We combined interviews with 113 purposively selected key informants, document analysis and triangulation to map how research and translation processes evolved and contributions to action were realized. After each case was analysed separately, a cross-case analysis was conducted to identify patterns in the association between features of research processes and the use of research.

RESULTS

The results of 20 of the 30 studies were used to contribute to action within 12 months. The priority setting and proposal selection process led to the funding of studies which were from the outset closely aligned with health sector priorities. Research was most likely to be used when it was initiated and conducted by people who were in a position to use their results in their own work. The results of 17 out of 18 of these user-initiated studies were translated into action. Other features of research that appeared to contribute to its use were involving potential key users in formulating proposals and developing recommendations.

CONCLUSIONS

Our study underlines the importance of supporting research that meets locally-expressed needs and that is led by people embedded in the contexts in which results can be used. Supporting the involvement of health sector professionals in the design, conduct and interpretation of research appears to be an especially worthwhile investment.

Health system guidance appraisal--concept evaluation and usability testing

BACKGROUND

Health system guidance (HSG) provides recommendations aimed to address health system challenges. However, there is a paucity of methods to direct, appraise, and report HSG. Earlier research identified 30 candidate criteria (concepts) that can be used to evaluate the quality of HSG and guide development and reporting requirements. The objective of this paper was to describe two studies aimed at evaluating the importance of these 30 criteria, design a draft HSG appraisal tool, and test its usability.

METHODS

This study involved a two-step survey process. In step 1, respondents rated the 30 concepts for appropriateness to, relevance to, and priority for health system decisions and HSG. This led to a draft tool. In step 2, respondents reviewed HSG documents, appraised them using the tool, and answered a series of questions. Descriptive analyses were computed.

RESULTS

Fifty participants were invited in step 1, and we had a response rate of 82 %. The mean response rates for each concept within each survey question were universally favorable. There was also an overall agreement about the need for a high-quality tool to systematically direct the development, appraisal, and reporting of HSG. Qualitative feedback and a consensus process by the team led to refinements to some of the concepts and the creation of a beta (draft) version of the HSG tool. In step 2, 35 participants were invited and we had a response rate of 74 %. Exploratory analyses showed that the quality of the HSGs reviewed varied as a function of the HSG item and the specific document assessed. A favorable consensus was reached with participants agreeing that the HSG items were easy to understand and easy to apply. Moreover, the overall agreement was high for the usability of the tool to systematically direct the development (85 %), appraisal (92 %), and reporting (81 %) of HSG. From this process, version 1.0 of the HSG appraisal tool was generated complete with 32 items (and their descriptions) and 4 domains.

CONCLUSIONS

The final tool, named the Appraisal of Guidelines for Research and Evaluation for Health Systems (AGREE-HS) (version 1), defines expectations of HSG and facilitates informed decisions among policymakers on health system delivery, financial, and governance arrangements.

Interventions for preventing, delaying the onset, or decreasing the burden of frailty: an overview of systematic reviews

BACKGROUND

Many systematic reviews have evaluated the effectiveness of interventions to prevent, delay, or decrease frailty symptoms, but no effort has been made to identify, map, and synthesize the findings from reviews across the full spectrum of interventions. Our objectives are to (1) synthesize findings from all existing systematic reviews evaluating interventions for preventing, delaying the onset, or decreasing the burden of frailty symptoms; (2) examine different conceptualizations of frailty that have been used in the development and implementation of interventions; and (3) inform policy by convening a stakeholder dialogue with Canadian health-system leaders.

METHODS/DESIGN

We will conduct an overview of systematic reviews to identify and synthesize all of the systematic reviews addressing interventions to preventing, delaying the onset, or decreasing the burden of frailty symptoms. To identify relevant systematic reviews, we will conduct database searches for published and grey literature as well as contact key experts and search reference lists of included reviews. Two reviewers will independently review all search results for inclusion and then conceptually map, extract key findings (including the conceptualization/definition of frailty used) and assess the methodological quality of all included reviews. We will then synthesize the findings by producing a 'gap map' (i.e. mapping reviews in a matrix according to the interventions and outcomes assessed), and narratively synthesize the key messages across reviews related to type of interventions.

DISCUSSION

Following the completion of the synthesis, we will use the findings to develop an evidence brief that mobilizes the best available evidence about the problem related to preventing, delaying the onset, or decreasing the burden of frailty symptoms in older adults, policy and programmatic options to address the problem and implementation considerations. The evidence brief will then be used as the input into a stakeholder dialogue, which will engage 18-22 Canadian health-system leaders (including policymakers, health providers, researchers, and other stakeholders) in 'off-the-record' deliberations to inform future actions and policymaking.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42015022082.

Supporting evidence-informed health policy making: the development and contents of an online repository of policy-relevant documents addressing healthcare renewal in Canada

OBJECTIVES

(1) To develop an online repository of policy-relevant documents, other than and complementary to those from the peer-reviewed scientific literature, addressing healthcare renewal in Canada; and (2) to describe the distribution of document contents.

METHODS

An iterative scoping review approach was undertaken. Documents were identified through website hand-searches and referrals from 19 Canadian health organizations. Descriptive frequencies were calculated, such as for document type.

FINDINGS

In July 2014, 1," documents were in the Evidence-Informed Healthcare Renewal Portal. The top three types of documents were situation analyses (n = 390, 38%), health and health system data (n = 191, 18%) and jurisdictional reviews (n = 115, 11%). The top three national priority areas addressed were health human resources (n = 778, 75%), quality as a performance indicator (n = 502, 49%) and information technology (n = 385, 37%).

CONCLUSION

The process of developing a systematic method for identifying these documents has yielded a new resource to support evidence-informed health policy making and has identified a large volume of policy-relevant documents addressing healthcare renewal priority areas in Canada.