Painting by numbers: a guide for systematically developing indicators of performance at any level of health care

Performance indicators on long-term care for older people in 43 high- and middle-income countries: literature review, web search and expert consultation

BACKGROUND

Long-term care (LTC) for older people is an area of focus for many health and social policies in high- and middle-income countries. Performance Indicators are used to provide national and subnational jurisdictions with insights to ensure quality of the provided LTC services for older people. Although LTC systems vary across jurisdictions, there is demand for internationally comparable indicators to support countries in monitoring LTC and facilitate mutual learning. The aim of this study was to provide an overview of indicators currently employed to monitor the performance of LTC systems and services in high- and middle- income countries and describe their key characteristics.

METHODS

A review of the literature in six scientific databases (literature review) and web searches of relevant sites across 43 selected countries (web search) was conducted. We asked country representatives from the Working Party on Health Care Quality and Outcomes of the Organization for Economic Cooperation and Development, where most of these countries are represented, to cross-validate the sources of information found (expert consultation). We then extracted and analysed the data from all obtained sources based on a predetermined set of characteristics.

RESULTS

The search of scientific databases yielded 12,960 records, from which forty papers were selected for inclusion. The scientific literature findings were complemented by 34 grey literature sources. In total, we identified performance indicators being used to monitor LTC systems and services across 29 national and subnational jurisdictions in 24 out of 43 countries. In total, 620 indicators were identified. All jurisdictions used indicators related to institutional LTC and 16 also used indicators on home care. The most frequently monitored structures, processes, and results were pressure ulcers, falls, use of restraints and pain management.

CONCLUSIONS

We identified LTC performance indicators currently being monitored in 29 jurisdictions across 24 countries. Many jurisdictions are monitoring similar structures, processes, and results. This presents an opportunity to develop internationally comparable LTC performance indicators based on existing efforts across countries.

Common Features of Selection Processes of Health System Performance Indicators in Primary Healthcare: A Systematic Review

BACKGROUND

Health system performance indicators are widely used to assess primary healthcare (PHC) performance. Despite the numerous tools and some convergence on indicator criteria, there is not a clear understanding of the common features of indicator selection processes. We aimed to review the literature to identify papers that document indicator selection processes for health system performance indicators in PHC.

METHODS

We searched the online databases Scopus, Medline, and CINAHL, as well as the grey literature, without time restrictions, initially on July 31, 2019 followed by an update November 13, 2020. Empirical studies or reports were included if they described the selection of health system performance indicators or frameworks, that included PHC indicators. A combination of the process focussed research question and qualitative analysis meant a quality appraisal tool or assessment of bias could not meaningfully be applied to assess individual studies. We undertook an inductive analysis based on potential indicator selection processes criteria, drawn from health system performance indicator appraisal tools reported in the literature.

RESULTS

We identified 16 503 records of which 28 were included in the review. Most studies used a descriptive case study design. We found no consistent variations between indicator selection processes of health systems of high income and low- or lower-middle income countries. Identified common features of selection processes for indicators in PHC include literature review or adaption of an existing framework as an initial step; a consensus building process with stakeholders; structuring indicators into categories; and indicator criteria focusing on validity and feasibility. The evidence around field testing with utility and consideration of reporting burden was less clear.

CONCLUSION

Our findings highlight several characteristics of health system indicator selection processes. These features provide the groundwork to better understand how to value indicator selection processes in PHC.

Selecting Performance Indicators and Targets in Health Care: An International Scoping Review and Standardized Process Framework

Objective

Health care organizations monitor hundreds of performance indicators. It is unclear what processes and criteria organizations use to identify the indicators they use, who is involved in these processes, how performance targets are set, and what the impacts of these processes are. The purpose of this study is to synthesize international approaches to indicator selection and develop a standardized process framework.

Methods

Using the PubMed and Web of Science search engines, a scoping review of peer reviewed and grey literature following PRISMA-ScR guidelines was conducted to identify documents describing indicator selection processes used by health systems. English-language papers from 11 countries published from 2010 to 2020 were included. Papers were thematically analyzed to develop a standardized process framework.

Results

The review included 33 peer-reviewed papers and 11 grey-literature documents. While there are common practices used in health care to select indicators, no single standardized process framework for indicator selection exists. Arbitrary or incomplete indicator selection processes risk over-measurement, lack of alignment with strategic and operational goals, lack of support by end-users, and paralyzed decision-making ability. By consolidating international practices, we developed the 5-P indicator selection process framework to mitigate process risks and support high-quality indicator selection processes.

Conclusion

The 5-P indicator selection process framework consists of five domains and 17 elements, and offers health care agencies a practical structure they can use to design indicator selection processes. The framework also provides researchers with a basis by which the implementation of these processes may be evaluated.

Steering by their own lights: Why regulators across Europe use different indicators to measure healthcare quality

•

Indicator sets differ in how they define, measure, and assess healthcare quality.

•

National sets shaped by varying governance traditions and healthcare system configuration.

•

Targeting of quality dimensions and hospital activities shaped by system-specific ‘demand-side’ pressures.

•

Measurement styles shaped by ‘supply-side’ constraints on data access and indicator construction.

•

International benchmarking is easier when healthcare systems and governance traditions are similar.

Despite widespread faith that quality indicators are key to healthcare improvement and regulation, surprisingly little is known about what is actually measured in different countries, nor how, nor why. To address that gap, this article compares the official indicator sets--comprising some 1100 quality measures-- used by statutory hospital regulators in England, Germany, France, and the Netherlands. The findings demonstrate that those countries’ regulators strike very different balances in: the dimensions of quality they assess (e.g. between safety, effectiveness, and patient-centredness); the hospital activities they target (e.g. between clinical and non-clinical activities and management); and the ‘Donabedian’ measurement style of their indicators (between structure, process and outcome indicators). We argue that these contrasts reflect: i) how the distinctive problems facing each country’s healthcare system create different ‘demand-side’ pressures on what national indicator sets measure; and ii) how the configuration of national healthcare systems and governance traditions create ‘supply-side’ constraints on the kinds of data that regulators can use for indicator construction. Our analysis suggests fundamental differences in the meaning of quality and its measurement across countries that are likely to impede international efforts to benchmark quality and identify best practice.

"Strangers in the ER": Quality indicators and third party interference in Dutch emergency care

RATIONALE, AIMS, AND OBJECTIVES

This paper examines a remarkable dispute between Dutch insurers, hospitals, doctors, and patients about a set of quality indicators. In 2013, private insurers planned to drastically reform Dutch emergency care using quality indicators they had formulated drawing from clinical guidelines, RCTs, and systematic reviews. Insurers' plans caused much debate in the field of emergency care. As quality indicators have come to play a more central role in health care governance, the questions what constitutes good evidence for them, how they ought to be used, and who controls them have become politically and morally charged. This paper is a case study of how a Dutch public knowledge institution, the National Health Care Institute, intervened in this dispute and how they addressed these questions.

METHOD

We conducted ethnographic research into the knowledge work of the National Health Care Institute. Research entailed document analysis, participant observation, in-depth conversations, and formal interviews with 5 key-informants.

RESULTS

The National Health Care Institute problematized not only the evidence supporting insurers' indicators, but also-and especially-the scope, purpose, and use of the indicators. Our analysis shows the institute's struggle to reconcile the technical rationality of quality indicators with their social and political implications in practice. The institute deconstructed quality indicators as national standards and, instead, promoted the use of indicators in dialogue with stakeholders and their local and contextual knowledge.

CONCLUSIONS

Even if quality indicators are based on scientific evidence, they are not axiomatically good or useful. Both proponents and critics of Evidence-based Medicine always feared uncritical use of evidence by third parties. For non-medical parties who have no access to primary care processes, the type of standardized knowledge professed by Evidence-based Medicine provides the easiest way to gain insights into "what works" in clinical practice. This case study reminds us that using standardized knowledge for the management of health care quality requires the involvement of stakeholders for the development and implementation of indicators, and for the interpretation of their results.

A methodology to design a performance management system in preventive care

Background

Preventive care has gained increasing attention in health reforms around the world due to its ability to reduce the burden of disease and to save health costs. Nevertheless, there is a gap in terms of the development of reliable systems to measure and evaluate performance of preventive care in order to support decision-making and increase service outcomes. The aim of this study is to define a methodology for designing a performance management system (PMS) in order to effectively support the planning, control and evaluation of preventive care and to identify the factors that influence such a process.

Methods

The methodology is based on the participatory action research approach, which implies collaboration between researchers and practitioners. The study was articulated in four phases and carried out in an Italian regional healthcare system that was undergoing a major reorganization process.

Results

The findings provide insights into the peculiarities that affect preventive care and highlight two categories of critical factors: general issues regarding the process and specific issues regarding preventive care. The first category includes the importance of interactions between academics, physicians and policy-makers, the impact of workloads and red tape on employee involvement and the increased conservation mechanisms during periods of institutional change. The second category concerns the strong heterogeneity of preventive activities within health organizations, the huge amount of regulations and the incompleteness of information systems.

Conclusion

The development of a PMS for preventive care can best be served by collaborative methods that involve academics, professionals and policy-makers, whose roles and responsibilities must be clearly defined, and by an improvement in transparency and communication within organizations in order to enhance the involvement of different professionals at appropriate times and in appropriate ways. Key recommendations that may improve the maintenance and use of information systems are proposed to policy-makers.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3837-8) contains supplementary material, which is available to authorized users.

Healthcare efficiency assessment using DEA analysis in the Slovak Republic

A regional disparity is becoming increasingly important growth constraint. Policy makers need quantitative knowledge to design effective and targeted policies. In this paper, the regional efficiency of healthcare facilities in Slovakia is measured (2008-2015) using data envelopment analysis (DEA). The DEA is the dominant approach to assessing the efficiency of the healthcare system but also other economic areas. In this study, the window approach is introduced as an extension to the basic DEA models to evaluate healthcare technical efficiency in individual regions and quantify the basic regional disparities and discrepancies. The window DEA method was chosen since it leads to increased discrimination on results especially when applied to small samples and it enables year-by-year comparisons of the results. Two stable inputs (number of beds, number of medical staff), three variable inputs (number of all medical equipment, number of magnetic resonance (MR) devices, number of computed tomography (CT) devices) and two stable outputs (use of beds, average nursing time) were chosen as production variable in an output-oriented 4-year window DEA model for the assessment of technical efficiency in 8 regions. The database was made available from the National Health Information Center and the Slovak Statistical Office, as well as from the online databases Slovstat and DataCube. The aim of the paper is to quantify the impact of the non-standard Data Envelopment Analysis (DEA) variables as the use of medical technologies (MR, CT) on the results of the assessment of the efficiency of the healthcare facilities and their adequacy in the evaluation of the monitored processes. The results of the analysis have shown that there is an indirect dependence between the values of the variables over time and the results of the estimated efficiency in all regions. The regions that had low values of the variables over time achieved a high degree of efficiency and vice versa. Interesting knowledge was that the gradual addition of variables number of MR, number of CT and number of medical devices together, to the input side did not have a significant impact on the overall estimated efficiency of healthcare facilities.

The effect of referral templates on out-patient quality of care in a hospital setting: a cluster randomized controlled trial

BACKGROUND

The assessment of quality of care is an integral part of modern medicine. The referral represents the handing over of care from the general practitioner to the specialist. This study aimed to assess whether an improved referral could lead to improved quality of care.

METHODS

A cluster randomized trial with the general practitioner surgery as the clustering unit was performed. Fourteen surgeries in the area surrounding the University Hospital of North Norway Harstad were randomized stratified by town versus countryside location. The intervention consisted of implementing referral templates for new referrals in four clinical areas: dyspepsia; suspected colorectal cancer; chest pain; and confirmed or suspected chronic obstructive pulmonary disease. The control group followed standard referral practice. Quality of treatment pathway as assessed by newly developed quality indicators was used as main outcome. Secondary outcomes included subjective quality assessment, positive predictive value of referral and adequacy of prioritization. Assessment of outcomes was done at the individual level. The patients, hospital doctors and outcome assessors were blinded to the intervention status.

RESULTS

A total of 500 patients were included, with 281 in the intervention and 219 in the control arm. From the multilevel regression model the effect of the intervention on the quality indicator score was insignificant at 1.80% (95% CI, -1.46 to 5.06, p = 0.280). No significant differences between the intervention and the control groups were seen in the secondary outcomes. Active use of the referral intervention was low, estimated at approximately 50%. There was also wide variation in outcome scoring between the different assessors.

CONCLUSIONS

In this study no measurable effect on quality of care or prioritization was revealed after implementation of referral templates at the general practitioner/hospital interface. The results were hindered by a limited uptake of the intervention at GP surgeries and inconsistencies in outcome assessment.

TRIAL REGISTRATION

The study was registered under registration number NCT01470963 on September 5th, 2011.