Design and Implementation of a Performance Measurement System in Swedish Health Care Services

Measuring performance in allied health professional role substitution models of care: a clinician survey

BACKGROUND

Professional role substitution models of care have emerged as a key strategy to address increasing healthcare demand. Gaining insights from those actively engaged in the process of these models' implementation and evaluation is pivotal to ensuring sustainability and further successful implementation. The purpose of this study was to describe allied-health clinicians' perceptions, practice, and experiences of healthcare performance evaluation in professional role substitution models of care.

METHODS

Data were collected via an online platform between 22 June - 22 July 2022 using a combination of convenience and network-based sampling of allied-health clinicians involved or interested in the implementation and evaluation of professional role substitution models of care. Clinicians answered 25 questions which consisted of demographic and targeted questions regarding performance evaluation across six domains of healthcare quality (effectiveness, safety, appropriateness, access & equity, continuity of care, and cost, efficiency, productivity & sustainability).

RESULTS

A total of 102 clinicians accessed the survey, with 72 providing complete survey data. Eleven allied-health professions were represented, working across twelve specialities in thirteen hospital and health services. Whilst most allied-health clinicians (93-100%) supported measuring performance in each of the six healthcare quality domains, only 26-58% were measuring these domains in practice. Allied-health leadership support (62.5%), clinician drive (62.5%), consumer engagement (50%) and medical support (46%) were enablers whilst a lack of resources (human, time, financial (47%)), healthcare performance frameworks and/or policies (40%) were identified as barriers. Given the opportunity, clinicians would invest the most financial resources in digital solutions as a core strategy to improve performance evaluation.

CONCLUSIONS

Allied-health professionals expressed strong support for principles of performance evaluation, however in practice, performance evaluation is still in its infancy in professional role substitution models of care. Organisations can implement strategies that maximise the enablers whilst addressing barriers identified to improve performance evaluation in these models of care.

Performance evaluation: a comparative study between public and private sectors

Purpose

This study analysed international publications related to the performance evaluation (PE) theme to compare the public and private sectors with a focus on the metrics and a reflection on the consideration of each sector's characteristics.

Design/methodology/approach

In total, two bibliographic portfolios (BPs) were selected with the aid of the Knowledge Development Process-Constructivist (ProKnow-C) instrument, and the literature was systematically and critically reviewed to construct a theme literature map; the metrics analysed the performance evaluation systems (PESs) used in the studies and identified each sector's characteristics and PES behaviour based on these characteristics.

Findings

The two sectors stand out because of the low incidence of articles with cardinal scales, which enable the measurement and identification of the consequences of performance that are below or above the target. The behaviour of the sector characteristics was unusual, especially regarding the user and the organisational structure; it was found that the presence or absence of critical factors, such as communication, may prove to be a differential of success or failure. An absence of the organisation's particularities when designing and using a PES was noted.

Originality/value

The contributions relate to (1) identification of the public and private sector characteristics based on the selected literature, which helps in understanding the critical success and failure factors of an organisation's PES, especially with regard to metrics adequacy for each context; and (2) verification of the paths taken by the literature in both sectors.

Quality and legal aspects in public care procurement

Purpose

– The purpose of this paper is to assess the applicability of care quality concepts as contract award criteria for public procurement of health and social care, using the case of Sweden.

Design/methodology/approach

– Based on a literature review, European and Swedish legal texts, government regulations as well as 26 Swedish court review cases concerning care procurement have been analysed.

Findings

– Methods used for assessing care quality are seldom useful for predicting the quality to be delivered by a potential contractor. Legal principles of transparency and equal treatment of tenderers make it necessary to apply strict requirements for verification.

Research limitations/implications

– Results refer primarily to a Swedish context but could be applicable throughout the EU. Further studies of relations between award criteria and public/private collaborative practices for improving care quality during contractual periods are desirable.

Practical implications

– Local and regional procurement officials should benefit from a better understanding of how quality criteria should be designed and applied to the award procedures for care contracts. Care providers in the private sector would also be able to develop their quality strategies and present their abilities more efficiently when tendering for public contracts.

Social implications

– Issues of quality of health and social care are of obvious importance for social sustainability. Public awareness of care quality problems is evident and often a cause of media concern.

Originality/value

– This investigation pinpoints the difference between traditional care quality thinking and the legal principles underlying contract award in public procurement of care services.

Assisting allied health in performance evaluation: a systematic review

Background

Performance evaluation raises several challenges to allied health practitioners and there is no agreed approach to measuring or monitoring allied health service performance. The aim of this review was to examine the literature on performance evaluation in healthcare to assist in the establishment of a framework that can guide the measurement and evaluation of allied health clinical service performance. This review determined the core elements of a performance evaluation system, tools for evaluating performance, and barriers to the implementation of performance evaluation.

Methods

A systematic review of the literature was undertaken. Five electronic databases were used to search for relevant articles: MEDLINE, Embase, CINAHL, PsychInfo, and Academic Search Premier. Articles which focussed on any allied health performance evaluation or those which examined performance in health care in general were considered in the review. Content analysis was used to synthesise the findings from individual articles.

Results

A total of 37 articles were included in the review. The literature suggests there are core elements involved in performance evaluation which include prioritising clinical areas for measurement, setting goals, selecting performance measures, identifying sources of feedback, undertaking performance measurement, and reporting the results to relevant stakeholders. The literature describes performance evaluation as multi-dimensional, requiring information or data from more than one perspective to provide a rich assessment of performance. A range of tools or instruments are available to capture various perspectives and gather a comprehensive picture of health care quality.

Conclusions

Every allied health care delivery system has different performance needs and will therefore require different approaches. However, there are core processes that can be used as a framework to evaluate allied health performance. A careful examination of barriers to performance evaluation and subsequent tailoring of strategies to overcome these barriers should be undertaken to achieve the aims of performance evaluation. The findings of this review should inform the development of a standardised framework that can be used to measure and evaluate allied health performance. Future research should explore the utility and overall impact of such framework in allied health service delivery.

Advancing the use of performance evaluation in health care

PURPOSE

The purpose of this paper is to develop a framework for health care performance evaluation that enables decision makers to identify areas indicative of corrective actions. The framework should provide information on strategic pro-/regress in an operational context that justifies the need for organizational adjustments.

DESIGN/METHODOLOGY/APPROACH

The study adopts qualitative methods for constructing the framework, subsequently implementing the framework in a Danish magnetic resonance imaging (MRI) unit. Workshops and interviews form the basis of the qualitative construction phase, and two internal and five external databases are used for a quantitative data collection.

FINDINGS

By aggregating performance outcomes, collective measures of performance are achieved. This enables easy and intuitive identification of areas not strategically aligned. In general, the framework has proven helpful in an MRI unit, where operational decision makers have been struggling with extensive amounts of performance information.

RESEARCH LIMITATIONS/IMPLICATIONS

The implementation of the framework in a single case in a public and highly political environment restricts the generalizing potential. The authors acknowledge that there may be more suitable approaches in organizations with different settings.

PRACTICAL IMPLICATIONS

The strength of the framework lies in the identification of performance problems prior to decision making. The quality of decisions is directly related to the individual decision maker. The only function of the framework is to support these decisions.

ORIGINALITY/VALUE

The study demonstrates a more refined and transparent use of performance reporting by combining strategic weight assignment and performance aggregation in hierarchies. In this way, the framework accentuates performance as a function of strategic progress or regress, thus assisting decision makers in exerting operational effort in pursuit of strategic alignment.

Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

RATIONALE, AIMS AND OBJECTIVES

We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting.

METHOD

We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions.

RESULTS

The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients.

CONCLUSIONS

Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

More than a score: a qualitative study of ancillary benefits of performance measurement

BACKGROUND

Prior research has examined clinical effects of performance measurement systems. To the extent that non-clinical effects have been researched, the focus has been on negative unintended consequences. Yet, these same systems may also have ancillary benefits for patients and providers--that is, benefits that extend beyond improvements on clinical measures. The purpose of this study is to identify and describe potential ancillary benefits of performance measures as perceived by primary care staff and facility leaders in a large US healthcare system.

METHODS

In-person individual semistructured interviews were conducted with 59 primary care staff and facility leaders at four Veterans Health Administration facilities. Transcribed interviews were coded and organised into thematic categories.

RESULTS

Interviewed staff observed that local performance measurement implementation practices can result in increased patient knowledge and motivation. These effects on patients can lead to improved performance scores and additional ancillary benefits. Performance measurement implementation can also directly result in ancillary benefits for the patients and providers. Patients may experience greater satisfaction with care and psychosocial benefits associated with increased provider-patient communication. Ancillary benefits of performance measurement for providers include increased pride in individual or organisational performance and greater confidence that one's practice is grounded in evidence-based medicine.

CONCLUSIONS

A comprehensive understanding of the effects of performance measurement systems needs to incorporate ancillary benefits as well as effects on clinical performance scores and negative unintended consequences. Although clinical performance has been the focus of most evaluations of performance measurement to date, both patient care and provider satisfaction may improve more rapidly if all three categories of effects are considered when designing and evaluating performance measurement systems.

Using quality measures for quality improvement: the perspective of hospital staff

Research objective

This study examines the perspectives of a range of key hospital staff on the use, importance, scientific background, availability of data, feasibility of data collection, cost benefit aspects and availability of professional personnel for measurement of quality indicators among Iranian hospitals. The study aims to facilitate the use of quality indicators to improve quality of care in hospitals.

Study design

A cross-sectional study was conducted over the period 2009 to 2010. Staff at Iranian hospitals completed a self-administered questionnaire eliciting their views on organizational, clinical process, and outcome (clinical effectiveness, patient safety and patient centeredness) indicators.

Population studied

93 hospital frontline staff including hospital/nursing managers, medical doctors, nurses, and quality improvement/medical records officers in 48 general and specialized hospitals in Iran.

Principal findings

On average, only 69% of respondents reported using quality indicators in practice at their affiliated hospitals. Respondents varied significantly in their reported use of organizational, clinical process and outcome quality indicators. Overall, clinical process and effectiveness indicators were reported to be least used. The reported use of indicators corresponded with their perceived level of importance. Quality indicators were reported to be used among clinical staff significantly more than among managerial staff. In total, 74% of the respondents reported to use obligatory indicators, while this was 68% for voluntary indicators (p<0.05).

Conclusions

There is a general awareness of the importance and usability of quality indicators among hospital staff in Iran, but their use is currently mostly directed towards external accountability purposes. To increase the formative use of quality indicators, creation of a common culture and feeling of shared ownership, alongside an increased uptake of clinical process and effectiveness indicators is needed to support internal quality improvement processes at hospital level.

Implementation of JACIE accreditation results in the establishment of new indicators that unevenly monitor processes contributing to the delivery of hematopoietic SCT

More than 145 European hematopoietic SCT programs have received JACIE (Joint Accreditation Committee for ISCT Europe and EBMT) accreditation since 2000, demonstrating compliance with FACT (Foundation for the Accreditation of Cell Therapy)-JACIE international standards. The association of JACIE with improved patient outcome was recently documented. However, conditions in which quality management systems were introduced and the actual benefits remain to be fully evaluated. Our study focuses on one aspect of quality management: introduction and use of indicators. Through a questionnaire sent to JACIE-accredited centers and responses from 32 programs (or 40%), we identified 293 indicators, including 224 (76%) that were introduced during the preparatory phase of JACIE accreditation. Indicators were associated with the following processes: measurement, analysis and improvement (54/293 or 18%); donor collection (49/293 or 16%); processing and storage of cell therapy products (37/293 or 12.5%); and administration of hematopoietic progenitor cells (67/293 or 23%). Mapping revealed an uneven distribution of indicators across the different subprocesses that contribute to this highly specialized medical procedure. Moreover, we found that only 101/293 indicators (34%) complied with the rules for implementation of a quality indicator, as defined by the FDX 50-171 standard. This suggests that risks to donors/recipients are unevenly monitored, leaving critical medical steps with low levels of monitoring.

What is the value and impact of quality and safety teams? A scoping review

BACKGROUND

The purpose of this study was to conduct a scoping review of the literature about the establishment and impact of quality and safety team initiatives in acute care.

METHODS

Studies were identified through electronic searches of Medline, Embase, CINAHL, PsycINFO, ABI Inform, Cochrane databases. Grey literature and bibliographies were also searched. Qualitative or quantitative studies that occurred in acute care, describing how quality and safety teams were established or implemented, the impact of teams, or the barriers and/or facilitators of teams were included. Two reviewers independently extracted data on study design, sample, interventions, and outcomes. Quality assessment of full text articles was done independently by two reviewers. Studies were categorized according to dimensions of quality.

RESULTS

Of 6,674 articles identified, 99 were included in the study. The heterogeneity of studies and results reported precluded quantitative data analyses. Findings revealed limited information about attributes of successful and unsuccessful team initiatives, barriers and facilitators to team initiatives, unique or combined contribution of selected interventions, or how to effectively establish these teams.

CONCLUSIONS

Not unlike systematic reviews of quality improvement collaboratives, this broad review revealed that while teams reported a number of positive results, there are many methodological issues. This study is unique in utilizing traditional quality assessment and more novel methods of quality assessment and reporting of results (SQUIRE) to appraise studies. Rigorous design, evaluation, and reporting of quality and safety team initiatives are required.

Evaluating instruments for regulation of health care in the Netherlands

OBJECTIVES

Reliable and valid judgments are necessary for regulatory authorities to merit confidence from care institutions and society and preserve authority. Moreover, limited reliability and validity of regulatory judgments increase the risk of limited improvement of the quality of health care. The goal of the study is to obtain insight in (dis) advantages of different regulatory instruments for regulation of health care.

METHOD

In this study, the reliability and validity of judgments generated by a lightly structured and highly structured regulatory instrument used by the Dutch Health Care Inspectorate are compared.

RESULTS

Results indicate that the lightly structured instrument causes a large variety in discussed topics in regulatory visits: indicators pointing out potential risks in care are not always part of these discussions, by which incentives to improve care remain unjustly undone. Both types of instruments show variations in the meaning of judgments, indicating validity problems.

CONCLUSION

The results of our study suggest that regulation of health care requires thorough appraisal of instruments. Several requirements are identified: first, an instrument that justifies the complexity of care with an accompanying explicit set of standards is necessary. Second, commitment of inspectors to the instrument is essential. And third, training of inspectors is indispensable.

Implementing quality indicators in intensive care units: exploring barriers to and facilitators of behaviour change

BACKGROUND

Quality indicators are increasingly used in healthcare but there are various barriers hindering their routine use. To promote the use of quality indicators, an exploration of the barriers to and facilitating factors for their implementation among healthcare professionals and managers of intensive care units (ICUs) is advocated.

METHODS

All intensivists, ICU nurses, and managers (n = 142) working at 54 Dutch ICUs who participated in training sessions to support future implementation of quality indicators completed a questionnaire on perceived barriers and facilitators. Three types of barriers related to knowledge, attitude, and behaviour were assessed using a five-point Likert scale (1 = strongly disagree to 5 = strongly agree).

RESULTS

Behaviour-related barriers such as time constraints were most prominent (Mean Score, MS = 3.21), followed by barriers related to knowledge and attitude (MS = 3.62; MS = 4.12, respectively). Type of profession, age, and type of hospital were related to knowledge and behaviour. The facilitating factor perceived as most important by intensivists was administrative support (MS = 4.3; p = 0.02); for nurses, it was education (MS = 4.0; p = 0.01), and for managers, it was receiving feedback (MS = 4.5; p = 0.001).

CONCLUSIONS

Our results demonstrate that healthcare professionals and managers are familiar with using quality indicators to improve care, and that they have positive attitudes towards the implementation of quality indicators. Despite these facts, it is necessary to lower the barriers related to behavioural factors. In addition, as the barriers and facilitating factors differ among professions, age groups, and settings, tailored strategies are needed to implement quality indicators in daily practice.