Health system performance assessment: embedding resilience through performance intelligence

Health systems around the world are facing challenges in achieving their goals. In the wake of the coronavirus disease pandemic, the need for resilient health systems has become even more apparent. This article argues that embedding resilience into health system performance assessment (HSPA) frameworks can be a valuable approach for improving health system performance. This perspective examines key challenges threatening health systems and makes a case for the continued relevance of HSPA by embedding resilience-related performance intelligence.

Acute care pathway assessed through performance indicators during the COVID-19 pandemic in OECD countries (2020-2021): a scoping review

BACKGROUND

The COVID-19 pandemic severely impacted care for non-COVID patients. Performance indicators to monitor acute care, timely reported and internationally accepted, lacked during the pandemic in OECD countries. This study aims to summarize the performance indicators available in the literature to monitor changes in the quality of acute care in OECD countries during the first year and a half of the pandemic (2020-July 2021) and to assess their trends.

METHODS

Scoping review. Search in Embase and MEDLINE (07-07-2022). Acute care performance indicators and indicators related to acute general surgery were collected and collated following a care pathway approach. Indicators assessing identical clinical measures were grouped under a common indicator title. The trends from each group of indicators were collated (increase/decrease/stable).

RESULTS

A total of 152 studies were included. 2354 indicators regarding general acute care and 301 indicators related to acute general surgery were included. Indicators focusing on pre-hospital services reported a decreasing trend in the volume of patients: from 225 indicators, 110 (49%) reported a decrease. An increasing trend in pre-hospital treatment times was reported by most of the indicators (n = 41;70%) and a decreasing trend in survival rates of out-of-hospital cardiac arrest (n = 61;75%). Concerning care provided in the emergency department, most of the indicators (n = 752;71%) showed a decreasing trend in admissions across all levels of urgency. Concerning the mortality rate after admission, most of the indicators (n = 23;53%) reported an increasing trend. The subset of indicators assessing acute general surgery showed a decreasing trend in the volume of patients (n = 50;49%), stability in clinical severity at admission (n = 36;53%), and in the volume of surgeries (n = 14;47%). Most of the indicators (n = 28;65%) reported no change in treatment approach and stable mortality rate (n = 11,69%).

CONCLUSION

This review signals relevant disruptions across the acute care pathway. A subset of general surgery performance indicators showed stability in most of the phases of the care pathway. These results highlight the relevance of assessing this care pathway more regularly and systematically across different clinical entities to monitor disruptions and to improve the resilience of emergency services during a crisis.

What can we learn from general practitioners who left Spain? A mixed methods international study

BACKGROUND

International mobility of health workforce affects the performance of health systems and has major relevance in human resources for health policy and planning. To date, there has been little research exploring the reasons why general practitioners (GPs) migrate. This mixed methods study aimed to investigate the reasons why Spain-trained GPs migrate and develop GP retention and recruitment health policy recommendations relevant to Spanish primary care.

METHODS

The study followed an explanatory sequential mixed methods study design combining surveys with semi-structured interviews and focus groups with GPs who qualified in Spain and were living overseas at the time of the study. The survey data examined the reasons why GPs left Spain and their intention to return and were analysed using quantitative methods. The transcripts from interviews and focus groups centred on GPs' insights to enhance retention and recruitment in Spain and were analysed thematically.

RESULTS

The survey had 158 respondents with an estimated 25.4% response rate. Insufficient salary (75.3%), job insecurity and temporality (67.7%), excessive workload (67.7%), poor primary care governance (55.7%), lack of flexibility in the workplace (43.7%) and personal circumstances (43.7%) were the main reasons for leaving Spain. Almost half of the respondents (48.7%) would consider returning to Spanish general practice if their working conditions improved. Interviews and focus groups with respondents (n = 24) pointed towards the need to improve the quality of employment contracts, working conditions, opportunities for professional development, and governance in primary care for effective retention and recruitment.

CONCLUSION

Efforts to improve GP retention and recruitment in Spain should focus on salary, job security, flexibility, protected workload, professional development, and governance. We draw ten GP retention and recruitment recommendations expected to inform urgent policy action to tackle existing and predicted GP shortages in Spanish primary care.

COVID-19 Disruption To Routine Health Care Services: How 8 Latin American And Caribbean Countries Responded

Latin America and the Caribbean was one of the regions hardest hit globally by SARS-CoV-2. This qualitative exploratory study examined how the COVID-19 pandemic disrupted the delivery of routine health services from the perspective of health care system decision makers and managers. Between May and December 2022, we conducted forty-two semistructured interviews with decision makers from ministries of health and health care managers with responsibilities during the COVID-19 pandemic in eight countries in Latin America and the Caribbean. On the basis of these interviews, we identified themes in three domains: impacts on the provision of routine health services, including postponed and forgone primary care and hospital services; barriers to maintaining routine health services due to preexisting structural health care system weaknesses and difficulties attributed to the pandemic; and innovative strategies to sustain and recover services such as public-private financing and coordination, telemedicine, and new roles for primary care. In the short term, policy efforts should focus on recovering postponed services, including those for noncommunicable diseases. Medium- and long-term health care system reforms should strengthen primary care and address structural issues, such as fragmentation, to promote more resilient health care systems.

Data-Driven Collaboration between Hospitals and Other Healthcare Organisations in Europe During the COVID-19 Pandemic: An Explanatory Sequential Mixed-Methods Study among Mid-Level Hospital Managers

Introduction

Data and digital infrastructure drive collaboration and help develop integrated healthcare systems and services. COVID-19 induced changes to collaboration between healthcare organisations, which previously often happened in fragmented and competitive ways. New collaborative practices relied on data and were crucial in managing coordinated responses to the pandemic. In this study, we explored data-driven collaboration between European hospitals and other healthcare organisations in 2021 by identifying common themes, lessons learned and implications going forward.

Methods

Study participants were recruited from an existing Europe-wide community of mid-level hospital managers. For data collection, we ran an online survey, conducted multi-case study interviews and organised webinars. Data were analysed using descriptive statistics, thematic analysis and cross-case synthesis.

Results

Mid-level hospital managers from 18 European countries reported an increase in data exchange between healthcare organisations during the COVID-19 pandemic. Data-driven collaborative practices were goal-oriented and focused on the optimisation of hospitals' governance functions, innovation in organisational models and improvements to data infrastructure. This was often made possible by temporarily overcoming system complexities, which would otherwise hinder collaboration and innovation. Sustainability of these developments remains a challenge.

Discussion

Mid-level hospital managers form a huge potential of reacting and collaborating when needed, including rapidly setting up novel partnerships and redefining established processes. Major post-COVID unmet medical needs are linked to hospital care provision, including diagnostic and therapeutic backlogs. Tackling these will require rethinking of the position of hospitals within healthcare systems, including their role in care integration.

Conclusion

Learning from COVID-19-induced developments in data-driven collaboration between hospitals and other healthcare organisations is important to address systemic barriers, sustain resilience and further build transformative capacity to help build better integrated healthcare systems.

Patient-Reported Outcomes and Experiences Assessment in Women with Breast Cancer: Portuguese Case Study

In 2020, female breast cancer was the most commonly diagnosed cancer worldwide, representing the type of cancer with the highest incidence among women and the second most common cause of cancer death among women in all OECD countries. The conventional measures addressing the burden of breast cancer by measuring mortality, incidence, and survival do not entirely reflect the quality of life and patients experience when receiving breast cancer care. The main objective of this study is to capture patient-reported outcomes and experiences in women with breast cancer in Portugal using methods developed for international benchmarking purposes, such as the OECD Patient-reported Indicators Surveys. The study included 378 women with breast cancer, with the age distribution being 19.8% aged 15 to 49 years and 80.2% aged 50 years and over. The data collection procedure and analysis followed the "OECD Breast Cancer Patient Reported Outcomes Working Group" protocol, allowing subsequent comparability with data from other OECD member countries. Most women were satisfied with the treatment outcome regarding the shape of their lumpectomy breast when wearing a bra (96.1%) and with the equal size of both breasts (78.3%). Findings on the WHO QOL-BREF showed that women manifest a lower score in well-being when compared with the general population or populations living with chronic diseases. This study shows the feasibility of implementing and using patient-reported metrics (PROM and PREM) in breast cancer services in Portugal. Measuring PROMs and PREMs from Portuguese women receiving breast cancer care provides insightful evidence into the quality and value of cancer care.

How are regulatory oversight organisations using long-term care performance indicators: a qualitative descriptive study in 10 high-income countries

OBJECTIVES

Regulatory oversight organisations play an important role in quality stewardship in long-term care (LTC) facilities. Performance indicators are a key tool for any quality-related work. Our aim was to better understand how and what performance indicators are used by regulatory oversight organisations for long-term care facilities oversight and which features are affecting their fitness for use.

DESIGN

Qualitative descriptive.

SETTING AND PARTICIPANTS

We explored the use of LTC facility performance indicators by 10 regulatory oversight organisations from England, Ireland, Malta, New Zealand, Norway, Scotland, Singapore, Slovenia, Sweden and the Netherlands. We collected information by means of a questionnaire, 13 follow-up interviews with 20 experts from these organisations and document review.

RESULTS

Performance indicators are used by participating oversight organisations to choose priority topics for audits, prioritise facilities to be audited and to identify areas within an audited facility, that require more attention. The indicators of most interest to oversight organisations are related to the dimensions of care articulated in the preset requirements on which audits are based. When the purpose of using indicators is to design a risk assessment model, the fitness for use of indicators ultimately depends on their ability to predict non-compliances on subsequent audits. When indicators are used directly by auditors, the ease of access, clear guidance to evaluate the data and the provision of contextual information are used by oversight organisations to increase fitness for use.

CONCLUSIONS

Oversight organisations do not use LTC facility performance indicators to assess quality, but rather to assess the risk of lower quality or of non-compliance with requirements. This risk-related purpose has to be considered when the indicators used in oversight are chosen and when other aspects of fitness for use, such as data analysis and displaying findings, are developed.

Citizen engagement in healthcare procurement decision-making by healthcare insurers: recent experiences in the Netherlands

BACKGROUND

In insurance-based healthcare systems, healthcare insurers are interested in engaging citizens in care procurement to contract healthcare services that matter to people. In the Netherlands, an amendment to the Health Insurance Act was set forth in 2021 to formalize and strengthen the engagement of the insured population with healthcare insurers' procurement cycles. This study explores the role of Dutch healthcare insurers in operationalizing citizen engagement in procurement cycles before changes occur linked to the amendment to the Health Insurance Act.

METHODS

A phenomenological qualitative design was employed in two phases: (1) we consulted academics and policy experts on the role of healthcare insurers regarding citizen engagement; (2) we conducted focus groups with representatives of healthcare insurers to understand how citizens' engagement is being operationalized. Transcripts of the interviews with experts and detailed notes of focus group meetings were analysed using a qualitative inductive approach. Selected excerpts were analysed on discourse and content and organized by a coding scheme following a rigorous and accelerated data reduction technique.

RESULTS

We identified four strategies used by healthcare insurers to operationalize citizen engagement: (1) broadening their population health orientation; (2) developing and improving mechanisms for engaging citizens; (3) strengthening features of data governance for effective use of value-driven data; (4) implementing financial and incentive mechanisms among healthcare providers in support of value-based healthcare. However, regulated market mechanisms and low institutional trust in healthcare insurers undermine their transition from merely funding healthcare towards becoming people-centred value-based healthcare purchasers.

CONCLUSION

Dutch healthcare insurers seem to be strengthening the community orientation of their functioning while enhancing the end-to-end experience of the insured. The expected practical effects of the amendment to the Health Insurance Act include broadening the role of the council of insurees in decision-making processes and systematically documenting the efforts set forth by healthcare insurers in engaging citizens. Further research is needed to better understand how the regulated competitive market could be hampering the engagement of citizens in healthcare procurement decision-making and value creation from the citizens' perspective.

Status of the health information system in Ireland and its fitness to support health system performance assessment: a multimethod assessment based on stakeholder involvement

Background

Between 2019 and 2021, the first Irish health system performance assessment (HSPA) framework was developed. As routinely collected health data are necessary to continuously populate indicators of an HSPA framework, a purpose-driven assessment of the health information system (HIS) in Ireland and its fitness to support the implementation of an HSPA framework was conducted. This study reports on the status of the Irish HIS through a multimethod assessment based on continuous broad stakeholder involvement.

Methods

Between May and November 2020, over 50 informants were engaged in individual and group interviews and stakeholder consultation workshops as part of the HIS assessment process. Descriptive themes and high-level data availability heatmaps were derived from interview and workshop data using thematic analysis. Indicator “passports” for the HSPA framework were populated during stakeholder consultation workshops and analysed using univariate descriptive statistics.

Results

The HIS in Ireland was able to provide administrative, survey and registry-based data for public sector acute care services, focusing on structure, process and output metrics. Significant data availability gaps, most notably from primary care, private hospitals and community care, were reported, with little availability of electronic health record and people-reported data. Data on outcome metrics were mostly missing, as were linkage possibilities across datasets for care pathway monitoring. The COVID-19 pandemic highlighted the national HIS’s shortcomings but also the capacity for rapid development and improvement.

Conclusions

A tailor-made assessment of the HIS in Ireland, involving a broad set of relevant stakeholders, revealed strengths, weaknesses and areas for improvement in the Irish health data landscape. It also contributed to the development of a national HSPA framework and momentum to further strengthen data infrastructure and governance, while working towards a more data-driven and person-centred healthcare system. This work demonstrates the utility of an inclusive HIS assessment process and is applicable beyond Ireland, where this case study was conducted.

Use of Performance Data by Mid-Level Hospital Managers in Ontario: Results of a Province-Wide Survey and a Comparison with Hospital Managers in Europe

This paper provides insights into the use of performance data by middle managerial staff in Ontario hospitals in 2019 and compares the results to a study conducted in Europe in the same year. A total of 236 managers working in 61 hospitals across Ontario provided responses to the survey. Compared to their European colleagues, Ontario respondents self-assessed using significantly more performance data for managerial decision making. The use of performance data in Ontario was mostly motivated by external accountability requirements, followed by internal quality improvement efforts. Ontario managers also reported accessibility, appropriateness and timeliness of data and human resources and engagement as the biggest barriers to further performance data utilization. Comparative studies, such as the one this paper is based on, provide the foundation for drawing lessons across jurisdictions. This paper also affirms the importance of hospital middle management in moving from quality assurance to quality improvement efforts and developing sustainable learning healthcare organizations and systems.

Assessing the outcomes and experiences of care from the perspective of people living with chronic conditions, to support countries in developing people-centred policies and practices: study protocol of the International Survey of People Living with Chronic Conditions (PaRIS survey)

INTRODUCTION

In view of growing populations with chronic conditions, many countries are redesigning their health systems. However, little information is available about how health systems perform from the perspective of people living with chronic conditions. The Organisation for Economic Co-operation and Development (OECD) Member States therefore mandated the OECD to initiate the International Survey of People Living with Chronic Conditions (PaRIS survey), which aims to provide insight in outcomes and experiences of care as reported by people living with chronic conditions. The PaRIS-SUR consortium has been tasked by the OECD to support the development and implementation of the survey.

METHODS AND ANALYSIS

As primary care services play a pivotal role in the management of chronic conditions, the PaRIS survey will be implemented in the primary care setting. Data will be collected with a survey among users of primary care services aged 45 years or older, of whom many have chronic conditions. An additional survey is conducted among their primary care providers. The nested study design will allow analysis of the patient-reported data in relation to characteristics of and care provided by primary care providers within and across countries. In 2022, the survey will be tested in a Field Trial in participating countries. Data for cross-country comparison will be collected by the Main Survey in 2023.

ETHICS AND DISSEMINATION

Informed consent will be obtained from primary care providers and service users. National Project Managers search ethical approval of the survey in their country, if required. Reporting by the OECD will focus on questions for international comparison. A secured information technology platform will be developed for participants and stakeholders in countries to receive feedback and answer their own questions. Findings will also be disseminated through an international OECD flagship report, conferences, scientific papers and policy briefs, to inform strategies to improve care for people living with chronic conditions throughout the world.

Mental Health Patient-Reported Outcomes and Experiences Assessment in Portugal

Mental ill-health is increasingly recognized by policymakers for its significant human and economic toll. The main objective of this study is to capture patient-reported outcomes and experiences on mental health care in Portugal using methods developed for international benchmarking purposes, such as the OECD Patient-reported Indicators Surveys. The study included 397 participants, 247 (62.2%) women, divided into four age groups: ages 16-24 years, ages 25-44 years, ages 45-65 years, and ages 66 years or older. The data collection procedure and analysis followed the OECD PaRIS Mental Health Working Group 2021 protocol allowing subsequent comparability with data from other OECD member countries. Findings on the WHO-5 Well-Being Index showed that women manifest a lower score in well-being following mental health care services use. This finding may be, at least in part, explained by the study population (mental health services users), including individuals with clinical depression which is more frequently observed in women. In terms of the level of satisfaction with treatment (provided by nurses, doctors, phycologists, etc.) the response "Yes, definitely" varied from 67% of answers regarding "time spent by care providers", 76.3% "involvement in decisions" to 79.7% regarding "clarity of explanations" and 84.4% regarding the item courtesy and respect. This study shows the feasibility of implementing and using patient-reported metrics (PROM and PREM) in mental health services in Portugal. The study results generate useful clinical information to help meet the expectations and needs of patients, contributing to a continuous improvement of mental health community services.

Optimising the secondary use of primary care prescribing data to improve quality of care: a qualitative analysis

OBJECTIVES

To explore available data sources, secondary uses and key considerations for optimising the actionability of primary care prescribing data to improve quality of care in the Dutch context.

DESIGN

An exploratory qualitative study was undertaken based on semi-structured interviews. We anchored our investigation around three tracer prescription types: antibiotics; benzodiazepines and opioids. Descriptive and explanatory themes were derived from interview data using thematic analysis.

SETTING

Stakeholders were sampled from across the micro (clinical), meso (organisational) and macro (policy) contexts of the Dutch primary care system.

PARTICIPANTS

The study involved 28 informants representing general practitioners (GPs), community pharmacists, regional chronic care networks (care groups), academia and research institutes, insurers, professional associations, electronic health record (EHR) vendors and national authorities.

RESULTS

In the Netherlands, three main sources of data for improving prescribing in primary care are in use: clinical data in the EHRs of GP practices; pharmacy data in community pharmacy databases and claims data of insurers. While the secondary use of pharmacy and claims data is well-established across levels, the use of these data together with EHR data is limited. Important differences in the types of prescribing information needed by micro-meso-macro context are found, though the extent to which current indicators address these varies by prescription type. Five main themes were identified as areas for optimising data use: (1) measuring what matters, (2) increasing data linkages, (3) improving data quality, (4) facilitating data sharing and (5) optimising fit for use analysis.

CONCLUSIONS

To make primary care prescribing data useful for improving quality, consolidated patient-specific data on the indication for a prescription and dispensed medicine, over time, is needed. In the Netherlands, the selection of indicators requires further prioritisation to better signal the appropriateness and long-term use of prescription drugs. Prioritising data linkages is critical towards more actionable use.

Changes in the quality of cancer care as assessed through performance indicators during the first wave of the COVID-19 pandemic in 2020: a scoping review

BACKGROUND

Cancer comprises a high burden on health systems. Performance indicators monitoring cancer outcomes are routinely used in OECD countries. However, the development of process and cancer-pathway based information is essential to guide health care delivery, allowing for better monitoring of changes in the quality of care provided. Assessing the changes in the quality of cancer care during the COVID-19 pandemic requires a structured approach considering the high volume of publications. This study aims to summarize performance indicators used in the literature to evaluate the impact of the COVID-19 pandemic on cancer care (January-June 2020) in OECD countries and to assess changes in the quality of care as reported via selected indicators.

METHODS

Search conducted in MEDLINE and Embase databases. Performance indicators and their trends were collated according to the cancer care pathway.

RESULTS

This study included 135 articles, from which 1013 indicators were retrieved. Indicators assessing the diagnostic process showed a decreasing trend: from 33 indicators reporting on screening, 30 (91%) signalled a decrease during the pandemic (n = 30 indicators, 91%). A reduction was also observed in the number of diagnostic procedures (n = 64, 58%) and diagnoses (n = 130, 89%). The proportion of diagnoses in the emergency setting and waiting times showed increasing trends (n = 8, 89% and n = 14, 56%, respectively). A decreasing trend in the proportion of earliest stage cancers was reported by 63% of indicators (n = 9), and 70% (n = 43) of indicators showed an increasing trend in the proportion of advanced-stage cancers. Indicators reflecting the treatment process signalled a reduction in the number of procedures: 79%(n = 82) of indicators concerning surgeries, 72%(n = 41) of indicators assessing radiotherapy, and 93%(n = 40) of indicators related to systemic therapies. Modifications in cancer treatment were frequently reported: 64%(n = 195) of indicators revealed changes in treatment.

CONCLUSIONS

This study provides a summary of performance indicators used in the literature to assess the cancer care pathway from January 2020 to June 2020 in OECD countries, and the changes in the quality of care signalled by these indicators. The trends reported inform on potential bottlenecks of the cancer care pathway. Monitoring this information closely could contribute to identifying moments for intervention during crises.

The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study

Background

Governments across the World Health Organization (WHO) European Region have prioritised dashboards for reporting COVID-19 data. The ubiquitous use of dashboards for public reporting is a novel phenomenon.

Objective

This study explores the development of COVID-19 dashboards during the first year of the pandemic and identifies common barriers, enablers and lessons from the experiences of teams responsible for their development.

Methods

We applied multiple methods to identify and recruit COVID-19 dashboard teams, using a purposive, quota sampling approach. Semi-structured group interviews were conducted from April to June 2021. Using elaborative coding and thematic analysis, we derived descriptive and explanatory themes from the interview data. A validation workshop was held with study participants in June 2021.

Results

Eighty informants participated, representing 33 national COVID-19 dashboard teams across the WHO European Region. Most dashboards were launched swiftly during the first months of the pandemic, February to May 2020. The urgency, intense workload, limited human resources, data and privacy constraints and public scrutiny were common challenges in the initial development stage. Themes related to barriers or enablers were identified, pertaining to the pre-pandemic context, pandemic itself, people and processes and software, data and users. Lessons emerged around the themes of simplicity, trust, partnership, software and data and change.

Conclusions

COVID-19 dashboards were developed in a learning-by-doing approach. The experiences of teams reveal that initial underpreparedness was offset by high-level political endorsement, the professionalism of teams, accelerated data improvements and immediate support with commercial software solutions. To leverage the full potential of dashboards for health data reporting, investments are needed at the team, national and pan-European levels.

Making Use of Comparable Health Data to Improve Quality of Care and Outcomes in Diabetes: The EUBIROD Review of Diabetes Registries and Data Sources in Europe

Background

Registries and data sources contain information that can be used on an ongoing basis to improve quality of care and outcomes of people with diabetes. As a specific task of the EU Bridge Health project, we carried out a survey of diabetes-related data sources in Europe.

Objectives

We aimed to report on the organization of different sources of diabetes information, including their governance, information infrastructure and dissemination strategies for quality control, service planning, public health, policy and research.

Methods

Survey using a structured questionnaire to collect targeted data from a network of collaborating institutions managing registries and data sources in 17 countries in the year 2017.

Results

The 18 data sources participating in the study were most frequently academic centres (44.4%), national (72.2%), targeting all types of diabetes (61.1%) covering no more than 10% of the target population (44.4%). Although population-based in over a quarter of cases (27.8%), sources relied predominantly on provider-based datasets (38.5%), fewer using administrative data (16.6%). Data collection was continuous in the majority of cases (61.1%), but 50% could not perform data linkage. Public reports were more frequent (72.2%) as well as quality reports (77.8%), but one third did not provide feedback to policy and only half published ten or more peer reviewed papers during the last 5 years.

Conclusions

The heterogeneous implementation of diabetes registries and data sources hampers the comparability of quality and outcomes across Europe. Best practices exist but need to be shared more effectively to accelerate progress and deliver equitable results for people with diabetes.

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study

BACKGROUND

Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration.

OBJECTIVE

We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time.

METHODS

We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point.

RESULTS

Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards.

CONCLUSIONS

Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

COVID-19 Preparedness and Perceived Safety in Nursing Homes in Southern Portugal: A Cross-Sectional Survey-Based Study in the Initial Phases of the Pandemic

(1) Background: Nursing homes’ preparedness in managing a public health emergency has been poor, with effects on safety culture. The objective of this study was to assess nursing homes’ COVID-19 preparedness in southern Portugal, including staff’s work experiences during the pandemic. (2) Methods: We used a COVID-19 preparedness checklist to be completed by management teams, followed by follow-up calls to nursing homes. Thereafter, a survey of staff was applied. Data analysis included descriptive statistics, exploratory factor analysis, and thematic analysis of open-end questions. (3) Results: In total, 71% (138/195) of eligible nursing homes returned the preparedness checklist. We conducted 83 follow-up calls and received 720 replies to the staff survey. On average, 25% of nursing homes did not have an adequate decision-making structure to respond to the pandemic. Outbreak capacity and training were areas for improvement among nursing homes’ contingency plans. We identified teamwork as an area of strength for safety culture, whereas compliance with procedures and nonpunitive response to mistakes need improvement. (4) Conclusions: To strengthen how nursing homes cope with upcoming phases of the COVID-19 pandemic or future public health emergencies, nursing homes’ preparedness and safety culture should be fostered and closely monitored.

Patient-reported indicators in mental health care: towards international standards among members of the OECD

BACKGROUND

Achieving people-centred health care systems requires new and innovative strategies to capture information about whether, and to what degree, health care is successful in improving health from the perspective of the patient. Patient-reported outcome measures (PROMs) and Patient-reported experience measures (PREMs) can bring some of these new insights, and are increasingly used in research, clinical care, and policymaking.

METHODS

This paper reflects the ongoing discussions and findings of the OECD PaRIS Working Group on Patient-reported Indicators for Mental Health Care.

RESULTS

The OECD has been measuring quality of care for mental health conditions over the last 14 years through the Health Care Quality and Outcomes (HCQO) program; nonetheless, information on how persons with mental health problems value the services they receive, and impact of the services, remains limited. As of 2018, a survey from the OECD showed that only five of the twelve countries surveyed (Australia, Israel, Netherlands, Sweden, United Kingdom) reported PROMs and PREMs collection on a regular basis in mental health settings. The paper details some of the challenges specific to the collection and use of PROMs and PREMs in mental health care, and examples from countries which have implemented comprehensive programmes to gather information about PROMs and PREMs for individuals receiving mental health services.

CONCLUSIONS

Given the health and economic impact of mental ill-health across all OECD countries, there is significant value to being able to assess the quality and outcomes of care in this area using internationally-comparable measures. Continued international harmonisation of PROMs and PREMs for mental health through international coordination is a key way to facilitate the sharing of national experiences, promote the use of PROMs and PREMs, and create meaningful indicators for national and international benchmarking.

Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards

Background

Since the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed.

Objective

The aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users (“why”), content and data (“what”), and analyses and displays (“how” they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards.

Methods

We conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question.

Results

A total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are “close to home”; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues.

Conclusions

COVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Measuring patient voice matters: setting the scene for patient-reported indicators

BACKGROUND

Achieving people-centred health care systems requires new and innovative strategies to capture information about whether, and to what degree, health care is successful in improving health from the perspective of the patient. Patient-reported outcome measures (PROMs) and Patient-reported experience measures (PREMs) can bring some of these new insights, and are increasingly used in research, clinical care, and policymaking.

METHODS

This paper reflects the ongoing discussions and findings of the OECD PaRIS Working Group on Patient-reported Indicators for Mental Health Care.

RESULTS

The OECD has been measuring quality of care for mental health conditions over the last 14 years through the Health Care Quality and Outcomes (HCQO) program; nonetheless, information on how persons with mental health problems value the services they receive, and impact of the services, remains limited. As of 2018, a survey from the OECD showed that only five of the twelve countries surveyed (Australia, Israel, Netherlands, Sweden, United Kingdom) reported PROMs and PREMs collection on a regular basis in mental health settings. The paper details some of the challenges specific to the collection and use of PROMs and PREMs in mental health care, and examples from countries which have implemented comprehensive programmes to gather information about PROMs and PREMs for individuals receiving mental health services.

CONCLUSIONS

Given the health and economic impact of mental ill-health across all OECD countries, there is significant value to being able to assess the quality and outcomes of care in this area using internationally-comparable measures. Continued international harmonisation of PROMs and PREMs for mental health through international coordination is a key way to facilitate the sharing of national experiences, promote the use of PROMs and PREMs, and create meaningful indicators for national and international benchmarking.

Associations between hospital deaths (HSMR), readmission and length of stay (LOS): a longitudinal assessment of performance results and facility characteristics of teaching and large-sized hospitals in Canada between 2013-2014 and 2017-2018

OBJECTIVES

To examine the association between hospital deaths (hospital standardised mortality ratio, HSMR), readmission, length of stay (LOS) and eight hospital characteristics.

DESIGN

Longitudinal observational study.

SETTING

A total of 119 teaching and large-sized hospitals in Canada between fiscal years 2013-2014 and 2017-2018.

PARTICIPANTS

Analysis focused on indicator results and characteristics of individual Canadian hospitals.

PRIMARY AND SECONDARY OUTCOMES

Hospital deaths (HSMR); all patients readmitted to hospital; average LOS and a series of eight hospital characteristic summary measures: number of acute care hospital stays; number of acute care beds; number of emergency department visits; average acute care resource intensity weight; total acute care resource intensity weight; hospital occupancy rate; patients admitted through the emergency department (%); patient days in alternate level of care (%).

RESULTS

Comparing 2013-2014 to 2017-2018, hospital deaths (HSMR) largely declined, while readmissions increased; 69% of hospitals decreased their hospital deaths (HSMR), while 65% of hospitals increased their readmissions rates. A greater proportion of community-large hospitals (31%, n=14) improved on both hospital deaths (HSMR) and readmission compared to Teaching hospitals (13.9%, n=5). Hospital deaths (HSMR), readmission and LOS largely showed very weak and non-significant correlations. LOS was largely positively and statistically significantly correlated with the suite of eight hospital characteristics. Hospital deaths (HSMR) was largely negatively (not statistically significantly) correlated with the hospital characteristics. Readmission was largely not statistically significantly correlated and showed no clear pattern of correlation (direction) with hospital characteristics.

CONCLUSIONS

Examining publicly reported hospital performance results can reveal meaningful insights into the association among outcome indicators and hospital characteristics. Good or bad hospital performance in one care domain does not necessarily reflect similar performance in other care domains. Thus, caution is warranted in a narrow use of outcome indicators in the design and operationalisation of hospital performance measurement and governance models (namely pay-for-performance schemes). Analysis such as this can also inform quality improvement strategies and targeted efforts to address domains of care experiencing declining performance over time; further granular subdivision of the analyses, for example, by hospital peer-groups, can reveal notable differences in performance.

Complex Governance Does Increase Both the Real and Perceived Registration Burden: The Case of the Netherlands Comment on "Perceived Burden Due to Registrations for Quality Monitoring and Improvement in Hospitals: A Mixed Methods Study"

The burden of registrations for professionals should be more firmly on the policy agenda. In a rigorous study, Marieke Zegers and colleagues make a compelling argument why that should be the case. In Dutch hospitals, the average professional spends 52.3 minutes a day on quality registries and monitoring instruments. Many more administrative duties exist. These represent substantial resources and ultimately could become a drag on the intrinsic motivation of the care professions. We agree with Zegers et al that we are in need for more operational efficiency. However, the issue at hand is very complex and also intensely connected to the entire healthcare system and its different levels. More operational efficiency alone will not solve this problem. We are also in need for better governance of data-issues at the macro-system level.

Understanding the use of patient-reported data by health care insurers: A scoping review

Background

Patient-reported data are widely used for many purposes by different actors within a health system. However, little is known about the use of such data by health insurers. Our study aims to map the evidence on the use of patient-reported data by health insurers; to explore how collected patient-reported data are utilized; and to elucidate the motives of why patient-reported data are collected by health insurers.

Methods

The study design is that of a scoping review. In total, 11 databases were searched on. Relevant grey literature was identified through online searches, reference mining and recommendations from experts. Forty-two documents were included. We synthesized the evidence on the uses of patient-reported data by insurers following a structure-process-outcome approach; we also mapped the use and function of those data by a health insurer.

Results

Health insurers use patient-reported data for assurance and improvement of quality of care and value-based health care. The patient-reported data most often collected are those of outcomes, experiences and satisfaction measures; structure indicators are used to a lesser extent and often combined with process indicators. These data are mainly used for the purposes of procurement and purchasing of services, quality assurance, improvement and reporting, and strengthening the involvement of insured people.

Conclusions

The breadth to which insurers use patient-reported data in their business models varies greatly. Some hindering factors to the uptake of such data are the varying and overlapping terminology in use in the field and the limited involvement of insured people in a health insurer’s business. Health insurers are advised to be more explicit in regard to the role they want to play within the health system and society at large, and accommodate implications for the use of patient-reported data accordingly.

More public health service providers are experiencing job burnout than clinical care providers in primary care facilities in China

BACKGROUND

Health workers are at high risk of job burnout. Primary care in China has recently expanded its scope of services to a broader range of public health services in addition to clinical care. This study aims to measure the prevalence of burnout and identify its associated factors among clinical care and public health service providers at primary care facilities.

METHODS

A cross-sectional survey (2018) was conducted among 17,816 clinical care and public health service providers at 701 primary care facilities from six provinces. Burnout was measured by the Chinese version of the Maslach Burnout Inventory-General Scale, and multilevel linear regression analysis was conducted to identify burnout's association with demographics, as well as occupational and organisational factors.

RESULTS

Overall, half of the providers (50.09%) suffered from burnout. Both the presence of burnout and the proportion of severe burnout among public health service providers (58.06% and 5.25%) were higher than among clinical care providers (47.55% and 2.26%, respectively). Similar factors were associated with burnout between clinical care and public health service providers. Younger, male, lower-educated providers and providers with intermediate professional title, permanent contract or higher working hours were related to a higher level of burnout. Organisational environment, such as the presence of a performance-based salary system, affected job burnout.

CONCLUSIONS

Job burnout is prevalent among different types of primary care providers in China, indicating the need for actions that encompass the entirety of primary care. We recommend strengthening the synergy between clinical care and public health services and transforming the performance-based salary system into a more quality-based system that includes teamwork incentives.

Exploring eHealth Literacy and Patient-Reported Experiences With Outpatient Care in the Hungarian General Adult Population: Cross-Sectional Study

Background

Digital health, which encompasses the use of information and communications technology in support of health, is a key driving force behind the cultural transformation of medicine toward people-centeredness. Thus, eHealth literacy, assisted by innovative digital health solutions, may support better experiences of care.

Objective

The purpose of this study is to explore the relationship between eHealth literacy and patient-reported experience measures (PREMs) among users of outpatient care in Hungary.

Methods

In early 2019, we conducted a cross-sectional survey on a large representative online sample recruited from the Hungarian general population. eHealth literacy was measured with the eHealth Literacy Scale (eHEALS). PREMs with outpatient care were measured with a set of questions recommended by the Organisation for Economic Co-operation and Development (OECD) for respondents who attended outpatient visit within 12 months preceding the survey. Bivariate relationships were explored via polychoric correlation, the Kruskal–Wallis test, and chi-square test. To capture nonlinear associations, after controlling covariates, we analyzed the relationship between eHEALS quartiles and PREMs using multivariate probit, ordinary least squares, ordered logit, and logistic regression models.

Results

From 1000 survey respondents, 666 individuals (364 females, 54.7%) were included in the study with mean age of 48.9 (SD 17.6) years and mean eHEALS score of 29.3 (SD 4.9). Respondents with higher eHEALS scores were more likely to understand the health care professionals’ (HCPs’) explanations (χ²₉=24.2, P=.002) and to be involved in decision making about care and treatment (χ²₉=18.2, P=.03). In multivariate regression, respondents with lowest (first quartile) and moderately high (third quartile) eHEALS scores differed significantly, where the latter were more likely to have an overall positive experience (P=.02) and experience fewer problems (P=.02). In addition, those respondents had better experiences in terms of how easy it was to understand the HCPs’ explanations (P<.001) and being able to ask questions during their last consultation (P=.04). Patient-reported experiences of individuals with highest (fourth quartile) and lowest (first quartile) eHEALS levels did not differ significantly in any items of the PREM instrument, and neither did composite PREM scores generated from the PREM items (P>.05 in all models).

Conclusions

We demonstrated the association between eHealth literacy and PREMs. The potential patient-, physician-, and system-related factors explaining the negative experiences among people with highest levels of eHealth literacy warrant further investigation, which may contribute to the development of efficient eHealth literacy interventions. Further research is needed to establish causal relationship between eHealth literacy and patient-reported experiences.

A problem-oriented systems approach to primary care system development: development and initial testing of the problem-oriented primary care system development record

BACKGROUND

There is significant global policy interest related to enabling a data-driven approach for evidence-based primary care system development. This paper describes the development and initial testing of a prototype tool (the Problem-Oriented Primary Care System Development Record, or PCSDR) that enables a data-driven and contextualized approach to primary care system development.

METHODS

The PCSDR is an electronic record that enables the systematic input, classification, structuring, storage, processing and analysis of different types of data related to the structure, function and performance of primary care systems over time. Data inputted into the PCSDR was coded using the WHO's PHC-IMPACT framework and classification system. The PCSDR's functionalities were tested by using a case study of primary care system development in Tajikistan.

RESULTS

Tajikistan's case study demonstrated that the PCSDR is a potentially effective and conceptually-sound tool for the input, classification, structuring and storage of different data types from myriad sources. The PCSDR is therefore a basic data entry and data management system that enables query and analytics functions for health services research and evidence-based primary care system development functions.

CONCLUSIONS

The PCSDR is a data system that enables a contextualized approach to evidence-based primary care system development. It represents a coherent and effective synthesis of the fields of primary care system development and performance assessment. The PCSDR enables analysts to leverage primary care performance assessment frameworks for a broad range of functions related to health systems analysis, improvement and the development of learning health systems.

Eliciting preferences for outpatient care experiences in Hungary: A discrete choice experiment with a national representative sample

Introduction

Patient-reported experience measures (PREMs) are central to inform on the responsiveness of health systems to citizens’ health care needs and expectations. At their current form, PREMs do not reflect the weights that patients assign to varying aspects of the care experience. We aimed to investigate patients’ preferences and willingness to pay (WTP) for attributes of the care experience in outpatient settings.

Methods

A discrete choice experiment was conducted among a representative sample of the general adult population of Hungary (n = 1000). Choice set attributes and levels were defined based on OECD’s standardized PREMs (e.g. a doctor spending enough time in consultation, providing easy to understand explanations, giving opportunity to ask questions, and involving in decision making) and a price attribute. Conditional and mixed logit analyses were conducted. WTP estimates were computed in preference and WTP space.

Results

The respondents most preferred attribute was that of a doctor spending enough time in consultation, followed by involvement in decision making. Moreover, waiting times had a less important effect on respondents’ choice preference compared with aspects of the doctor-patient relationship. Estimates in the WTP space varied from €4.38 (2.85–5.90) for waiting an hour less at a doctor’s office to €36.13 (32.07–40.18) for a consultation where a doctor spends enough time with a patient relative to a consultation where a doctor does not.

Conclusions

A preference-based PREMs approach provide insight on the value patients assign to different aspects of their care experience. This can inform the decisions of policy-makers and other stakeholders to coordinate efforts and resource allocation in a more targeted manner, by acting on attributes of the care experience that have a greater impact on the implementation of patient-centered care.

Managing COVID-19 within and across health systems: why we need performance intelligence to coordinate a global response

Background

The COVID-19 pandemic is a complex global public health crisis presenting clinical, organisational and system-wide challenges. Different research perspectives on health are needed in order to manage and monitor this crisis. Performance intelligence is an approach that emphasises the need for different research perspectives in supporting health systems’ decision-makers to determine policies based on well-informed choices. In this paper, we present the viewpoint of the Innovative Training Network for Healthcare Performance Intelligence Professionals (HealthPros) on how performance intelligence can be used during and after the COVID-19 pandemic.

Discussion

A lack of standardised information, paired with limited discussion and alignment between countries contribute to uncertainty in decision-making in all countries. Consequently, a plethora of different non-data-driven and uncoordinated approaches to address the outbreak are noted worldwide. Comparative health system research is needed to help countries shape their response models in social care, public health, primary care, hospital care and long-term care through the different phases of the pandemic. There is a need in each phase to compare context-specific bundles of measures where the impact on health outcomes can be modelled using targeted data and advanced statistical methods. Performance intelligence can be pursued to compare data, construct indicators and identify optimal strategies. Embracing a system perspective will allow countries to take coordinated strategic decisions while mitigating the risk of system collapse.A framework for the development and implementation of performance intelligence has been outlined by the HealthPros Network and is of pertinence. Health systems need better and more timely data to govern through a pandemic-induced transition period where tensions between care needs, demand and capacity are exceptionally high worldwide. Health systems are challenged to ensure essential levels of healthcare towards all patients, including those who need routine assistance.

Conclusion

Performance intelligence plays an essential role as part of a broader public health strategy in guiding the decisions of health system actors on the implementation of contextualised measures to tackle COVID-19 or any future epidemic as well as their effect on the health system at large. This should be based on commonly agreed-upon standardised data and fit-for-purpose indicators, making optimal use of existing health information infrastructures. The HealthPros Network can make a meaningful contribution.

Appropriateness, effectiveness and safety of care delivered in Canadian hospitals: a longitudinal assessment on the utility of publicly reported performance trend data between 2012-2013 and 2016-2017

OBJECTIVES

To assess the utility of publicly reported performance trend results of Canadian hospitals (by hospital size/type and jurisdiction).

DESIGN

Longitudinal observational study.

SETTING

489 hospitals in Canada between fiscal years 2012-2013 and 2016-2017.

PARTICIPANTS

Analysis focused on indicator results of individual Canadian hospitals.

PRIMARY AND SECONDARY OUTCOMES

Eight outcome indicators of hospital performance: in-hospital mortality (2), readmissions (4) and adverse events (2). Performance trend outcomes of improving, weakening or no change over time. Comparators in performance by hospital size/type of above, below or same as average.

RESULTS

At the national level, between 2012-2013 and 2016-2017, Canadian hospitals largely reduced in-hospital mortality: hospital deaths (hospital standardised mortality ratio) -9%; hospital deaths following major surgery -11.1%. Conversely, readmission to hospital increased nationwide: medical 1.5%; obstetric 5%; patients aged 19 years and younger 4.6% and surgical 3%. In-hospital sepsis declined -7.1%. Approximately 10% of the 489 hospitals in this study had a trend of improving performance over time (n=49) in one or more indicators, and a similar number showed a weakening performance over time (n=52). Roughly half of the hospitals in this study (n=224) had no change in performance over time for at least four out of the eight indicators. No single hospital had an improving or weakening trend in more than two indicators. Teaching and larger-sized hospitals showed a higher ratio of improving performance compared with smaller-sized hospitals.

CONCLUSIONS

Analysis of Canadian hospital performance through eight indicators shows improvement of in-hospital mortality and in-hospital sepsis, but rising rates of readmissions. Subdividing the analysis by hospital size/type shows greater instances of improvement in teaching and larger-sized hospitals. There is no clear pattern of a particular province/territory with a significant number of hospitals with improving or weakening trends. The overall assessment of trends of improving and weakening as presented in this study can be used more systematically in monitoring progress.

A Multi-sourced Data Analytics Approach to Measuring and Assessing Biopsychosocial Complexity: The Vancouver Community Analytics Tool Complexity Module (VCAT-CM)

Operationalization of the fundamental building blocks of primary care (i.e. empanelment, team-based care and population management) within the context of Community Health Centers requires accurate and real-time measures of biopsychosocial complexity, at both client and population-levels. This article describes the conceptualization, design and development of a novel software tool (the VCAT-Complexity Module) that can calculate and report real-time person-oriented biopsychosocial complexity profiles, using multiple data sources. The tool aligns with a profile approach to conceptualizing health outcomes, and represents a potentially significant advance over disease-oriented complexity assessment tools. The results and face validity of the software's complexity score outputs are discussed, along with their practical implications on functions related to the development of primary care within Vancouver Coastal Health, a Canadian Regional Health Authority.

Patient experiences with outpatient care in Hungary: results of an online population survey

BACKGROUND

Health systems are undertaking efforts to make health care more patient centered and value based. To achieve this goal, the use of patient-reported experience measures (PREMs) is increasing, especially across OECD countries. However, in Hungary, data on patients' experiences are still lacking. Thus, our aim was twofold: first, to collect data on outpatient experience in Hungary on patient-doctor communication and patient involvement in decision making and compare it with that of other OECD countries; second, to assess associations of outpatient experience with patients' socioeconomic characteristics.

METHODS

In early 2019, we conducted a cross-sectional, online, self-administered survey in a national representative sample of Hungary's population (n = 1000). The sample was weighted considering gender, age, highest education level attained, type of settlement, and region of residence. The survey questions were based on a set of recommended questions by the OECD.

RESULTS

Our findings show that the proportion of reported positive experiences is as follows: doctors providing easy-to-understand explanations (93.1%) followed by time spent on the consultation (87.5%), opportunities to raise questions (85.8%), and doctors involving patients in decision making about care and treatment (80.1%). The share of positive experiences falls behind OECD's average regarding patient-doctor communication and patient involvement in decision making, which signals room for improvement in these areas.

CONCLUSIONS

Women, younger people, people with a paid job, and patients with consultations with allied health professionals reported significant lesser positive care experiences and, hence, more targeted policies can be initiated based on our findings.

Unmet medical needs in ambulatory care in Hungary: forgone visits and medications from a representative population survey

BACKGROUND

The objective of this paper is to explore unmet health care needs in Hungary in ambulatory care due to costs and difficulties in travelling, and to analyze how unmet needs relate to socio-demographic characteristics.

METHODS

The quantitative analysis is based on a national, representative online survey carried out in Hungary on a sample of 1000 respondents in early 2019 using a proposed set of questions developed by the OECD. We present and compare unmet medical needs in different socio-demographic groups, and we use multivariate logistic regression analysis to identify the main determinants of unmet medical needs.

RESULTS

Among responders who had medical problems in the last 12 months, 27.3% reported forgone medical visit due to difficulties in travelling, 24.2% had unfilled prescription for medicine due to costs, 21.4% reported forgone medical visit or follow-up visit due to costs and 16.6% reported skipped medical test, treatment or other follow-up due to costs. These shares are much higher than presented previously in international databases. The logistic model indicates that respondents were significantly more likely to report unmet needs if they were women, younger or belonged to first and second income quintiles.

CONCLUSIONS

Policy makers need to address the issue of high prevalence of forgone medical care among the Hungarian population to avoid deterioration of population health and inequalities in access. As a first step, policies should try to decrease financial burden of vulnerable groups to improve access.

Mapping standard ophthalmic outcome sets to metrics currently reported in eight eye hospitals

Background

To determine alignment of proposed international standard outcomes sets for ophthalmic conditions to metrics currently reported by eye hospitals.

Methods

Mixed methods comparative benchmark study, including eight eye hospitals in Australia, India, Singapore, Sweden, U.K., and U.S. All are major international tertiary care and training centers in ophthalmology. Main outcome measure is consistency of ophthalmic outcomes measures reported.

Results

International agreed standard outcomes (ICHOM) sets are available for cataract surgery (10 metrics) and macular degeneration (7 metrics). The eight hospitals reported 22 different metrics for cataract surgery and 2 for macular degeneration, which showed only limited overlap with the proposed ICHOM metrics. None of the hospitals reported patient reported visual functioning or vision-related quality of life outcomes measures (PROMs). Three hospitals (38%) reported rates for uncomplicated cataract surgeries only. There was marked variation in how and at what point postoperatively visual outcomes following cataract, cornea, glaucoma, strabismus and oculoplastics procedures were reported. Seven (87.5%) measured post-operative infections and four (50%) measured 30 day unplanned reoperation rates.

Conclusions

Outcomes reporting for ophthalmic conditions currently widely varies across hospitals internationally and does not include patient-reported outcomes. Reaching consensus on measures and consistency in data collection will allow meaningful comparisons and provide an evidence base enabling improved sharing of “best practices” to improve eye care globally. Implementation of international standards is still a major challenge and practice-based knowledge on measures should be one of the inputs of the international standardization process.

Electronic supplementary material

The online version of this article (doi: 10.1186/s12886-017-0667-0) contains supplementary material, which is available to authorized users.

Monitoring compliance with standards of care for chronic diseases using healthcare administrative databases in Italy: Strengths and limitations

Background

A recent comprehensive report on healthcare quality in Italy published by the Organization of Economic Co-operation and Development (OECD) recommended that regular monitoring of quality of primary care by means of compliance with standards of care for chronic diseases is performed. A previous ecological study demonstrated that compliance with standards of care could be reliably estimated on regional level using administrative databases. This study compares estimates based on administrative data with estimates based on GP records for the same persons, to understand whether ecological fallacy played a role in the results of the previous study.

Methods

We compared estimates of compliance with diagnostic and therapeutic standards of care for type 2 diabetes (T2DM), hypertension and ischaemic heart disease (IHD) from administrative data (IAD) with estimates from medical records (MR) for the same persons registered with 24 GP’s in 2012. Data were linked at an individual level.

Results

32,688 persons entered the study, 12,673 having at least one of the three diseases according to at least one data source. Patients not detected by IAD were many, for all three conditions: adding MR increased the number of cases of T2DM, hypertension, and IHD by +40%, +42%, and +104%, respectively. IAD had imperfect sensitivity in detecting population compliance with therapies (adding MR increased the estimate, from +11.5% for statins to +14.7% for antithrombotics), and, more substantially, with diagnostic recommendations (adding MR increased the estimate, from +23.7% in glycated hemoglobin tests, to +50.5% in electrocardiogram). Patients not detected by IAD were less compliant with respect to those that IAD correctly identified (from -4.8 percentage points in proportion of IHD patients compliant with a yearly glycated hemoglobin test, to -40.1 points in the proportion of T2DM patients compliant with the same recommendation). IAD overestimated indicators of compliance with therapeutic standards (significant differences ranged from 3.3. to 3.6 percentage points) and underestimated indicators of compliance with diagnostic standards (significant differences ranged from -2.3 to -14.1 percentage points).

Conclusion

IAD overestimated the percentage of patients compliant with therapeutic standards by less than 6 percentage points, and underestimated the percentage of patients compliant with diagnostic standards by a maximum of 14 percentage points. Therefore, both discussions at local level between GP's and local health unit managers and discussions at central level between national and regional policy makers can be informed by indicators of compliance estimated by IAD, which, based on those results, have the ability of signalling critical or excellent clusters. However, this study found that estimates are partly flawed, because a high number of patients with chronic diseases are not detected by IAD, patients detected are not representative of the whole population of patients, and some categories of diagnostic tests are markedly underrecorded in IAD (up to 50% in the case of electrocardiograms). Those results call to caution when interpreting IAD estimates. Audits based on medical records, on the local level, and an interpretation taking into account information external to IAD, on the central level, are needed to assess a more comprehensive compliance with standards.

Volume-outcome revisited: The effect of hospital and surgeon volumes on multiple outcome measures in oesophago-gastric cancer surgery

Background

Most studies showing a volume outcome effect in resection surgery for oesophago-gastric cancer were conducted before the centralisation of clinical services. This study evaluated the relation between hospital- and surgeon volume and different risk-adjusted outcomes after oesophago-gastric (OG) cancer surgery in England between 2011 and 2013.

Methods

In data from the National Oesophago-Gastric Cancer Audit from the UK, multivariable random-effects logistic regression models were used to quantify the effect of surgeon and hospital volume on three outcomes: 30-day and 90-day mortality and anastomotic leakage. The models included patient risk factors to adjust for differences in case-mix among hospitals and surgeons. The between-cluster heterogeneity was estimated with the median odds ratio (MOR).

Results

The study included patients treated at 42 hospitals and 329 surgeons. The median (interquartile range) of the annual hospital and surgeon volumes were 110 patients (82 to 137) and 13 patients (8 to 19), respectively. The overall rates for 30-day and 90-day mortality were 2.3% and 4.4% respectively, and the anastomotic leakage was 6.3%. Higher hospital volume was associated with lower 30-day mortality (OR: 0.94; 95% CI: 0.91–0.98) and lower anastomotic leakage rates (OR: 0.96; 95% CI: 0.93–0.98) but not 90-day mortality. Higher surgeon volume was only associated with lower anastomotic leakage rates (OR: 0.81; 95% CI: 0.72–0.92). Hospital volume explained a part of the between-hospital variation in 30-day mortality whereas surgeon volume explained part of the between-hospital variation in anastomotic leakage.

Conclusions

In the setting of centralized O-G cancer surgery in England, we could still observe an effect of volume on short-term outcomes. However, the effect is inconsistent, depending on the type of outcome measure under consideration, and much smaller than in previous studies. Efforts to centralise O-G cancer services further should carefully address the effects of both hospital and surgeon volume on the range of outcome measures that are relevant to patients.

Automatic identification of type 2 diabetes, hypertension, ischaemic heart disease, heart failure and their levels of severity from Italian General Practitioners' electronic medical records: a validation study

OBJECTIVES

The Italian project MATRICE aimed to assess how well cases of type 2 diabetes (T2DM), hypertension, ischaemic heart disease (IHD) and heart failure (HF) and their levels of severity can be automatically extracted from the Health Search/CSD Longitudinal Patient Database (HSD). From the medical records of the general practitioners (GP) who volunteered to participate, cases were extracted by algorithms based on diagnosis codes, keywords, drug prescriptions and results of diagnostic tests. A random sample of identified cases was validated by interviewing their GPs.

SETTING

HSD is a database of primary care medical records. A panel of 12 GPs participated in this validation study.

PARTICIPANTS

300 patients were sampled for each disease, except for HF, where 243 patients were assessed.

OUTCOME MEASURES

The positive predictive value (PPV) was assessed for the presence/absence of each condition against the GP's response to the questionnaire, and Cohen's κ was calculated for agreement on the severity level.

RESULTS

The PPV was 100% (99% to 100%) for T2DM and hypertension, 98% (96% to 100%) for IHD and 55% (49% to 61%) for HF. Cohen's kappa for agreement on the severity level was 0.70 for T2DM and 0.69 for hypertension and IHD.

CONCLUSIONS

This study shows that individuals with T2DM, hypertension or IHD can be validly identified in HSD by automated identification algorithms. Automatic queries for levels of severity of the same diseases compare well with the corresponding clinical definitions, but some misclassification occurs. For HF, further research is needed to refine the current algorithm.

Testing the construct validity of hospital care quality indicators: a case study on hip replacement

Background

Quality indicators are increasingly used to measure the quality of care and compare quality across hospitals. In the Netherlands over the past few years numerous hospital quality indicators have been developed and reported. Dutch indicators are mainly based on expert consensus and face validity and little is known about their construct validity. Therefore, we aim to study the construct validity of a set of national hospital quality indicators for hip replacements.

Methods

We used the scores of 100 Dutch hospitals on national hospital quality indicators looking at care delivered over a two year period. We assessed construct validity by relating structure, process and outcome indicators using chi-square statistics, bootstrapped Spearman correlations, and independent sample t-tests. We studied indicators that are expected to associate as they measure the same clinical construct.

Result

Among the 28 hypothesized correlations, three associations were significant in the direction hypothesized. Hospitals with low scores on wound infections had high scores on scheduling postoperative appointments (p-value = 0.001) and high scores on not transfusing homologous blood (correlation coefficient = -0.28; p-value = 0.05). Hospitals with high scores on scheduling complication meetings, also had high scores on providing thrombosis prophylaxis (correlation coefficient = 0.21; p-value = 0.04).

Conclusion

Despite the face validity of hospital quality indicators for hip replacement, construct validity seems to be limited. Although the individual indicators might be valid and actionable, drawing overall conclusions based on the whole indicator set should be done carefully, as construct validity could not be established. The factors that may explain the lack of construct validity are poor data quality, no adjustment for case-mix and statistical uncertainty.

Data Extraction and Management in Networks of Observational Health Care Databases for Scientific Research: A Comparison of EU-ADR, OMOP, Mini-Sentinel and MATRICE Strategies

Introduction:

We see increased use of existing observational data in order to achieve fast and transparent production of empirical evidence in health care research. Multiple databases are often used to increase power, to assess rare exposures or outcomes, or to study diverse populations. For privacy and sociological reasons, original data on individual subjects can’t be shared, requiring a distributed network approach where data processing is performed prior to data sharing.

Case Descriptions and Variation Among Sites:

We created a conceptual framework distinguishing three steps in local data processing: (1) data reorganization into a data structure common across the network; (2) derivation of study variables not present in original data; and (3) application of study design to transform longitudinal data into aggregated data sets for statistical analysis. We applied this framework to four case studies to identify similarities and differences in the United States and Europe: Exploring and Understanding Adverse Drug Reactions by Integrative Mining of Clinical Records and Biomedical Knowledge (EU-ADR), Observational Medical Outcomes Partnership (OMOP), the Food and Drug Administration’s (FDA’s) Mini-Sentinel, and the Italian network—the Integration of Content Management Information on the Territory of Patients with Complex Diseases or with Chronic Conditions (MATRICE).

Findings:

National networks (OMOP, Mini-Sentinel, MATRICE) all adopted shared procedures for local data reorganization. The multinational EU-ADR network needed locally defined procedures to reorganize its heterogeneous data into a common structure. Derivation of new data elements was centrally defined in all networks but the procedure was not shared in EU-ADR. Application of study design was a common and shared procedure in all the case studies. Computer procedures were embodied in different programming languages, including SAS, R, SQL, Java, and C++.

Conclusion:

Using our conceptual framework we found several areas that would benefit from research to identify optimal standards for production of empirical knowledge from existing databases.an opportunity to advance evidence-based care management. In addition, formalized CM outcomes assessment methodologies will enable us to compare CM effectiveness across health delivery settings.

Appropriateness of Prescriptions of Recommended Treatments in Organisation for Economic Co-operation and Development Health Systems: Findings Based on the Long-Term Registry of the European Society of Cardiology on Heart Failure

OBJECTIVE

This observational study aimed to identify clinical variables and health system characteristics associated with incomplete guideline application in drug treatment of patients with chronic heart failure (HF) across 15 countries.

METHODS

Three data sets were used: European Society of Cardiology Heart Failure Registry, Organisation for Economic Co-operation and Development's Health System Characteristics Survey, and Organisation for Economic Co-operation and Development Health Statistics 2013. Patient and country variables were examined by multilevel, multiple logistic regression. The study population consisted of ambulatory patients with chronic HF and reduced ejection fraction. Inappropriateness of prescription of pharmacological treatments was defined as patients not prescribed at least one of the two recommended treatments (angiotensin-converting enzyme inhibitors/angiotensin-receptor blockers and beta-blockers) or treated with both medications but at suboptimal dosage and in absence of documented contraindication/intolerance.

RESULTS

Of 4605 patients, 1097 (23.8%) received inappropriate drug prescriptions with a large variation within and across countries, with 18.5% of the total variability accounted for by between-country health structure characteristics. Patient-level characteristics such as having mitral regurgitation (odds ratio 1.4; 95% confidence interval 1.1-1.7) was significantly associated with inappropriate prescription of recommended drugs, whereas chronic obstructive pulmonary disease (odds ratio 0.7; 95% confidence interval 0.5-0.9) was associated with more appropriate prescriptions. Among the country-level variables, incentives or obligation to comply with guidelines increased the probability of prescription appropriateness.

CONCLUSIONS

Combining clinical variables with health system characteristics is a promising exercise to explain the appropriateness of recommended drug prescriptions. Such an understanding can help decision makers to design more effective policies to improve adherence to guidelines, improve health care outcomes, and potentially reduce costs.

Deepening our understanding of quality improvement in Europe (DUQuE): overview of a study of hospital quality management in seven countries

Introduction and Objective

This paper provides an overview of the DUQuE (Deepening our Understanding of Quality Improvement in Europe) project, the first study across multiple countries of the European Union (EU) to assess relationships between quality management and patient outcomes at EU level. The paper describes the conceptual framework and methods applied, highlighting the novel features of this study.

Design

DUQuE was designed as a multi-level cross-sectional study with data collection at hospital, pathway, professional and patient level in eight countries.

Setting and Participants

We aimed to collect data for the assessment of hospital-wide constructs from up to 30 randomly selected hospitals in each country, and additional data at pathway and patient level in 12 of these 30.

Main outcome measures

A comprehensive conceptual framework was developed to account for the multiple levels that influence hospital performance and patient outcomes. We assessed hospital-specific constructs (organizational culture and professional involvement), clinical pathway constructs (the organization of care processes for acute myocardial infarction, stroke, hip fracture and deliveries), patient-specific processes and outcomes (clinical effectiveness, patient safety and patient experience) and external constructs that could modify hospital quality (external assessment and perceived external pressure).

Results

Data was gathered from 188 hospitals in 7 participating countries. The overall participation and response rate were between 75% and 100% for the assessed measures.

Conclusions

This is the first study assessing relation between quality management and patient outcomes at EU level. The study involved a large number of respondents and achieved high response rates. This work will serve to develop guidance in how to assess quality management and makes recommendations on the best ways to improve quality in healthcare for hospital stakeholders, payers, researchers, and policy makers throughout the EU.

The effect of certification and accreditation on quality management in 4 clinical services in 73 European hospitals

OBJECTIVE

To investigate the relationship between ISO 9001 certification, healthcare accreditation and quality management in European hospitals.

DESIGN

A mixed method multi-level cross-sectional design in seven countries. External teams assessed clinical services on the use of quality management systems, illustrated by four clinical pathways.

SETTING AND PARTICIPANTS

Seventy-three acute care hospitals with a total of 291 services managing acute myocardial infarction (AMI), hip fracture, stroke and obstetric deliveries, in Czech Republic, France, Germany, Poland, Portugal, Spain and Turkey.

MAIN OUTCOME MEASURE

Four composite measures of quality and safety [specialized expertise and responsibility (SER), evidence-based organization of pathways (EBOP), patient safety strategies (PSS) and clinical review (CR)] applied to four pathways.

RESULTS

Accreditation in isolation showed benefits in AMI and stroke more than in deliveries and hip fracture; the greatest significant association was with CR in stroke. Certification in isolation showed little benefit in AMI but had more positive association with the other conditions; greatest significant association was in PSS with stroke. The combination of accreditation and certification showed least benefit in EBOP, but significant benefits in SER (AMI), in PSS (AMI, hip fracture and stroke) and in CR (AMI and stroke).

CONCLUSIONS

Accreditation and certification are positively associated with clinical leadership, systems for patient safety and clinical review, but not with clinical practice. Both systems promote structures and processes, which support patient safety and clinical organization but have limited effect on the delivery of evidence-based patient care. Further analysis of DUQuE data will explore the association of certification and accreditation with clinical outcomes.

Feasibility of using administrative data to compare hospital performance in the EU

OBJECTIVE

To describe hospitals' organizational arrangements relevant to the abstraction of administrative data, to report on the completeness of administrative data collected and to assess associations between organizational arrangements and completeness of data submission.

DESIGN

A cross-sectional

STUDY DESIGN

utilizing administrative data.

SETTING AND PARTICIPANTS

Randomly selected hospitals from seven European countries (The Czech Republic, France, Germany, Poland, Portugal, Spain, and Turkey).

MAIN OUTCOME MEASURES

Completeness of data submission for four quality indicators: mortality after acute myocardial infarction, stroke and hip fractures and complications after normal delivery.

RESULTS

In general, hospitals were able to produce data on the four indicators required for this research study. A substantial proportion had missing data on one or more data items. The proportion of hospitals that was able to produce more detailed indicators of relevance for quality monitoring and improvement was low and ranged from 40.1% for thrombolysis performed on patients with acute ischemic stroke to 63.8% for hip-fracture operations performed within 48 h after admission for patients aged 65 or older. National factors were strong predictors of data completeness on the studied indicators.

CONCLUSIONS

At present, hospital administrative databases do not seem to be an appropriate source of information for comparison of hospital performance across the countries of the EU. However, given that this is a dynamic field, changes to administrative databases may make this possible in the near future. Such changes could be accelerated by an in-depth comparative analysis of the issues of using administrative data for comparisons of hospital performances in EU countries.

Measuring and reporting on quality of care and patient safety in Canada: focusing on what matters

Quality of Healthcare in Canada: A Chartbook is a comprehensive, useful reference that organizes the multiple measures it contains around a well-established framework. It documents Canada's performance relative to several comparator countries and over time on a large number of indicators of care quality. However, the chartbook does not identify where the gaps are in the measurement of quality of care and patient safety, and it is limited in its ability to represent the measurement required to monitor current and upcoming policy initiatives in the field of healthcare quality. Further, it fails to represent fully the perspective of the patient or to incorporate the progress made in measuring patient-centred quality of care. The authors propose four ways forward for strengthening the measurement of healthcare quality and patient safety in Canada: (1) the standardized collection of patient-reported outcome measures; (2) a focus on the standardized measurement of patient safety across the country at micro-, meso- and macro-levels; (3) the measurement of multiple morbidities and of the quality of care provided to patients with multiple chronic conditions; and (4) a design of measurement systems in ways that reflect the perspectives of patients and citizens.

A systematic approach for uptake of evidence on sex-specific issues in guidelines--a pilot study

RATIONALE, AIMS AND OBJECTIVES

Increasing evidence indicates that sex-specific issues may have impact on prevention, diagnosis, or treatment. These issues are not systematically considered during the development of Dutch clinical practice guidelines. The aim of this study is to identify how members of guideline development groups discuss sex-specific evidence, and whether and how the outcomes of these discussions are reflected in the guideline.

METHODS

Six guideline development committees (GDCs) were studied. Each committee was supported by a staff member from the guideline organization who was trained and received feedback to facilitate uptake of evidence on sex differences in the process of guideline development. Non-participant observation and transcription of audio recordings from 22 GDC meetings were performed. Content analysis of meeting transcripts and guidelines were studied to analyse characteristics of discussion episodes on sex-specific research data-based issues (subject matter, initiator and group approach towards the topic and themes) and whether or not conclusions on evidence were reflected in the final guideline text.

RESULTS

Of the 87 identified discussion episodes, 68 dealt with sex-specific research evidence potentially relevant to guidelines. Respectively 51%, 28% and 21% of the latter episodes were initiated by committee members, staff members and chairpersons. Group approaches towards the subject matter were generally positive. Data from 60% of those episodes were reflected in the final guideline text. Sex-specific data on reproductive issues were more often discussed and reflected in guideline texts than data on other health issues. Discussion episodes on sex-specific evidence initiated by chairpersons were most often reflected in the guidelines.

CONCLUSIONS

This pilot study indicates that GDCs regularly focused on sex-specific issues. The participation of a trained staff member contributed to this.

Quality legislation: lessons for Ontario from abroad

While the Excellent Care for All Act, 2010 (ECFA Act) provides a comprehensive approach to stimulating quality improvement in healthcare, there are other examples of legislations articulating strategies aimed at the same goal but proposing different approaches. This paper reviews quality of care legislations in the Netherlands, the United States, England and Australia, compares those pieces of legislation with the ECFA Act and suggests lessons for Ontario in planning the next stages of its healthcare quality strategy. Notable among the commonalities that the EFCA Act shares with the selected examples of legislation are mandatory reporting of performance results at an organizational level and furthering quality improvement, evidence generation and performance monitoring. However, the EFCA Act does not include any elements of restructuring or competition, unlike some of the other examples. Key to successful transformation of the Ontario healthcare system will be to propose a package of changes that will deal systematically with all aspects of transformation sought (including structural changes, payments systems and elements of competition), will garner support from all the actors, and will be implemented consistently and persistently. Benchmarking on the implementation and impact of reforms with the countries presented in this paper may be an additional important step. Quality of care is a key focus of health system reforms, and in recent years many countries in the Organisation of Economic Co-operation and Development (OECD), including Canada, have developed strategies aimed at improving healthcare quality and patient safety (OECD 2010). Øvretveit and Klazinga propose that national strategies for quality of care can be targeted at different types of health system stakeholders: professionals, healthcare organizations, medical products and technologies, patients and financers (World Health Organization Regional Office for Europe 2008). The generic elements of these strategies relate to legislation and regulation, monitoring and measurement; assuring and improving the quality and safety of individual healthcare services, and assuring and improving the quality of the healthcare system as a whole. Various combinations of quality improvement approaches (such as quality assessment, standards-based quality management, team problem solving, and patient and community participation) are suitable for these functions as part of the respective quality strategies.