The multiple causal pathways between performance measures' use and effects

The elicitation of patient and physician preferences for calculating consumer-based composite measures on hospital report cards: results of two discrete choice experiments

PURPOSE

The calculation of aggregated composite measures is a widely used strategy to reduce the amount of data on hospital report cards. Therefore, this study aims to elicit and compare preferences of both patients as well as referring physicians regarding publicly available hospital quality information METHODS: Based on systematic literature reviews as well as qualitative analysis, two discrete choice experiments (DCEs) were applied to elicit patients' and referring physicians' preferences. The DCEs were conducted using a fractional factorial design. Statistical data analysis was performed using multinomial logit models RESULTS: Apart from five identical attributes, one specific attribute was identified for each study group, respectively. Overall, 322 patients (mean age 68.99) and 187 referring physicians (mean age 53.60) were included. Our models displayed significant coefficients for all attributes (p < 0.001 each). Among patients, "Postoperative complication rate" (20.6%; level range of 1.164) was rated highest, followed by "Mobility at hospital discharge" (19.9%; level range of 1.127), and ''The number of cases treated" (18.5%; level range of 1.045). In contrast, referring physicians valued most the ''One-year revision surgery rate'' (30.4%; level range of 1.989), followed by "The number of cases treated" (21.0%; level range of 1.372), and "Postoperative complication rate" (17.2%; level range of 1.123) CONCLUSION: We determined considerable differences between both study groups when calculating the relative value of publicly available hospital quality information. This may have an impact when calculating aggregated composite measures based on consumer-based weighting.

Patient experience and nurse staffing level in South Korea

Patient experience has recently become a key driver for hospital quality improvement in South Korea, marked by the introduction of the Patient Experience Assessment (PXA) within its National Health Insurance in 2017. While the PXA has garnered special attention from the media and hospitals, there has been a lack of focus on its structural determinants, hindering continuous and sustained improvement in patient experience. Given the relatively low number of practicing nurses per 1000 population in South Korea and the significant variation in nurse staffing levels across hospitals, the staffing level of nurses in hospitals could be a crucial structural determinant of patient experience. This study examines the association between patient experience and hospital nurse staffing levels in South Korea. We used individual- and hospital-level data from the 2019 PXA, encompassing 7250 patients from 42 tertiary hospitals and 16 235 patients from 109 non-tertiary general hospitals with 300 or more beds. The dependent variables were derived from the complete set of 21 proper questions on patient experience in the Nurse and other domains. The main explanatory variable was the hospital-level Nurse Staffing Grade (NSG), employed by the National Health Insurance to adjust reimbursement to hospitals. Multilevel ordered/binomial logistic or linear regression was conducted accounting for other hospital- and patient-level characteristics as well as acknowledging the nested nature of the data. A clear, positive association was observed between patient experience in the Nurse domain and NSG, even after accounting for other characteristics. For example, the predicted probability of reporting the top-box category of "Always" to the question "How often did nurses treat you with courtesy and respect?" was 70.3% among patients from non-tertiary general hospitals with the highest NSG, compared to 63.1% among patients from their peer hospitals with the lowest NSG. Patient experience measured in other domains that were likely to be affected by nurse staffing levels also showed similar associations, although generally weaker and less consistent than in the Nurse domain. Better patient experience was associated with higher hospital nurse staffing levels in South Korea. Alongside current initiatives focused on measuring and publicly reporting patient experience, strengthening nursing and other hospital workforce should also be included in policy efforts to improve patient experience.

Referring physicians' intention to use hospital report cards for hospital referral purposes in the presence or absence of patient-reported outcomes: a randomized trial

PURPOSE

This study aims to determine the intention to use hospital report cards (HRCs) for hospital referral purposes in the presence or absence of patient-reported outcomes (PROs) as well as to explore the relevance of publicly available hospital performance information from the perspective of referring physicians.

METHODS

We identified the most relevant information for hospital referral purposes based on a literature review and qualitative research. Primary survey data were collected (May-June 2021) on a sample of 591 referring orthopedists in Germany and analyzed using structural equation modeling. Participating orthopedists were recruited using a sequential mixed-mode strategy and randomly allocated to work with HRCs in the presence (intervention) or absence (control) of PROs.

RESULTS

Overall, 420 orthopedists (mean age 53.48, SD 8.04) were included in the analysis. The presence of PROs on HRCs was not associated with an increased intention to use HRCs (p = 0.316). Performance expectancy was shown to be the most important determinant for using HRCs (path coefficient: 0.387, p < .001). However, referring physicians have doubts as to whether HRCs can help them. We identified "complication rate" and "the number of cases treated" as most important for the hospital referral decision making; PROs were rated slightly less important.

CONCLUSIONS

This study underpins the purpose of HRCs, namely to support referring physicians in searching for a hospital. Nevertheless, only a minority would support the use of HRCs for the next hospital search in its current form. We showed that presenting relevant information on HRCs did not increase their use intention.

Public Reporting of Performance Indicators in Long-Term Care in Canada: Does it Make a Difference?

Evidence of the impact of public reporting of health care performance on quality improvement is not yet sufficient for definitive conclusions to be drawn, despite the important policy implications. This study explored the association of public reporting of performance indicators of long-term care facilities in Canada with performance trends. We considered 16 performance indicators in long-term care in Canada, 8 of which are publicly reported at a facility level, whereas the other 8 are not publicly reported, between the fiscal years 2011-2012 and 2018-2019. Data from 1,087 long-term care facilities were included. Improving trends were observed among publicly reported indicators more often than among indicators that were not publicly reported. Our analysis also suggests that the association between publication of data and improvement is stronger among indicators for which there was no improvement prior to publication and among the worst performing facilities.

Characterising performance information use in the primary healthcare systems of El Salvador, Lebanon and Malawi: multiple qualitative case study protocol

INTRODUCTION

Governments in low-income and middle-income countries (LMICs) and official development assistance agencies use a variety of performance measurement and management approaches to improve the performance of healthcare systems. The effectiveness of such approaches is contingent on the extent to which managers and care providers use performance information. To date, major knowledge gaps exist about the contextual factors that contribute, or not, to performance information use by primary healthcare (PHC) decision-makers in LMICs. This study will address three research questions: (1) How do decision-makers use performance information, and for what purposes? (2) What are the contextual factors that influence the use or non-use of performance information? and (3) What are the proximal outcomes reported by PHC decision-makers from performance information use?

METHODS AND ANALYSIS

We present the protocol of a theory-driven, qualitative study with a multiple case study design to be conducted in El Salvador, Lebanon and Malawi.Data sources include semi structured in-depth interviews and document review. Interviews will be conducted with approximately 60 respondents including PHC system decision-makers and providers. We follow an interdisciplinary theoretical framework that draws on health policy and systems research, public administration, organisational science and health service research. Data will be analysed using thematic analysis to explore how respondents use performance information or not, and for what purposes as well as barriers and facilitators of use.

ETHICS AND DISSEMINATION

The ethical boards of the participating universities approved the protocol presented here. Study results will be disseminated through peer-reviewed journals and global health conferences.

Determinants of inpatient satisfaction: evidence from Switzerland

PURPOSE

A common way to handle quality problems and increasing costs of modern health care systems is more transparency through public reporting. Thereby, patient satisfaction is seen as one main reported outcome. Previous studies proposed several associated factors. Only a few of them included organizational determinants with potential to inform the health care provider's management. Therefore, the aim of this study was to investigate the influence of organizational contingency factors on patient satisfaction.

DESIGN/METHODOLOGY/APPROACH

As a case, Switzerland's inpatient rehabilitation sector was used. Therein, a cross-sectional study of public released secondary data with an exploratory multiple linear regression (MLR) modeling approach was conducted.

FINDINGS

Five significant influencing factors on patient satisfaction were found. They declared 42.2% of the variance in satisfaction on provider level. The organizations' supplementary insured patients, staff payment, outpatients, extracantonal patients and permanent resident population revealed significant correlations with patient satisfaction.

RESEARCH LIMITATIONS/IMPLICATIONS

Drawing on publicly available cross-sectional data, statistically no causality can be proved. However, integration of routine data and organization theory can be useful for further studies.

PRACTICAL IMPLICATIONS

Regarding inpatient satisfaction, improvement levers for providers' managers are as follow: first, service provision should be customized to patients' needs, expectations and context; second, employees' salary should be adequate to prevent dissatisfaction; third, the main business should be prioritized to avoid frittering.

ORIGINALITY/VALUE

Former studies regarding public reporting are often atheoretical and rarely used organizational variables as determinants for relevant outcomes. Therefore, uniformed data are useful.

Effects of the CMS' Public Reporting Program for Inpatient Psychiatric Facilities on Targeted and Nontargeted Safety: Differences Between For-Profits and Nonprofits

The Centers for Medicare and Medicaid Services implemented the Inpatient Psychiatric Facility Quality Reporting Program in 2012, which publicly reports facilities' performance on restraint and seclusion (R-S) measures. Using data from Massachusetts, we examined whether nonprofits and for-profits responded differently to the program on targeted indicators, and if the program had a differential spillover effect on nontargeted indicators of quality by ownership. Episodes of R-S (targeted), complaints (nontargeted), and discharges were obtained for 2008-2017 through public records requests to the Commonwealth of Massachusetts. Using difference-in-differences estimators, we found no differential changes in R-S between for-profits and nonprofits. However, for-profits had larger increases in overall complaints, safety-related complaints, abuse-related complaints, and R-S-related complaints compared with nonprofits. This is the first study to examine the effects of a national public reporting program among psychiatric facilities on nontargeted measures. Researchers and policymakers should further scrutinize intended and unintended consequences of performance-reporting programs.

Impact of primary care funding on patient satisfaction: a retrospective longitudinal study of English general practice, 2013-2016

BACKGROUND

Providing high-quality clinical care and good patient experience are priorities for most healthcare systems.

AIM

To understand the relationship between general practice funding and patient-reported experience.

DESIGN AND SETTING

Retrospective longitudinal study of English general practice-level data for the financial years 2013-2014 to 2016-2017.

METHOD

Data for all general practices in England from the General and Personal Medical Services database were linked to patient experience data from the GP Patient Survey (GPPS). Panel data multivariate regression was used to estimate the impact of general practice funding (current or lagged 1 year) per patient on GPPS-reported patient experience of access, continuity of care, professionalism, and overall satisfaction. Confounding was controlled for by practice, demographic, and GPPS responder characteristics, and for year effects.

RESULTS

Inflation-adjusted mean total annual funding per patient was £133.66 (standard deviation [SD] = £39.46). In all models, higher funding was associated with better patient experience. In the model with lagged funding and practice fixed effects (model 6), a 1 SD increase in funding was associated with increases in scores in the domains of access (1.18%; 95% confidence interval [CI] = 0.89 to 1.47), continuity (0.86%; 95% CI = 0.19 to 1.52), professionalism of GP (0.47%; 95% CI = 0.22 to 0.71), professionalism of nurse (0.51%; 95% CI = 0.24 to 0.77), professionalism of receptionist (0.51%; 95% CI = 0.24 to 0.78), and in overall satisfaction (0.88%; 95% CI = 0.52 to 1.24).

CONCLUSION

Better-funded general practices were more likely to have higher reported patient experience ratings across a wide range of domains.

The Effect of Centers for Medicare and Medicaid's Inpatient Psychiatric Facility Quality Reporting Program on the Use of Restraint and Seclusion

BACKGROUND

Patients in inpatient psychiatry settings are uniquely vulnerable to harm. As sources of harm, research and policy efforts have specifically focused on minimizing and eliminating restraint and seclusion. The Centers for Medicare and Medicaid's Inpatient Psychiatric Facility Quality Reporting (IPFQR) program attempts to systematically measure and reduce restraint and seclusion. We evaluated facilities' response to the IPFQR program and differences by ownership, hypothesizing that facilities reporting these measures for the first time will show a greater reduction and that ownership will moderate this effect.

METHODS

Using a difference-in-differences design and exploiting variation among facilities that previously reported on these measures to The Joint Commission, we examined the effect of the IPFQR public reporting program on the use and duration of restraint and seclusion from the end of 2012 through 2017.

RESULTS

There were a total of 9705 observations of facilities among 1841 unique facilities. Results suggest the IPFQR program reduced duration of restraint by 48.96% [95% confidence interval (95% CI), 16.69%-68.73%] and seclusion by 53.54% (95% CI, 19.71%-73.12%). There was no change in odds of zero restraint and, among for-profits only, a decrease of 36.89% (95% CI, 9.32%-56.07%) in the odds of zero seclusion.

CONCLUSIONS

This is the first examination of the effect of the IPFQR program on restraint and seclusion, suggesting the program was successful in reducing their use. We did not find support for ownership moderating this effect. Additional research is needed to understand mechanisms of response and the impact of the program on nontargeted aspects of quality.

Public reporting of hospital performance data: views of senior medical directors in Victoria, Australia

Objective The aim of the present study was to better understand senior medical directors' perceptions of public reporting of hospital performance data, how public reporting affects institutional behavioural change towards quality improvement and how it could be improved. Methods Interviews were undertaken with 17 medical directors representing 26 metropolitan and regional public hospitals in Victoria, Australia, between June and August 2016. Data were analysed thematically. Results Medical directors are well placed to comment on clinical and administrative aspects of quality, safety and performance monitoring in public hospitals. Their responses largely suggested that public reporting of hospital performance data in Australia is immature and not fulfilling its potential. There was little consensus among informants around what public reporting is, who it is for or its purpose. Although public reporting was considered to have important functions for hospitals and consumers, it was generally considered to lack robustness and have underutilised potential to inform consumers, build trust and drive quality and performance improvements within hospitals. Conclusions The next steps needed to advance public reporting of hospital performance data in Australia include engaging clinicians and patients in selection and development of metrics, improving timeliness of reporting, and improving communication of information so that it is accessible and meaningful for different audiences. What is known about the topic? Public reporting of hospital performance data is a mechanism increasingly used in the Australian health system, but it has attracted little research. What does this paper add? This paper reveals a lack of shared understanding among medical directors in Victoria, Australia, on what public reporting of hospital performance data is, who it is for and its purpose. The paper highlights the potential importance of public reporting of hospital performance data for rural and regional healthcare consumers and how it may be strengthened. What are the implications for practitioners? Stronger systems of public reporting of hospital performance data have the potential to increase consumer engagement and improve hospital performance, quality and safety. Awareness of the discourse around public reporting of hospital performance data can increase practitioners' engagement in debate and development of reporting systems.

"What is meant by public?": Stakeholder views on strengthening impacts of public reporting of hospital performance data

Public reporting of hospital performance data is a developing area that is gaining increased attention. This is the first study to explore a range of stakeholder opinions on how such public reporting could be strengthened in Australia. Thirty-four semi-structured interviews were conducted with a purposive sample of expert healthcare consumer, provider and purchaser informants who worked in a variety of senior roles and had knowledge of or involvement in public reporting of hospital data within the public or private healthcare sectors. Informants from all Australian states, territory and national jurisdictions participated. Thematic analysis was used to gain an overview of experts' opinions to inform policy and systems-development for strengthening foundational frameworks for public reporting of health services performance. Themes arising were synthesised to generate explanatory figures to highlight key areas for strengthening public reporting. Our findings suggest that in Australia there is a lack of agreement on what the objectives and who the audience are for public reporting of hospital performance data. Without this shared understanding it is difficult to strengthen frameworks and impacts of public reporting. When developing frameworks for public reporting of hospital data in Australia, more explicit definition of what or who are the 'public' is needed along with identification of barriers, desired impacts, data needs, and data collection/reporting/feedback mechanisms. All relevant stakeholders should be involved in design of public reporting frameworks. Offering multiple systems of public reporting, each tailored to particular audiences, might enable greater impact of reporting towards improved hospital quality and safety, and consumer knowledge to inform treatment decisions. This study provides an overview of perspectives, but further research is warranted to develop PR frameworks that can generate greatest impacts for the needs of various audiences.

The German Quality Network Sepsis: study protocol for the evaluation of a quality collaborative on decreasing sepsis-related mortality in a quasi-experimental difference-in-differences design

BACKGROUND

While sepsis-related mortality decreased substantially in other developed countries, mortality of severe sepsis remained as high as 44% in Germany. A recent German cluster randomized trial was not able to improve guideline adherence and decrease sepsis-related mortality within the participating hospitals, partly based on lacking support by hospital management and lacking resources for documentation of prospective data. Thus, more pragmatic approaches are needed to improve quality of sepsis care in Germany. The primary objective of the study is to decrease sepsis-related hospital mortality within a quality collaborative relying on claims data.

METHOD

The German Quality Network Sepsis (GQNS) is a quality collaborative involving 75 hospitals. This study protocol describes the conduction and evaluation of the start-up period of the GQNS running from March 2016 to August 2018. Democratic structures assure participatory action, a study coordination bureau provides central support and resources, and local interdisciplinary quality improvement teams implement changes within the participating hospitals. Quarterly quality reports focusing on risk-adjusted hospital mortality in cases with sepsis based on claims data are provided. Hospitals committed to publish their individual risk-adjusted mortality compared to the German average. A complex risk-model is used to control for differences in patient-related risk factors. Hospitals are encouraged to implement a bundle of interventions, e.g., interdisciplinary case analyses, external peer-reviews, hospital-wide staff education, and implementation of rapid response teams. The effectiveness of the GQNS is evaluated in a quasi-experimental difference-in-differences design by comparing the change of hospital mortality of cases with sepsis with organ dysfunction from a retrospective baseline period (January 2014 to December 2015) and the intervention period (April 2016 to March 2018) between the participating hospitals and all other German hospitals. Structural and process quality indicators of sepsis care as well as efforts for quality improvement are monitored regularly.

DISCUSSION

The GQNS is a large-scale quality collaborative using a pragmatic approach based on claims data. A complex risk-adjustment model allows valid quality comparisons between hospitals and with the German average. If this study finds the approach to be useful for improving quality of sepsis care, it may also be applied to other diseases.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02820675.

How do aggregated patient-reported outcome measures data stimulate health care improvement? A realist synthesis

Objectives Internationally, there has been considerable debate about the role of data in supporting quality improvement in health care. Our objective was to understand how, why and in what circumstances the feedback of aggregated patient-reported outcome measures data improved patient care. Methods We conducted a realist synthesis. We identified three main programme theories underlying the use of patient-reported outcome measures as a quality improvement strategy and expressed them as nine 'if then' propositions. We identified international evidence to test these propositions through searches of electronic databases and citation tracking, and supplemented our synthesis with evidence from similar forms of performance data. We synthesized this evidence through comparing the mechanisms and impact of patient-reported outcome measures and other performance data on quality improvement in different contexts. Results Three programme theories were identified: supporting patient choice, improving accountability and enabling providers to compare their performance with others. Relevant contextual factors were extent of public disclosure, use of financial incentives, perceived credibility of the data and the practicality of the results. Available evidence suggests that patients or their agents rarely use any published performance data when selecting a provider. The perceived motivation behind public reporting is an important determinant of how providers respond. When clinicians perceived that performance indicators were not credible but were incentivized to collect them, gaming or manipulation of data occurred. Outcome data do not provide information on the cause of poor care: providers needed to integrate and interpret patient-reported outcome measures and other outcome data in the context of other data. Lack of timeliness of performance data constrains their impact. Conclusions Although there is only limited research evidence to support some widely held theories of how aggregated patient-reported outcome measures data stimulate quality improvement, several lessons emerge from interventions sharing the same programme theories to help guide the increasing use of these measures.

Performance associated effect variations of public reporting in promoting antibiotic prescribing practice: a cluster randomized-controlled trial in primary healthcare settings

Aim To evaluate the variations in effect of public reporting in antibiotic prescribing practice among physicians with different performance in primary healthcare settings.

BACKGROUND

Overprovision of antibiotics is a major public health concern. Public reporting has been adopted to encourage good antibiotic prescribing practices. However, which group, for instance, high, average or low antibiotic prescribers, accounted for antibiotic prescription reduction has not been fully understood.

METHODS

A cluster randomized-controlled trial was conducted. In total, 20 primary healthcare institutions in Qianjiang city were paired through a six indicators-synthesized score. Coin flipping was used to assign control-intervention status; 10 were then subjected to intervention where prescription indicators were publicly reported monthly over a one-year period. Prescriptions for upper respiratory tract infections (URTIs) before and after the intervention were collected. Physicians were divided into high, average and low antibiotic prescribers based on their antibiotic prescribing rates last month, which were publicly reported in intervention arm. Multilevel difference-in-differences logit regressions were performed to estimate intervention effect in each physician group on three outcome indicators: prescriptions containing antibiotics, two or more antibiotics and antibiotic injections. Findings In total, 31 460 URTI prescriptions were collected (16 170 in intervention arm and 15 290 in control arm). Reduction in antibiotic prescription attributed to intervention was 2.82% [95% confidence intervals (CI): -4.09, -1.54%, P<0.001], least significant in low prescribers (-1.41%, 95% CI: -3.81, 0.99%, P=0.249) and most significant in average prescribers (-5.01%, 95% CI: -6.94, -3.07%, P<0.001). Reduction in combined antibiotics prescriptions attributed to intervention was 3.81% (95% CI: -5.23, -2.39%, P<0.001), least significant in low prescribers (-2.42%, 95% CI: -4.39, -0.45%, P=0.016) and most significant in average prescribers (-5.01%, 95% CI: -7.47, -2.56%, P<0.001).

CONCLUSION

Public reporting can positively influence antibiotic prescribing patterns of physicians for URTIs in primary care settings, with reduction in antibiotic and combined antibiotic prescriptions. The reduction was mainly attributed to average and high antibiotic prescribers.

Impact of State Reporting Laws on Central Line-Associated Bloodstream Infection Rates in U.S. Adult Intensive Care Units

OBJECTIVE

To examine the effect of mandated state health care-associated infection (HAI) reporting laws on central line-associated bloodstream infection (CLABSI) rates in adult intensive care units (ICUs).

DATA SOURCES

We analyzed 2006-2012 adult ICU CLABSI and hospital annual survey data from the National Healthcare Safety Network. The final analytic sample included 244 hospitals, 947 hospital years, 475 ICUs, 1,902 ICU years, and 16,996 ICU months.

STUDY DESIGN

We used a quasi-experimental study design to identify the effect of state mandatory reporting laws. Several secondary models were conducted to explore potential explanations for the plausible effects of HAI laws.

PRINCIPAL FINDINGS

Controlling for the overall time trend, ICUs in states with laws had lower CLABSI rates beginning approximately 6 months prior to the law's effective date (incidence rate ratio = 0.66; p < .001); this effect persisted for more than 6 1/2 years after the law's effective date. These findings were robust in secondary models and are likely to be attributed to changes in central line usage and/or resources dedicated to infection control.

CONCLUSIONS

Our results provide valuable evidence that state reporting requirements for HAIs improved care. Additional studies are needed to further explore why and how mandatory HAI reporting laws decreased CLABSI rates.

Improving patient experience in primary care: a multimethod programme of research on the measurement and improvement of patient experience

Background

There has been an increased focus towards improving quality of care within the NHS in the last 15 years; as part of this, there has been an emphasis on the importance of patient feedback within policy, through National Service Frameworks and the Quality and Outcomes Framework. The development and administration of large-scale national patient surveys to gather representative data on patient experience, such as the national GP Patient Survey in primary care, has been one such initiative. However, it remains unclear how the survey is used by patients and what impact the data may have on practice.

Objectives

Our research aimed to gain insight into how different patients use surveys to record experiences of general practice; how primary care staff respond to feedback; and how to engage primary care staff in responding to feedback.

Methods

We used methods including quantitative survey analyses, focus groups, interviews, an exploratory trial and an experimental vignette study.

Results

(1)Understanding patient experience data. Patients readily criticised their care when reviewing consultations on video, although they were reluctant to be critical when completing questionnaires. When trained raters judged communication during a consultation to be poor, a substantial proportion of patients rated the doctor as ‘good’ or ‘very good’. Absolute scores on questionnaire surveys should be treated with caution; they may present an overoptimistic view of general practitioner (GP) care. However, relative rankings to identify GPs who are better or poorer at communicating may be acceptable, as long as statistically reliable figures are obtained. Most patients have a particular GP whom they prefer to see; however, up to 40% of people who have such a preference are unable regularly to see the doctor of their choice. Users of out-of-hours care reported worse experiences when the service was run by a commercial provider than when it was run by a not-for profit or NHS provider. (2)Understanding patient experience in minority ethnic groups. Asian respondents to the GP Patient Survey tend to be registered with practices with generally low scores, explaining about half of the difference in the poorer reported experiences of South Asian patients than white British patients. We found no evidence that South Asian patients used response scales differently. When viewing the same consultation in an experimental vignette study, South Asian respondents gave higher scores than white British respondents. This suggests that the low scores given by South Asian respondents in patient experience surveys reflect care that is genuinely worse than that experienced by their white British counterparts. We also found that service users of mixed or Asian ethnicity reported lower scores than white respondents when rating out-of-hours services. (3)Using patient experience data. We found that measuring GP–patient communication at practice level masks variation between how good individual doctors are within a practice. In general practices and in out-of-hours centres, staff were sceptical about the value of patient surveys and their ability to support service reconfiguration and quality improvement. In both settings, surveys were deemed necessary but not sufficient. Staff expressed a preference for free-text comments, as these provided more tangible, actionable data. An exploratory trial of real-time feedback (RTF) found that only 2.5% of consulting patients left feedback using touch screens in the waiting room, although more did so when reminded by staff. The representativeness of responding patients remains to be evaluated. Staff were broadly positive about using RTF, and practices valued the ability to include their own questions. Staff benefited from having a facilitated session and protected time to discuss patient feedback.

Conclusions

Our findings demonstrate the importance of patient experience feedback as a means of informing NHS care, and confirm that surveys are a valuable resource for monitoring national trends in quality of care. However, surveys may be insufficient in themselves to fully capture patient feedback, and in practice GPs rarely used the results of surveys for quality improvement. The impact of patient surveys appears to be limited and effort should be invested in making the results of surveys more meaningful to practice staff. There were several limitations of this programme of research. Practice recruitment for our in-hours studies took place in two broad geographical areas, which may not be fully representative of practices nationally. Our focus was on patient experience in primary care; secondary care settings may face different challenges in implementing quality improvement initiatives driven by patient feedback. Recommendations for future research include consideration of alternative feedback methods to better support patients to identify poor care; investigation into the factors driving poorer experiences of communication in South Asian patient groups; further investigation of how best to deliver patient feedback to clinicians to engage them and to foster quality improvement; and further research to support the development and implementation of interventions aiming to improve care when deficiencies in patient experience of care are identified.

Funding

The National Institute for Health Research Programme Grants for Applied Research programme.

Hospital Compare and Hospital Choice: Public Reporting and Hospital Choice by Hip Replacement Patients in Texas

The Centers for Medicare & Medicaid Services publicizes comparative performance data on Hospital Compare, a website maintained to support consumer decision making. Given the agency’s goal, this study investigates the relationship between public reporting and hospital choices of hip replacement patients in Texas. Estimating individual-level valuations of provider characteristics allowing for heterogeneity across patients, we find consumer selections and hospitals’ displayed performance vary together in time. Comparing associations involving public reporting with those associated with more readily observable hospital attributes, we conclude relationships coinciding with release of comparative performance data are modest, but not inconsequential. Our use of an empirical strategy novel for evaluation of public reporting has methodological implications, while the study’s affirmative result is of potential interest to policy makers and administrators.

Investigating the meaning of 'good' or 'very good' patient evaluations of care in English general practice: a mixed methods study

OBJECTIVE

To examine concordance between responses to patient experience survey items evaluating doctors' interpersonal skills, and subsequent patient interview accounts of their experiences of care.

DESIGN

Mixed methods study integrating data from patient questionnaires completed immediately after a video-recorded face-to-face consultation with a general practitioner (GP) and subsequent interviews with the same patients which included playback of the recording.

SETTING

12 general practices in rural, urban and inner city locations in six areas in England.

PARTICIPANTS

50 patients (66% female, aged 19-96 years) consulting face-to-face with 32 participating GPs.

MAIN OUTCOME MEASURES

Positive responses to interpersonal skills items in a postconsultation questionnaire ('good' and 'very good') were compared with experiences reported during subsequent video elicitation interview (categorised as positive, negative or neutral by independent clinical raters) when reviewing that aspect of care.

RESULTS

We extracted 230 textual statements from 50 interview transcripts which related to the evaluation of GPs' interpersonal skills. Raters classified 70.9% (n=163) of these statements as positive, 19.6% (n=45) neutral and 9.6% (n=22) negative. Comments made by individual patients during interviews did not always express the same sentiment as their responses to the questionnaire. Where questionnaire responses indicated that interpersonal skills were 'very good', 84.6% of interview statements concerning that item were classified as positive. However, where patients rated interpersonal skills as 'good', only 41.9% of interview statements were classified as positive, and 18.9% as negative.

CONCLUSIONS

Positive responses on patient experience questionnaires can mask important negative experiences which patients describe in subsequent interviews. The interpretation of absolute patient experience scores in feedback and public reporting should be done with caution, and clinicians should not be complacent following receipt of 'good' feedback. Relative scores are more easily interpretable when used to compare the performance of providers.

Public reporting as a prescriptions quality improvement measure in primary care settings in China: variations in effects associated with diagnoses

The overprovision and irrational use of antibiotics and injections are a major public health concern. Public reporting has been adopted as a strategy to encourage good prescribing practices. This study evaluated the effects of public reporting on antibiotic and injection prescriptions in urban and rural primary care settings in Hubei province, China. A randomized control trial was conducted, with 10 primary care institutions being subject to public reporting and another 10 serving as controls. Prescription indicators were publicly reported monthly over a one-year period. Prescriptions for bronchitis, gastritis and hypertension before and after the intervention were collected. Difference-in-difference tests were performed to estimate the effect size of the intervention on five prescription indicators: percentage of prescriptions containing antibiotics; percentage of prescriptions containing two or more antibiotics; percentage of prescriptions containing injections; percentage of prescriptions containing antibiotic injections; and average prescription cost. Public reporting had varied effects on prescriptions for different diagnoses. It reduced antibiotic prescribing for gastritis. Prescriptions containing injections, especially antibiotic injections, also declined, but only for gastritis. A reduction of prescription costs was noted for bronchitis and gastritis. Public reporting has the potential to encourage good prescribing practices. Its effects vary with different disease conditions.

Are performance indicators used for hospital quality management: a qualitative interview study amongst health professionals and quality managers in The Netherlands

BACKGROUND

Hospitals are under increasing pressure to share indicator-based performance information. These indicators can also serve as a means to promote quality improvement and boost hospital performance. Our aim was to explore hospitals' use of performance indicators for internal quality management activities.

METHODS

We conducted a qualitative interview study among 72 health professionals and quality managers in 14 acute care hospitals in The Netherlands. Concentrating on orthopaedic and oncology departments, our goal was to gain insight into data collection and use of performance indicators for two conditions: knee and hip replacement surgery and breast cancer surgery. The semi-structured interviews were recorded and summarised. Based on the data, themes were synthesised and the analyses were executed systematically by two analysts independently. The findings were validated through comparison.

RESULTS

The hospitals we investigated collect data for performance indicators in different ways. Similarly, these hospitals have different ways of using such data to support their quality management, while some do not seem to use the data for this purpose at all. Factors like 'linking pin champions', pro-active quality managers and engaged medical specialists seem to make a difference. In addition, a comprehensive hospital data infrastructure with electronic patient records and robust data collection software appears to be a prerequisite to produce reliable external performance indicators for internal quality improvement.

CONCLUSIONS

Hospitals often fail to use performance indicators as a means to support internal quality management. Such data, then, are not used to its full potential. Hospitals are recommended to focus their human resource policy on 'linking pin champions', the engagement of professionals and a pro-active quality manager, and to invest in a comprehensive data infrastructure. Furthermore, the differences in data collection processes between Dutch hospitals make it difficult to draw comparisons between outcomes of performance indicators.

Keys to successful implementation of a French national quality indicator in health care organizations: a qualitative study

Background

Several countries have launched public reporting systems based on quality indicators (QIs) to increase transparency and improve quality in health care organizations (HCOs). However, a prerequisite to quality improvement is successful local QI implementation. The aim of this study was to explore the pathway through which a mandatory QI of the French national public reporting system, namely the quality of the anesthesia file (QAF), was put into practice.

Method

Seven ethnographic case studies in French HCOs combining in situ observations and 37 semi-structured interviews.

Results

A significant proportion of potential QAF users, such as anesthetists or other health professionals were often unaware of quality data. They were, however, involved in improvement actions to meet the QAF criteria. In fact, three intertwined factors influenced QAF appropriation by anesthesia teams and impacted practice. The first factor was the action of clinical managers (chief anesthetists and head of department) who helped translate public policy into local practice largely by providing legitimacy by highlighting the scientific evidence underlying QAF, achieving consensus among team members, and pointing out the value of QAF as a means of work recognition. The two other factors related to the socio-material context, namely the coherence of information systems and the quality of interpersonal ties within the department.

Conclusions

Public policy tends to focus on the metrological validity of QIs and on ranking methods and overlooks QI implementation. However, effective QI implementation depends on local managerial activity that is often invisible, in interaction with socio-material factors. When developing national quality improvement programs, health authorities might do well to specifically target these clinical managers who act as invaluable mediators. Their key role should be acknowledged and they ought to be provided with adequate resources.

Rating Communication in GP Consultations: The Association Between Ratings Made by Patients and Trained Clinical Raters

Patient evaluations of physician communication are widely used, but we know little about how these relate to professionally agreed norms of communication quality. We report an investigation into the association between patient assessments of communication quality and an observer-rated measure of communication competence. Consent was obtained to video record consultations with Family Practitioners in England, following which patients rated the physician’s communication skills. A sample of consultation videos was subsequently evaluated by trained clinical raters using an instrument derived from the Calgary-Cambridge guide to the medical interview. Consultations scored highly for communication by clinical raters were also scored highly by patients. However, when clinical raters judged communication to be of lower quality, patient scores ranged from “poor” to “very good.” Some patients may be inhibited from rating poor communication negatively. Patient evaluations can be useful for measuring relative performance of physicians’ communication skills, but absolute scores should be interpreted with caution.

Understanding high and low patient experience scores in primary care: analysis of patients' survey data for general practices and individual doctors

OBJECTIVES

To determine the extent to which practice level scores mask variation in individual performance between doctors within a practice.

DESIGN

Analysis of postal survey of patients' experience of face-to-face consultations with individual general practitioners in a stratified quota sample of primary care practices.

SETTING

Twenty five English general practices, selected to include a range of practice scores on doctor-patient communication items in the English national GP Patient Survey.

PARTICIPANTS

7721 of 15,172 patients (response rate 50.9%) who consulted with 105 general practitioners in 25 practices between October 2011 and June 2013.

MAIN OUTCOME MEASURE

Score on doctor-patient communication items from post-consultation surveys of patients for each participating general practitioner. The amount of variance in each of six outcomes that was attributable to the practices, to the doctors, and to the patients and other residual sources of variation was calculated using hierarchical linear models.

RESULTS

After control for differences in patients' age, sex, ethnicity, and health status, the proportion of variance in communication scores that was due to differences between doctors (6.4%) was considerably more than that due to practices (1.8%). The findings also suggest that higher performing practices usually contain only higher performing doctors. However, lower performing practices may contain doctors with a wide range of communication scores.

CONCLUSIONS

Aggregating patients' ratings of doctors' communication skills at practice level can mask considerable variation in the performance of individual doctors, particularly in lower performing practices. Practice level surveys may be better used to "screen" for concerns about performance that require an individual level survey. Higher scoring practices are unlikely to include lower scoring doctors. However, lower scoring practices require further investigation at the level of the individual doctor to distinguish higher and lower scoring general practitioners.

The role of patient experience surveys in quality assurance and improvement: a focus group study in English general practice

Background

Despite widespread adoption of patient feedback surveys in international health‐care systems, including the English NHS, evidence of a demonstrable impact of surveys on service improvement is sparse.

Objective

To explore the views of primary care practice staff regarding the utility of patient experience surveys.

Design

Qualitative focus groups.

Setting and participants

Staff from 14 English general practices.

Results

Whilst participants engaged with feedback from patient experience surveys, they routinely questioned its validity and reliability. Participants identified surveys as having a number of useful functions: for patients, as a potentially therapeutic way of getting their voice heard; for practice staff, as a way of identifying areas of improvement; and for GPs, as a source of evidence for professional development and appraisal. Areas of potential change stimulated by survey feedback included redesigning front‐line services, managing patient expectations and managing the performance of GPs. Despite this, practice staff struggled to identify and action changes based on survey feedback alone.

Discussion

Whilst surveys may be used to endorse existing high‐quality service delivery, their use in informing changes in service delivery is more challenging for practice staff. Drawing on the Utility Index framework, we identified concerns relating to reliability and validity, cost and feasibility acceptability and educational impact, which combine to limit the utility of patient survey feedback.

Conclusions

Feedback from patient experience surveys has great potential. However, without a specific and renewed focus on how to translate feedback into action, this potential will remain incompletely realized.