Development, implementation, and public reporting of the HCAHPS survey

Who's behind an HCAHPS score?

BACKGROUND

The Hospital Consumer Assessment of Health Care Providers and Systems (HCAHPS) survey asks patients how frequently their physicians treated them with courtesy and respect, listened carefully, and explained things in a way they could understand. Such summary reports may obscure differences among the types of physicians involved. A study was conducted to examine the association between ratings for different physician types and the overall HCAHPS rating of physicians.

METHODS

A mixed-methods study included closed-ended surveys and in-depth interviews of patients on a hospitalist teaching service. The three HCAHPS physician communication items were used to interview patients about their communication experiences with emergency medicine (EM) physicians, hospitalists, and specialists. The association between the overall score and the scores of each physician type was examined using Spearman correlation coefficients and linear regression. Qualitative data from additional in-depth interviews were analyzed using the constant comparative method to identify recurrent themes.

RESULTS

Ninety-six patients were recruited for the survey, and additional in-depth interviews were conducted with the first 30 patients. Hospitalist and specialist scores were significantly associated (p values < .05) with overall scores. Recurrent themes regarding determinants of patients' ratings were categorized in three broad domains: individual physician behavior, team communication, and system issues. The influence of each domain differed across physician types.

DISCUSSION

Physician communication scores may be most strongly influenced by patient experiences with hospitalists and specialists rather than with EM physicians. Several team communication and system issues represent opportunities for improving physician communication.

Nurses' widespread job dissatisfaction, burnout, and frustration with health benefits signal problems for patient care

Job dissatisfaction among nurses contributes to costly labor disputes, turnover, and risk to patients. Examining survey data from 95,499 nurses, we found much higher job dissatisfaction and burnout among nurses who were directly caring for patients in hospitals and nursing homes than among nurses working in other jobs or settings, such as the pharmaceutical industry. Strikingly, nurses are particularly dissatisfied with their health benefits, which highlights the need for a benefits review to make nurses' benefits more comparable to those of other white-collar employees. Patient satisfaction levels are lower in hospitals with more nurses who are dissatisfied or burned out-a finding that signals problems with quality of care. Improving nurses' working conditions may improve both nurses' and patients' satisfaction as well as the quality of care.

How can health care organizations be reliably compared?: Lessons from a national survey of patient experience

BACKGROUND

Patient experience is increasingly used to assess organizational performance, for example in public reporting or pay-for-performance schemes. Conventional approaches using 95% confidence intervals are commonly used to determine required survey samples or to report performance but these may result in unreliable organizational comparisons.

METHODS

We analyzed data from 2.2 million patients who responded to the English 2009 General Practice Patient Survey, which included 45 patient experience questions nested within 6 different care domains (access, continuity of care, communication, anticipatory care planning, out-of-hours care, and overall care satisfaction). For each question, unadjusted and case-mix adjusted (for age, sex, and ethnicity) organization-level reliability, and intraclass correlation coefficients were calculated.

RESULTS

Mean responses per organization ranged from 23 to 256 for questions evaluating primary care practices, and from 1454 to 2758 for questions evaluating out-of-hours care organizations. Adjusted and unadjusted reliability values were similar. Twenty-six questions had excellent reliability (≥0.90). Seven nurse communication questions had very good reliability (≥0.85), but 3 anticipatory care planning questions had lower reliability (<0.70). Reliability was typically <0.70 for questions with <100 mean responses per practice, usually indicating questions which only a subset of patients were eligible to answer. Nine questions had both excellent reliability and high intraclass correlation coefficients (≥0.10) indicating both reliable measurement and substantial performance variability.

CONCLUSIONS

High reliability is a necessary property of indicators used to compare health care organizations. Using the English General Practice Patient Survey as a case study, we show how reliability and intraclass correlation coefficients can be used to select measures to support robust organizational comparisons, and to design surveys that will both provide high-quality measurement and optimize survey costs.

The Generic Short Patient Experiences Questionnaire (GS-PEQ): identification of core items from a survey in Norway

Background

Questionnaires are commonly used to collect patient, or user, experiences with health care encounters; however, their adaption to specific target groups limits comparison between groups. We present the construction of a generic questionnaire (maximum of ten questions) for user evaluation across a range of health care services.

Methods

Based on previous testing of six group-specific questionnaires, we first constructed a generic questionnaire with 23 items related to user experiences. All questions included a "not applicable" response option, as well as a follow-up question about the item's importance. Nine user groups from one health trust were surveyed. Seven groups received questionnaires by mail and two by personal distribution. Selection of core questions was based on three criteria: applicability (proportion "not applicable"), importance (mean scores on follow-up questions), and comprehensiveness (content coverage, maximum two items per dimension).

Results

1324 questionnaires were returned providing subsample sizes ranging from 52 to 323. Ten questions were excluded because the proportion of "not applicable" responses exceeded 20% in at least one user group. The number of remaining items was reduced to ten by applying the two other criteria. The final short questionnaire included items on outcome (2), clinician services (2), user involvement (2), incorrect treatment (1), information (1), organisation (1), and accessibility (1).

Conclusion

The Generic Short Patient Experiences Questionnaire (GS-PEQ) is a short, generic set of questions on user experiences with specialist health care that covers important topics for a range of groups. It can be used alone or with other instruments in quality assessment or in research. The psychometric properties and the relevance of the GS-PEQ in other health care settings and countries need further evaluation.

Hospital survey shows improvements in patient experience

Hospitals are improving the inpatient care experience. A government survey that measures patients' experiences with a range of issues from staff responsiveness to hospital cleanliness-the Hospital Consumer Assessment of Healthcare Providers and Systems survey-is showing modest but meaningful gains. Using data from the surveys reported in March 2008 and March 2009, we present the first comprehensive national assessment of changes in patients' experiences with inpatient care since public reporting of the results began. We found improvements in all measures of patient experience, except doctors' communication. These improvements were fairly uniform across hospitals. The largest increases were in measures related to staff responsiveness and the discharge information that patients received.

Semi-customizing patient surveys: linking results and organizational conditions

OBJECTIVE

The study investigated the needs and consequences of semi-customizing patient satisfaction surveys to low organizational levels and explored whether patient satisfaction was correlated with local organizational conditions.

DESIGN

From 1999 to 2006, the County of Aarhus carried out 398 surveys during four rounds in eight hospitals. To explain differences between the wards, data on the 40 wards with the best and the 40 wards with the worst evaluations (identified by patient surveys) were compared with the data from job satisfaction surveys and management information systems.

SETTING

Eight public hospitals in a Danish county.

PARTICIPANTS

32,809 inpatients and 1842 nurses on 84 wards.

MAIN OUTCOME MEASURE

Optimal organizational level for measuring patient satisfaction and correlations between overall patient satisfaction and organizational context.

RESULTS

In all, 71.4% of the departments chose to have the survey results specified at the subunit level or for specific diagnostic groups. Substantial differences in patient satisfaction between wards are illustrated. On the wards with the highest improvement potential, we found significantly higher occupancy rates, acute rates, rates of sickness absenteeism, staff perceptions of high workload and low experience of professionalism.

CONCLUSIONS

The study confirmed that departments desired individual, detailed descriptions of the results. Differences in patient satisfaction were associated with differences in organizational conditions. Establishing a link between patient satisfaction and organizational variables broadens the quality development focus to include more than simply analysis of specific questions. Semi-customizing patient surveys are recommended.

Rasch analysis of the Psychiatric Out-Patient Experiences Questionnaire (POPEQ)

BACKGROUND

The Psychiatric Out-Patient Experiences Questionnaire (POPEQ) is an 11-item core measure of psychiatric out-patients experiences of the perceived outcome of the treatment, the quality of interaction with the clinician, and the quality of information provision. The POPEQ was found to have evidence for reliability and validity following the application of classical test theory but has not previously been assessed by Rasch analysis.

METHODS

Two national postal surveys of psychiatric outpatients took place in Norway in 2004 and 2007. The performance of the POPEQ, including item functioning and differential item functioning, was assessed by Rasch analysis. Principal component analysis of item residuals was used to assess the presence of subdimensions.

RESULTS

6,677 (43.3%) and 11,085 (35.2%) psychiatric out patients responded to the questionnaire in 2004 and 2007, respectively. All items in the scale were retained after the Rasch analysis. The resulting scale had reasonably good fit to the Rasch model. The items performed the same for the two survey years and there was no differential item functioning relating to patient characteristics. Principal component analysis of the residuals confirmed that the measure to a high degree is unidimensional. However, the data also reflects three potential subscales, each relating to one of the three included aspects of health care.

CONCLUSIONS

The POPEQ had excellent psychometric properties and Rasch analysis further supported the construct validity of the scale by also identifying the three subdimensions originally included as components in the instrument development. The 11-item instrument is recommended in future research on psychiatric out-patient experiences. Future development may lead to the construction of more precise measures of the three subdomains that the POPEQ is based on.

Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England

OBJECTIVE

To assess the robustness of patient responses to a new national survey of patient experience as a basis for providing financial incentives to doctors.

DESIGN

Analysis of the representativeness of the respondents to the GP Patient Survey compared with those who were sampled (5.5 million patients registered with 8273 general practices in England in January 2009) and with the general population. Analysis of non-response bias looked at the relation between practice response rates and scores on the survey. Analysis of the reliability of the survey estimated the proportion of the variance of practice scores attributable to true differences between practices.

RESULTS

The overall response rate was 38.2% (2.2 million responses), which is comparable to that in surveys using similar methodology in the UK. Men, young adults, and people living in deprived areas were under-represented among respondents. However, for questions related to pay for performance, there was no systematic association between response rates and questionnaire scores. Two questions which triggered payments to general practitioners were reliable measures of practice performance, with average practice-level reliability coefficients of 93.2% and 95.0%. Less than 3% and 0.5% of practices had fewer than the number of responses required to achieve conventional reliability levels of 90% and 70%. A change to the payment formula in 2009 resulted in an increase in the average impact of random variation in patient scores on payments to general practitioners compared with payments made in 2007 and 2008.

CONCLUSIONS

There is little evidence to support the concern of some general practitioners that low response rates and selective non-response bias have led to systematic unfairness in payments attached to questionnaire scores. The study raises issues relating to the validity and reliability of payments based on patient surveys and provides lessons for the UK and for other countries considering the use of patient experience as part of pay for performance schemes.