Criterion-Based Measurements of Patient Experience in Health Care: Eliminating Winners and Losers to Create a New Moral Ethos

Reliability of a Measure of Admission Intensity for Emergency Physicians

STUDY OBJECTIVE

We assess the stability of a measure of emergency department (ED) admission intensity for value-based care programs designed to reduce variation in ED admission rates. Measure stability is important to accurately assess admission rates across sites and among physicians.

METHODS

We sampled data from 358 EDs in 41 states (January 2018 to December 2021), separate from sites where the measure was derived. The measure is the ED admission rate per 100 ED visits for 16 clinical conditions and 535 included International Classification of Disease 10 diagnosis codes. We used descriptive plots and multilevel linear probability models to assess stability over time across EDs and among physicians.

RESULTS

Across included 3,571 ED-quarters, the average admission rate was 27.6% (95% confidence interval [CI] 26.0% to 28.2%). The between-facility standard deviation was 9.7% (95% CI 9.0% to 10.6%), and the within-facility standard deviation was 3.0% (95% CI 2.95% to 3.10%), with an intraclass correlation coefficient of 0.91. At the physician-quarter level, the average admission rate was 28.3% (95% CI 28.0% to 28.5%) among 7,002 physicians. Relative to their site's mean in each quarter, the between-physician standard deviation was 6.7% (95% CI 6.6% to 6.8%), and the within-physician standard deviation was 5.5% (95% CI 5.5% to 5.6%), with an intraclass correlation coefficient of 0.59. Moreover, 2.9% of physicians were high-admitting in 80%+ of their practice quarters relative to their peers in the same ED and in the same quarter, whereas 3.9% were low-admitting.

CONCLUSION

The measure exhibits stability in characterizing ED-level admission rates and reliably identifies high- and low-admitting physicians.

When peer comparison information harms physician well-being

Policymakers and business leaders often use peer comparison information—showing people how their behavior compares to that of their peers—to motivate a range of behaviors. Despite their widespread use, the potential impact of peer comparison interventions on recipients’ well-being is largely unknown. We conducted a 5-mo field experiment involving 199 primary care physicians and 46,631 patients to examine the impact of a peer comparison intervention on physicians’ job performance, job satisfaction, and burnout. We varied whether physicians received information about their preventive care performance compared to that of other physicians in the same health system. Our analyses reveal that our implementation of peer comparison did not significantly improve physicians’ preventive care performance, but it did significantly decrease job satisfaction and increase burnout, with the effect on job satisfaction persisting for at least 4 mo after the intervention had been discontinued. Quantitative and qualitative evidence on the mechanisms underlying these unanticipated negative effects suggest that the intervention inadvertently signaled a lack of support from leadership. Consistent with this account, providing leaders with training on how to support physicians mitigated the negative effects on well-being. Our research uncovers a critical potential downside of peer comparison interventions, highlights the importance of evaluating the psychological costs of behavioral interventions, and points to how a complementary intervention—leadership support training—can mitigate these costs.