Using estimated true safety event rates versus flagged safety event rates: does it change hospital profiling and payment?

A Keyword Approach to Identify Adverse Events Within Narrative Documents From 4 Italian Institutions

OBJECTIVES

Existing methods for measuring adverse events in hospitals intercept a restricted number of events. Text mining refers to a range of techniques to extract data from narrative sources. The goal of this study was to evaluate the performance of an automated approach for extracting adverse event keywords from within electronic health records.

METHODS

The study involved 4 medical centers in the Region of Lombardy. A starting set of keywords was trained in an iterative process to develop queries for 7 adverse events, including those used by the Agency for Healthcare Research and Quality as patient safety indicators. We calculated positive predictive values of the 7 queries and performed an error analysis to detect reasons for false-positive cases of pulmonary embolism, deep vein thrombosis, and urinary tract infection.

RESULTS

Overall, 397,233 records were collected (34,805 discharge summaries, 292,593 emergency department notes, and 69,835 operation reports). Positive predictive values were higher for postoperative wound dehiscence (83.83%) and urinary tract infection (73.07%), whereas they were lower for deep vein thrombosis (5.37%), pulmonary embolism (13.63%), and postoperative sepsis (12.28%). The most common reasons for false positives were reporting of past events (42.25%), negations (22.80%), and conditions suspected by physicians but not confirmed by a diagnostic test (11.25%).

CONCLUSIONS

The results of our study demonstrated the feasibility of using an automated approach to detect multiple adverse events in several data sources. More sophisticated techniques, such as natural language processing, should be tested to evaluate the feasibility of using text mining as a routine method for monitoring adverse events in hospitals.

Daily review of AHRQ patient safety indicators has important impact on value-based purchasing, reimbursement, and performance scores

BACKGROUND

The Patient Safety Indicators (PSIs) Composite (PSI 90) of the Agency for Healthcare Research and Quality has been found to have low positive predictive values. Because scores can affect hospital reimbursement and ranking, our institution designed a review process to ensure accurate data and incur minimal penalties under the Hospital Value-Based Purchasing Program.

METHODS

A multidisciplinary team was assembled to review PSI 90 within a performance period. The positive predictive value of each PSI was calculated. Weight-adjusted PSI rates were used to recalculate the PSI 90 Performance Period Index Value (PPIV). The adjusted PPIV was used to estimate what the achievement points and financial impact would have been if PSI review had not been implemented. Differences in PPIV, achievement points, and financial impact before and after PSI review were calculated.

RESULTS

A total of 1,470 cases were flagged for PSI over a 2-year period. The positive predictive value was 63.3%. Refuting 36.7% of PSIs resulted in a decrease in the PPIV from 0.696 to 0.508, an increase in achievement points from 5 to 10, resulting in a decreased net loss of $111,773.

CONCLUSION

Multidisciplinary review processes are practical and effective in identifying false-positive patient safety events. The real-time process affects hospital performance and resultant Medicare reimbursement substantially.

Navigating a ship with a broken compass: evaluating standard algorithms to measure patient safety

Objective

Agency for Healthcare Research and Quality (AHRQ) software applies standardized algorithms to hospital administrative data to identify patient safety indicators (PSIs). The objective of this study was to assess the validity of PSI flags and report reasons for invalid flagging.

Material and Methods

At a 6-hospital academic medical center, a retrospective analysis was conducted of all PSIs flagged in fiscal year 2014. A multidisciplinary PSI Quality Team reviewed each flagged PSI based on quarterly reports. The positive predictive value (PPV, the percent of clinically validated cases) was calculated for 12 PSI categories. The documentation for each reversed case was reviewed to determine the reasons for PSI reversal.

Results

Of 657 PSI flags, 185 were reversed. Seven PSI categories had a PPV below 75%. Four broad categories of reasons for reversal were AHRQ algorithm limitations (38%), coding misinterpretations (45%), present upon admission (10%), and documentation insufficiency (7%). AHRQ algorithm limitations included 2 subcategories: an "incident" was inherent to the procedure, or highly likely (eg, vascular tumor bleed), or an "incident" was nonsignificant, easily controlled, and/or no intervention was needed.

Discussion

These findings support previous research highlighting administrative data problems. Additionally, AHRQ algorithm limitations was an emergent category not considered in previous research. Herein we present potential solutions to address these issues.

Conclusions

If, despite poor validity, US policy continues to rely on PSIs for incentive and penalty programs, improvements are needed in the quality of administrative data and the standardized PSI algorithms. These solutions require national motivation, research attention, and dissemination support.

A critical review of patient safety indicators attributed to trauma surgeons

BACKGROUND

The Agency for Health Care Research and Quality (AHRQ) developed patient safety indicators (PSIs) to identify events with a high likelihood of representing medical error. The purpose of this study was to validate PSIs attributed to trauma surgeons and compare validated PSIs to performance improvement (PI) and morbidity and mortality (M&M) data. We hypothesized that PSIs are not an indicator of quality of care in trauma.

METHODS

PSI's attributed to trauma surgeons (n=9) at our institution were reviewed (Jan-Dec 2015). An initial review was conducted to ensure they met inclusion and exclusion criteria (valid). "Valid" PSIs were distributed to the trauma division for secondary review.

RESULTS

48 PSIs were identified (17.2 per 1000 cases) during the study period. 19 were false positives yielding a positive predictive value of 60% (95% CI 45-74%). False positive PSIs were the result of coding error (78%), present on admission status (17%) and documentation error (5%). Valid PSIs (n=29) were further analyzed. The most common were post-op PE/DVT (n=14), failure to rescue (n=6) and accidental puncture/laceration (n=3). 60% of patients with a post-op PE/DVT were started on chemoprophylaxis on admission and 40% had significant intracranial hemorrhage; all were deemed non-preventable through trauma PI. All deaths considered failure to rescue were classified as expected mortalities during M&M review. Although not clinically significant, all cases of accidental puncture/laceration (10% of valid PSIs) represented opportunities for improvement.

CONCLUSION

Overall, PSIs have low validity and do not reflect quality of care in trauma.

Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator for Postoperative Respiratory Failure (PSI 11) does not identify accurately patients who received unsafe care

BACKGROUND

The Agency for Healthcare Research and Quality Patient Safety Indicator 11 is used to identify postoperative respiratory failure events and detect areas for quality improvement. This study examines the accuracy of Patient Safety Indicator 11 in identifying clinically valid patient safety events.

METHODS

All cases flagged for Patient Safety Indicator 11 from July 2013 to July 2015 by Agency for Healthcare Research and Quality QI Version 4.5 including International Classification of Diseases-9 codes were evaluated. Code-confirmed cases underwent independent review by 2 physicians. Inpatient electronic medical records were used to identify clinical factors for postoperative respiratory failure in each case to determine if postoperative respiratory failure was a result of unsafe care. The clinical true-positive rate and positive predictive value were calculated.

RESULTS

A total of 166 postoperative respiratory failure cases were reviewed; 51 were recoded and reversed due to coding or documentation errors; 115 cases met the Agency for Healthcare Research and Quality definition of postoperative respiratory failure. A total of 71 (61.7%) of the 115 cases were false positives and did not reflect unsafe care, while 44 cases were true positives with a positive predictive value of 38.3%. χ(2) analysis did not reveal an association between demographics, clinical characteristics, or operative procedure with true-positive cases.

CONCLUSION

Administrative coding data for Agency for Healthcare Research and Quality Patient Safety Indicator 11 do not identify accurately patients who received unsafe care when taking into account unpreventable clinical factors causing postoperative respiratory failure. The use of Agency for Healthcare Research and Quality Patient Safety Indicator 11 as a hospital performance measure should be reconsidered until inclusion and exclusion criteria are revised.

Patient Safety Indicators for Judging Hospital Performance

Patient Safety Indicators (PSIs) were originally intended for use as a screen for quality of care but are now being used to rank hospitals and to modify hospital reimbursement. PSI data are dependent on accuracy of clinical documentation and coding. Information on whether a PSI event is inherent to the nature of the operation or posed a significant impact on the outcome is lacking. Cases for one year at a single academic center were queried. Cases with target PSIs were included (n = 136). Cases were evaluated for both the inherent nature and significance of injury. Both patient safety officers agreed that the PSI event was inherent to the disease process, and thus, the procedure and was not a marker of patient safety (false positive) in 11.8% to 33.3% of cases. Both reviewers agreed that the events were not clinically significant in 11.8% to 30.4% of cases. This study found high false-positive rates and only moderate interrater reliability for 3 PSIs. PSIs as currently reported are not reliable enough to be utilized for ranking.