A national clinical indicator database: issues of reliability and validity

Influences of hospital information systems, indicator data collection and computation on reported Dutch hospital performance indicator scores

Background

For health care performance indicators (PIs) to be reliable, data underlying the PIs are required to be complete, accurate, consistent and reproducible. Given the lack of regulation of the data-systems used in the Netherlands, and the self-report based indicator scores, one would expect heterogeneity with respect to the data collection and the ways indicators are computed. This might affect the reliability and plausibility of the nationally reported scores.

Methods

We aimed to investigate the extent to which local hospital data collection and indicator computation strategies differ and how this affects the plausibility of self-reported indicator scores, using survey results of 42 hospitals and data of the Dutch national quality database.

Results

The data collection and indicator computation strategies of the hospitals were substantially heterogenic. Moreover, the Hip and Knee replacement PI scores can be regarded as largely implausible, which was, to a great extent, related to a limited (computerized) data registry. In contrast, Breast Cancer PI scores were more plausible, despite the incomplete data registry and limited data access. This might be explained by the role of the regional cancer centers that collect most of the indicator data for the national cancer registry, in a standardized manner. Hospitals can use cancer registry indicator scores to report to the government, instead of their own locally collected indicator scores.

Conclusions

Indicator developers, users and the scientific field need to focus more on the underlying (heterogenic) ways of data collection and conditional data infrastructures. Countries that have a liberal software market and are aiming to implement a self-report based performance indicator system to obtain health care transparency, should secure the accuracy and precision of the heath care data from which the PIs are calculated. Moreover, ongoing research and development of PIs and profound insight in the clinical practice of data registration is warranted.

Influences of definition ambiguity on hospital performance indicator scores: examples from The Netherlands

RESEARCH OBJECTIVE

Reliable and unambiguously defined performance indicators are fundamental to objective and comparable measurements of hospitals' quality of care. In two separate case studies (intensive care and breast cancer care), we investigated if differences in definition interpretation of performance indicators affected the indicator scores.

DESIGN

Information about possible definition interpretations was obtained by a short telephone survey and a Web survey. We quantified the interpretation differences using a patient-level dataset from a national clinical registry (Case I) and a hospital's local database (Case II). In Case II, there was additional textual information available about the patients' status, which was reviewed to get more insight into the origin of the differences.

PARTICIPANTS

For Case I, we investigated 15 596 admissions of 33 intensive care units in 2009. Case II consisted of 144 admitted patients with a breast tumour surgically treated in one hospital in 2009.

RESULTS

In both cases, hospitals reported different interpretations of the indicators, which lead to significant differences in the indicator values. Case II revealed that these differences could be explained by patient-related factors such as severe comorbidity and patients' individual preference in surgery date.

CONCLUSIONS

With this article, we hope to increase the awareness on pitfalls regarding the indicator definitions and the quality of the underlying data. To enable objective and comparable measurements of hospitals' quality of care, organizations that request performance information should formalize the indicators they use, including standardization of all data elements of which the indicator is composed (procedures, diagnoses).

The epidemiology of cardiac arrests in a Sydney hospital

AIM

To examine the epidemiology of cardiac arrest (CA) in New South Wales (NSW), Australia, and a large teaching hospital in Sydney and to identify predictors of survival.

METHODS

Data from the 1996/97 NSW inpatient statistics collection were analysed. Logistic regression was used to determine predictors of mortality from CA.

RESULTS

In 1996/97 in public hospitals, Eastern Sydney Area Health Service (AHS) (47.5), and in private hospitals of Macleay-Hastings District Health Services (DHSs), recorded the highest crude CA rates in NSW with figures of 47.5 and 21.3 per 10,000 hospital separations. Standardised CA rates were highest in Western Sydney, Illawara, Macleay-Hasting, Mid North Coast and Orana DHSs. Most CAs in hospital 'X' occurred after admission and 55.7% occurred in patients admitted with a non-cardiac principal diagnosis, mainly pneumonia. Acute myocardial infarction was the leading diagnosis (28.9%) in patients who suffered CA after admission. Only 32% of CA patients survived to discharge. Age above 65 (odds ratio (OR)=2.284, P=0.006) had the highest effect on cardiac mortality. The longer the patients were in the intensive care unit (OR=0.997, P=0.037), the more likely they were to survive.

CONCLUSION

We describe considerable variation in CA rates in NSW. The majority of hospital CAs occur at or soon after admission, and are associated with underlying non-cardiac conditions. Survival following hospital CA is low, but there may be preventable elements. Knowledge of risk factors and epidemiology of hospital CAs may help identify patients at risk of CA.