Reliability study of the European appropriateness evaluation protocol

Reliability and Validity of the Appropriateness Evaluation Protocol for Public Hospitals in Korea

Objectives

This study was conducted to assess the applicability of the Appropriateness Evaluation Protocol (AEP) for public hospitals in Korea.

Methods

In May 2016, 1500 admission claims were collected from Korean public district hospitals using stratified random sampling. Of these claims, 560 admissions to 37 hospitals were retrieved for analysis. Medical records administrators determined the appropriateness of admission using the criteria detailed in the AEP, and a physician separately assessed the appropriateness of admission based on her clinical judgment. To examine the applicability of the AEP, the concordance of the decisions made between a pair of AEP reviewers and between an AEP reviewer and a physician reviewer was compared.

Results

The results showed an almost perfect inter-rater agreement between the AEP reviewers and a moderate agreement between the AEP reviewers and the physician. The sensitivity and specificity of the AEP were calculated as 0.86 and 0.56, respectively.

Conclusions

Our findings suggest that the AEP could potentially be applied to Korean public hospitals as a reliable and valid instrument for assessing the appropriateness of admissions.

Evaluation of the Appropriateness of Acute Hospitalisations in Hong Kong

Objective

To test the validity of the Hong Kong version of Appropriateness Evaluation Protocol and estimate the prevalence of inappropriate acute hospitalisations in Hong Kong.

Methods

A retrospective chart review of two hundred randomly selected patients admitted to the specialty of Internal Medicine and General Surgery via the Accident & Emergency department of 2 regional hospitals in 2008. Comparison between the Hong Kong version of Appropriateness Evaluation Protocol and the consensus of an expert panel on appropriateness of admissions was made. The extent of agreement between the reviewer using the protocol and the expert panel was measured.

Results

The kappa coefficient for agreement was 0.73 (95% confidence interval: 0.63-0.83). The prevalence of inappropriate acute hospitalisations was 29%.

Conclusions

The Hong Kong version of Appropriateness Evaluation Protocol is a valid tool for assessing the appropriateness of acute hospitalisations.

Reliability and Validity of the Chinese Version Appropriateness Evaluation Protocol

Objective

To adapt the Appropriateness Evaluation Protocol (AEP) to the specific settings of health care in China and to validate the Chinese version AEP (C-AEP).

Methods

Forward and backward translations were carried out to the original criteria. Twenty experts participated in the consultancy to form a preliminary version of the C-AEP. To ensure applicability, tests of reliability and validity were performed on 350 admissions and 3,226 hospital days of acute myocardial infraction patients and total hip replacement patients in two tertiary hospitals by two C-AEP reviewers and two physician reviewers. Overall agreement, specific agreement, and Cohen’s Kappa were calculated to compare the concordance of decisions between pairs of reviewers to test inter-rater reliability and convergent validity. The use of “overrides” and opinions of experts were recorded as measurements of content validity. Face validity was tested through collecting perspectives of nonprofessionals. Sensitivity, specificity, and predictive values were also reported.

Results

There are 14 admission and 24 days of care criteria in the initial version of C-AEP. Kappa coefficients indicate substantial agreement between reviewers: with regard to inter-rater reliability, Kappa (κ) coefficients are 0.746 (95% confidence interval [CI] 0.644–0.834) and 0.743 (95% CI 0.698–0.767) of admission and hospital days, respectively; for convergent validity, the κ statistics are 0.678 (95% CI 0.567–0.778) and 0.691 (95% CI 0.644–0.717), respectively. Overrides account for less than 2% of all judgments. Content validity and face validity were confirmed by experts and nonprofessionals, respectively. According to the C-AEP reviewers, 18.3% of admissions and 28.5% of inpatient days were deemed inappropriate.

Conclusions

The C-AEP is a reliable and valid screening tool in China’s tertiary hospitals. The prevalence of inappropriateness is substantial in our research. To reduce inappropriate utilization, further investigation is needed to elucidate the reasons and risk factors for this inappropriateness.

Does clinical governance influence the appropriateness of hospital stay?

Background

Clinical Governance provides a framework for assessing and improving clinical quality through a single coherent program. Organizational appropriateness is aimed at achieving the best health outcomes and the most appropriate use of resources. The goal of the present study is to verify the likely relationship between Clinical Governance and appropriateness of hospital stay.

Methods

A cross-sectional study was conducted in 2012 in an Italian Teaching Hospital. The OPTIGOV^© (Optimizing Health Care Governance) methodology was used to quantify the level of implementation of Clinical Governance globally and in its main dimensions. Organizational appropriateness was measured retrospectively using the Italian version of the Appropriateness Evaluation Protocol to analyze a random sample of medical records for each clinical unit.

Pearson-correlation and multiple linear regression were used to test the relationship between the percentage of inappropriate days of hospital stay and the Clinical Governance implementation levels.

Results

47 Units were assessed. The percentage of inappropriate days of hospital stay showed an inverse correlation with almost all the main Clinical Governance dimensions. Adjusted multiple regression analysis resulted in a significant association between the percentage of inappropriate days and the overall Clinical Governance score (β = −0.28; p < 0.001; R-squared = 0.8). EBM and Clinical Audit represented the Clinical Governance dimensions which had the strongest association with organizational appropriateness.

Conclusions

This study suggests that the evaluation of both Clinical Governance and organizational appropriateness through standardized and repeatable tools, such as OPTIGOV^© and AEP, is a key strategy for healthcare quality. The relationship between the two underlines the central role of Clinical Governance, and especially of EBM and Clinical Audit, in determining a rational improvement of appropriateness levels.

Length of stay as risk factor for inappropriate hospital days: interaction with patient age and co-morbidity

RATIONALE, AIMS AND OBJECTIVES

The likelihood of a hospital day being inappropriate depends on patient characteristics, on the organization of in-hospital care and on the co-ordination between hospital care and the rest of the health care sector. The aim of the study was to assess if certain socio-demographic and medical factors affect inappropriate hospital stay including possible interactions between age and co-morbidity.

METHODS

To determine the appropriateness of length of hospitalization, a prospective study was carried out using the European version of the Appropriateness Evaluation Protocol (AEP). A total of 438 hospital days of stay was analysed in medical wards of a university-affiliated teaching hospital in the North of Italy for 3 days in September 2010.

RESULTS

44.6% of hospitalization days were classified as inappropriate. Unjustified hospital use was more frequent in patients whose hospital length of stay exceeded 10 days. Age and co-morbidity were not per se risk factors for inappropriateness; however, in young patients hospitalized for more than 10 days, absence of chronic illness was a predictor. Conservative patient management, lack of discharge planning and delays in scheduling diagnostic tests or therapeutic interventions were the most common causal or contributory doctor- and hospital-related factors.

CONCLUSIONS

Doctor attitudes and hospital organization are still among the most common reasons for inappropriate in-hospital days of care. Monitoring whether the length of stay is appropriate combined with protocol interventions for scheduling of diagnosis, treatment and discharge are likely to improve efficiency in this area of medical care.

Reduction in inappropriate hospital use based on analysis of the causes

Background

To reduce inappropriate admissions and stays with the application of an improvement cycle in patients admitted to a University Hospital. The secondary objective is to analyze the hospital cost saved by reducing inadequacy after the implementation of measures proposed by the group for improvement.

Methods

Pre- and post-analysis of a sample of clinical histories studied retrospectively, in which the Appropriateness Evaluation Protocol (AEP) was applied to a representative hospital sample of 1350 clinical histories in two phases. In the first phase the AEP was applied retrospectively to 725 admissions and 1350 stays. The factors associated with inappropriateness were analysed together with the causes, and specific measures were implemented in a bid to reduce inappropriateness. In the second phase the AEP was reapplied to a similar group of clinical histories and the results of the two groups were compared. The cost of inappropriate stays was calculated by cost accounting. Setting: General University Hospital with 426 beds serving a population of 320,000 inhabitants in the centre of Murcia, a city in south-eastern Spain.

Results

Inappropriate admissions were reduced significantly: 7.4% in the control group and 3.2% in the intervention group. Likewise, inappropriate stays decreased significantly from 24.6% to 10.4%. The cost of inappropriateness in the study sample fell from 147,044 euros to 66,642 euros. The causes of inappropriateness for which corrective measures were adopted were those that showed the most significant decrease.

Conclusions

It is possible to reduce inadequacy by applying measures based on prior analysis of the situation in each hospital.

Inappropriate hospital admission: interaction between patient age and co-morbidity

The aim of the study is to determine the prevalence of inappropriate admission, and to identify the factors that influence appropriateness of hospital admission. Data were prospectively collected from all 345 consecutive patients admitted during the period of 1 month for acute hospital care at a 110-bed division of internal medicine using socio-demographic and medical information. Statistical analyses included χ2 tests, t tests, and logistic regression analyses. According to the European version of the Appropriateness Evaluation Protocol of hospital admission, 28.1% of medical admissions for acute care in the Central Hospital of Bolzano, Italy, have been classified as inappropriate. Factors that reduced appropriateness included female gender, age and chronic illness that are significantly associated with appropriateness of medical admission, whereas time of day or day of week of the emergency department (ED) visit does not influence appropriateness. In multiple logistic regression analyses, age and co-morbidity are not independently related to appropriateness, however, when tested for interaction, inappropriateness is significantly more frequent at a young age in the absence of co-morbidities, and, numerically most relevant, in elderly patients presenting with co-morbidities. In this evaluation of a single centre North Italian hospital admission, co-morbidity turns out to be an important age-dependent determinant of appropriateness. Although in the young age group, co-morbidity increases the likelihood of being appropriately admitted, the presence of chronic illness in the elderly increases the risk of inappropriate hospital use.

Associated factors and cost of inappropriate hospital admissions and stays in a second-level hospital

The objective of this study was to analyze the variables related to inappropriate admissions and hospital stays and their financial repercussions. This was a descriptive retrospective study in which the Appropriateness Evaluation Protocol (AEP) was applied to a representative sample of 725 hospital admissions and 1350 hospital stays. The cost of inappropriate stays was calculated by cost accounting. The study found that 7.4% of admissions and 24.6% of stays were inappropriate. Inappropriate admissions were significantly related to medical specialties and younger patient age. Inappropriate stays were related to these factors, plus patients being outside their corresponding areas, nonurgent admissions, and low occupancy rate. The cost of the inappropriate admissions and stays was 147 044 euros. Extrapolated to the hospital as a whole this would represent 2125638 euros per year. Steps must be taken to reduce inappropriate admissions and stays to lower health care costs and to reserve hospital resources for patients who genuinely need them.

[Not Available]

OBJECTIVES

To assess the main features and determining factors of inappropriate admission and hospital stay in relationship with a new clinical management model.

METHODS

Study population included all patients discharged from the Hospital Clinic-University of Barcelona. The review tool was the medical and surgical Appropriateness Evaluation Protocol (AEP). We reviewed the clinical records of a representative sample of all discharged patients. Two independent and well trained AEP reviewers were used for each record review. A multivariate analysis was performed, using admission and inappropriate hospital stay as independent variables.

RESULTS

A total of 401 admissions were reviewed. Twenty-eight (6.9%) were deemed to be inappropriate. Elective admission, admission over the week-end and being admitted to a medical ward were the predictive variables identified for inappropriate admissions. Of all 2,187 hospital stays reviewed, 267 (12.2%) were deemed to be inappropriate. An inappropriate admission, total length of stay, type of insurance coverage and discharge type were the predictive variables identified for inappropriate hospital stays.

CONCLUSIONS

The new organization and clinical management model, developed by the Patient Focused Care Institutes, showed very acceptable levels of inappropriate admissions and hospital stays. Surgical services have benefited most from this new organization. Care provided to patients with medical conditions, notably those patients with multiple and chronic conditions, make up most of the inappropriateness identified.

Impact of a physician's order entry (POE) system on physicians' ordering patterns and patient length of stay

OBJECTIVE

To investigate the impact of a physician's order entry (POE) system on physicians' ordering patterns and patient length of stay.

DESIGN

Prospective time series study at pre-POE, 3 months and 6 months after POE at a tertiary teaching hospital in Korea. The study period was from June 1999 to May 2000.

MEASUREMENTS

The number of orders (doctor's, PRN, medication, changed, canceled orders), number of tests (complete blood count, chemistry, chest X-ray, stat laboratory, serum electrolytes tests), appropriateness and length of patient stay were measured through chart review of 171 in-patients (liver disease, renal disease, gastrectomy, simple mastectomy).

RESULTS

The number of doctors' orders, PRN, and medication orders significantly increased after POE. The numbers of changed and canceled orders were not significantly different between pre- and post-POE. The number of stat lab tests significantly decreased after POE. There was no change in appropriateness of patients' hospital stay between pre- and post-POE. Length of stay significantly decreased (P=0.049).

CONCLUSION

POE contributed to improving the quality of care in two ways: improvement of auditability by recording the medical services for patients in more precise and transparent manner, and more appropriate utilization of resources by decreasing the number of stat diagnostic tests and length of stay.

Reliability and validity of the Intercollegiate Stroke Audit Package

The aim of this study was to assure the validity and reliability of the Intercollegiate Stroke Audit Package as used in the National Sentinel Audit of Stroke. The Intercollegiate Working Party for Stroke, which included most stakeholders, including patients, devised the audit standards. These were submitted to a formal consensus (modified Delphi) survey before the audit questions were developed and piloted for validity and reliability. Following the pilot, Help Booklets were developed to promote the involvement of all disciplines as auditors in the national sentinel audit of stroke and ensure inter-rater reliability. During the national audit each Trust was asked to double rate the first five cases with auditors of different disciplines working independently. A total of 886 case notes were double-rated in 184 separate sites (median 5, range 1-5 per site). Trusts used auditors from different disciplines in 77% of cases. After excluding the 'No answer' cases the kappa score for items ranged from 0.49 to 0.87 (median 0.70, IQR 0.63-0.78). Very good agreement was found for seven of the 45 items, good agreement for 30 items, and moderate agreement for eight items. This large study, across a range of hospital sites and involving many disciplines, demonstrates that careful piloting of audit tools, with use of clear instructions to auditors, promotes the reliability of data.