Ranking hospital performance based on individual indicators: can we increase reliability by creating composite indicators?

Differential gene expression analysis pipelines and bioinformatic tools for the identification of specific biomarkers: A review

In recent years, the role of bioinformatics and computational biology together with omics techniques and transcriptomics has gained tremendous importance in biomedicine and healthcare, particularly for the identification of biomarkers for precision medicine and drug discovery. Differential gene expression (DGE) analysis is one of the most used techniques for RNA-sequencing (RNA-seq) data analysis. This tool, which is typically used in various RNA-seq data processing applications, allows the identification of differentially expressed genes across two or more sample sets. Functional enrichment analyses can then be performed to annotate and contextualize the resulting gene lists. These studies provide valuable information about disease-causing biological processes and can help in identifying molecular targets for novel therapies. This review focuses on differential gene expression (DGE) analysis pipelines and bioinformatic techniques commonly used to identify specific biomarkers and discuss the advantages and disadvantages of these techniques.

Biases in Electronic Health Records Data for Generating Real-World Evidence: An Overview

Electronic Health Records (EHR) are increasingly being perceived as a unique source of data for clinical research as they provide unprecedentedly large volumes of real-time data from real-world settings. In this review of the secondary uses of EHR, we identify the anticipated breadth of opportunities, pointing out the data deficiencies and potential biases that are likely to limit the search for true causal relationships. This paper provides a comprehensive overview of the types of biases that arise along the pathways that generate real-world evidence and the sources of these biases. We distinguish between two levels in the production of EHR data where biases are likely to arise: (i) at the healthcare system level, where the principal source of bias resides in access to, and provision of, medical care, and in the acquisition and documentation of medical and administrative data; and (ii) at the research level, where biases arise from the processes of extracting, analyzing, and interpreting these data. Due to the plethora of biases, mainly in the form of selection and information bias, we conclude with advising extreme caution about making causal inferences based on secondary uses of EHRs.

Stability of hospital quality indicators over time: A multi-year observational study of German hospital data

BACKGROUND

Retrospective hospital quality indicators can only be useful if they are trustworthy signals of current or future quality. Despite extensive longitudinal quality indicator data and many hospital quality public reporting initiatives, research on quality indicator stability over time is scarce and skepticism about their usefulness widespread.

OBJECTIVE

Based on aggregated, widely available hospital-level quality indicators, this paper sought to determine whether quality indicators are stable over time. Implications for health policy were drawn and the limited methodological foundation for stability assessments of hospital-level quality indicators enhanced.

METHODS

Two longitudinal datasets (self-reported and routine data), including all hospitals in Germany and covering the period from 2004 to 2017, were analysed. A logistic regression using Generalized Estimating Equations, a time-dependent, graphic quintile representation of risk-adjusted rates and Spearman's rank correlation coefficient were used.

RESULTS

For a total of eight German quality indicators significant stability over time was demonstrated. The probability of remaining in the best quality cluster in the future across all hospitals reached from 46.9% (CI: 42.4-51.6%) for hip replacement reoperations to 80.4% (CI: 76.4-83.8%) for decubitus. Furthermore, graphical descriptive analysis showed that the difference in adverse event rates for the 20% top performing compared to the 20% worst performing hospitals in the two following years is on average between 30% for stroke and AMI and 79% for decubitus. Stability over time has been shown to vary strongly between indicators and treatment areas.

CONCLUSION

Quality indicators were found to have sufficient stability over time for public reporting. Potentially, increasing case volumes per hospital, centralisation of medical services and minimum-quantity regulations may lead to more stable and reliable quality of care indicators. Finally, more robust policy interventions such as outcome-based payment, should only be applied to outcome indicators with a higher level of stability over time. This should be subject to future research.

Effects of different palliative care models on decedents with kidney failure receiving maintenance dialysis: a nationwide population-based retrospective observational study in Taiwan

OBJECTIVES

Patients with kidney failure receiving maintenance dialysis are a particularly important population and carry a heavy disease burden. However, evidence related to palliative care for patients with kidney failure receiving maintenance dialysis remains scarce, especially in regard to palliative care consultation services and palliative home care. This study aimed to evaluate the effects of different palliative care models on aggressive treatment among patients with kidney failure receiving maintenance dialysis during the end of life.

DESIGN

A population-based retrospective observational study.

SETTING

This study used a population database maintained by Taiwan's Ministry of Health and Welfare in combination with Taiwan's National Health Research Insurance Database.

PARTICIPANTS

We enrolled all decedents who were patients with kidney failure receiving maintenance dialysis from the period 1 January 2017 to 31 December 2017 in Taiwan.

MAIN EXPOSURE MEASURE

Hospice care during the 1-year period before death.

MAIN OUTCOME MEASURES

Eight aggressive treatments within 30 days before death, more than one emergency department visit, more than one admission, a longer than 14-day admission, admission to an intensive care unit, death in hospital, endotracheal tube use, ventilator use and need for cardiopulmonary resuscitation.

RESULTS

A total of 10 083 patients were enrolled, including 1786 (17.7%) patients with kidney failure who received palliative care 1 year before death. Compared with patients without palliative care, patients with palliative care had significantly less aggressive treatments within 30 days before death (Estimates: -0.09, CI: -0.10 to -0.08). Patients with inpatient palliative care, palliative home care or a mixed model experienced significantly lower treatment aggressiveness within 30 days before death.

CONCLUSIONS

Palliative care, particularly use of a mixed care model, inpatient palliative care and palliative home care in patients with kidney failure receiving dialysis, could all significantly reduce the aggressiveness of treatment within 30 days before death.

The impact of surgical volume on hospital ranking using the standardized infection ratio

The Centers for Medicare and Medicaid Services require hospitals to report on quality metrics which are used to financially penalize those that perform in the lowest quartile. Surgical site infections (SSIs) are a critical component of the quality metrics that target healthcare-associated infections. However, the accuracy of such hospital profiling is highly affected by small surgical volumes which lead to a large amount of uncertainty in estimating standardized hospital-specific infection rates. Currently, hospitals with less than one expected SSI are excluded from rankings, but the effectiveness of this exclusion criterion is unknown. Tools that can quantify the classification accuracy and can determine the minimal surgical volume required for a desired level of accuracy are lacking. We investigate the effect of surgical volume on the accuracy of identifying poorly performing hospitals based on the standardized infection ratio and develop simulation-based algorithms for quantifying the classification accuracy. We apply our proposed method to data from HCA Healthcare (2014-2016) on SSIs in colon surgery patients. We estimate that for a procedure like colon surgery with an overall SSI rate of 3%, to rank hospitals in the HCA colon SSI dataset, hospitals that perform less than 200 procedures have a greater than 10% chance of being incorrectly assigned to the worst performing quartile. Minimum surgical volumes and predicted events criteria are required to make evaluating hospitals reliable, and these criteria vary by overall prevalence and between-hospital variability.

The iterative bisection procedure: a useful tool for determining parameter values in data-generating processes in Monte Carlo simulations

BACKGROUND

Data-generating processes are key to the design of Monte Carlo simulations. It is important for investigators to be able to simulate data with specific characteristics.

METHODS

We described an iterative bisection procedure that can be used to determine the numeric values of parameters of a data-generating process to produce simulated samples with specified characteristics. We illustrated the application of the procedure in four different scenarios: (i) simulating binary outcome data from a logistic model such that the prevalence of the outcome is equal to a specified value; (ii) simulating binary outcome data from a logistic model based on treatment status and baseline covariates so that the simulated outcomes have a specified treatment relative risk; (iii) simulating binary outcome data from a logistic model so that the model c-statistic has a specified value; (iv) simulating time-to-event outcome data from a Cox proportional hazards model so that treatment induces a specified marginal or population-average hazard ratio.

RESULTS

In each of the four scenarios the bisection procedure converged rapidly and identified parameter values that resulted in the simulated data having the desired characteristics.

CONCLUSION

An iterative bisection procedure can be used to identify numeric values for parameters in data-generating processes to generate data with specified characteristics.

Operating Room Performance Optimization Metrics: a Systematic Review

Literature proposes numerous initiatives for optimization of the Operating Room (OR). Despite multiple suggested strategies for the optimization of workflow on the OR, its patients and (medical) staff, no uniform description of 'optimization' has been adopted. This makes it difficult to evaluate the proposed optimization strategies. In particular, the metrics used to quantify OR performance are diverse so that assessing the impact of suggested approaches is complex or even impossible. To secure a higher implementation success rate of optimisation strategies in practice we believe OR optimisation and its quantification should be further investigated. We aim to provide an inventory of the metrics and methods used to optimise the OR by the means of a structured literature study. We observe that several aspects of OR performance are unaddressed in literature, and no studies account for possible interactions between metrics of quality and efficiency. We conclude that a systems approach is needed to align metrics across different elements of OR performance, and that the wellbeing of healthcare professionals is underrepresented in current optimisation approaches.

Application of power analysis to determine the optimal reporting time frame for use in statewide trauma system quality reporting

BACKGROUND

Meaningful reporting of quality metrics relies on detecting a statistical difference when a true difference in performance exists. Larger cohorts and longer time frames can produce higher rates of statistical differences. However, older data are less relevant when attempting to enact change in the clinical setting. The selection of time frames must reflect a balance between being too small (type II errors) and too long (stale data). We explored the use of power analysis to optimize time frame selection for trauma quality reporting.

METHODS

Using data from 22 Level III trauma centers, we tested for differences in 4 outcomes within 4 cohorts of patients. With bootstrapping, we calculated the power for rejecting the null hypothesis that no difference exists amongst the centers for different time frames. From the entire sample for each site, we simulated randomly generated datasets. Each simulated dataset was tested for whether a difference was observed from the average. Power was calculated as the percentage of simulated datasets where a difference was observed. This process was repeated for each outcome.

RESULTS

The power calculations for the 4 cohorts revealed that the optimal time frame for Level III trauma centers to assess whether a single site's outcomes are different from the overall average was 2 years based on an 80% cutoff.

CONCLUSION

Power analysis with simulated datasets allows testing of different time frames to assess outcome differences. This type of analysis allows selection of an optimal time frame for benchmarking of Level III trauma center data.

Composite measures of quality of health care: Evidence mapping of methodology and reporting

Background

Quality indicators are used to quantify the quality of care. A large number of quality indicators makes assessment of overall quality difficult, time consuming and impractical. There is consequently an increasing interest for composite measures based on a combination of multiple indicators.

Objective

To examine the use of different approaches to construct composite measures of quality of care and to assess the use of methodological considerations and justifications.

Methods

We conducted a literature search on PubMed and EMBASE databases (latest update 1 December 2020). For each publication, we extracted information on the weighting and aggregation methodology that had been used to construct composite indicator(s).

Results

A total of 2711 publications were identified of which 145 were included after a screening process. Opportunity scoring with equal weights was the most used approach (86/145, 59%) followed by all-or-none scoring (48/145, 33%). Other approaches regarding aggregation or weighting of individual indicators were used in 32 publications (22%). The rationale for selecting a specific type of composite measure was reported in 36 publications (25%), whereas 22 papers (15%) addressed limitations regarding the composite measure.

Conclusion

Opportunity scoring and all-or-none scoring are the most frequently used approaches when constructing composite measures of quality of care. The attention towards the rationale and limitations of the composite measures appears low.

Discussion

Considering the widespread use and the potential implications for decision-making of composite measures, a high level of transparency regarding the construction process of the composite and the functionality of the measures is crucial.

A more comprehensive evaluation of quality of care after total hip and knee arthroplasty: combining 4 indicators in an ordered composite outcome

Background and purpose - Most arthroplasty registers give hospital-specific feedback on revision rates after total hip and knee arthroplasties (THA/TKA). However, due to the low number of events per hospital, multiple years of data are required to reliably detect worsening performance, and any single indicator provides only part of the quality of care delivered. Therefore, we developed an ordered composite outcome including revision, readmission, complications, and long length-of-stay (LOS) for a more comprehensive view on quality of care and assessed the ability to reliably differentiate between hospitals in their performance (rankability) with fewer years of data. Methods - All THA and TKA performed between 2017 and 2019 in 20 Dutch hospitals were included. All combinations of the 4 indicators were ranked from best to worst to create the ordinal composite outcome for THA and TKA separately. Between-hospital variation for the composite outcome was compared with individual indicators standardized for case-mix differences, and we calculated the statistical rankability using fixed and random effects models. Results - 22,908 THA and 20,423 TKA were included. Between-hospital variation for the THA and TKA composite outcomes was larger when compared with revision, readmission, and complications, and similar to long LOS. Rankabilities for the composite outcomes were above 80% even with 1 year of data, meaning that largely true hospital differences were detected rather than random variation. Interpretation - The ordinal composite outcome gives a more comprehensive overview of quality of delivered care and can reliably differentiate between hospitals in their performance using 1 year of data, thereby allowing earlier introduction of quality improvement initiatives.

Comparison of cost and complication rates for profiling hospital performance in lumbar fusion for spondylolisthesis

BACKGROUND CONTEXT

There is growing interest among payers in profiling hospital value and quality-of-care, including both the cost and safety of common surgeries, such as lumbar fusion. Nonetheless, there is sparse evidence describing the statistical reliability of such measures when applied to lumbar fusion for spondylolisthesis.

PURPOSE

To evaluate the reliability of 90-day inpatient hospital costs, overall complications, and rates of serious complications for profiling hospital performance in lumbar fusion surgery for spondylolisthesis.

STUDY DESIGN/SETTING

Data for this analysis came from State Inpatient Databases from nine states made available through the Healthcare Cost and Utilization Project.

PATIENT SAMPLE

Patients undergoing elective lumbar spine fusion for spondylolisthesis from 2010 to 2017 in participating states.

OUTCOME MEASURES

Statistical reliability, defined as the ability to distinguish true performance differences across hospitals relative to statistical noise. Reliability was assessed separately for 90-day inpatient costs (standardized across years to 2019 dollars), overall complications, and serious complication rates.

METHODS

Statistical reliability was measured as the amount of variation between hospitals relative to the total amount of variation for each measure. Total variation includes both between-hospital variation ("signal") and within-hospital variation ("statistical noise"). Thus, reliability equals signal over (signal plus noise) and ranges from 0 to 1. To adjust for differences in patient-level risk and procedural characteristics, hierarchical linear and logistic regression models were created for the cost and complication outcomes. Random hospital intercepts were used to assess between-hospital variation. We evaluated the reliability of each measure by study year and examined the number of hospitals meeting different thresholds of reliability by year.

RESULTS

We included a total of 66,571 elective lumbar fusion surgeries for spondylolisthesis performed at 244 hospitals during the study period. The mean 90-day hospital cost was $30,827 (2019 dollars). 12.0% of patients experienced a complication within 90 days of surgery, including 7.8% who had a serious complication. The median reliability of 90-day cost ranged from 0.97to 0.99 across study years, and there was a narrow distribution of reliability values. By comparison, the median reliability for the overall complication metric ranged from 0.22 to 0.44, and the reliability of the serious complication measure ranged from 0.30 to 0.49 across the study years. At least 96% of hospitals had high (> 0.7) reliability for cost in any year, whereas only 0-9% and 0-11% of hospitals reached this cutoff for the overall and serious complication rate in any year, respectively. By comparison, 10%-69% of hospitals per year achieved a more moderate threshold of 0.4 reliability for overall complications, compared to 21%-80% of hospitals who achieved this lower reliability threshold for serious complications.

CONCLUSIONS

90-day inpatient costs are highly reliable for assessing variation across hospitals, whereas overall and serious complications are only moderately reliable for profiling performance. These results support the viability of emerging bundled payment programs that assume true differences in costs of care exist across hospitals.

Administrative Data Are Unreliable for Ranking Hospital Performance Based on Serious Complications After Spine Fusion

STUDY DESIGN

Retrospective analysis of administrative billing data.

OBJECTIVE

To evaluate the extent to which a metric of serious complications determined from administrative data can reliably profile hospital performance in spine fusion surgery.

SUMMARY OF BACKGROUND DATA

While payers are increasingly focused on implementing pay-for-performance measures, quality metrics must reliably reflect true differences in performance among the hospitals profiled.

METHODS

We used State Inpatient Databases from nine states to characterize serious complications after elective cervical and thoracolumbar fusion. Hierarchical logistic regression was used to risk-adjust differences in case mix, along with variability from low case volumes. The reliability of this risk-stratified complication rate (RSCR) was assessed as the variation between hospitals that was not due to chance alone, calculated separately by fusion type and year. Finally, we estimated the proportion of hospitals that had sufficient case volumes to obtain reliable (>0.7) complication estimates.

RESULTS

From 2010 to 2017 we identified 154,078 cervical and 213,133 thoracolumbar fusion surgeries. 4.2% of cervical fusion patients had a serious complication, and the median RSCR increased from 4.2% in 2010 to 5.5% in 2017. The reliability of the RSCR for cervical fusion was poor and varied substantially by year (range 0.04-0.28). Overall, 7.7% of thoracolumbar fusion patients experienced a serious complication, and the RSCR varied from 6.8% to 8.0% during the study period. Although still modest, the RSCR reliability was higher for thoracolumbar fusion (range 0.16-0.43). Depending on the study year, 0% to 4.5% of hospitals had sufficient cervical fusion case volume to report reliable (>0.7) estimates, whereas 15% to 36% of hospitals reached this threshold for thoracolumbar fusion.

CONCLUSION

A metric of serious complications was unreliable for benchmarking cervical fusion outcomes and only modestly reliable for thoracolumbar fusion. When assessed using administrative datasets, these measures appear inappropriate for high-stakes applications, such as public reporting or pay-for-performance.Level of Evidence: 3.

Textbook outcome after rectal cancer surgery as a composite measure for quality of care: A population-based study

BACKGROUND

Textbook outcome is a composite measure of combined outcome indicators, which has been suggested to be of additional value over single outcome parameters in clinical auditing of surgical treatment. This study aimed to assess textbook outcome after rectal cancer surgery as short-term marker for quality of care.

MATERIALS AND METHODS

Patients who underwent elective rectal cancer surgery between 2012 and 2019 and registered in the Dutch ColoRectal Audit were included. Textbook outcome was achieved when the following criteria were met: 30-day and primary hospital admission survival, no reintervention, tumour-free margins, no postoperative complications, a hospital stay of less than 14 days and no readmission. Hospital variation was evaluated in case-mix corrected funnel-plots. A multilevel logistic regression analysis was performed to identify associated factors with textbook outcome.

RESULTS

The study population consisted of 20,521 patients who underwent primary rectal cancer surgery, of whom 56.3% achieved textbook outcome. Postoperative complications were the main contributor to not achieving textbook outcome. Case-mix corrected funnel plots demonstrated that underperforming hospitals in 2012-2015 were no underperformers in 2016-2019 anymore. Female sex, laparoscopic surgery, and rectal resection without defunctioning stoma creation were positively associated with textbook outcome.

CONCLUSION

Textbook outcome after rectal cancer resection is mainly driven by postoperative complications. Although textbook outcome showed some discriminating value for identifying underperforming hospitals, it does not fit the plan-do-check-act cycle of clinical auditing. In our opinion, textbook outcome has little added value to the current outcome indicators for rectal cancer surgery.

Assessing Nursing Homes Quality Indicators' Between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability

Nursing home quality indicators are often used to publicly report the quality of nursing home care. In Switzerland, six national nursing home quality indicators covering four clinical domains (polypharmacy, pain, use of physical restraints and weight loss) were recently developed. To allow for meaningful comparisons, these indicators must reliably show differences in quality of care levels between nursing homes. This study’s objectives were to assess nursing home quality indicators’ between-provider variability and reliability using intraclass correlations and rankability. This approach has not yet been used in long-term care contexts but presents methodological advantages. This cross-sectional multicenter study uses data of 11,412 residents from a convenience sample of 152 Swiss nursing homes. After calculating intraclass correlation 1 (ICC1) and rankability, we describe between-provider variability for each quality indicator using empirical Bayes estimate-based caterpillar plots. To assess reliability, we used intraclass correlation 2 (ICC2). Overall, ICC1 values were high, ranging from 0.068 (95% confidence interval (CI) 0.047–0.086) for polypharmacy to 0.396 (95% CI 0.297–0.474) for physical restraints, with quality indicator caterpillar plots showing sufficient between-provider variability. However, testing for rankability produced mixed results, with low figures for two indicators (0.144 for polypharmacy; 0.471 for self-reported pain) and moderate to high figures for the four others (from 0.692 for observed pain to 0.976 for physical restraints). High ICC2 figures, ranging from 0.896 (95% CI 0.852–0.917) (self-reported pain) to 0.990 (95% CI 0.985–0.993) (physical restraints), indicated good reliability for all six quality indicators. Intraclass correlations and rankability can be used to assess nursing home quality indicators’ between-provider variability and reliability. The six selected quality indicators reliably distinguish care differences between nursing homes and can be recommended for use, although the variability of two—polypharmacy and self-reported pain—is substantially chance-driven, limiting their utility.

Variation in clinical outcomes and process of care measures in community acquired pneumonia: a systematic review

Background

Variation in outcomes of patients with community acquired pneumonia (CAP) has been reported in some, but not all, studies. Although some variation is expected, unwarranted variation in healthcare impacts patient outcomes and equity of care. The aim of this systematic review was to: i) summarise current evidence on regional and inter-hospital variation in the clinical outcomes and process of care measures of patients hospitalised with CAP and ii) assess the strength of this evidence.

Methods

Databases were systematically searched from inception to February 2018 for relevant studies and data independently extracted by two investigators in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Included studies enrolled adults hospitalised with CAP and reported a measure of variation between two or more units in healthcare outcomes or process of care measures. Outcomes of interest were mortality, length of hospital stay (LOS) and re-admission rates. A structured synthesis of the studies was performed.

Results

Twenty-two studies were included in the analysis. The median number of units compared across studies was five (IQR 4–15). Evidence for variation in mortality between units was inconsistent; of eleven studies that performed statistical significance testing, five found significant variation. For LOS, of nine relevant studies, all found statistically significant variation. Four studies reported site of admission accounted for 1–24% of the total observed variation in LOS. A shorter LOS was not associated with increased mortality or readmission rates. For readmission, evidence was mixed; of seven studies, 4 found statistically significant variation. There was consistent evidence for variation in the use of intensive care, obtaining blood cultures on admission, receiving antibiotics within 8 h of admission and duration of intravenous antibiotics. Across all outcome measures, only one study accounted for natural variation between units in their analysis.

Conclusion

There is consistent evidence of moderate quality for significant variation in length of stay and process of care measures but not for in-patient mortality or hospital re-admission. Evidence linking variation in outcomes with variation in process of care measures was limited; where present no difference in mortality was detected despite POC variation. Adjustment for natural variation within studies was lacking; the proportion of observed variation due to chance is not quantified by existing evidence.

Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability

OBJECTIVES

Hospital comparisons to improve quality of care require valid and reliable quality indicators. We aimed to test the validity and reliability of 6 breast cancer indicators by quantifying the influence of case-mix and random variation.

METHODS

The nationwide population-based database included 79 690 patients with breast cancer from 91 Dutch hospitals between 2011 and 2016. The indicator-scores calculated were: (1) irradical breast-conserving surgery (BCS) for invasive disease, (2) irradical BCS for ductal carcinoma-in-situ, (3) breast contour-preserving treatment, (4) magnetic resonance imaging (MRI) before neo-adjuvant chemotherapy, (5) radiotherapy for locally advanced disease, and (6) surgery within 5 weeks from diagnosis. Case-mix and random variation adjustments were performed by multivariable fixed and random effect logistic regression models. Rankability quantified the between-hospital variation, representing unexplained differences that might be the result of the level of quality of care, as low (<50%), moderate (50%-75%), or high (>75%).

RESULTS

All of the indicators showed between-hospital variation with wide (interquartile) ranges. Case-mix adjustment reduced variation in indicators 1 and 3 to 5. Random variation adjustment (further) reduced the variation for all indicators. Case-mix and random variation adjustments influenced the indicator-scores of individual hospitals and their ranking. Rankability was poor for indicator 1, 2, and 5, and moderate for 3, 4, and 6.

CONCLUSIONS

The 6 indicators lacked validity and/or reliability to a certain extent. Although measuring quality indicators may stimulate quality improvement in general, comparisons and judgments of individual hospital performance should be made with caution if based on indicators that have not been tested or adjusted for validity and reliability, especially in benchmarking.