Funnel plots for comparing institutional performance

Considerable interlaboratory variation in PD-L1 positivity for head and neck squamous cell carcinoma in the Netherlands- A nationwide evaluation study

AIMS

Patients with recurrent or metastatic head and neck squamous cell carcinoma (HNSCC) are eligible for first-line immune checkpoint inhibition if their tumour is positive for programmed death ligand 1 (PD-L1) determined by the combined positive score (CPS). This nationwide study, using real-world data, investigated the developing PD-L1 testing landscape in the first 3 years after introduction of the test in HNSCC and examined interlaboratory variation in PD-L1 positivity rates.

METHODS

Pathology reports of HNSCC patients mentioning PD-L1 were extracted from the Dutch Pathology Registry (Palga). Tumour and PD-L1 testing characteristics were analysed per year and interlaboratory variation in PD-L1 positivity rates was assessed using funnel plots with 95% confidence limits around the overall mean.

RESULTS

A total of 817 PD-L1 tests were reported in 702 patients among 19 laboratories; 85.2% of the tests on histological material were stated to be positive. The national PD-L1 positivity rate differed significantly per year during the study period (79.7-89.9%). The use of the recommended 22C3 antibody increased from 59.9 to 74.3%. A total of 673 PD-L1 tests on histological material from 12 laboratories were analysed to investigate interlaboratory variation. Four (33%) deviated significantly from the national mean of PD-L1-positive cases using CPS ≥ 1 cut-off, while two (17%) deviated significantly for CPS ≥ 20 cut-off.

CONCLUSION

In the first 3 years of PD-L1 assessment in HNSCC, the testing landscape became more uniform. However, interlaboratory variation in PD-L1 positivity rates between Dutch laboratories was substantial. This implies that there is a need for further test standardisation to reduce this variation.

Individualized Surgeon Reports in a Statewide Registry

ABSTRACT

Despite progress with the development of joint replacement registries in the United States, surgeons may have limited opportunities to determine the cumulative outcome of their own patients or understand how those outcomes compare with their peers; this information is important for quality improvement. In order to provide surgeons with accurate data, it is first necessary to have a registry with complete coverage and patient matching. Some international registries have accomplished this. Building on a comprehensive statewide registry in the United States, a surgeon-specific report has been developed to provide surgeons with survivorship and complication data, which allows comparisons with other surgeons in the state. This article describes funnel plots, cumulative sum reports, complication-specific data, and patient-reported outcome measure data, which are provided to hip and knee arthroplasty surgeons with the goal of improving quality, decreasing variability in the delivery of care, and leading to improved value and outcomes for hip and knee arthroplasty in the state of Michigan.

Individualized empirical null estimation for exact tests of healthcare quality

United States federal agencies evaluate healthcare providers to identify, flag, and potentially penalize those that deliver low-quality care compared to national expectations. In practice, evaluation metrics are inevitably impacted by unobserved confounding factors, which reduce flagging accuracy and cause the statistics to be overdispersed relative to the theoretical null distributions. In response to this issue, several authors have proposed individualized empirical null (IEN) methods to estimate an appropriate null distribution for each provider's evaluation statistic while taking into account the provider's effective size. However, existing IEN methods require that the statistics asymptotically follow normal distributions, which often does not hold in applications with small providers or misspecified models. In this article, we develop an IEN framework for exact hypothesis tests that accounts for the impact of unobserved confounding without making any asymptotic assumptions. Simulations show that the proposed IEN method has greater flagging accuracy compared to conventional approaches. We apply these methods to evaluate dialysis facilities and transplant centers that are monitored by the Centers for Medicare and Medicaid Services.

Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study

BACKGROUND

Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.

METHODS

We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.

RESULTS

All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.

CONCLUSION

Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.

Factors associated with conversion from day-case to in-patient elective inguinal hernia repair surgery across England: an observational study using administrative data

PURPOSE

Elective primary inguinal hernia repair surgery is increasingly being conducted as a day-case procedure. However, some patients planned for day-case surgery have to stay in hospital for at least one night. The aim of this study was to identify the factors associated with conversion from day-case to in-patient management for elective inguinal hernia repair surgery.

METHODS

This was an exploratory retrospective analysis of observational data from the Hospital Episode Statistics dataset for England. All patients aged ≥ 17 years undergoing a first elective inguinal hernia repair between 1st April 2014 and 31st March 2022 that was planned as day-case surgery were identified. The exposure of interest was discharged on the day of admission (day-case) or requiring overnight stay. The primary outcome of interest was 30-day emergency readmission with an overnight stay. For reporting, providers were aggregated to an Integrated Care Board (ICB) level.

RESULTS

A total of 351,528 planned day-case elective primary inguinal hernia repairs were identified over the eight-year study period. Of these, 45,305 (12.9%) stayed in hospital for at least one night and were classed as day-case to in-patient stay conversions. Patients who converted to in-patient stay were older, had more comorbidities, and were more likely to have bilateral surgery and be operated on by a low-annual volume surgeon. Post-procedural complications were strongly associated with conversion. Across the 42 ICBs in England, model-adjusted conversion rates varied from 3.3% to 21.3%.

CONCLUSIONS

There was considerable variation in conversion to in-patient stay rates for inguinal hernia repair across ICBs in England. Our findings should help surgical teams to better identify patients suitable for day-case inguinal hernia repair and plan discharge services more effectively. This should help to reduce the variation in conversion rates.

Outcome variation and the role of caseload in certified colorectal cancer centers - a retrospective cohort analysis of 90,000 cases

BACKGROUND

Studies have shown that surgical treatment of colorectal carcinomas in certified centers leads to improved outcomes. However, there were considerable fluctuations in outcome parameters. It has not yet been examined whether this variability is due to continuous differences between hospitals or variability within a hospital over time.

MATERIALS AND METHODS

In this retrospective observational cohort study, administrative quality assurance data of 153 German-certified colorectal cancer centers between 2010 and 2019 were analyzed. Six outcome quality indicators (QI) were studied: 30-day postoperative mortality rate (POM), surgical site infection rate (SSI), anastomotic insufficiency rate (AI), and revision surgery rate (RS). AI and RS were also analyzed for colon (C) and rectal cancer operations (R). Variability was analyzed by funnel plots with 95% and 99% control limits and modified Cleveland dot plots.

RESULTS

In the 153 centers 90,082 patients with colon cancer and 47,623 patients with rectal cancer were treated. Average QI scores were 2.7% POM, 6.2% SSI, 4.8% AI-C, 8.5% AI-R, 9.1% RS-C, and 9.8% RS-R. The funnel plots revealed that for every QI about 10.1% of hospitals lay above the upper 99% and about 8.7% below the lower 99% control limit. In POM, SSI, and AI-R, a significant negative correlation with the average annual caseload was observed.

CONCLUSION

The analysis showed high variability in outcome quality between and within the certified colorectal cancer centers. Only a small number of hospitals had a high performance on all six quality indicators, suggesting that significant quality variation exists even within the group of certified centers.

Improving risk models for patients having emergency bowel cancer surgery using linked electronic health records: a national cohort study

BACKGROUND

Life-saving emergency major resection of colorectal cancer (CRC) is a high-risk procedure. Accurate prediction of postoperative mortality for patients undergoing this procedure is essential for both healthcare performance monitoring and preoperative risk assessment. Risk-adjustment models for CRC patients often include patient and tumour characteristics, widely available in cancer registries and audits. The authors investigated to what extent inclusion of additional physiological and surgical measures, available through linkage or additional data collection, improves accuracy of risk models.

METHODS

Linked, routinely-collected data on patients undergoing emergency CRC surgery in England between December 2016 and November 2019 were used to develop a risk model for 90-day mortality. Backwards selection identified a 'selected model' of physiological and surgical measures in addition to patient and tumour characteristics. Model performance was assessed compared to a 'basic model' including only patient and tumour characteristics. Missing data was multiply imputed.

RESULTS

Eight hundred forty-six of 10 578 (8.0%) patients died within 90 days of surgery. The selected model included seven preoperative physiological and surgical measures (pulse rate, systolic blood pressure, breathlessness, sodium, urea, albumin, and predicted peritoneal soiling), in addition to the 10 patient and tumour characteristics in the basic model (calendar year of surgery, age, sex, ASA grade, TNM T stage, TNM N stage, TNM M stage, cancer site, number of comorbidities, and emergency admission). The selected model had considerably better discrimination compared to the basic model (C-statistic: 0.824 versus 0.783, respectively).

CONCLUSION

Linkage of disease-specific and treatment-specific datasets allowed the inclusion of physiological and surgical measures in a risk model alongside patient and tumour characteristics, which improves the accuracy of the prediction of the mortality risk for CRC patients having emergency surgery. This improvement will allow more accurate performance monitoring of healthcare providers and enhance clinical care planning.

[Hospital standardized mortality ratio: limits and potential of the indicator for assessing hospital performance in the Brazilian Unified National Health System]

Comparative analyses based on clinical performance indicators to monitor the quality of hospital care have been carried out for decades in several countries, most notably the hospital standardized mortality ratio (HSMR). In Brazil, studies and the adoption of methodological tools that allow regular analysis of the performance of institutions are still scarce. This study aimed to assess the use of HSMR to compare the performance of hospitals funded by the Brazilian Unified National Health System (SUS). The Hospital Information System was the source of data on adult hospitalizations in Brazil from 2017 to 2019. The methodological approach to estimate HSMR was adapted to the available data and included the causes of hospitalization (main diagnosis) responsible for 80% of deaths. The number of expected deaths was estimated using a logistic regression model that included predictor variables widely described in the literature. The analysis was conducted in two stages: (i) hospitalization level and (ii) hospital level. The final risk adjustment model showed a C-statistic of 0.774, which is considered adequate. The variation in HSMR was wide, especially among the worst-performing hospitals (1.54 to 6.77). Private hospitals performed better than public hospitals. Although the limits of the available data and the challenges still face its more refined use, HSMR is applicable and has the potential to become an important tool for assessing hospital performance in the SUS.

Variations in surgical practice and short-term outcomes for degenerative lumbar scoliosis and spondylolisthesis: do surgeon training and experience matter?

For diverse procedures, sizable geographic variation exists in rates and outcomes of surgery, including for degenerative lumbar spine conditions. Little is known about how surgeon training and experience are associated with surgeon-level variations in spine surgery practice and short-term outcomes. This retrospective observational analysis characterized variations in surgical operations for degenerative lumbar scoliosis or spondylolisthesis, two common age-related conditions. The study setting was two large spine surgery centers in one region during 2017-19. Using data (International Classification of Diseases-10th edition and current procedural terminology codes) extracted from electronic health record systems, we characterized surgeon-level variations in practice (use of instrumented fusion - a more extensive procedure that involves device-related risks) and short-term postoperative outcomes (major in-hospital complications and readmissions). Next, we tested for associations between surgeon training (specialty and spine fellowship) and experience (career stage and operative volume) and use of instrumented fusion as well as outcomes. Eighty-nine surgeons performed 2481 eligible operations. For the study diagnoses, spine surgeons exhibited substantial variation in operative volume, use of instrumented fusion, and postoperative outcomes. Among surgeons above the median operative volume, use of instrumented fusion ranged from 0% to >90% for scoliosis and 9% to 100% for spondylolisthesis, while rates of major in-hospital complications ranged from 0% to 25% for scoliosis and from 0% to 14% for spondylolisthesis. For scoliosis, orthopedic surgeons were more likely than neurosurgeons to perform instrumented fusion for scoliosis [49% vs. 33%, odds ratio (OR) = 2.3, 95% confidence interval (95% CI) 1.3-4.2, P-value = .006] as were fellowship-trained surgeons (49% vs. 25%, OR = 3.0, 95% CI 1.6-5.8; P = .001). Fellowship-trained surgeons had lower readmission rates. Surgeons with higher operative volumes used instrumented fusion more often (OR = 1.1, 95% CI 1.0-1.2, P < .05 for both diagnoses) and had lower rates of major in-hospital complications (OR = 0.91, 95% CI 0.85-0.97; P = .006). Surgical practice can vary greatly for degenerative spine conditions, even within the same region and among colleagues at the same institution. Surgical specialty and subspecialty, in addition to recent operative volume, can be linked to variations in spine surgeons' practice patterns and outcomes. These findings reinforce the notion that residency and fellowship training may contribute to variation and present important opportunities to optimize surgical practice over the course of surgeons' careers. Future efforts to reduce unexplained variation in surgical practice could test interventions focused on graduate medical education. Graphical Abstract.

Regional variation and temporal trends in transcatheter and surgical aortic valve replacement in Switzerland: A population-based small area analysis

BACKGROUND

Aortic valve stenosis (AS) is the most common valvular heart disease and if severe, is treated with either transcatheter (TAVR) or surgical aortic valve replacement (SAVR). We assessed temporal trends and regional variation of these interventions in Switzerland and examined potential determinants of geographic variation.

METHODS

We conducted a population-based analysis using patient discharge data from all Swiss public and private acute care hospitals from 2013 to 2018. We generated hospital service areas (HSAs) based on patient flows for TAVR. We calculated age-standardized mean procedure rates and variation indices (extremal quotient [EQ] and systematic component of variation [SCV]). Using multilevel regression, we calculated the influence of calendar year and regional demographics, socioeconomic factors (language, insurance status), burden of disease, and number of cardiologists/cardiovascular surgeons on geographic variation.

RESULTS

Overall, 8074 TAVR and 11,825 SAVR procedures were performed in 8 HSAs from 2013 to 2018. Whereas the age-/sex-standardized rate of TAVR increased from 12 to 22 procedures/100,000 persons, the SAVR rate decreased from 33 to 24 procedures during this period. After full adjustment, the predicted TAVR and SAVR rates varied from 12 to 22 and 20 to 35 per 100,000 persons across HSAs, respectively. The regional procedure variation was low to moderate over time, with a low overall variation in TAVR (EQ 1.9, SCV 3.9) and SAVR (EQ 1.6, SCV 2.2). In multilevel regression, TAVR rates increased annually by 10% and SAVR rates decreased by 5%. Determinants of higher TAVR rates were older age, male sex, living in a German speaking area, and higher burden of disease. A higher proportion of (semi)private insurance was also associated with higher TAVR and lower SAVR rates. After full adjustment, 10.6% of the variance in TAVR and 18.4% of the variance in SAVR remained unexplained. Most variance in TAVR and SAVR rates was explained by language region and insurance status.

CONCLUSION

The geographic variation in TAVR and SAVR rates was low to moderate across Swiss regions and largely explained by differences in regional demographics and socioeconomic factors. The use of TAVR increased at the expense of SAVR over time.

Large regional variation in cardiac closure procedures to prevent ischemic stroke in Switzerland a population-based small area analysis

BACKGROUND

Percutaneous closure of a patent foramen ovale (PFO) or the left atrial appendage (LAA) are controversial procedures to prevent stroke but often used in clinical practice. We assessed the regional variation of these interventions and explored potential determinants of such a variation.

METHODS

We conducted a population-based analysis using patient discharge data from all Swiss hospitals from 2013-2018. We derived hospital service areas (HSAs) using patient flows for PFO and LAA closure. We calculated age-standardized mean procedure rates and variation indices (extremal quotient [EQ] and systematic component of variation [SCV]). SCV values >5.4 indicate a high and >10 a very high variation. Because the evidence on the efficacy of PFO closure may differ in patients aged <60 years and ≥60 years, age-stratified analyses were performed. We assessed the influence of potential determinants of variation using multilevel regression models with incremental adjustment for demographics, cultural/socioeconomic, health, and supply factors.

RESULTS

Overall, 2574 PFO and 2081 LAA closures from 10 HSAs were analyzed. The fully adjusted PFO and LAA closure rates varied from 3 to 8 and from 1 to 9 procedures per 100,000 persons per year across HSAs, respectively. The regional variation was high with respect to overall PFO closures (EQ 3.0, SCV 8.3) and very high in patients aged ≥60 years (EQ 4.0, SCV 12.3). The variation in LAA closures was very high (EQ 16.2, SCV 32.1). In multivariate analysis, women had a 28% lower PFO and a 59% lower LAA closure rate than men. French/Italian language areas had a 63% lower LAA closure rate than Swiss German speaking regions and areas with a higher proportion of privately insured patients had a 86% higher LAA closure rate. After full adjustment, 44.2% of the variance in PFO closure and 30.3% in LAA closure remained unexplained.

CONCLUSIONS

We found a high to very high regional variation in PFO closure and LAA closure rates within Switzerland. Several factors, including sex, language area, and insurance status, were associated with procedure rates. Overall, 30-45% of the regional procedure variation remained unexplained and most probably represents differing physician practices.

Physician Variation and the Impact of Payment Model in Cardiac Imaging

BACKGROUND

The influence of fee-for-service reimbursement on cardiac imaging has not been compared with other payment models. Furthermore, variation in ordering practices is not well understood.

METHODS AND RESULTS

This retrospective, population-based cohort study using linked administrative data from Alberta, Canada included adults with chronic heart disease (atrial fibrillation, coronary artery disease, and heart failure) seen by cardiac specialists for a new outpatient consultation April 2012 to December 2018. Generalized linear mixed-effects models estimated the association of payment model (including the ability to bill to interpret imaging tests) and the use of cardiac imaging and quantified variation in cardiac imaging. Among 31 685 adults seen by 308 physicians at 136 sites, patients received an observed mean of 0.67 (95% CI, 0.67-0.68) imaging tests per consultation. After adjustment, patients seeing fee-for-service physicians had 2.07 (95% CI, 1.68-2.54) and fee-for-service physicians with ability to interpret had 2.87 (95% CI, 2.16-3.81) times the rate of receiving a test than those seeing salaried physicians. Measured patient, physician, and site effects accounted for 31% of imaging variation and, following adjustment, reduced unexplained site-level variation 40% and physician-level variation 29%.

CONCLUSIONS

We identified substantial variation in the use of outpatient cardiac imaging related to physician and site factors. Physician payment models have a significant association with imaging use. Our results raise concern that payment models may influence cardiac imaging practice. Similar methods could be applied to identify the source and magnitude of variation in other health care processes and outcomes.

CGR-CUSUM: a continuous time generalized rapid response cumulative sum chart

Rapidly detecting problems in the quality of care is of utmost importance for the well-being of patients. Without proper inspection schemes, such problems can go undetected for years. Cumulative sum (CUSUM) charts have proven to be useful for quality control, yet available methodology for survival outcomes is limited. The few available continuous time inspection charts usually require the researcher to specify an expected increase in the failure rate in advance, thereby requiring prior knowledge about the problem at hand. Misspecifying parameters can lead to false positive alerts and large detection delays. To solve this problem, we take a more general approach to derive the new Continuous time Generalized Rapid response CUSUM (CGR-CUSUM) chart. We find an expression for the approximate average run length (average time to detection) and illustrate the possible gain in detection speed by using the CGR-CUSUM over other commonly used monitoring schemes on a real-life data set from the Dutch Arthroplasty Register as well as in simulation studies. Besides the inspection of medical procedures, the CGR-CUSUM can also be used for other real-time inspection schemes such as industrial production lines and quality control of services.

Geographical Variation in Underlying Social Deprivation, Cardiovascular and Other Comorbidities in Patients with Potentially Curable Cancers in England: Results from a National Registry Dataset Analysis

AIMS

To describe the prevalence of cardiovascular disease (CVD), multiple comorbidities and social deprivation in patients with a potentially curable cancer in 20 English Cancer Alliances.

MATERIALS AND METHODS

This National Registry Dataset Analysis used national cancer registry data and CVD databases to describe rates of CVD, comorbidities and social deprivation in patients diagnosed with a potentially curable malignancy (stage I-III breast cancer, stage I-III colon cancer, stage I-III rectal cancer, stage I-III prostate cancer, stage I-IIIA non-small cell lung cancer, stage I-IV diffuse large B-cell lymphoma, stage I-IV Hodgkin lymphoma) between 2013 and 2018. Outcome measures included observation of CVD prevalence, other comorbidities (evaluated by the Charlson Comorbidity Index) and deprivation (using the Index of Multiple Deprivation) according to tumour site and allocation to Cancer Alliance. Patients were allocated to CVD prevalence tertiles (minimum: <33.3rd percentile; middle: 33.3rd to 66.6th percentile; maximum: >66.6th percentile).

RESULTS

In total, 634 240 patients with a potentially curable malignancy were eligible. The total CVD prevalence for all cancer sites varied between 13.4% (CVD n = 2058; 95% confidence interval 12.8, 13.9) and 19.6% (CVD n = 7818; 95% confidence interval 19.2, 20.0) between Cancer Alliances. CVD prevalence showed regional variation both for male (16-26%) and female patients (8-16%) towards higher CVD prevalence in northern Cancer Alliances. Similar variation was observed for social deprivation, with the proportion of cancer patients being identified as most deprived varying between 3.3% and 32.2%, depending on Cancer Alliance. The variation between Cancer Alliance for total comorbidities was much smaller.

CONCLUSION

Social deprivation, CVD and other comorbidities in patients with a potentially curable malignancy in England show significant regional variations, which may partly contribute to differences observed in treatments and outcomes.

The association of demographics, frailty and multiple health conditions with outcomes from acute medical admissions to hospitals in England: exploratory analysis of an administrative dataset

Emergency and acute hospital services in England are under increasing pressure. The aim of this study was to investigate the association between key case-mix indicators and outcomes for adults admitted to hospital with an acute medical condition in England. All patients aged ≥16 years admitted to hospital in England as an acute unselected medical admission and who survived to discharge during the financial year 2021-2022 were included. Length of hospital stay was the primary outcome of interest. Data were available for 1,586,168 unique patients. A case-mix index was developed with a score that ranged from 0 to 12. Frailty was the most important variable in the index, followed by multiple health conditions and patient age. The mean case-mix score across hospital trusts in England ranged from 5.3 to 7.8. The case-mix index will support initiatives to better understand factors contributing to outcomes from acute medical admissions to hospital.

Mastectomy patterns among older women with early invasive breast cancer in England and Wales: A population-based cohort study

INTRODUCTION

Older women with early invasive breast cancer (EIBC) are more likely to receive a mastectomy compared with younger women. This study assessed factors associated with receiving a mastectomy among older women with EIBC, with a particular focus on comorbidity and frailty.

MATERIALS AND METHODS

Women diagnosed with EIBC (stages I-IIIa) aged ≥50 years from 2014 to 2019 in English and Welsh NHS organisations who received breast surgery were identified from cancer registration datasets linked to routine hospital data. Separate multivariable logistic regression models explored factors associated with mastectomy use, within each tumour stage (T1-T3). For each tumour stage, risk-adjusted rates of mastectomy were calculated for each NHS organisation and displayed using funnel plots.

RESULTS

We included 106,952 women with EIBC: 23.4% received a mastectomy as their first breast cancer surgery. Receipt of mastectomy was more common among patients with a higher tumour stage (T1: 12.3%; T2: 37.6%; T3: 77.5%), and mastectomy use increased with age within each tumour stage category (50-59 vs 80 + years: 11.8% vs 26.3% for T1; 31.5% vs 56.9% for T2; 73.4% vs 90.3% for T3). Results from a multivariable regression model showed that more severe frailty was associated with mastectomy use for women with T1 (p = 0.002) or T2 (p = 0.003) tumours, but may not be for women with T3 tumours (p = 0.041). There was no association between comorbidity and mastectomy use after accounting for frailty (all p > 0.1). Adjusting for clinical and patient factors only slightly reduced the association between age and mastectomy use. Variation in mastectomy use between NHS organisations was greatest for women with T2 EIBC (unadjusted range: 17.7% to 68.4%).

DISCUSSION

Older women with EIBC are more commonly treated with mastectomy. This could not be explained by tumour characteristics or physical fitness, raising questions about whether surgical decision-making inconsistently incorporates information on patient fitness and functional age.

Variation in Anesthesiology Provider-Volume for Complex Gastrointestinal Cancer Surgery: A Population-Based Study

OBJECTIVE

Examine between-hospital and between-anesthesiologist variation in anesthesiology provider-volume (PV) and delivery of high-volume anesthesiology care.

BACKGROUND

Better outcomes for anesthesiologists with higher PV of complex gastrointestinal cancer surgery have been reported. The factors linking anesthesiology practice and organization to volume are unknown.

METHODS

We identified patients undergoing elective esophagectomy, hepatectomy, and pancreatectomy using linked administrative health data sets (2007-2018). Anesthesiology PV was the annual number of procedures done by the primary anesthesiologist in the 2 years before the index surgery. High-volume anesthesiology was PV>6 procedures/year. Funnel plots to described variation in anesthesiology PV and delivery of high-volume care. Hierarchical regression models examined between-anesthesiologist and between-hospital variation in delivery of high-volume care use with variance partition coefficients (VPCs) and median odds ratios (MORs).

RESULTS

Among 7893 patients cared for at 17 hospitals, funnel plots showed variation in anesthesiology PV (median ranging from 1.5, interquartile range: 1-2 to 11.5, interquartile range: 8-16) and delivery of HV care (ranging from 0% to 87%) across hospitals. After adjustment, 32% (VPC 0.32) and 16% (VPC: 0.16) of the variation were attributable to between-anesthesiologist and between-hospital differences, respectively. This translated to an anesthesiologist MOR of 4.81 (95% CI, 3.27-10.3) and hospital MOR of 3.04 (95% CI, 2.14-7.77).

CONCLUSIONS

Substantial variation in anesthesiology PV and delivery of high-volume anesthesiology care existed across hospitals. The anesthesiologist and the hospital were key determinants of the variation in high-volume anesthesiology care delivery. This suggests that targeting anesthesiology structures of care could reduce variation and improve delivery of high-volume anesthesiology care.

Assessing national patterns and outcomes of pituitary surgery: is hospital administrative data good enough?

PURPOSE

Patterns of surgical care, outcomes, and quality of care can be assessed using hospital administrative databases but this requires accurate and complete data. The aim of this study was to explore whether the quality of hospital administrative data was sufficient to assess pituitary surgery practice in England.

METHODS

The study analysed Hospital Episode Statistics (HES) data from April 2013 to March 2018 on all adult patients undergoing pituitary surgery in England. A series of data quality indicators examined the attribution of cases to consultants, the coding of sellar and parasellar lesions, associated endocrine and visual disorders, and surgical procedures. Differences in data quality over time and between neurosurgical units were examined.

RESULTS

A total of 5613 records describing pituitary procedures were identified. Overall, 97.3% had a diagnostic code for the tumour or lesion treated, with 29.7% (n = 1669) and 17.8% (n = 1000) describing endocrine and visual disorders, respectively. There was a significant reduction from the first to the fifth year in records that only contained a pituitary tumour code (63.7%-47.0%, p < .001). The use of procedure codes that attracted the highest tariff increased over time (66.4%-82.4%, p < .001). Patterns of coding varied widely between the 24 neurosurgical units.

CONCLUSION

The quality of HES data on pituitary surgery has improved over time but there is wide variation in the quality of data between neurosurgical units. Research studies and quality improvement programmes using these data need to check it is of sufficient quality to not invalidate their results.

The Impact of a Liver Transplant Program on the Outcomes of Hepatocellular Carcinoma

OBJECTIVE

We sought to evaluate the impact of liver transplantation (LT) programs on the prognosis of hepatocellular carcinoma (HCC) patients who underwent liver resection (LR) and noncurative intent treatment.

BACKGROUND

LT programs have an array of resources and services that would positively affect the prognosis of patients with HCC.

METHODS

Patients who underwent LT, LR, radiotherapy (RT), or chemotherapy (CTx) for HCC between 2004 and 2018 were included in the National Cancer Database. Institutions with LT programs were defined as those that performed 1 or more LT for at least 5 years. Centers were stratified by hospital volume. The impact of LT programs was assessed after propensity score matching to achieve covariate balance.

RESULTS

A total of 71,735 patients were identified, of which 7997 received LT (11.1%), 12,683 LR (17.7%), 15,675 RT (21.9%), and 35,380 CTx (49.3%). Among a total of 1267 distinct institutions, 94 (7.4%) were categorized as LT programs. Designation as an LT program was also associated with a high volume of LR and noncurative intent treatment (both P <0.001). After propensity score matching, LT programs were associated with better survival among LR and noncurative intent treatment patients. Although hospital volume was also associated with improved prognosis, LT programs were associated with additional survival benefits in noncurative intent treatment. On the other hand, no such benefit was noted in patients who underwent LR.

CONCLUSIONS

The presence of an LT program was associated with a higher volume of LR and noncurative intent treatment. Furthermore, designation as an LT program had a "halo effect" on the prognosis of patients undergoing RT/CTx that went beyond the procedure-volume effect.

Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Total Hip and Knee Arthroplasty Implant Revision Risk to 5 Years From a State-wide Arthroplasty Registry in Michigan

Background

Information on the revision risk of implants is useful for improving the quality of care for elective hip and knee arthroplasty. The purpose of this study was to report on the revision risk of implants using a state-wide registry in the United States.

Methods

The Michigan Arthroplasty Registry Collaborative Quality Initiative systematically collects data on elective primary and revision hip and knee arthroplasty cases in Michigan. It contained data on 139,970 hip and 245,499 knee arthroplasty cases from February 15, 2012, to December 31, 2021. Kaplan-Meier estimates of revision risk were computed using time to first revision as the dependent variable, and the results were computed and expressed as the cumulative percent revision (CPR). CPR estimates were computed for all implants having at least 500 cases in the Michigan Arthroplasty Registry Collaborative Quality Initiative dataset.

Results

At 5-years postoperatively, elective primary conventional total hip arthroplasty implant stem/cup combinations had CPR values from 0.95% (0.39%-2.30%, 95% confidence intervals [CI]) to 5.77% (4.22%-7.85%, 95% CI), and elective primary total knee arthroplasty CPR ranged from 1.10% (0.64%-1.89%, 95% CI) to 12.52% (8.37%-18.50%, 95% CI). Unicondylar knee arthroplasty CPR at 5-years went from 4.23% (3.54%-5.06%, 95% CI) to 7.13% (6.20%-8.20%, 95% CI).

Conclusions

The wide variation in CPR points to the need for surgeons to choose implants wisely to improve quality of care.

Characteristics of home oxygen therapy for preterm infants with bronchopulmonary dysplasia in China: results of a multicenter cohort study

BACKGROUND

Home oxygen therapy (HOT) is indicated upon discharge in some preterm infants with severe bronchopulmonary dysplasia (BPD). There is a lack of evidence-based consensus on the indication for HOT among these infants. Because wide variation in the institutional use of HOT exists, little is known about the role of regional social-economic level in the wide variation of HOT.

METHODS

This was a secondary analysis of Chinese Neonatal Network (CHNN) data from January 1, 2019 to December 31, 2019. Infants at gestational ages < 32 weeks, with a birth weight < 1500 g, and with moderate or severe BPD who survived to discharge from tertiary hospitals located in 25 provinces were included in this study. Infants with major congenital anomalies and those who were discharged against medical advice were excluded.

RESULTS

Of 1768 preterm infants with BPD, 474 infants (26.8%) were discharged to home with oxygen. The proportion of HOT use in participating member hospitals varied from 0 to 89%, with five of 52 hospitals' observing proportions of HOT use that were significantly greater than expected, with 14 hospitals with observing proportions significantly less than expected, and with 33 hospitals with appropriate proportions. We noted a negative correlation between different performance groups of HOT and median GDP per capita (P = 0.04).

CONCLUSIONS

The use of HOT varied across China and was negatively correlated with the levels of provincial economic levels. A local HOT guideline is needed to address the wide variation in HOT use with respect to different regional economic levels in countries like China.

Use of antenatal corticosteroids among infants with gestational age at 24 to 31 weeks in 57 neonatal intensive care units of China: a cross-sectional study

BACKGROUND

Antenatal corticosteroids (ACS) can significantly improve the outcomes of preterm infants. This study aimed to describe the ACS use rates among preterm infants admitted to Chinese neonatal intensive care units (NICU) and to explore perinatal factors associated with ACS use, using the largest contemporary cohort of very preterm infants in China.

METHODS

This cross-sectional study enrolled all infants born at 24 +0 to 31 +6 weeks and admitted to 57 NICUs of the Chinese Neonatal Network from January 1st, 2019 to December 30th, 2019. The ACS administration was defined as at least one dose of dexamethasone and betamethasone given before delivery. Multiple logistic regressions were applied to determine the association between perinatal factors and ACS usage.

RESULTS

A total of 7828 infants were enrolled, among which 6103 (78.0%) infants received ACS. ACS use rates increased with increasing gestational age (GA), from 177/259 (68.3%) at 24 to 25 weeks' gestation to 3120/3960 (78.8%) at 30 to 31 weeks' gestation. Among infants exposed to ACS, 2999 of 6103 (49.1%) infants received a single complete course, and 33.4% (2039/6103) infants received a partial course. ACS use rates varied from 30.2% to 100% among different hospitals. Multivariate regression showed that increasing GA, born in hospital (inborn), increasing maternal age, maternal hypertension and premature rupture of membranes were associated with higher likelihood to receive ACS.

CONCLUSIONS

The use rate of ACS remained low for infants at 24 to 31 weeks' gestation admitted to Chinese NICUs, with fewer infants receiving a complete course. The use rates varied significantly among different hospitals. Efforts are urgently needed to propose improvement measures and thus improve the usage of ACS.

Delivery of biannual ultrasound surveillance for individuals with cirrhosis and cured hepatitis C in the UK

BACKGROUND

Previous studies show the uptake of biannual ultrasound (US) surveillance in patients with cirrhosis is suboptimal. Here, our goal was to understand in broader terms how surveillance is being delivered to cirrhosis patients with cured hepatitis C in the UK.

METHODS

Hepatitis C cirrhosis patients achieving a sustained viral response (SVR) to antiviral therapies were identified from the national Hepatitis-C-Research-UK resource. Data on (i) liver/abdominal US examinations, (ii) HCC diagnoses, and (iii) HCC curative treatment were obtained through record-linkage to national health registries. The rate of US uptake was calculated by dividing the number of US episodes by follow-up time.

RESULTS

A total of 1908 cirrhosis patients from 31 liver centres were followed for 3.8 (IQR: 3.4-4.9) years. Overall, 10 396 liver/abdominal USs were identified. The proportion with biannual US was 19% in the first 3 years after SVR and 9% for all follow-up years. Higher uptake of biannual US was associated with attending a liver transplant centre; older age and cirrhosis decompensation. Funnel plot analysis indicated significant inter-centre variability in biannual US uptake, with 6/29 centres outside control limits. Incident HCC occurred in 133 patients, of which 49/133 (37%) were treated with curative intent. The number of US episodes in the two years prior to HCC diagnosis was significantly associated with higher odds of curative-intent treatment (aOR: 1.53; 95% CI: 1.12-2,09; p = .007).

CONCLUSIONS

This study provides novel data on the cascade of care for HCC in the UK. Our findings suggest biannual US is poorly targeted, inefficient and is not being delivered equitably to all patients.

Surgeon-level versus hospital-level quality variance in kidney cancer surgery

PURPOSE

To determine whether variance in kidney cancer surgery quality indicators (QIs) is most impacted by surgeon-level or hospital-level factors in order to inform quality improvement initiatives.

MATERIALS AND METHODS

The ICES and Veterans Affairs (VA) databases were queried for patients undergoing surgery for localized kidney cancer. Kidney cancer surgery QIs were defined within each cohort. Quality of care was benchmarked at a surgeon- vs. hospital-level to identify statistical outliers, using available clinicopathological data to adjust for differences in case-mix. Variance between surgeons and hospitals was calculated for each QI using a random-effects model.

RESULTS

The QI with the greatest amount of variance explained by hospital and surgeon-level factors was proportion of cases performed with minimally invasive surgery (MIS). The majority of this variance was due to surgeon-level factors for both the VA and ICES cohorts. The proportion of cases performed using an MIS approach was also the QI with the greatest number of outlier hospitals and surgeons compared to the average performance. The proportion of partial nephrectomies performed for patients at risk of chronic kidney disease was the QI with the greatest amount of variance due to hospital-level factors for the ICES cohort.

CONCLUSIONS

The proportion of localized kidney cancer cases performed using an MIS approach is the QI requiring the greatest attention. Quality improvement initiatives should focus on surgeon-level factors to increase the number of MIS cases being performed for patients with localized renal masses.

[Three-year revision rate of certified centres for joint replacement according to EndoCert : Risk-adjusted analysis of outcome quality and comparison with other quality assurance systems]

BACKGROUND

The EndoCert initiative does not yet allow a long-term assessment of outcome quality. The assessment cannot be achieved without cooperation with the German arthroplasty registry (EPRD) and other quality assurance infrastructure, such as the quality assurance system of the nationwide healthcare insurance data for inpatient hospital treatment (QSR) by the German local healthcare fund (AOK). Therefore, the quality of care of all certified centres for joint replacement (EPZ) after primary hip and knee arthroplasty was to be examined for the first time. These data were subsequently compared to the data of the EPRD.

MATERIALS AND METHODS

In EPZ that provided care to at least one AOK-insured patient in 2016, the risk-adjusted 3‑year revision rate and the SMR-value (standardised mortality or morbidity ratio), which is the quotient of the observed and expected revision rate, were analysed as markers for the quality of care. Annual hospital volume, type of centre and audit results were examined as possible influencing factors.

RESULTS

In the group comparison, significant differences (p = 0.042) for the SMR value of the 3‑year revision rate were demonstrated for hip arthroplasty with regard to the EPZ type. The annual number of primary hip arthroplasties, however, did not influence the 3‑year revision rate. For knee arthroplasties, no effect of the defined categories on the 3‑year revision rate and its SMR value was observed. The comparison of our 3‑year revision rates with those of the EPRD showed similar results for the hip but indicated significant differences for the knee.

CONCLUSION

We did not observe a correlation between quality of care and annual hospital volume in certified EPZ. However, different quality assurance procedures can lead to different results with respect to the outcome quality. Therefore, a considerably improved interaction of the German quality systems must be achieved. Participation in the EPRD is not sufficient for this. Rather, a complete report of all arthroplasties must be required, at least with the achievement of a minimum reporting rate per participating hospital. Uniform inclusion and exclusion criteria should be defined.

Is there variation between hospitals within each region in postoperative mortality for lung cancer surgery in France? A nationwide study from 2013 to 2020

Introduction

The practice of thoracic surgery for lung cancer is subject to authorization in France. We evaluated the performance of hospitals using 30-day post-operative mortality as a quality indicator, estimating its distribution within each region and measuring its variability between regions.

Material and methods

All data for patients who underwent pulmonary resection for lung cancer in France (2013–2020) were collected from the national hospital administrative database. Thirty-day mortality was defined as any patient who died in hospital (including transferred patients) within the first 30 days after the operation and those who died later during the initial hospitalization. The Standardized Mortality ratio (SMR) was the smoothed, adjusted, hospital-specific mortality rate divided by the expected mortality. To describe the variation in hospital mortality between hospitals in each region, we used different commonly used indicators of variation such as coefficients of variation (CV), interquartile interval or range (IQR), extreme ratio, and systematic component of variance (SCV).

Results

In 2013–2020, 87,232 patients underwent lung resection for cancer in France. The number of deaths was 2,537, a rate of 2.91%. The median SMR of 199 hospitals was 0.99 with an IQR of 0.86 to 1.18 and a CV of 0.25. Among the regions that had the most hospitals performing lung resections for cancer, the extreme ratio was &gt;2, which means that the maximum value is twice as high as the minimum value. The SCV between hospitals was &gt;10 for two of these regions, which is considered indicative of very high variation. For the other regions (with few hospitals performing lung resections for cancer), the variation between hospitals was lower. Globally, the variability between regions concerning the SMR was moderate, 6% of the variance was due to differences across regions. On the contrary, the hospital volume was significantly related to the SMR (p = 0.003) with a negative linear trend, whatever the region.

Conclusion

This work shows significant differences in the practices of the various hospitals within regions. However, overall, the variability in the 30-day mortality rate between regions was moderate. Our findings raises questions regarding the regionalization of major surgical procedures in France.

Test-specific funnel plots for healthcare provider profiling leveraging individual- and summary-level information

In addition to applications in meta-analysis, funnel plots have emerged as an effective graphical tool for visualizing the detection of health care providers with unusual performance. Although there already exist a variety of approaches to producing funnel plots in the literature of provider profiling, limited attention has been paid to elucidating the critical relationship between funnel plots and hypothesis testing. Within the framework of generalized linear models, here we establish methodological guidelines for creating funnel plots specific to the statistical tests of interest. Moreover, we show that the test-specific funnel plots can be created merely leveraging summary statistics instead of individual-level information. This appealing feature inhibits the leak of protected health information and reduces the cost of inter-institutional data transmission. Two data examples, one for surgical patients from Michigan hospitals and the other for Medicare-certified dialysis facilities, demonstrate the applicability to different types of providers and outcomes with either individual- or summary-level information.

Practice variation in opioid prescribing for non-cancer pain in Dutch primary care: A retrospective database study

BACKGROUND

Prescription opioid use has increased steadily in many Western countries over the past two decades, most notably in the US, Canada, and most European countries, including the Netherlands. Especially the increasing use of prescription opioids for chronic non-cancer pain has raised concerns. Most opioids in the Netherlands are prescribed in general practices. However, little is known about variation in opioid prescribing between general practices. To better understand this, we investigated practice variation in opioid prescribing for non-cancer pain between Dutch general practices.

METHODS

Data from 2017-2019 of approximately 10% of all Dutch general practices was used. Each year included approximately 1000000 patients distributed over approximately 380 practices. The primary outcome was the proportion of patients with chronic (>90 days) high-dose (≥90 oral morphine equivalents) opioid prescriptions. The secondary outcome was the proportion of patients with chronic (<90 oral morphine equivalents) opioid prescriptions. Practice variation was expressed as the ratio of the 95th/5th percentiles and the ratio of mean top 10/bottom 10. Funnel plots were used to identify outliers. Potential factors associated with unwarranted variation were investigated by comparing outliers on practice size, patient neighbourhood socioeconomic status, and urbanicity.

RESULTS

Results were similar across all years. The magnitude of variation for chronic high-dose opioid prescriptions in 2019 was 7.51-fold (95%/5% ratio), and 15.1-fold (top 10/bottom 10 ratio). The percentage of outliers in the funnel plots varied between 13.8% and 21.7%. Practices with high chronic high-dose opioid prescription proportions were larger, and had more patients from lower income and densely populated areas.

CONCLUSIONS

There might be unwarranted practice variation in chronic high-dose opioid prescriptions in primary care, pointing at possible inappropriate use of opioids. This appears to be related to socioeconomic status, urbanicity, and practice size. Further investigation of the factors driving practice variation can provide target points for quality improvement and reduce inappropriate care and unwarranted variation.

ERCP for bile duct stones across a national service, demonstrating a high requirement for repeat procedures

Background and study aims  Bile duct stones (BDS) represent approximately 50 % of the requirement for endoscopic retrograde cholangiopancreatography (ERCP) within most services. Significant variation in outcome rates for BDS clearance at ERCP has been reported, and endoscopy societies have set standards for expected clearance rates. The aim of this study was to analyze procedure outcomes across a national service. Patients and methods  Using verified hospital episode statistics (HES) data for the National Health Service (NHS) in England, we analyzed all patients having first ERCPs for BDS from 2015 to 2017, and followed these patients for at least 2 years. Results  In total 37,468 patients underwent a first ERCP for BDS, with 69.8 % undergoing only one procedure. This figure of less than 70 % of BDS cleared at first ERCP is below the Key Performance Indicators as set by the British Society of Gastroenterology (> 75 %) and the European Society of Gastrointestinal Endoscopy (> 90 %). Of 55,556 ERCPs done for BDS, 52.9 % were repeat procedures, with 11,322 patients needing multiple procedures. For hospitals performing significant numbers of ERCPs (more than 600 for BDS during the study period) patients undergoing repeat ERCPs for BDS ranged from 9 % to 50 %. Conclusions  In this nationwide study, the performance at clearing BDS at first ERCP was suboptimal, with high numbers of repeat procedures required. This may have a negative impact on both patient outcomes and experience, and increase pressure on endoscopy services. Apparent variation of outcome between acute hospital care providers requires further analysis.

Funnel plots a graphical instrument for the evaluation of population performance and quality of trauma care: a blueprint of implementation

BACKGROUND

Using patient outcomes to monitor medical centre performance has become an essential part of modern health care. However, classic league tables generally inflict stigmatization on centres rated as "poor performers", which has a negative effect on public trust and professional morale. In the present study, we aim to illustrate that funnel plots, including trends over time, can be used as a method to control the quality of data and to monitor and assure the quality of trauma care. Moreover, we aimed to present a set of regulations on how to interpret and act on underperformance or overperformance trends presented in funnel plots.

METHODS

A retrospective observational cohort study was performed using the Dutch National Trauma Registry (DNTR). Two separate datasets were created to assess the effects of healthy and multiple imputations to cope with missing values. Funnel plots displaying the performance of all trauma-receiving hospitals in 2020 were generated, and in-hospital mortality was used as the main indicator of centre performance. Indirect standardization was used to correct for differences in the types of cases. Comet plots were generated displaying the performance trends of two level-I trauma centres since 2017 and 2018.

RESULTS

Funnel plots based on data using healthy imputation for missing values can highlight centres lacking good data quality. A comet plot illustrates the performance trend over multiple years, which is more indicative of a centre's performance compared to a single measurement. Trends analysis offers the opportunity to closely monitor an individual centres' performance and direct evaluation of initiated improvement strategies.

CONCLUSION

This study describes the use of funnel and comet plots as a method to monitor and assure high-quality data and to evaluate trauma centre performance over multiple years. Moreover, this is the first study to provide a regulatory blueprint on how to interpret and act on the under- or overperformance of trauma centres. Further evaluations are needed to assess its functionality.

LEVEL OF EVIDENCE

Retrospective study, level III.

Recalibration of prediction model was needed for monitoring health care quality in subgroups: a retrospective cohort study

OBJECTIVES

To evaluate the predictive ability of a mortality prediction model in subgroups of intensive care unit (ICU) patients and test the validity for monitoring the outcome.

STUDY DESIGN AND SETTING

A Japanese ICU database was used for the analyses. Adults admitted to an ICU between April 1, 2019, and March 31, 2020, were included. Nine clinically relevant subgroups were selected, and we evaluated the discrimination and calibration of the Japan Risk of Death model, a recalibrated Acute Physiology and Chronic Health Evaluation III-j model. Funnel plots and exponentially weighted moving average (EWMA) charts were used to check its validity for monitoring in-hospital mortality. If the predictive performance was poor, the model was recalibrated and model performance was reassessed.

RESULTS

The study population comprised 14,513 patients across nine subgroups. The in-hospital mortality rate ranged from 11.3% to 30.9%. The calibration was poor in most subgroups, and the funnel plots and EWMA charts frequently revealed "out-of-control" signals crossing the control limit of three standard deviations (SDs). The calibration improved after recalibration, and the number of "out-of-control" signals decreased.

CONCLUSION

When monitoring the quality of care among subgroups of patients, testing the predictive ability and recalibration of the risk model are needed.

Early detection of variants of concern via funnel plots of regional reproduction numbers

Early detection of the emergence of a new variant of concern (VoC) is essential to develop strategies that contain epidemic outbreaks. For example, knowing in which region a VoC starts spreading enables prompt actions to circumscribe the geographical area where the new variant can spread, by containing it locally. This paper presents 'funnel plots' as a statistical process control method that, unlike tools whose purpose is to identify rises of the reproduction number ([Formula: see text]), detects when a regional [Formula: see text] departs from the national average and thus represents an anomaly. The name of the method refers to the funnel-like shape of the scatter plot that the data take on. Control limits with prescribed false alarm rate are derived from the observation that regional [Formula: see text]'s are normally distributed with variance inversely proportional to the number of infectious cases. The method is validated on public COVID-19 data demonstrating its efficacy in the early detection of SARS-CoV-2 variants in India, South Africa, England, and Italy, as well as of a malfunctioning episode of the diagnostic infrastructure in England, during which the Immensa lab in Wolverhampton gave 43,000 incorrect negative tests relative to South West and West Midlands territories.

Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data

BACKGROUND

Institutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance.

METHODS

We provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the 'Normal-Poisson' random effects model and the 'Logistic' random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI).

RESULTS

In general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well.

CONCLUSION

If the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.

Variation in hospital morbidities in an Australian neonatal intensive care unit network

OBJECTIVE

There is an expectation among the public and within the profession that the performance and outcome of neonatal intensive care units (NICUs) should be comparable between centres with a similar setting. This study aims to benchmark and audit performance variation in a regional Australian network of eight NICUs.

DESIGN

Cohort study using prospectively collected data.

SETTING

All eight perinatal centres in New South Wales and the Australian Capital Territory, Australia.

PATIENTS

All live-born infants born between 23⁺⁰ and 31⁺⁶ weeks gestation admitted to one of the tertiary perinatal centres from 2007 to 2020 (n=12 608).

MAIN OUTCOME MEASURES

Early and late confirmed sepsis, intraventricular haemorrhage, medically and surgically treated patent ductus arteriosus, chronic lung disease (CLD), postnatal steroid for CLD, necrotising enterocolitis, retinopathy of prematurity (ROP), surgery for ROP, hospital mortality and home oxygen.

RESULTS

NICUs showed variations in maternal and neonatal characteristics and resources. The unadjusted funnel plots for neonatal outcomes showed apparent variation with multiple centres outside the 99.8% control limits of the network values. The hierarchical model-based risk-adjustment accounting for differences in patient characteristics showed that discharged home with oxygen is the only outcome above the 99.8% control limits.

CONCLUSIONS

Hierarchical model-based risk-adjusted estimates of morbidity rates plotted on funnel plots provide a robust and straightforward visual graphical tool for presenting variations in outcome performance to detect aberrations in healthcare delivery and guide timely intervention. We propose using hierarchical model-based risk adjustment and funnel plots in real or near real-time to detect aberrations and start timely intervention.

Measuring variation in the quality of systemic anti-cancer therapy delivery across hospitals: A national population-based evaluation

AIM

To date, there has been little systematic assessment of the quality of care associated with systemic anti-cancer therapy (SACT) delivery across national healthcare systems. We evaluated hospital-level toxicity rates during SACT treatment as a means of identifying variation in care quality.

METHODS

All colorectal cancer (CRC) patients receiving SACT within 106 English National Health Service (NHS) hospitals between 2016 and 2019 were included. Severe acute toxicity rates were derived from hospital administrative data using a validated coding framework. Variation in hospital-level toxicity rates was assessed separately in the adjuvant and metastatic settings. Toxicity rates were adjusted for age, sex, comorbidity, performance status, tumour site, and TNM staging.

RESULTS

Eight thousand one hundred and seventy three patients received SACT in the adjuvant setting, and 7,683 patients in the metastatic setting. Adjusted severe acute toxicity rates varied between hospitals from 11% to 49% for the adjuvant cohort, and from 25% to 67% for the metastatic cohort. Compared to the national mean toxicity rate in the adjuvant cohort, six hospitals were more than two standard deviations (2SD) above, and four hospitals were more than 2SD below. In the metastatic cohort, six hospitals were more than 2SD above, and seven hospitals were more than 2SD below the national mean toxicity rate. Overall, 12 hospitals (12%) had toxicity rates more than 2SD above the national mean, and 11 (10%) had rates more than 2SD below.

CONCLUSION

There is substantial variation in hospital-level severe acute toxicity rates in both the adjuvant and metastatic settings, despite risk-adjustment. Ongoing reporting of this performance indicator can be used to focus further investigation of toxicity rates and stimulate quality improvement initiatives to improve care.

Technique failure in peritoneal dialysis: Modifiable causes and patient-specific risk factors

BACKGROUND

Technique survival is a core outcome for peritoneal dialysis (PD), according to Standardized Outcomes in Nephrology-Peritoneal Dialysis. This study aimed to identify modifiable causes and risk factors of technique failure in a large Dutch cohort using standardised definitions.

METHODS

Patients who participated in the retrospective Dutch nOcturnal and hoME dialysis Study To Improve Clinical Outcomes cohort study and started PD between 2012 and 2016 were included and followed until 1 January 2017. The primary outcome was technique failure, defined as transfer to in-centre haemodialysis for ≥ 30 days or death. Death-censored technique failure was analysed as secondary outcome. Cox regression models and competing risk models were used to assess the association between potential risk factors and technique failure.

RESULTS

A total of 695 patients were included, of whom 318 experienced technique failure during follow-up. Technique failure rate in the first year was 29%, while the death-censored technique failure rate was 23%. Infections were the most common modifiable cause for technique failure, accounting for 20% of all causes during the entire follow-up. Leakage and catheter problems were important causes within the first 6 months of PD treatment (both accounting for 15%). APD use was associated with a lower risk of technique failure (hazard ratio 0.66, 95% confidence interval 0.53-0.83).

CONCLUSION

Infections, leakage and catheter problems were important modifiable causes for technique failure. As the first-year death-censored technique failure rate remains high, future studies should focus on infection prevention and catheter access to improve technique survival.

Evaluation of the ICD-10 system in coding revascularisation procedures in patients with peripheral arterial disease in England: A retrospective cohort study using national administrative and clinical databases

BACKGROUND

Many studies evaluating care in hospitals in England use the Hospital Episode Statistics (HES) administrative database. The aim of this study was to explore whether the International Classification of Diseases 10th Revision (ICD-10) system used by HES supported the evaluation of care received by patients with peripheral arterial disease (PAD) who had revascularisation.

METHODS

This retrospective cohort study used records on patients who had revascularisation for PAD between 1st January 2017 and 31st December 2019 in England, collected prospectively in the National Vascular Registry (NVR) and linked to HES. Patients were excluded if their NVR record did not have a match in HES, due to lack of consent or different admission and procedure dates. Agreement between different presentations of PAD recorded in the NVR and the ICD-10 diagnostic codes recorded in HES was evaluated using the unweighted Kappa statistic and sensitivity and specificity. Agreement between the NVR and HES was also assessed for gender, age, comorbidities, mode of admission, and procedure type and side.

FINDINGS

In total, 20,603 patients who had 24,621 admissions were included in the study. Agreement between NVR and HES on patient gender (Kappa = 0.98), age (Kappa = 0.98), mode of admission (Kappa = 0.80), and procedure type and side (Kappa = 0.92 and 0.87, respectively) was excellent. When all diagnostic fields in HES were explored, substantial agreement was observed for chronic ischaemia with tissue loss (Kappa = 0.63), but it was lower for chronic ischaemia without tissue loss (Kappa = 0.32) and acute limb ischaemia (Kappa = 0.15). Agreement on comorbidities was mixed; excellent for diabetes (Kappa = 0.82), moderate for chronic lung disease (Kappa = 0.56), chronic kidney disease (Kappa = 0.56), and ischaemic heart disease (Kappa = 0.45) and fair for chronic heart failure (Kappa = 0.35).

INTERPRETATION

The diagnostic ICD-10 codes currently used in HES cannot accurately differentiate between stages of PAD. Therefore, studies using HES to examine patterns of care and outcomes for patients with PAD are likely to suffer from misclassification bias. Adopting an extended ICD-10 system or the ICD-11 version released to the World Health Organisation member states in 2022, may overcome this problem.

FUNDING

Healthcare Quality Improvement Partnership (HQIP).

Hospital Variation in Feeding Jejunostomy Policy for Minimally Invasive Esophagectomy: A Nationwide Cohort Study

The purpose of this study was to investigate hospital variation in the placement, surgical techniques, and safety of feeding jejunostomies (FJ) during minimally invasive esophagectomy (MIE) in the Netherlands. This nationwide cohort study analyzed patients registered in the Dutch Upper Gastrointestinal Cancer Audit (DUCA) that underwent MIE for cancer. Hospital variation in FJ placement rates were investigated using case-mix corrected funnel plots. Short-term outcomes were compared between patients with and without FJ using multilevel multivariable logistic regression analysis. The incidence of FJ-related complications was described and compared between hospitals performing routine and non-routine placement (≥90%−<90% of patients). Between 2018−2020, an FJ was placed in 1481/1811 (81.8%) patients. Rates ranged from 11−100% among hospitals. More patients were discharged within 10 days (median hospital stay) without FJ compared to patients with FJ (64.5% vs. 50.4%; OR: 0.62, 95% CI: 0.42−0.90). FJ-related complications occurred in 45 (3%) patients, of whom 23 (1.6%) experienced severe complications (≥Clavien−Dindo IIIa). The FJ-related complication rate was 13.7% in hospitals not routinely placing FJs vs. 1.7% in hospitals performing routine FJ placement (p < 0.001). Significant hospital variation in the use of FJs after MIE exists in the Netherlands. No effect of FJs on complications was observed. FJs can be placed safely, with lower FJ-related complication rates, in centers performing routine placement.

Mortality as an indicator of quality of neurosurgical care in England: a retrospective cohort study

OBJECTIVES

Postoperative mortality is a widely used quality indicator, but it may be unreliable when procedure numbers and/or mortality rates are low, due to insufficient statistical power. The objective was to investigate the statistical validity of postoperative 30-day mortality as a quality metric for neurosurgical practice across healthcare providers.

DESIGN

Retrospective cohort study.

SETTING

Hospital Episode Statistics data from all neurosurgical units in England.

PARTICIPANTS

Patients who underwent neurosurgical procedures between April 2013 and March 2018. Procedures were grouped using the National Neurosurgical Audit Programme classification.

OUTCOMES MEASURED

National 30-day postoperative mortality rates were calculated for elective and non-elective neurosurgical procedural groups. The study estimated the proportion of neurosurgeons and NHS trusts in England that performed sufficient procedures in 3-year and 5-year periods to detect unusual performance (defined as double the national rate of mortality). The actual difference in mortality rates that could be reliably detected based on procedure volumes of neurosurgeons and units over a 5-year period was modelled.

RESULTS

The 30-day mortality rates for all elective and non-elective procedures were 0.4% and 6.1%, respectively. Only one neurosurgeon in England achieved the minimum sample size (n=2402) of elective cases in 5 years needed to detect if their mortality rate was double the national average. All neurosurgical units achieved the minimum sample sizes for both elective (n=2402) and non-elective (n=149) procedures. In several neurosurgical subspecialties, approximately 80% of units (or more) achieved the minimum sample sizes needed to detect if their mortality rate was double the national rate, including elective neuro-oncology (baseline mortality rate=2.3%), non-elective neuro-oncology (rate=5.7%), neurovascular (rate=6.7%) and trauma (rate=11%).

CONCLUSION

Postoperative mortality lacks statistical power as a measure of individual neurosurgeon performance. Neurosurgical units in England performed sufficient procedure numbers overall and in several subspecialty areas to support the use of mortality as a quality indicator.

Analysis of hospital readmissions with competing risks

The 30-day hospital readmission rate has been used in provider profiling for evaluating inter-provider care coordination, medical cost effectiveness, and patient quality of life. Current profiling analyzes use logistic regression to model 30-day readmission as a binary outcome, but one disadvantage of this approach is that this outcome is strongly affected by competing risks (e.g., death). Thus, one, perhaps unintended, consequence is that if two facilities have the same rates of readmission, the one with the higher rate of competing risks will have the lower 30-day readmission rate. We propose a discrete time competing risk model wherein the cause-specific readmission hazard is used to assess provider-level effects. This approach takes account of the timing of events and focuses on the readmission rates which are of primary interest. The quality measure, then is a standardized readmission ratio, akin to a standardized mortality ratio. This measure is not systematically affected by the rate of competing risks. To facilitate the estimation and inference of a large number of provider effects, we develop an efficient Blockwise Inversion Newton algorithm, and a stabilized robust score test that overcomes the conservative nature of the classical robust score test. An application to dialysis patients demonstrates improved profiling, model fitting, and outlier detection over existing methods.

Development of binational radiation therapy quality indicator reports for prostate cancer treatment using registry data

INTRODUCTION

Quality indicators (QIs) are metrics which seek to allow comparison of clinicians' and institutes' practice to best evidence-based practice. The Australia and New Zealand Prostate Cancer Outcomes Registry (PCOR-ANZ) is a bi-national clinical quality registry with coverage estimated to be over 60% of the men newly diagnosed with prostate cancer. We outline the production and ambition of institute-level QI reports to benchmark performance for radiation therapy in the treatment of prostate cancer.

METHODS

An expert clinician panel was assembled to create a list of candidate QIs based on a comprehensive literature review, and on modified Delphi-method and expert-consensus voting. A separate implementation group-including, clinicians, epidemiologists, data managers and data scientists-employed an evidence- and consensus- based approach to generate an effective QI report designed for automated production and regular distribution to participating institutes. Feedback from the recipient clinicians was sought to enable refinement of these reports.

RESULTS

Seven QIs, including three related to post-treatment symptoms, were deemed feasible to analyse with the currently available data. Utilising an existing report template employed for benchmarking of surgical indicators, a novel radiation therapy report was generated using registry data in a secure analytical environment. The first, beta version of these reports have been produced and confidentially distributed. It is planned to automatically generate these reports biannually and iteratively refine them based on the clinician input.

CONCLUSION

QI reports for the treatment of prostate cancer by radiation oncologists have been produced using data from Australia and New Zealand patients. These are being disseminated to institutes on a six-monthly basis allowing comparisons to de-identified peers. The reports aim to facilitate improving patient outcomes, deepen engagement with the radiation oncology community and increase the breadth of PCOR-ANZ coverage. Additional QIs will be included in future iterations of these reports as data matures.

Surgeon volume and patient-reported urinary incontinence after radical prostatectomy. Population-based register study in Sweden

OBJECTIVE

To investigate the association between surgeon volume and urinary incontinence after radical prostatectomy.

METHODS

A total of 8326 men in The National Prostate Cancer Register of Sweden (NPCR) underwent robot-assisted radical prostatectomy (RARP) between 2017 and 2019 of whom 56% (4668/8 326) had responded to a questionnaire one year after RARP. The questionnaire included the question: 'How much urine leakage do you experience?' with the response alternatives 'Not at all', 'A little', defined as continence and 'Moderately', 'Much/Very much' as incontinence. Association between incontinence and mean number of RARPs/year/surgeon was analysed with multivariable logistic regression including age, Charlson Comorbidity Index (CCI), PSA, prostate volume, number of biopsy cores with cancer, cT stage, Gleason score, lymph node dissection, nerve sparing intent and response rate to the questionnaire.

RESULTS

14% (659/4 668) of the men were incontinent one year after RARP. There was no statistically significant association between surgeon volume and incontinence. Older age (>75 years vs. < 65 years, OR 2.29 [95% CI 1.48-3.53]), higher CCI (CCI 2+ vs. CCI 0, OR 1.37 [95% CI 1.04-1.80]) and no nerve sparing intent (no vs. yes OR 1.53 [95% CI 1.26-1.85]) increased risk of incontinence. There were large differences in the proportion of incontinent men between surgeons with similar annual volumes, which remained after adjustment.

CONCLUSIONS

The lack of association between surgeon volume and incontinence and the wide range in outcome between surgeons with similar volumes underline the importance of individual feedback to surgeons on functional results.

Epidemiology of post-hemorrhagic ventricular dilatation in very preterm infants

OBJECTIVE

To describe the incidence, trends, management's variability and short-term outcomes of preterm infants with severe post-hemorrhagic ventricular dilatation (sPHVD).

METHODS

We reviewed infants &lt;33 weeks' gestation who had PHVD and were admitted to the Canadian Neonatal Network between 2010 and 2018. We compared perinatal characteristics and short-term outcomes between those with sPHVD and those with mild/moderate PHVD and those with and without ventriculo-peritoneal (VP) shunt.

RESULTS

Of 29,417 infants, 2439 (8%) had PHVD; rate increased from 7.3% in 2010 to 9.6% in 2018 (P = 0.005). Among infants with PHVD, sPHVD (19%) and VP shunt (29%) rates varied significantly across Canadian centers and between geographic regions (P &lt; 0.01 and P = 0.0002). On multivariable analysis, sPHVD was associated with greater mortality, seizures and meningitis compared to mild/moderate PHVD.

CONCLUSIONS

Significant variability in sPHVD and VP shunt rates exists between centers and regions in Canada. sPHVD was associated with increased mortality and morbidities.

Textbook oncologic outcomes and regionalization among patients undergoing hepatic resection for intrahepatic cholangiocarcinoma

BACKGROUND AND OBJECTIVES

Textbook oncologic outcome (TOO) and its association with regionalization of care for intrahepatic cholangiocarcinoma (ICC) have not been evaluated.

METHODS

We identified patients who underwent hepatic resection for ICC between 2004 and 2018 from the National Cancer Database. Facilities were categorized by annual hepatectomy volume for ICC. TOO was defined as no 90-day mortality, margin-negative resection, no prolonged hospitalization, no 30-day readmission, receipt of appropriate adjuvant therapy, and adequate lymphadenectomy. Multivariable regression was used to evaluate the association between annual hepatectomy volume and TOO.

RESULTS

A total of 5359 patients underwent liver resection for ICC. TOO was achieved in 11.2% (n = 599) of patients. Inadequate lymphadenectomy was the largest impediment to achieving TOO. After adjusting for patient, pathologic, and facility characteristics, high volume facilities had 67% increased odds of achieving TOO (Ref.: low volume; high volume: odds ratio 1.67, 95% confidence interval: 1.24-2.25; p < 0.001). Patients treated at high-volume centers who achieved a TOO had better overall survival (OS) versus patients treated at low-volume facilities (low volume vs. high volume; median OS, 47.3 vs. 71.1 months, p < 0.05).

CONCLUSIONS

A composite oncologic measure, TOO, provides a comprehensive insight into the performance of liver resection and regionalization of surgical care for ICC.

MEASURING PERFORMANCE FOR END-OF-LIFE CARE

Although not without controversy, readmission is entrenched as a hospital quality metric with statistical analyses generally based on fitting a logistic-Normal generalized linear mixed model. Such analyses, however, ignore death as a competing risk, although doing so for clinical conditions with high mortality can have profound effects; a hospital's seemingly good performance for readmission may be an artifact of it having poor performance for mortality. in this paper we propose novel multivariate hospital-level performance measures for readmission and mortality that derive from framing the analysis as one of cluster-correlated semi-competing risks data. We also consider a number of profiling-related goals, including the identification of extreme performers and a bivariate classification of whether the hospital has higher-/lower-than-expected readmission and mortality rates via a Bayesian decision-theoretic approach that characterizes hospitals on the basis of minimizing the posterior expected loss for an appropriate loss function. in some settings, particularly if the number of hospitals is large, the computational burden may be prohibitive. To resolve this, we propose a series of analysis strategies that will be useful in practice. Throughout, the methods are illustrated with data from CMS on N = 17,685 patients diagnosed with pancreatic cancer between 2000-2012 at one of J = 264 hospitals in California.

End-of-Life Use of Systemic Therapy in Patients With Advanced Melanoma: A Nationwide Cohort Study

PURPOSE:

The introduction of immune checkpoint inhibitors and targeted therapies improved the overall survival of patients with advanced melanoma. It is not known how often these costly treatments with potential serious side effects are ineffectively applied in the last phase of life. This study aimed to investigate the start of a new systemic therapy within 45 and 90 days of death in Dutch patients with advanced melanoma.

METHODS:

We selected patients who were diagnosed with unresectable IIIC or stage IV melanoma, registered in the Dutch Melanoma Treatment Registry, and died between 2013 and 2019. Primary outcome was the probability of starting a new systemic therapy 45 and 90 days before death. Secondary outcomes were type of systemic therapy started, grade 3/4 adverse events (AEs), and the total costs of systemic therapies.

RESULTS:

Between 2013 and 2019, 3,797 patients with unresectable IIIC or stage IV melanoma were entered in the registry and died. The percentage of patients receiving a new systemic therapy within 45 and 90 days before death was significantly different between Dutch melanoma centers (varying from 6% to 23% and 20% to 46%, respectively). Thirteen percent of patients (n = 146) developed grade 3/4 AEs in the last period before death. The majority of patients with an AE required hospital admission (n = 102, 69.6%). Mean total costs of systemic therapy per cohort year of the patients who received a new systemic therapy within 90 days before death were 2.3%-2.8% of the total costs spent on melanoma therapies.

CONCLUSION:

The minority of Dutch patients with metastatic melanoma started a new systemic therapy in the last phase of life. However, the percentages varied between Dutch melanoma centers. Financial impact of these therapies in the last phase of life is relatively small.