Overweight-years and cancer risk: A prospective study of the association and comparison of predictive performance with body mass index (Atherosclerosis Risk in Communities Study)

Excess body mass index (BMI) is associated with a higher risk of at least 13 cancers, but it is usually measured at a single time point. We tested whether the overweight-years metric, which incorporates exposure time to BMI ≥25 kg/m2 , is associated with cancer risk and compared this with a single BMI measure. We used adulthood BMI readings in the Atherosclerosis Risk in Communities (ARIC) study to derive the overweight-years metric. We calculated associations between the metric and BMI and the risk of cancers using Cox proportional hazards models. Models that either included the metric or BMI were compared using Harrell's C-statistic. We included 13,463 participants, with 3,876 first primary cancers over a mean of 19 years (SD 7) of cancer follow-up. Hazard ratios for obesity-related cancers per standard deviation overweight-years were 1.15 (95% CI: 1.05-1.25) in men and 1.14 (95% CI: 1.08-1.20) in women. The difference in the C-statistic between models that incorporated BMI, or the overweight-years metric was non-significant in men and women. Overweight-years was associated with the risk of obesity-related cancers but did not outperform a single BMI measure in association performance characteristics.

Indirect impact of the war in Ukraine on primary percutaneous coronary interventions for ST-elevation myocardial infarction in Poland

INTRODUCTION

The Russian invasion of Ukraine in February 2022 resulted in the displacement of approximately 12.5 million refugees to adjacent countries including Poland, that may have strained healthcare service delivery.

OBJECTIVES

Using the ST-elevation myocardial infarction (STEMI) data, we aimed to evaluate whether the Russian invasion of Ukraine has indirectly impacted the delivery of acute cardiovascular care in Poland.

PATIENTS AND METHODS

We analyzed all adult patients undergoing percutaneous coronary interventions (PCI) for STEMI across Poland between 25th February 2017 to 24th May 2022. Centers were allocated to regions of <100km and >100km of the Polish-Ukraine border. Mixed effect generalized linear regression models with random effects per hospital were used to explore the associations between the war in Ukraine starting with several outcomes of interest, and whether these associations differed across regions of 100km from the Polish-Ukraine border.

RESULTS

A total of 90,115 procedures were included in the analysis. The average number of procedures per-month was similar to predicted volume for centers in the >100km region, while the average number of PCI was higher than expected (by an estimated 15 (11-19)) for the <100km region. There was no difference in adjusted fatality rate or quality of care outcomes pre- vs. during-war in both <100 and >100 km regions, with no evidence of a difference-in-difference across regions.

CONCLUSIONS

Following the Russian invasion of Ukraine, there was only a modest and temporary increase in primary PCI predominantly in centers situated within 100km of the border, although no significant impact on in-hospital fatality rate.

Making predictions under interventions: a case study from the PREDICT-CVD cohort in New Zealand primary care

Background

Most existing clinical prediction models do not allow predictions under interventions. Such predictions allow predicted risk under different proposed strategies to be compared and are therefore useful to support clinical decision making. We aimed to compare methodological approaches for predicting individual level cardiovascular risk under three interventions: smoking cessation, reducing blood pressure, and reducing cholesterol.

Methods

We used data from the PREDICT prospective cohort study in New Zealand to calculate cardiovascular risk in a primary care setting. We compared three strategies to estimate absolute risk under intervention: (a) conditioning on hypothetical interventions in non-causal models; (b) combining existing prediction models with causal effects estimated using observational causal inference methods; and (c) combining existing prediction models with causal effects reported in published literature.

Results

The median absolute cardiovascular risk among smokers was 3.9%; our approaches predicted that smoking cessation reduced this to a median between a non-causal estimate of 2.5% and a causal estimate of 2.8%, depending on estimation methods. For reducing blood pressure, the proposed approaches estimated a reduction of absolute risk from a median of 4.9% to a median between 3.2% and 4.5% (both derived from causal estimation). Reducing cholesterol was estimated to reduce median absolute risk from 3.1% to between 2.2% (non-causal estimate) and 2.8% (causal estimate).

Conclusions

Estimated absolute risk reductions based on non-causal methods were different to those based on causal methods, and there was substantial variation in estimates within the causal methods. Researchers wishing to estimate risk under intervention should be explicit about their causal modelling assumptions and conduct sensitivity analysis by considering a range of possible approaches.

Rapid systematic review on risks and outcomes of sepsis: the influence of risk factors associated with health inequalities

BACKGROUND AND AIMS

Sepsis is a serious and life-threatening condition caused by a dysregulated immune response to an infection. Recent guidance issued in the UK gave recommendations around recognition and antibiotic treatment of sepsis, but did not consider factors relating to health inequalities. The aim of this study was to summarise the literature investigating associations between health inequalities and sepsis.

METHODS

Searches were conducted in Embase for peer-reviewed articles published since 2010 that included sepsis in combination with one of the following five areas: socioeconomic status, race/ethnicity, community factors, medical needs and pregnancy/maternity.

RESULTS

Five searches identified 1,402 studies, with 50 unique studies included in the review after screening (13 sociodemographic, 14 race/ethnicity, 3 community, 3 care/medical needs and 20 pregnancy/maternity; 3 papers examined multiple health inequalities). Most of the studies were conducted in the USA (31/50), with only four studies using UK data (all pregnancy related). Socioeconomic factors associated with increased sepsis incidence included lower socioeconomic status, unemployment and lower education level, although findings were not consistent across studies. For ethnicity, mixed results were reported. Living in a medically underserved area or being resident in a nursing home increased risk of sepsis. Mortality rates after sepsis were found to be higher in people living in rural areas or in those discharged to skilled nursing facilities while associations with ethnicity were mixed. Complications during delivery, caesarean-section delivery, increased deprivation and black and other ethnic minority race were associated with post-partum sepsis.

CONCLUSION

There are clear correlations between sepsis morbidity and mortality and the presence of factors associated with health inequalities. To inform local guidance and drive public health measures, there is a need for studies conducted across more diverse setting and countries.

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration

OBJECTIVES

In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.

STUDY DESIGN AND SETTING

A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).

RESULTS

The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.

CONCLUSION

We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

Identifying the prevalence of genital injuries amongst patients attending Saint Mary's sexual assault referral centre following an allegation of digital penetration

This study aimed to (1) add to the limited evidence base regarding genital injury associated with digital vaginal penetration and (2) identify predisposing or protective factors to the identification of a genital injury. Data collection was performed retrospectively on the paper case files of 120 female adult (>18 years) patients alleging digital vaginal penetration with no penile vaginal penetration that had an acute FME at Saint Mary's Sexual Assault Referral Centre (SARC) Manchester. Descriptive statistics were used to investigate differences in the demographics of those reporting digital penetration, with and without injuries. Overall, 18% had genital injuries noted at the time of the FME. Posterior fourchette was the most common location of genital injury and abrasion was the most common injury type. It is worth further noting that all 22 patients where an injury was noted were of white ethnicity, only 12 patients in the sample were not white so caution is needed in interpretating this finding of a non-significant difference. Future research should consider injury and ethnicity more specifically. The findings from this study add to the existing evidence base and should prove useful to expert witnesses when called upon to interpret examination findings of sexual assault complainants as they relate to an allegation of digital penetration.

Comparing Predictive Performance of Time Invariant and Time Variant Clinical Prediction Models in Cardiac Surgery

Clinical prediction models are increasingly used across healthcare to support clinical decision making. Existing methods and models are time-invariant and thus ignore the changes in populations and healthcare practice that occur over time. We aimed to compare the performance of time-invariant with time-variant models in UK National Adult Cardiac Surgery Audit data from Manchester University NHS Foundation Trust between 2009 and 2019. Data from 2009-2011 were used for initial model fitting, and data from 2012-2019 for validation and updating. We fitted four models to the data: a time-invariant logistic regression model (not updated), a logistic model which was updated every year and validated it in each subsequent year, a logistic regression model where the intercept is a function of calendar time (not updated), and a continually updating Bayesian logistic model which was updated with each new observation and continuously validated. We report predictive performance over the complete validation cohort and for each year in the validation data. Over the complete validation data, the Bayesian model had the best predictive performance.

A scoping review finds a growing trend in studies validating multimorbidity patterns and identifies five broad types of validation methods

OBJECTIVES

Multimorbidity, the presence of two or more long-term conditions, is a growing public health concern. Many studies use analytical methods to discover multimorbidity patterns from data. We aimed to review approaches used in published literature to validate these patterns.

STUDY DESIGN AND SETTING

We systematically searched PubMed and Web of Science for studies published between July 2017 and July 2023 that used analytical methods to discover multimorbidity patterns.

RESULTS

Out of 31,617 studies returned by the searches, 172 were included. Of these, 111 studies (64%) conducted validation, the number of studies with validation increased from 53.13% (17 out of 32 studies) to 71.25% (57 out of 80 studies) in 2017-2019 to 2022-2023, respectively. Five types of validation were identified: assessing the association of multimorbidity patterns with clinical outcomes (n = 79), stability across subsamples (n = 26), clinical plausibility (n = 22), stability across methods (n = 7) and exploring common determinants (n = 2). Some studies used multiple types of validation.

CONCLUSION

The number of studies conducting a validation of multimorbidity patterns is clearly increasing. The most popular validation approach is assessing the association of multimorbidity patterns with clinical outcomes. Methodological guidance on the validation of multimorbidity patterns is needed.

Development and Internal Validation of a Multivariable Prediction Model to Predict Repeat Attendances in the Pediatric Emergency Department: A Retrospective Cohort Study

OBJECTIVE

Unplanned reattendances to the pediatric emergency department (PED) occur commonly in clinical practice. Multiple factors influence the decision to return to care, and understanding risk factors may allow for better design of clinical services. We developed a clinical prediction model to predict return to the PED within 72 hours from the index visit.

METHODS

We retrospectively reviewed all attendances to the PED of Royal Manchester Children's Hospital between 2009 and 2019. Attendances were excluded if they were admitted to hospital, aged older than 16 years or died in the PED. Variables were collected from Electronic Health Records reflecting triage codes. Data were split temporally into a training (80%) set for model development and a test (20%) set for internal validation. We developed the prediction model using LASSO penalized logistic regression.

RESULTS

A total of 308,573 attendances were included in the study. There were 14,276 (4.63%) returns within 72 hours of index visit. The final model had an area under the receiver operating characteristic curve of 0.64 (95% confidence interval, 0.63-0.65) on temporal validation. The calibration of the model was good, although with some evidence of miscalibration at the high extremes of the risk distribution. After-visit diagnoses codes reflecting a nonspecific problem ("unwell child") were more common in children who went on to reattend.

CONCLUSIONS

We developed and internally validated a clinical prediction model for unplanned reattendance to the PED using routinely collected clinical data, including markers of socioeconomic deprivation. This model allows for easy identification of children at the greatest risk of return to PED.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Clinical and health inequality risk factors for non-COVID-related sepsis during the global COVID-19 pandemic: a national case-control and cohort study

Background

Sepsis, characterised by significant morbidity and mortality, is intricately linked to socioeconomic disparities and pre-admission clinical histories. This study aspires to elucidate the association between non-COVID-19 related sepsis and health inequality risk factors amidst the pandemic in England, with a secondary focus on their association with 30-day sepsis mortality.

Methods

With the approval of NHS England, we harnessed the OpenSAFELY platform to execute a cohort study and a 1:6 matched case-control study. A sepsis diagnosis was identified from the incident hospital admissions record using ICD-10 codes. This encompassed 248,767 cases with non-COVID-19 sepsis from a cohort of 22.0 million individuals spanning January 1, 2019, to June 31, 2022. Socioeconomic deprivation was gauged using the Index of Multiple Deprivation score, reflecting indicators like income, employment, and education. Hospitalisation-related sepsis diagnoses were categorised as community-acquired or hospital-acquired. Cases were matched to controls who had no recorded diagnosis of sepsis, based on age (stepwise), sex, and calendar month. The eligibility criteria for controls were established primarily on the absence of a recorded sepsis diagnosis. Associations between potential predictors and odds of developing non-COVID-19 sepsis underwent assessment through conditional logistic regression models, with multivariable regression determining odds ratios (ORs) for 30-day mortality.

Findings

The study included 224,361 (10.2%) cases with non-COVID-19 sepsis and 1,346,166 matched controls. The most socioeconomic deprived quintile was associated with higher odds of developing non-COVID-19 sepsis than the least deprived quintile (crude OR 1.80 [95% CI 1.77-1.83]). Other risk factors (after adjusting comorbidities) such as learning disability (adjusted OR 3.53 [3.35-3.73]), chronic liver disease (adjusted OR 3.08 [2.97-3.19]), chronic kidney disease (stage 4: adjusted OR 2.62 [2.55-2.70], stage 5: adjusted OR 6.23 [5.81-6.69]), cancer, neurological disease, immunosuppressive conditions were also associated with developing non-COVID-19 sepsis. The incidence rate of non-COVID-19 sepsis decreased during the COVID-19 pandemic and rebounded to pre-pandemic levels (April 2021) after national lockdowns had been lifted. The 30-day mortality risk in cases with non-COVID-19 sepsis was higher for the most deprived quintile across all periods.

Interpretation

Socioeconomic deprivation, comorbidity and learning disabilities were associated with an increased odds of developing non-COVID-19 related sepsis and 30-day mortality in England. This study highlights the need to improve the prevention of sepsis, including more precise targeting of antimicrobials to higher-risk patients.

Funding

The UK Health Security Agency, Health Data Research UK, and National Institute for Health Research.

Use of SARC services by victims of sexual violence: Auditing the ethnicity of Saint Mary's Sexual Assault Referral Centre's clients

INTRODUCTION

Saint Mary's Sexual Assault Referral Centre (SARC) in Manchester provides services to adults and children who have suffered sexual assault. The ethnic composition of those who attended the centre was audited in 2001 and 2003 to measure how well it serves different ethnic groups. This paper provides an updated audit using 2019 data.

METHODOLOGY

Census data for Greater Manchester, and data from the Crime Survey of England and Wales (CSEW) showing rates of sexual assault for different ethnic groups, were used to predict the ethnic composition of sexual assault victims in Greater Manchester. These predicted figures were then compared with the ethnic composition of Saint Mary's SARC 2019 client base to measure how well the SARC is serving different groups. This comparison was repeated using data from the 2001 SARC client base to explore change over time.

RESULTS

The analysis shows that South Asians and Chinese individuals are underrepresented in the SARC client base, and that this issue has become more pronounced over time. Every other group is overrepresented.

CONCLUSION

The underrepresentation of South Asian and Chinese clients at Saint Mary's SARC is concerning. Making the service more accessible to those from these ethnic groups should be a priority for the centre.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques

INTRODUCTION

This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis.

METHODS

We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring.

RESULTS

Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors.

DISCUSSION

We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

External validation of the Manchester Acute Coronary Syndromes ECG risk model within a pre-hospital setting

OBJECTIVES

The Manchester Acute Coronary Syndromes ECG (MACS-ECG) prediction model calculates a score based on objective ECG measurements to give the probability of a non-ST elevation myocardial infarction (NSTEMI). The model showed good performance in the emergency department (ED), but its accuracy in the pre-hospital setting is unknown. We aimed to externally validate MACS-ECG in the pre-hospital environment.

METHODS

We undertook a secondary analysis from the Pre-hospital Evaluation of Sensitive Troponin (PRESTO) study, a multi-centre prospective study to validate decision aids in the pre-hospital setting (26 February 2019 to 23 March 2020). Patients with chest pain where the treating paramedic suspected acute coronary syndrome were included. Paramedics collected demographic and historical data and interpreted ECGs contemporaneously (as 'normal' or 'abnormal'). After completing recruitment, we analysed ECGs to calculate the MACS-ECG score, using both a pre-defined threshold and a novel threshold that optimises sensitivity to differentiate AMI from non-AMI. This was compared with subjective ECG interpretation by paramedics. The diagnosis of AMI was adjudicated by two investigators based on serial troponin testing in hospital.

RESULTS

Of 691 participants, 87 had type 1 AMI and 687 had complete data for paramedic ECG interpretation. The MACS-ECG model had a C-index of 0.68 (95% CI: 0.61 to 0.75). At the pre-determined cut-off, MACS-ECG had 2.3% (95% CI: 0.3% to 8.1%) sensitivity, 99.5% (95% CI: 98.6% to 99.9%) specificity, 40.0% (95% CI: 10.2% to 79.3%) positive predictive value (PPV) and 87.6% (87.3% to 88.0%) negative predictive value (NPV). At the optimal threshold for sensitivity, MACS-ECG had 50.6% sensitivity (39.6% to 61.5%), 83.1% specificity (79.9% to 86.0%), 30.1% PPV (24.7% to 36.2%) and 92.1% NPV (90.4% to 93.5%). In comparison, paramedics had a sensitivity of 71.3% (95% CI: 60.8% to 80.5%) with 53.8% (95% CI: 53.8% to 61.8%) specificity, 19.7% (17.2% to 22.45%) PPV and 93.3% (90.8% to 95.1%) NPV.

CONCLUSION

Neither MACS-ECG nor paramedic ECG interpretation had a sufficiently high PPV or NPV to 'rule in' or 'rule out' NSTEMI alone.

Abstract 3034: Excess weight by degree and duration and cancer risk: an individual participant data (IPD) meta - analyses of over 1.4 million participants (ABACus2 consortium)

Background: Excess body fatness, approximated by body mass index (BMI), is associated with higher risk of at least 13 cancer types and is the second-largest avoidable cause of cancer in many populations. Current epidemiologic evidence linking body fatness with cancer risk is largely based on a ‘once-only’ BMI measure which may fail to capture the life-course exposure to body fatness. Here, we test whether a novel metric that combines repeated BMI measures over an individual’s adult lifetime, the overweight-years metric, improves the performance characteristics for associations with cancer compared with a ‘once-only’ BMI measure.

Methods: Within the ABACus 2 Consortium (4 US cohorts; 1 European cohort), we derived the overweight-years metric - a product of the degree of overweight (BMI minus 24.9 kg/m2) and the duration of overweight (in years). Using a random effects two-stage meta-analysis, we calculated the association between the overweight-years metric, and separately the cumulative degree and cumulative duration of overweight exposure with incident cancer by fitting multivariable-adjusted Cox proportional hazards models and comparing each metrics performance with BMI measured at a single time using Harrell’s C-statistic.

Results: Out of 1,419,850 participants in the ABACus 2 consortium, 716,909 participants were included. Per standard deviation overweight-years, the multivariable-adjusted hazard ratio for obesity-related cancers in men was 1.15 (95% CI: 1.13, 1.17, I2: 0) and for women was 1.11 (95% CI: 1.04, 1.18, I2: 0.94). For ‘once-only’ BMI, the per standard deviation multivariable-adjusted hazard ratio for obesity-related cancers in men was 1.16 (95% CI: 1.15,1.18, I2: 0) and for women was 1.13 (95% CI: 1.09,1.18, I2: 0.82). For most obesity-related cancers, both degree and duration of overweight were significantly associated with risk. The overweight-years metric had a C-statistic of 0.610 (95% CI: 0.569, 0.649) and once-only BMI had a C-statistic of 0.608 (95% CI: 0.566, 0.648) for combined obesity-related cancers in men. In

women, the C-statistic for overweight-years was 0.562 (95% CI: 0.537, 0.587) and once-only BMI had a C-statistic of 0.566 (95% CI: 0.544, 0.588) for combined obesity-related cancers. The C-statistic of the overweight-years metric and BMI measured at a single time combined was 0.609 (95% CI: 0.567, 0.650) in men and 0.573 (95% CI: 0.546, 0.600) in women for combined obesity-related cancers.

Conclusion: Overall, there were marginal differences in the predictive performance between the overweight-years metric and a ‘once-only’ baseline-BMI measure. These findings show that excess weight throughout adulthood is important in cancer development.

Funding: CRUK, NIHR, NHLBI, NCI, NPCR.

Citation Format: Nadin Hawwash, Matthew Sperrin, Glen P. Martin, Michael Cook, Charles E. Matthews, Marian L. Neuhouser, Corinne E. Joshu, Elizabeth A. Platz, Heinz Freisling, Marc Gunter, Rob Bristow, Andrew G. Renehan. Excess weight by degree and duration and cancer risk: an individual participant data (IPD) meta - analyses of over 1.4 million participants (ABACus2 consortium) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 3034.

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

AIMS

Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n ) is appropriate relative to the number of events (E k ) and the number of predictor parameters (p k ) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes.

PROPOSED CRITERIA

The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R 2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R 2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R 2 of the multinomial logistic regression.

EVALUATION OF CRITERIA

We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation.

SUMMARY

We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

Targeted validation: validating clinical prediction models in their intended population and setting

Clinical prediction models must be appropriately validated before they can be used. While validation studies are sometimes carefully designed to match an intended population/setting of the model, it is common for validation studies to take place with arbitrary datasets, chosen for convenience rather than relevance. We call estimating how well a model performs within the intended population/setting "targeted validation". Use of this term sharpens the focus on the intended use of a model, which may increase the applicability of developed models, avoid misleading conclusions, and reduce research waste. It also exposes that external validation may not be required when the intended population for the model matches the population used to develop the model; here, a robust internal validation may be sufficient, especially if the development dataset was large.

Use of a device-based remote management heart failure care pathway is associated with reduced hospitalization and improved patient outcomes: TriageHF Plus real-world evaluation

Background

Heart failure (HF) is a leading cause of hospital admission. However, prompt identification of worsening HF using implantable device data and proactive intervention may reduce hospitalizations.

The validated TriageHF algorithm in enabled ICD/CRT devices uses sensor data to risk stratify patients for HF hospitalization in the next 30 days. TriageHF Plus is a novel device-based HF care pathway (DHFP) that uses “high” risk status as the trigger for remote intervention (see Figure 1 for pathway overview). Outcomes after DHFP implementation in a clinical setting have not been examined.

Purpose

To evaluate the impact of TriageHF Plus clinical pathway on hospitalisation rates.

Methods

A prospective, multi-center evaluation comparing monthly hospitalization rates for patients enrolled in a DHFP with a concurrent standard of care (SoC) cohort and characterizing staffing resources necessary to implement the DHFP. The DHFP cohort received telephonic assessment and guideline-directed clinical care upon transition to high-risk status. Propensity scores (PS) were applied to DHFP and SoC cohorts to allow unbiased comparison. A negative binomial model was fitted to the monthly number of all-cause hospitalizations with treatment group (DHFP vs. SoC) as a covariate, using PS as weights.

Results

Between 09/11/2019 and 06/24/2021, 758 patients were included in the study (443 DHFP, 315 SoC). Proportion CRT 76%/ 89% and LVEF <50% 78%/ 66% for DHFP/ SoC, respectively.

196 high risk transmissions prompted telephone assessment, with successful contact in 182; of which, 79 (43%) identified an explanatory acute medical issue. A secondary intervention was undertaken in 44/79 (56%). High risk transmissions took on average 19 minutes per clinical assessment (initial telephone triage and 30 day follow up). The rate of hospitalizations was 58% lower in the DHFP group, compared with SoC, after PS adjustment (IRR 0.42, 95% CI: 0.23, 0.76, p=0.004), see Figure 2. Sensitivity analyses showed Covid-19 had little effect on results.

Conclusions

This is the first prospective, real-world evaluation of a device-based HF care pathway to report a reduction in hospitalizations and does so with minimal staffing time. Integrated into existing HF services, device-based remote monitoring of HF patients can improve outcomes.

Funding Acknowledgement

Type of funding sources: Private company. Main funding source(s): Medtronic

Remotely Monitored Cardiac Implantable Electronic Device Data Predict All-Cause and Cardiovascular Unplanned Hospitalization

Background Unplanned hospitalizations are common in patients with cardiovascular disease. The "Triage Heart Failure Risk Status" (Triage-HFRS) algorithm in patients with cardiac implantable electronic devices uses data from up to 9 device-derived physiological parameters to stratify patients as low/medium/high risk of 30-day heart failure (HF) hospitalization, but its use to predict all-cause hospitalization has not been explored. We examined the association between Triage-HFRS and risk of all-cause, cardiovascular, or HF hospitalization. Methods and Results A prospective observational study of 435 adults (including patients with and without HF) with a Medtronic Triage-HFRS-enabled cardiac implantable electronic device (cardiac resynchronization therapy device, implantable cardioverter-defibrillator, or pacemaker). Cox proportional hazards models explored association between Triage-HFRS and time to hospitalization; a frailty term at the patient level accounted for repeated measures. A total of 274 of 435 patients (63.0%) transmitted ≥1 high HFRS transmission before or during the study period. The remaining 161 patients never transmitted a high HFRS. A total of 153 (32.9%) patients had ≥1 unplanned hospitalization during the study period, totaling 356 nonelective hospitalizations. A high HFRS conferred a 37.3% sensitivity and an 86.2% specificity for 30-day all-cause hospitalization; and for HF hospitalizations, these numbers were 62.5% and 85.6%, respectively. Compared with a low Triage-HFRS, a high HFRS conferred a 4.2 relative risk of 30-day all-cause hospitalization (8.5% versus 2.0%), a 5.0 relative risk of 30-day cardiovascular hospitalization (3.6% versus 0.7%), and a 7.7 relative risk of 30-day HF hospitalization (2.0% versus 0.3%). Conclusions In patients with cardiac implantable electronic devices, remotely monitored Triage-HFRS data discriminated between patients at high and low risk of all-cause hospitalization (cardiovascular or noncardiovascular) in real time.

Same-Day Discharge After Elective Percutaneous Coronary Intervention for Chronic Total Occlusion in the UK

OBJECTIVES

This study examines the safety and feasibility of same-day discharge (SDD) in patients undergoing percutaneous coronary intervention (PCI) to coronary chronic total occlusions (CTOs) and explores independent associations of clinical and procedural characteristics with SDD.

BACKGROUND

While the recently published consensus statements recommend SDD following uncomplicated CTO-PCI, there are limited studies to support this approach.

METHODS

Data were obtained from the British Cardiovascular Intervention Society (BCIS) registry dataset including 21,330 patients who underwent CTO-PCI electively from 2007 to 2014 in England and Wales. We used multiple logistic regression to evaluate associations with SDD and the BCIS national risk model to examine for safety of SDD.

RESULTS

Although overnight stay remained the standard of care following elective CTO-PCI, SDD practice increased from 21.7% to 44.7%. Women were less likely to have SDD than males. SDD was more common in higher CTO volume centers (36.3%) than low CTO volume centers (31.6%), and SDD patient profiles grew riskier over time, with the average age of SDD patients increasing from 61.4 years to 63.2 years. Transradial PCI was most strongly independently associated with SDD (odds ratio [OR], 1.94; 95% confidence interval [CI], 1.80-2.09). Finally, the SDD observed 30-day mortality rates were not different vs those predicted by the BCIS risk model, and SDD was not independently associated with 30-day mortality (OR, 0.54; 95% CI, 0.25-1.15).

CONCLUSION

This study illustrates that SDD is safe in selected patients undergoing CTO-PCI.

Excess years of life lost to COVID-19 and other causes of death by sex, neighbourhood deprivation, and region in England and Wales during 2020: A registry-based study

BACKGROUND

Deaths in the first year of the Coronavirus Disease 2019 (COVID-19) pandemic in England and Wales were unevenly distributed socioeconomically and geographically. However, the full scale of inequalities may have been underestimated to date, as most measures of excess mortality do not adequately account for varying age profiles of deaths between social groups. We measured years of life lost (YLL) attributable to the pandemic, directly or indirectly, comparing mortality across geographic and socioeconomic groups.

METHODS AND FINDINGS

We used national mortality registers in England and Wales, from 27 December 2014 until 25 December 2020, covering 3,265,937 deaths. YLLs (main outcome) were calculated using 2019 single year sex-specific life tables for England and Wales. Interrupted time-series analyses, with panel time-series models, were used to estimate expected YLL by sex, geographical region, and deprivation quintile between 7 March 2020 and 25 December 2020 by cause: direct deaths (COVID-19 and other respiratory diseases), cardiovascular disease and diabetes, cancer, and other indirect deaths (all other causes). Excess YLL during the pandemic period were calculated by subtracting observed from expected values. Additional analyses focused on excess deaths for region and deprivation strata, by age-group. Between 7 March 2020 and 25 December 2020, there were an estimated 763,550 (95% CI: 696,826 to 830,273) excess YLL in England and Wales, equivalent to a 15% (95% CI: 14 to 16) increase in YLL compared to the equivalent time period in 2019. There was a strong deprivation gradient in all-cause excess YLL, with rates per 100,000 population ranging from 916 (95% CI: 820 to 1,012) for the least deprived quintile to 1,645 (95% CI: 1,472 to 1,819) for the most deprived. The differences in excess YLL between deprivation quintiles were greatest in younger age groups; for all-cause deaths, a mean of 9.1 years per death (95% CI: 8.2 to 10.0) were lost in the least deprived quintile, compared to 10.8 (95% CI: 10.0 to 11.6) in the most deprived; for COVID-19 and other respiratory deaths, a mean of 8.9 years per death (95% CI: 8.7 to 9.1) were lost in the least deprived quintile, compared to 11.2 (95% CI: 11.0 to 11.5) in the most deprived. For all-cause mortality, estimated deaths in the most deprived compared to the most affluent areas were much higher in younger age groups, but similar for those aged 85 or over. There was marked variability in both all-cause and direct excess YLL by region, with the highest rates in the North West. Limitations include the quasi-experimental nature of the research design and the requirement for accurate and timely recording.

CONCLUSIONS

In this study, we observed strong socioeconomic and geographical health inequalities in YLL, during the first calendar year of the COVID-19 pandemic. These were in line with long-standing existing inequalities in England and Wales, with the most deprived areas reporting the largest numbers in potential YLL.

Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

Recent minimum sample size formula (Riley et al.) for developing clinical prediction models help ensure that development datasets are of sufficient size to minimise overfitting. While these criteria are known to avoid excessive overfitting on average, the extent of variability in overfitting at recommended sample sizes is unknown. We investigated this through a simulation study and empirical example to develop logistic regression clinical prediction models using unpenalised maximum likelihood estimation, and various post-estimation shrinkage or penalisation methods. While the mean calibration slope was close to the ideal value of one for all methods, penalisation further reduced the level of overfitting, on average, compared to unpenalised methods. This came at the cost of higher variability in predictive performance for penalisation methods in external data. We recommend that penalisation methods are used in data that meet, or surpass, minimum sample size requirements to further mitigate overfitting, and that the variability in predictive performance and any tuning parameters should always be examined as part of the model development process, since this provides additional information over average (optimism-adjusted) performance alone. Lower variability would give reassurance that the developed clinical prediction model will perform well in new individuals from the same population as was used for model development.

Revascularisation strategies in patients with significant left main coronary disease during the COVID-19 pandemic

Background

There are limited data on the impact of the COVID‐19 pandemic on left main (LM) coronary revascularisation activity, choice of revascularisation strategy, and post‐procedural outcomes.

Methods

All patients with LM disease (≥50% stenosis) undergoing coronary revascularisation in England between January 1, 2017 and August 19, 2020 were included (n = 22,235), stratified by time‐period (pre‐COVID: 01/01/2017–29/2/2020; COVID: 1/3/2020–19/8/2020) and revascularisation strategy (percutaneous coronary intervention (PCI) vs. coronary artery bypass grafting (CABG). Logistic regression models were performed to examine odds ratio (OR) of 1) receipt of CABG (vs. PCI) and 2) in‐hospital and 30‐day postprocedural mortality, in the COVID‐19 period (vs. pre‐COVID).

Results

There was a decline of 1,354 LM revascularisation procedures between March 1, 2020 and July 31, 2020 compared with previous years' (2017–2019) averages (−48.8%). An increased utilization of PCI over CABG was observed in the COVID period (receipt of CABG vs. PCI: OR 0.46 [0.39, 0.53] compared with 2017), consistent across all age groups. No difference in adjusted in‐hospital or 30‐day mortality was observed between pre‐COVID and COVID periods for both PCI (odds ratio (OR): 0.72 [0.51. 1.02] and 0.83 [0.62, 1.11], respectively) and CABG (OR 0.98 [0.45, 2.14] and 1.51 [0.77, 2.98], respectively) groups.

Conclusion

LM revascularisation activity has significantly declined during the COVID period, with a shift towards PCI as the preferred strategy. Postprocedural mortality within each revascularisation group was similar in the pre‐COVID and COVID periods, reflecting maintenance in quality of outcomes during the pandemic. Future measures are required to safely restore LM revascularisation activity to pre‐COVID levels.

Adequacy of Mediastinal Lymph Node Sampling in Patients With Lung Cancer Undergoing Lung Resection

BACKGROUND

Intraoperative mediastinal lymph node sampling (MLNS) is a crucial component of lung cancer surgery. Whilst several sampling strategies have been clearly defined in guidelines from international organizations, reports of adherence to these guidelines are lacking. We aimed to assess our center's adherence to guidelines and determine whether adequacy of sampling is associated with survival.

MATERIALS AND METHODS

A single-center retrospective review of consecutive patients undergoing lung resection for primary lung cancer between January 2013 and December 2018 was undertaken. Sampling adequacy was assessed against standards outlined in the International Association for the Study of Lung Cancer 2009 guidelines. Multivariable logistic and Cox proportional hazards regression analyses were used to assess the impact of specific variables on adequacy and of specific variables on overall survival, respectively.

RESULTS

A total of 2380 patients were included in the study. Overall adequacy was 72.1% (n= 1717). Adherence improved from 44.8% in 2013 to 85.0% in 2018 (P< 0.001). Undergoing a right-sided resection increased the odds of adequate MLNS on multivariable logistic regression (odds ratio 1.666, 95% confidence interval [CI]: 1.385-2.003, P< 0.001). Inadequate MLNS was not significantly associated with reduced overall survival on log rank analysis (P= 0.340) or after adjustment with multivariable Cox proportional hazards (hazard ratio 0.839, 95% CI 0.643-1.093).

CONCLUSIONS

Adherence to standards improved significantly over time and was significantly higher for right-sided resections. We found no evidence of an association between adequate MLNS and overall survival in this cohort. A pressing need remains for the introduction of national guidelines defining acceptable performance.

The Association Between Socioeconomic Status, Sex, Race / Ethnicity and In-Hospital Mortality Among Patients Hospitalized for Heart Failure

BACKGROUND

The association between socioeconomic status (SES), sex, race / ethnicity and outcomes during hospitalization for heart failure (HF) has not previously been investigated.

METHODS AND RESULTS

We analyzed HF hospitalizations in the United States National Inpatient Sample between 2015 and 2017. Using a hierarchical, multivariable Poisson regression model to adjust for hospital- and patient-level factors, we assessed the association between SES, sex, and race / ethnicity and all-cause in-hospital mortality. We estimated the direct costs (USD) across SES groups. Among 4,287,478 HF hospitalizations, 40.8% were in high SES, 48.7% in female, and 70.0% in White patients. Relative to these comparators, low SES (homelessness or lowest quartile of median neighborhood income) (relative risk [RR] 1.02, 95% confidence interval [CI] 1.00-1.05) and male sex (RR 1.09, 95% CI 1.07-1.11) were associated with increased risk, whereas Black (RR 0.79, 95% CI 0.76-0.81) and Hispanic (RR 0.90, 95% CI 0.86-0.93) race / ethnicity were associated with a decreased risk of in-hospital mortality (5.1% of all hospitalizations). There were significant interactions between race / ethnicity and both, SES (P < .01) and sex (P = .04), such that racial/ethnic differences in outcome were more pronounced in low SES groups and in male patients. The median direct cost of admission was lower in low vs high SES groups ($9324.60 vs $10,940.40), female vs male patients ($9866.60 vs $10,217.10), and Black vs White patients ($9077.20 vs $10,019.80). The median costs increased with SES in all demographic groups primarily related to greater procedural utilization.

CONCLUSIONS

SES, sex, and race / ethnicity were independently associated with in-hospital mortality during HF hospitalization, highlighting possible care disparities. Racial/ethnic differences in outcome were more pronounced in low SES groups and in male patients.

Advanced cardiovascular risk prediction in the emergency department: updating a clinical prediction model - a large database study protocol

BACKGROUND

Patients presenting with chest pain represent a large proportion of attendances to emergency departments. In these patients clinicians often consider the diagnosis of acute myocardial infarction (AMI), the timely recognition and treatment of which is clinically important. Clinical prediction models (CPMs) have been used to enhance early diagnosis of AMI. The Troponin-only Manchester Acute Coronary Syndromes (T-MACS) decision aid is currently in clinical use across Greater Manchester. CPMs have been shown to deteriorate over time through calibration drift. We aim to assess potential calibration drift with T-MACS and compare methods for updating the model.

METHODS

We will use routinely collected electronic data from patients who were treated using TMACS at two large NHS hospitals. This is estimated to include approximately 14,000 patient episodes spanning June 2016 to October 2020. The primary outcome of acute myocardial infarction will be sourced from NHS Digital's admitted patient care dataset. We will assess the calibration drift of the existing model and the benefit of updating the CPM by model recalibration, model extension and dynamic updating. These models will be validated by bootstrapping and one step ahead prequential testing. We will evaluate predictive performance using calibrations plots and c-statistics. We will also examine the reclassification of predicted probability with the updated TMACS model.

DISCUSSION

CPMs are widely used in modern medicine, but are vulnerable to deteriorating calibration over time. Ongoing refinement using routinely collected electronic data will inevitably be more efficient than deriving and validating new models. In this analysis we will seek to exemplify methods for updating CPMs to protect the initial investment of time and effort. If successful, the updating methods could be used to continually refine the algorithm used within TMACS, maintaining or even improving predictive performance over time.

TRIAL REGISTRATION

ISRCTN number: ISRCTN41008456.

Remote monitoring data from cardiac implantable electronic devices predicts all-cause mortality

Aims

To determine if remotely monitored physiological data from cardiac implantable electronic devices (CIEDs) can be used to identify patients at high risk of mortality.

Methods and results

This study evaluated whether a risk score based on CIED physiological data (Triage-Heart Failure Risk Status, ‘Triage-HFRS’, previously validated to predict heart failure (HF) events) can identify patients at high risk of death. Four hundred and thirty-nine adults with CIEDs were prospectively enrolled. Primary observed outcome was all-cause mortality (median follow-up: 702 days). Several physiological parameters [including heart rate profile, atrial fibrillation/tachycardia (AF/AT) burden, ventricular rate during AT/AF, physical activity, thoracic impedance, therapies for ventricular tachycardia/fibrillation] were continuously monitored by CIEDs and dynamically combined to produce a Triage-HFRS every 24 h. According to transmissions patients were categorized into ‘high-risk’ or ‘never high-risk’ groups. During follow-up, 285 patients (65%) had a high-risk episode and 60 patients (14%) died (50 in high-risk group; 10 in never high-risk group). Significantly more cardiovascular deaths were observed in the high-risk group, with mortality rates across groups of high vs. never-high 10.3% vs. <4.0%; P = 0.03. Experiencing any high-risk episode was associated with a substantially increased risk of death [odds ratio (OR): 3.07, 95% confidence interval (CI): 1.57–6.58, P = 0.002]. Furthermore, each high-risk episode ≥14 consecutive days was associated with increased odds of death (OR: 1.26, 95% CI: 1.06–1.48; P = 0.006).

Conclusion

Remote monitoring data from CIEDs can be used to identify patients at higher risk of all-cause mortality as well as HF events. Distinct from other prognostic scores, this approach is automated and continuously updated.

Missing data was handled inconsistently in UK prediction models: a review of method used

OBJECTIVES

No clear guidance exists on handling missing data at each stage of developing, validating and implementing a clinical prediction model (CPM). We aimed to review the approaches to handling missing data that underly the CPMs currently recommended for use in UK healthcare.

STUDY DESIGN AND SETTING

A descriptive cross-sectional meta-epidemiological study aiming to identify CPMs recommended by the National Institute for Health and Care Excellence (NICE), which summarized how missing data is handled across their pipelines.

RESULTS

A total of 23 CPMs were included through "sampling strategy." Six missing data strategies were identified: complete case analysis (CCA), multiple imputation, imputation of mean values, k-nearest neighbours imputation, using an additional category for missingness, considering missing values as risk-factor-absent. 52% of the development articles and 48% of the validation articles did not report how missing data were handled. CCA was the most common approach used for development (40%) and validation (44%). At implementation, 57% of the CPMs required complete data entry, whilst 43% allowed missing values. Three CPMs had consistent paths in their pipelines.

CONCLUSION

A broad variety of methods for handling missing data underly the CPMs currently recommended for use in UK healthcare. Missing data handling strategies were generally inconsistent. Better quality assurance of CPMs needs greater clarity and consistency in handling of missing data.

Excess deaths from COVID-19 and other causes by region, neighbourhood deprivation level and place of death during the first 30 weeks of the pandemic in England and Wales: A retrospective registry study

BACKGROUND

Excess deaths during the COVID-19 pandemic compared with those expected from historical trends have been unequally distributed, both geographically and socioeconomically. Not all excess deaths have been directly related to COVID-19 infection. We investigated geographical and socioeconomic patterns in excess deaths for major groups of underlying causes during the pandemic.

METHODS

Weekly mortality data from 27/12/2014 to 2/10/2020 for England and Wales were obtained from the Office of National Statistics. Negative binomial regressions were used to model death counts based on pre-pandemic trends for deaths caused directly by COVID-19 (and other respiratory causes) and those caused indirectly by it (cardiovascular disease or diabetes, cancers, and all other indirect causes) over the first 30 weeks of the pandemic (7/3/2020-2/10/2020).

FINDINGS

There were 62,321 (95% CI: 58,849 to 65,793) excess deaths in England and Wales in the first 30 weeks of the pandemic. Of these, 46,221 (95% CI: 45,439 to 47,003) were attributable to respiratory causes, including COVID-19, and 16,100 (95% CI: 13,410 to 18,790) to other causes. Rates of all-cause excess mortality ranged from 78 per 100,000 in the South West of England and in Wales to 130 per 100,000 in the West Midlands; and from 93 per 100,000 in the most affluent fifth of areas to 124 per 100,000 in the most deprived. The most deprived areas had the highest rates of death attributable to COVID-19 and other indirect deaths, but there was no socioeconomic gradient for excess deaths from cardiovascular disease/diabetes and cancer.

INTERPRETATION

During the first 30 weeks of the COVID-19 pandemic there was significant geographic and socioeconomic variation in excess deaths for respiratory causes, but not for cardiovascular disease, diabetes and cancer. Pandemic recovery plans, including vaccination programmes, should take account of individual characteristics including health, socioeconomic status and place of residence.

FUNDING

None.

Development and internal validation of a clinical prediction model for 90-day mortality after lung resection: the RESECT-90 score

OBJECTIVES

The ability to accurately estimate the risk of peri-operative mortality after lung resection is important. There are concerns about the performance and validity of existing models developed for this purpose, especially when predicting mortality within 90 days of surgery. The aim of this study was therefore to develop a clinical prediction model for mortality within 90 days of undergoing lung resection.

METHODS

A retrospective database of patients undergoing lung resection in two UK centres between 2012 and 2018 was used to develop a multivariable logistic risk prediction model, with bootstrap sampling used for internal validation. Apparent and adjusted measures of discrimination (area under receiving operator characteristic curve) and calibration (calibration-in-the-large and calibration slope) were assessed as measures of model performance.

RESULTS

Data were available for 6600 lung resections for model development. Predictors included in the final model were age, sex, performance status, percentage predicted diffusion capacity of the lung for carbon monoxide, anaemia, serum creatinine, pre-operative arrhythmia, right-sided resection, number of resected bronchopulmonary segments, open approach and malignant diagnosis. Good model performance was demonstrated, with adjusted area under receiving operator characteristic curve, calibration-in-the-large and calibration slope values (95% confidence intervals) of 0.741 (0.700, 0.782), 0.006 (-0.143, 0.156) and 0.870 (0.679, 1.060), respectively.

CONCLUSIONS

The RESECT-90 model demonstrates good statistical performance for the prediction of 90-day mortality after lung resection. A project to facilitate large-scale external validation of the model to ensure that the model retains accuracy and is transferable to other centres in different geographical locations is currently underway.

A systematic review of risk prediction models for perioperative mortality after thoracic surgery

OBJECTIVES

Guidelines advocate that patients being considered for thoracic surgery should undergo a comprehensive preoperative risk assessment. Multiple risk prediction models to estimate the risk of mortality after thoracic surgery have been developed, but their quality and performance has not been reviewed in a systematic way. The objective was to systematically review these models and critically appraise their performance.

METHODS

The Cochrane Library and the MEDLINE database were searched for articles published between 1990 and 2019. Studies that developed or validated a model predicting perioperative mortality after thoracic surgery were included. Data were extracted based on the checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies.

RESULTS

A total of 31 studies describing 22 different risk prediction models were identified. There were 20 models developed specifically for thoracic surgery with two developed in other surgical specialties. A total of 57 different predictors were included across the identified models. Age, sex and pneumonectomy were the most frequently included predictors in 19, 13 and 11 models, respectively. Model performance based on either discrimination or calibration was inadequate for all externally validated models. The most recent data included in validation studies were from 2018. Risk of bias (assessed using Prediction model Risk Of Bias ASsessment Tool) was high for all except two models.

CONCLUSIONS

Despite multiple risk prediction models being developed to predict perioperative mortality after thoracic surgery, none could be described as appropriate for contemporary thoracic surgery. Contemporary validation of available models or new model development is required to ensure that appropriate estimates of operative risk are available for contemporary thoracic surgical practice.

Indirect Impact of the COVID-19 Pandemic on Activity and Outcomes of Transcatheter and Surgical Treatment of Aortic Stenosis in England

Supplemental Digital Content is available in the text.

Aortic stenosis requires timely treatment with either surgical aortic valve replacement (SAVR) or transcatheter aortic valve replacement (TAVR). This study aimed to investigate the indirect impact of coronavirus disease 2019 (COVID-19) on national SAVR and TAVR activity and outcomes.

Adaptive Symptom Monitoring Using Hidden Markov Models - An Application in Ecological Momentary Assessment

Wearable and mobile technology provides new opportunities to manage health conditions remotely and unobtrusively. For example, healthcare providers can repeatedly sample a person's condition to monitor progression of symptoms and intervene if necessary. There is usually a utility-tolerability trade-off between collecting information at sufficient frequencies and quantities to be useful, and over-burdening the user or the underlying technology, particularly when active input is required from the user. Selecting the next sampling time adaptively using previous responses, so that people are only sampled at high frequency when necessary, can help to manage this trade-off. We present a novel approach to adaptive sampling using clustered continuous-time hidden Markov models. The model predicts, at any given sampling time, the probability of moving to an 'alert' state, and the next sample time is scheduled when this probability has exceeded a given threshold. The clusters, each representing a distinct sub-model, allow heterogeneity in states and state transitions. The work is illustrated using longitudinal mental-health symptom data in 49 people collected using ClinTouch, a mobile app designed to monitor people with a diagnosis of schizophrenia. Using these data, we show how the adaptive sampling scheme behaves under different model parameters and risk thresholds, and how the average sampling can be substantially reduced whilst maintaining a high sampling frequency during high-risk periods.

Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small

OBJECTIVES

When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square prediction error in new individuals. However, shrinkage and penalty terms ('tuning parameters') are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance.

STUDY DESIGN AND SETTING

This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net.

RESULTS

In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell R2 is low. The problem can lead to considerable miscalibration of model predictions in new individuals.

CONCLUSION

Penalization methods are not a 'carte blanche'; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters.

A scoping review of causal methods enabling predictions under hypothetical interventions

BACKGROUND

The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions.

AIMS

We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges.

METHODS

We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies.

RESULTS

We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation.

CONCLUSIONS

There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.

Informative presence and observation in routine health data: A review of methodology for clinical risk prediction

Objective

Informative presence (IP) is the phenomenon whereby the presence or absence of patient data is potentially informative with respect to their health condition, with informative observation (IO) being the longitudinal equivalent. These phenomena predominantly exist within routinely collected healthcare data, in which data collection is driven by the clinical requirements of patients and clinicians. The extent to which IP and IO are considered when using such data to develop clinical prediction models (CPMs) is unknown, as is the existing methodology aiming at handling these issues. This review aims to synthesize such existing methodology, thereby helping identify an agenda for future methodological work.

Materials and Methods

A systematic literature search was conducted by 2 independent reviewers using prespecified keywords.

Results

Thirty-six articles were included. We categorized the methods presented within as derived predictors (including some representation of the measurement process as a predictor in the model), modeling under IP, and latent structures. Including missing indicators or summary measures as predictors is the most commonly presented approach amongst the included studies (24 of 36 articles).

Discussion

This is the first review to collate the literature in this area under a prediction framework. A considerable body relevant of literature exists, and we present ways in which the described methods could be developed further. Guidance is required for specifying the conditions under which each method should be used to enable applied prediction modelers to use these methods.

Conclusions

A growing recognition of IP and IO exists within the literature, and methodology is increasingly becoming available to leverage these phenomena for prediction purposes. IP and IO should be approached differently in a prediction context than when the primary goal is explanation. The work included in this review has demonstrated theoretical and empirical benefits of incorporating IP and IO, and therefore we recommend that applied health researchers consider incorporating these methods in their work.

Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, "living" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.

Descriptive analysis of the context of child sexual abuse reportedly perpetrated by female suspects: Insights from Saint Mary's Sexual Assault Referral Centre

BACKGROUND

Determining the prevalence and characteristics of female-perpetrated child sexual abuse (CSA) is fraught with difficultly. There is a historical lack of empirical research and a discrepancy between the number of cases that reach the attention of the authorities and its suspected prevalence in society. It is also noted that for a myriad of reasons many CSA reports do not progress through the criminal justice process so many remain as allegations rather than proven or disproven crimes.

OBJECTIVES

The study set out to answer the research questions: 'What are the characteristics and context of CSA reportedly perpetrated by females, and what are the similarities and differences in the context of alleged CSA committed by male and female suspects?'

PARTICIPANTS AND SETTING

This study presents data from all service users aged 0-17 years (n = 986) that attended Saint Mary's Sexual Assault Referral Centre (SARC) for a forensic medical examination over a three-year period.

METHODS

Data collection was performed retrospectively from the paper case files recorded at the time of attendance. Due to the small number of female suspects, analysis was restricted to frequency calculations.

RESULTS

Results show females were reportedly involved in the alleged abuse of less than 4% of the children attending SARC. Females appeared more likely to be associated with the alleged abuse of younger children and abuse occurring within the child's home.

CONCLUSIONS

This study's most arresting feature is that despite the large number of CSA cases examined, it was rare to have a female suspect. This study demonstrates how much is still unknown about female-perpetrated CSA.

Sample sizes of prediction model studies in prostate cancer were rarely justified and often insufficient

OBJECTIVE

Developing clinical prediction models (CPMs) on data of sufficient sample size is critical to help minimize overfitting. Using prostate cancer as a clinical exemplar, we aimed to investigate to what extent existing CPMs adhere to recent formal sample size criteria, or historic rules of thumb of events per predictor parameter (EPP)≥10.

STUDY DESIGN AND SETTING

A systematic review to identify CPMs related to prostate cancer, which provided enough information to calculate minimum sample size. We compared the reported sample size of each CPM against the traditional 10 EPP rule of thumb and formal sample size criteria.

RESULTS

About 211 CPMs were included. Three of the studies justified the sample size used, mostly using EPP rules of thumb. Overall, 69% of the CPMs were derived on sample sizes that surpassed the traditional EPP≥10 rule of thumb, but only 48% surpassed recent formal sample size criteria. For most CPMs, the required sample size based on formal criteria was higher than the sample sizes to surpass 10 EPP.

CONCLUSION

Few of the CPMs included in this study justified their sample size, with most justifications being based on EPP. This study shows that, in real-world data sets, adhering to the classic EPP rules of thumb is insufficient to adhere to recent formal sample size criteria.

Performance of prediction models for COVID-19: the Caudine Forks of the external validation

Healthcare systems worldwide have observed significant changes to meet demands due to the coronavirus disease 2019 (COVID-19) pandemic. The uncertainty surrounding optimal treatment, the rapid public health urgency and clinical emergencies have caused a chaotic disruption of the cases and their related contacts at inpatient and outpatient settings. Developing more tailored healthcare plans based on the currently available scientific evidence, could help improve clinical efficacy, treatment outcomes, prognosis, and health efficiency.

Existing evidence suggests that none of the COVID-19 prediction models can be supported for clinical use. Here we discuss “what next” in COVID-19 prediction.https://bit.ly/2SMtoLV

External validation of six existing multivariable clinical prediction models for short-term mortality in patients undergoing lung resection

 

OBJECTIVES

National guidelines advocate the use of clinical prediction models to estimate perioperative mortality for patients undergoing lung resection. Several models have been developed that may potentially be useful but contemporary external validation studies are lacking. The aim of this study was to validate existing models in a multicentre patient cohort.

METHODS

The Thoracoscore, Modified Thoracoscore, Eurolung, Modified Eurolung, European Society Objective Score and Brunelli models were validated using a database of 6600 patients who underwent lung resection between 2012 and 2018. Models were validated for in-hospital or 30-day mortality (depending on intended outcome of each model) and also for 90-day mortality. Model calibration (calibration intercept, calibration slope, observed to expected ratio and calibration plots) and discrimination (area under receiver operating characteristic curve) were assessed as measures of model performance.

RESULTS

Mean age was 66.8 years (±10.9 years) and 49.7% (n = 3281) of patients were male. In-hospital, 30-day, perioperative (in-hospital or 30-day) and 90-day mortality were 1.5% (n = 99), 1.4% (n = 93), 1.8% (n = 121) and 3.1% (n = 204), respectively. Model area under the receiver operating characteristic curves ranged from 0.67 to 0.73. Calibration was inadequate in five models and mortality was significantly overestimated in five models. No model was able to adequately predict 90-day mortality.

CONCLUSIONS

Five of the validated models were poorly calibrated and had inadequate discriminatory ability. The modified Eurolung model demonstrated adequate statistical performance but lacked clinical validity. Development of accurate models that can be used to estimate the contemporary risk of lung resection is required.

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.

Pride and prejudice - What can we learn from peer review?

Objectives: Peer review is a powerful tool that steers the education and practice of medical researchers but may allow biased critique by anonymous reviewers. We explored factors unrelated to research quality that may influence peer review reports, and assessed the possibility that sub-types of reviewers exist. Our findings could potentially improve the peer review process.Methods: We evaluated the harshness, constructiveness and positiveness in 596 reviews from journals with open peer review, plus 46 reviews from colleagues' anonymously reviewed manuscripts. We considered possible influencing factors, such as number of authors and seasonal trends, on the content of the review. Finally, using machine-learning we identified latent types of reviewer with differing characteristics.Results: Reviews provided during a northern-hemisphere winter were significantly harsher, suggesting a seasonal effect on language. Reviews for articles in journals with an open peer review policy were significantly less harsh than those with an anonymous review process. Further, we identified three types of reviewers: nurturing, begrudged, and blasé.Conclusion: Nurturing reviews were in a minority and our findings suggest that more widespread open peer reviewing could improve the educational value of peer review, increase the constructive criticism that encourages researchers, and reduce pride and prejudice in editorial processes.