Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

Predicting Flare in Patients With Rheumatoid Arthritis in Biologic Induced Remission, on Tapering, and on Stable Therapy

OBJECTIVE

The tapering of biologic disease-modifying antirheumatic drug (b-DMARD) therapy for patients with rheumatoid arthritis (RA) in stable remission is frequently undertaken, but specific guidance on how to successfully taper is lacking. The objective of this study is to identify predictors of flare in patients in stable b-DMARD-induced clinical remission, who did or did not follow structured b-DMARD tapering.

METHODS

Patients with RA receiving b-DMARD treatment who had achieved sustained remission according to a Disease Activity Score in 28 joints using the C-reactive protein level (DAS28-CRP) <2.6 for ≥6 months were offered tapering. Clinical, ultrasound (US) (total power Doppler [PD]/grayscale abnormalities), CD4+ T cell subsets, and patient-reported outcomes (PROs) were collected at inclusion. The primary endpoint was the occurrence of flare (loss of DAS28-CRP remission) over 12 months. Logistic regression analyses identified predictors of flare. Dichotomization into high/low-risk groups was based on 80% specificity using the area under the receiving operator curve (AUROC).

RESULTS

Of 63 patients choosing tapering, 23 (37%) flared compared with 12 of 60 (20%) on stable treatment (P = 0.043). All patients who flared regained remission upon reinstating treatment. In the tapering group, flare was associated with lower regulatory T cell (Treg) (P < 0.0001) and higher CRP levels (P < 0.0001), erythrocyte sedimentation rate (P < 0.035), and inflammation-related cells (IRCs) (P = 0.054); stepwise modeling selected Tregs (odds ratio [OR] = 0.350, P = 0.004), IRCs (OR = 1.871, P = 0.007), and CRP level (OR = 1.577, P = 0.004) with 81.7% accuracy and AUROC = 0.890. In the continued therapy group, modeling retained the tender joint count, total PD, and visual analog scale pain score, with 82.1% accuracy and AUROC = 0.899. Most patients in the study were considered low risk of flare (80 of 123 patients [65%]). Only 5 of 37 (13.5%) of the low-risk patients who tapered flared, which was notable compared with the continued therapy group (20% flare).

CONCLUSION

Flare on tapering b-DMARDs was predicted by lower Tregs and elevated inflammation biomarkers (IRCs/CRP level); flare on continued b-DMARDs was associated with raised pain parameters and US inflammation. Knowledge of these biomarkers should improve outcomes by targeted selection for tapering, and by increased monitoring of those on continued therapy predicted to flare.

Development and application of an optimised Bayesian shrinkage prior for spectroscopic biomedical diagnostics

BACKGROUND AND OBJECTIVE

Classification of vibrational spectra is often challenging for biological substances containing similar molecular bonds, interfering with spectral outputs. To address this, various approaches are widely studied. However, whilst providing powerful estimations, these techniques are computationally extensive and frequently overfit the data. Shrinkage priors, which favour models with relatively few predictor variables, are often applied in Bayesian penalisation techniques to avoid overfitting.

METHODS

Using the logit-normal continuous analogue of the spike-and-slab (LN-CASS) as the shrinkage prior and modelling, we have established classification for accurate analysis, with the established system found to be faster than conventional least absolute shrinkage and selection operator, horseshoe or spike-and-slab. These were examined versus coefficient data based on a linear regression model and vibrational spectra produced via density functional theory calculations. Then applied to Raman spectra from saliva to classify the sample sex.

RESULTS

Subsequently applied to the acquired spectra from saliva, the evaluated models exhibited high accuracy (AUC>90 %) even when number of parameters was higher than the number of observations. Analyses of spectra for all Bayesian models yielded high-classification accuracy upon cross-validation. Further, for saliva sensing, LN-CASS was found to be the only classifier with 100 %-accuracy in predicting the output based on a leave-one-out cross validation.

CONCLUSIONS

With potential applications in aiding diagnosis from small spectroscopic datasets and are compatible with a range of spectroscopic data formats. As seen with the classification of IR and Raman spectra. These results are highly promising for emerging developments of spectroscopic platforms for biomedical diagnostic sensing systems.

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration

OBJECTIVES

In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.

STUDY DESIGN AND SETTING

A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).

RESULTS

The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.

CONCLUSION

We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

Development and Internal Validation of a Multivariable Prediction Model to Predict Repeat Attendances in the Pediatric Emergency Department: A Retrospective Cohort Study

OBJECTIVE

Unplanned reattendances to the pediatric emergency department (PED) occur commonly in clinical practice. Multiple factors influence the decision to return to care, and understanding risk factors may allow for better design of clinical services. We developed a clinical prediction model to predict return to the PED within 72 hours from the index visit.

METHODS

We retrospectively reviewed all attendances to the PED of Royal Manchester Children's Hospital between 2009 and 2019. Attendances were excluded if they were admitted to hospital, aged older than 16 years or died in the PED. Variables were collected from Electronic Health Records reflecting triage codes. Data were split temporally into a training (80%) set for model development and a test (20%) set for internal validation. We developed the prediction model using LASSO penalized logistic regression.

RESULTS

A total of 308,573 attendances were included in the study. There were 14,276 (4.63%) returns within 72 hours of index visit. The final model had an area under the receiver operating characteristic curve of 0.64 (95% confidence interval, 0.63-0.65) on temporal validation. The calibration of the model was good, although with some evidence of miscalibration at the high extremes of the risk distribution. After-visit diagnoses codes reflecting a nonspecific problem ("unwell child") were more common in children who went on to reattend.

CONCLUSIONS

We developed and internally validated a clinical prediction model for unplanned reattendance to the PED using routinely collected clinical data, including markers of socioeconomic deprivation. This model allows for easy identification of children at the greatest risk of return to PED.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

External validation of a United Kingdom primary-care Cushing's prediction tool in a population of referred Dutch dogs

BACKGROUND

A prediction tool was developed and internally validated to aid the diagnosis of Cushing's syndrome in dogs attending UK primary-care practices. External validation is an important part of model validation to assess model performance when used in different populations.

OBJECTIVES

To assess the original prediction model's transportability, applicability, and diagnostic performance in a secondary-care practice in the Netherlands.

ANIMALS

Two hundred thirty client-owned dogs.

METHODS

Retrospective observational study. Medical records of dogs under investigation of Cushing's syndrome between 2011 and 2020 were reviewed. Dogs diagnosed with Cushing's syndrome by the attending internists and fulfilling ALIVE criteria were defined as cases, others as non-cases. All dogs were scored using the aforementioned prediction tool. Dog characteristics and predictor-outcome effects in development and validation data sets were compared to assess model transportability. Calibration and discrimination were examined to assess model performance.

RESULTS

Eighty of 230 dogs were defined as cases. Significant differences in dog characteristics were found between UK primary-care and Dutch secondary-care populations. Not all predictors from the original model were confirmed to be significant predictors in the validation sample. The model systematically overestimated the probability of having Cushing's syndrome (a = -1.10, P < .001). Calibration slope was 1.35 and discrimination proved excellent (area under the receiver operating curve = 0.83).

CONCLUSIONS AND CLINICAL IMPORTANCE

The prediction model had moderate transportability, excellent discriminatory ability, and overall overestimated probability of having Cushing's syndrome. This study confirms its utility, though emphasizes that ongoing validation efforts of disease prediction tools are a worthwhile effort.

Cost Analysis of Magnetic Resonance Imaging and Computed Tomography in Cardiology: A Case Study of a University Hospital Complex in the Euro Region

INTRODUCTION

In recent years, several hospitals have incorporated MRI equipment managed directly by their cardiology departments. The aim of our work is to determine the total cost per test of both CT and MRI in the setting of a Cardiology Department of a tertiary hospital.

MATERIALS AND METHODS

The process followed for estimating the costs of CT and MRI tests consists of three phases: (1) Identification of the phases of the testing process; (2) Identification of the resources consumed in carrying out the tests; (3) Quantification and assessment of inputs.

RESULTS

MRI involves higher personnel (EUR 66.03 vs. EUR 49.17) and equipment (EUR 89.98 vs. EUR 33.73) costs, while CT consumes higher expenditures in consumables (EUR 93.28 vs. EUR 22.95) and overheads (EUR 1.64 vs. EUR 1.55). The total cost of performing each test is higher in MRI (EUR 180.60 vs. EUR 177.73).

CONCLUSIONS

We can conclude that the unit cost of each CT and MRI performed in that unit are EUR 177.73 and EUR 180.60, respectively, attributable to consumables in the case of CT and to amortization of equipment and staff time in the case of MRI.

Targeted validation: validating clinical prediction models in their intended population and setting

Clinical prediction models must be appropriately validated before they can be used. While validation studies are sometimes carefully designed to match an intended population/setting of the model, it is common for validation studies to take place with arbitrary datasets, chosen for convenience rather than relevance. We call estimating how well a model performs within the intended population/setting "targeted validation". Use of this term sharpens the focus on the intended use of a model, which may increase the applicability of developed models, avoid misleading conclusions, and reduce research waste. It also exposes that external validation may not be required when the intended population for the model matches the population used to develop the model; here, a robust internal validation may be sufficient, especially if the development dataset was large.