Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

Predicting hospital admissions, ICU utilization, and prolonged length of stay among febrile pediatric emergency department patients using incomplete and imbalanced electronic health record (EHR) data strategies

OBJECTIVE

Determine the efficacy of commonly used approaches to handling missing and/or imbalanced Electronic Health Record (EHR) data on the performance of predictive models targeting risk of admission, intensive care unit (ICU) use, or prolonged length of stay (PLOS) among presenting febrile pediatric emergency department (ED) patients.

MATERIALS AND METHODS

Historical ED EHR data was used to train a series of XGBoost (XGB) and logistic regression (LR) classifiers. Data handling strategies included imputation methods (multiple imputation (MI), median imputation, complete case (CC) analysis), and imbalanced data corrections (minority oversampling, stratified sub-group analysis). Model performance was evaluated using discriminative (AUC, AUPRC) and calibration metrics (Brier score, Z-scores, p-values).

RESULTS

Among the study population, 34 % were admitted, 2 % utilized the ICU, and 7 % had a PLOS. Significant data missingness was observed and determined to be not at random (MNAR). In predicting admissions using data recorded within the first two hours of presentation, LR trained using full cohort with median imputation was comparable to MI yielding well-calibrated admissions models with an AUC/AUPRC of 0.82/0.73 while CC analysis yielded an AUC/AUPRC of 0.76/0.78. XGB, trained with unimputed data, produced a well-calibrated admissions classifier with an AUC/AUPRC of 0.85/0.78. In contrast, imbalanced data correction techniques, including synthetic minority oversampling (SMOTE), risk stratification, or the use of XGB did not significantly improve the poor AUPRC and calibration performance of LR models predicting ICU and PLOS.

CONCLUSION

Both XGB and LR with median imputation demonstrated robust performance in predicting admissions in the presence of missing data. However, deriving clinically useful models for rare outcomes, such as ICU use or PLOS, remains a challenge due to poor precision/recall and calibration performance. Further research is needed to improve the prediction of rare outcomes in this population.

Development and validation of a nomogram to predict the risk of in-hospital MACE for emergence NSTE-ACS: A retrospective multicenter study based on the Chinese population

PURPOSE

Our study aims to develop and validate an effective in-hospital major adverse cardiovascular events(MACE) prediction model for patients with emergency Non-ST elevation acute coronary syndrome(NSTE-ACS).

METHODS

We retrospectively collected NSTE-ACS patients in three tertiary hospitals in Chongqing. In-hospital MACE was the predicted outcome. Patients from one hospital were divided into training set and internal validation set according to the ratio of 7:3. Besides, 662 patients from two other tertiary hospitals were for external validation. Patient information including demographics, laboratory tests results and disease course records were for comprehensive analysis. Finally, LASSO were used to identify the predictors and develop the model. This model was subsequently visualized as a nomogram, followed by both internal and external validations.The receiver operating characteristic curve, calibration curve and clinical decision curve analysis were used to assess the model's discrimination, calibration and clinical applicability, respectively.

RESULTS

A total of 3,308 patients were included, 375 of whom developed in-hospital MACE. The LR model demonstrated that length of stay, neutrophils, myoglobin, NYHA, CCI, NT-proBNP, LVEF and respiratory failure were risk factors for in-hospital MACE in emergence NSTE-ACS patients. In the training set, the AUC was 0.860 (95%CI:0.831-0.889). In external validation,the AUC was 0.855(95%CI:0.808-0.902), and both the calibration curve and DCA in validation set also revealed stable predictive accuracy and clinical validity.Additionally,it is available to calculate the MACE risk online via the web page (https://cocozhou99.shinyapps.io/DynNomapp/).

CONCLUSION

The prediction model we constructed has good predictive performance and can help healthcare professionals accurately assess the risk of in-hospital MACE in emergence NSTE-ACS patients.

Evaluating the sample size requirements of tree-based ensemble machine learning techniques for clinical risk prediction

Machine learning techniques (MLTs) are increasingly being used to develop clinical risk prediction models for binary health outcomes but the sample size requirements for developing and validating such models remain unclear. This study investigates whether sample size guidelines that target mean absolute prediction error (MAPE) for logistic regression models can be applied to tree-based ensemble MLTs (bagging, random forests, and boosting). Simulations based on two large cardiovascular datasets were used to evaluate the performance of MLTs in terms of MAPE, calibration, the C-statistic and Brier score, across six data-generating mechanisms (DGMs) and varying sample sizes. When the DGM and analysis model matched, boosting required a sample size 2-3 times larger than recommended; random forests and bagging did not achieve the target MAPE even with a 12-fold increase. For a neutral DGM that did not match any of the analysis models, logistic regression with only main effects and boosting resulted in target MAPE values with a 12-fold increase in the recommended sample size. For external validation, our simulations showed that sample size guidelines to achieve a target precision of the estimated C-statistic were suitable, and thus may be used to inform sample size calculations for MLTs.

Development of a nomogram for root caries risk assessment in a Chinese elderly population

BACKGROUND

Root caries is a common oral disease among the elderly, and its prevalence increases year by year with the deepening of population aging. Existing caries risk assessment models do not fully consider the specific risk factors for root caries in the elderly population. This study aims to develop a predictive model for root caries risk in the elderly to facilitate early identification and personalized intervention.

METHODS

This study included 310 individuals over the age of 60. Data on oral health status, frailty level, cognitive function, and sociodemographic factors were collected through oral examinations and six questionnaires, including the Fried Frailty Phenotype, MMSE, Barthel Index, SXI, OHIP-14, and a demographic questionnaire. A root caries prediction model was constructed using multivariate logistic regression analysis, and the model's performance was evaluated using the Hosmer-Lemeshow test, ROC curves, and Decision Curve Analysis (DCA).

RESULTS

Key risk factors for root caries included age, history of coronal caries, secondary caries, use of removable partial dentures, and the number of exposed root surfaces. The nomogram model demonstrated good predictive accuracy and clinical utility in both the training cohort (AUC=0.840) and the validation cohort (AUC=0.834).

CONCLUSION

The nomogram developed in this study provides an effective tool for the early assessment of root caries risk in the elderly. Future studies should conduct larger sample and multicenter validation research. The study was retrospectively registered in the Chinese Clinical Trial Registry (http://www.chictr.org.cn/) with the registration number ChiCTR2500099297.

CLINICAL SIGNIFICANCE

This nomogram prediction model can help clinicians identify high-risk elderly patients and take early preventive measures, enhancing personalized oral health management and improving the oral health of the elderly population.

Larger sample sizes are needed when developing a clinical prediction model using machine learning in oncology: methodological systematic review

BACKGROUND AND OBJECTIVES

Having a sufficient sample size is crucial when developing a clinical prediction model. We reviewed details of sample size in studies developing prediction models for binary outcomes using machine learning (ML) methods within oncology and compared the sample size used to develop the models with the minimum required sample size needed when developing a regression-based model (Nmin).

METHODS

We searched the Medline (via OVID) database for studies developing a prediction model using ML methods published in December 2022. We reviewed how sample size was justified. We calculated Nmin, which is the Nmin, and compared this with the sample size that was used to develop the models.

RESULTS

Only one of 36 included studies justified their sample size. We were able to calculate Nmin for 17 (47%) studies. 5/17 studies met Nmin, allowing to precisely estimate the overall risk and minimize overfitting. There was a median deficit of 302 participants with the event (n = 17; range: -21,331 to 2298) when developing the ML models. An additional three out of the 17 studies met the required sample size to precisely estimate the overall risk only.

CONCLUSION

Studies developing a prediction model using ML in oncology seldom justified their sample size and sample sizes were often smaller than Nmin. As ML models almost certainly require a larger sample size than regression models, the deficit is likely larger. We recommend that researchers consider and report their sample size and at least meet the minimum sample size required when developing a regression-based model.

Lipoprotein(a), high-sensitivity c-reactive protein, homocysteine and cardiovascular disease in the Multi-Ethnic Study of Atherosclerosis

Background and aims

Elevated lipoprotein(a) [Lp(a)], high-sensitivity C-Reactive Protein (hs-CRP), and total homocysteine (tHcy) are associated with atherosclerotic cardiovascular disease (ASCVD) risk. This study investigated the individual and joint associations of Lp(a), hs-CRP and tHcy with coronary heart disease (CHD) and stroke.

Methods

This study was conducted in the Multi-Ethnic Study of Atherosclerosis (MESA) cohort (2000-2017) (CHD analytic N = 6,676; stroke analytic N = 6,674 men and women). Associations between Lp(a) (<50 vs. ≥50 mg/dL), hs-CRP (<2 vs. ≥2 mg/L) and tHcy (<12 vs. ≥12 µmol/L) and CHD and stroke incidence were evaluated individually and jointly using Cox proportional hazards regression.

Results

Individually, elevated tHcy was associated with CHD and stroke incidence, Lp(a) with CHD only and hs-CRP with stroke only. In combined analyses, CHD risk was higher when multiple biomarkers were elevated [hs-CRP+Lp(a), hazard ratio (HR)=1.39, 95 % confidence interval (CI): 1.06, 1.82; hs-CRP+ tHcy, HR = 1.34, 95 % CI: 1.02, 1.75; Lp(a)+ tHcy HR = 1.58, 95 % CI: 1.08, 2.30; hs-CRP+Lp(a)+ tHcy HR = 2.02, 95 % CI: 1.26, 3.24]. Stroke risk was elevated when hs-CRP and either Lp(a) (HR = 1.51, 95 % CI: 1.02, 2.23) or tHcy (HR = 2.10, 95 % CI: 1.44, 3.06) was also high, when all three biomarkers were elevated (HR = 2.99, 95 % CI: 1.61, 5.58), or when hs-CRP and tHcy (HR = 1.79, 95 % CI: 1.16, 2.76) were both high.

Conclusions

Risk of ASCVD was highest with concomitant elevation of tHcy, hs-CRP and Lp(a). Inclusion of tHcy and consideration of biomarker combination rather than individual biomarker levels may help better identify individuals at greatest risk for ASCVD events.

Overlooked and underpowered: a meta-research addressing sample size in radiomics prediction models for binary outcomes

OBJECTIVES

To investigate how studies determine the sample size when developing radiomics prediction models for binary outcomes, and whether the sample size meets the estimates obtained by using established criteria.

METHODS

We identified radiomics studies that were published from 01 January 2023 to 31 December 2023 in seven leading peer-reviewed radiological journals. We reviewed the sample size justification methods, and actual sample size used. We calculated and compared the actual sample size used to the estimates obtained by using three established criteria proposed by Riley et al. We investigated which characteristics factors were associated with the sufficient sample size that meets the estimates obtained by using established criteria proposed by Riley et al. RESULTS: We included 116 studies. Eleven out of one hundred sixteen studies justified the sample size, in which 6/11 performed a priori sample size calculation. The median (first and third quartile, Q1, Q3) of the total sample size is 223 (130, 463), and those of sample size for training are 150 (90, 288). The median (Q1, Q3) difference between total sample size and minimum sample size according to established criteria are -100 (-216, 183), and those differences between total sample size and a more restrictive approach based on established criteria are -268 (-427, -157). The presence of external testing and the specialty of the topic were associated with sufficient sample size.

CONCLUSION

Radiomics studies are often designed without sample size justification, whose sample size may be too small to avoid overfitting. Sample size justification is encouraged when developing a radiomics model.

KEY POINTS

Question Sample size justification is critical to help minimize overfitting in developing a radiomics model, but is overlooked and underpowered in radiomics research. Findings Few of the radiomics models justified, calculated, or reported their sample size, and most of them did not meet the recent formal sample size criteria. Clinical relevance Radiomics models are often designed without sample size justification. Consequently, many models are too small to avoid overfitting. It should be encouraged to justify, perform, and report the considerations on sample size when developing radiomics models.

Development and internal validation of a clinical risk tool to predict chronic postsurgical pain in adults: a prospective multicentre cohort study

ABSTRACT

Chronic postsurgical pain (CPSP) is a highly prevalent condition. To improve CPSP management, we aimed to develop and internally validate generalizable point-of-care risk tools for preoperative and postoperative prediction of CPSP 3 months after surgery. A multicentre, prospective, cohort study in adult patients undergoing elective surgery was conducted between May 2021 and May 2023. Prediction models were developed for the primary outcome according to the International Association for the Study of Pain criteria and a secondary threshold-based CPSP outcome. Models were developed with multivariable logistic regression and backward stepwise selection. Internal validation was conducted using bootstrap resampling, and optimism was corrected by shrinkage of predictor weights. Model performance was assessed by discrimination and calibration. Clinical utility was assessed by decision curve analysis. The final cohort included 960 patients, 16.3% experienced CPSP according to the primary outcome and 33.6% according to the secondary outcome. The primary CPSP model included age and presence of other preoperative pain. Predictors in the threshold-based models associated with an increased risk of CPSP included younger age, female sex, preoperative pain in the surgical area, other preoperative pain, orthopedic surgery, minimally invasive surgery, expected surgery duration, and acute postsurgical pain intensity. Optimism-corrected area-under-the-receiver-operating curves for preoperative and postoperative threshold-based models were 0.748 and 0.747, respectively. These models demonstrated good calibration and clinical utility. The primary CPSP model demonstrated fair predictive performance including 2 significant predictors. Derivation of a generalizable risk tool with point-of-care predictors was possible for the threshold-based CPSP models but requires independent validation.

Putting computational models of immunity to the test-An invited challenge to predict B.pertussis vaccination responses

Systems vaccinology studies have been used to build computational models that predict individual vaccine responses and identify the factors contributing to differences in outcome. Comparing such models is challenging due to variability in study designs. To address this, we established a community resource to compare models predicting B. pertussis booster responses and generate experimental data for the explicit purpose of model evaluation. We here describe our second computational prediction challenge using this resource, where we benchmarked 49 algorithms from 53 scientists. We found that the most successful models stood out in their handling of nonlinearities, reducing large feature sets to representative subsets, and advanced data preprocessing. In contrast, we found that models adopted from literature that were developed to predict vaccine antibody responses in other settings performed poorly, reinforcing the need for purpose-built models. Overall, this demonstrates the value of purpose-generated datasets for rigorous and open model evaluations to identify features that improve the reliability and applicability of computational models in vaccine response prediction.

Prognostic models for prediction of perioperative allogeneic red blood cell transfusion in adult cardiac surgery: A systematic review and meta-analysis

OBJECTIVES

Identifying cardiac surgical patients at risk of requiring red blood cell (RBC) transfusion is crucial for optimizing their outcome. We critically appraised prognostic models preoperatively predicting perioperative exposure to RBC transfusion in adult cardiac surgery and summarized model performance.

METHODS

Design: Systematic review and meta-analysis.

STUDY ELIGIBILITY CRITERIA

Studies developing and/or externally validating models preoperatively predicting perioperative RBC transfusion in adult cardiac surgery. Information sources MEDLINE, CENTRAL & CDSR, Embase, Transfusion Evidence Library, Web of Science, Scopus, ClinicalTrials.gov, and WHO ICTRP. Risk of bias and applicability: Quality of reporting was assessed with the Transparent Reporting of studies on prediction models for Individual Prognosis or Diagnosis adherence form, and risk of bias and applicability with the Prediction model Risk of Bias ASsessment Tool.

SYNTHESIS METHODS

Random-effects meta-analyses of concordance-statistics and total observed:expected ratios for models externally validated ≥5 times.

RESULTS

Nine model development, and 27 external validation studies were included. The average TRIPOD adherence score was 66.4% (range 44.1%-85.2%). All studies but 1 were rated high risk of bias. For TRUST and TRACK, the only models externally validated ≥5 times, summary c-statistics were 0.74 (95% CI: 0.65-0.84; 6 contributing studies) and 0.72 (95% CI: 0.68-0.75; 5 contributing studies) respectively, and summary total observed:expected ratios were 0.86 (95% CI: 0.71-1.05; 5 contributing studies) and 0.94 (95% CI: 0.74-1.19; 5 contributing studies), respectively. Considerable heterogeneity was observed in all meta-analyses.

DISCUSSION

Future high quality external validation and model updating studies which strictly adhere to reporting guidelines, are warranted.

The definition of predictor and outcome variables in mortality prediction models: a scoping review and quality of reporting study

BACKGROUND AND OBJECTIVES

Mortality prediction models are promising tools for guiding clinical decision-making and resource allocation in intensive care units (ICUs). Clearly specified predictor and outcome variables are necessary to enable external validation and safe clinical application of prediction models. The objective of this study was to identify the predictor and outcome variables used in different mortality prediction models in the ICU and investigate their reporting.

METHODS

For this scoping review, MEDLINE, EMBASE, Web of Science, and the Cochrane Central Register of Controlled Trials were searched. Studies developed within a general ICU population reporting on prediction models with mortality as a primary or secondary outcome were eligible. The selection criteria were adopted from a review by Keuning et al. Predictor and outcome variables, variable characteristics (defined as units, definitions, moments of measurement, and methods of measurement), and publication details (defined as first author, year of publication and title) were extracted from the included studies. Predictor and outcome variable categories were demographics, chronic disease, care logistics, acute diagnosis, clinical examination and physiological derangement, laboratory assessment, additional diagnostics, support and therapy, risk scores, and (mortality) outcomes.

RESULTS

A total of 56 mortality prediction models, containing 204 unique predictor and outcome variables, were included. The predictor variables most frequently included in the models were age (40 times), admission type (27 times), and mechanical ventilation (21 times). We observed that single variables were measured with different units, according to different definitions, at a different moment, and with a different method of measurement in different studies. The reporting of the unit was mostly complete (98% overall, 95% in the laboratory assessment category), whereas the definition of the variable (74% overall, 63% in the chronic disease category) and method of measurement (70% overall, 34% in the demographics category) were most often lacking.

CONCLUSION

Accurate and transparent reporting of predictor and outcome variables is paramount to enhance reproducibility, model performance in different contexts, and validity. Since unclarity about the required input data may introduce bias and thereby affect model performance, this study advocates that prognostic ICU models can be improved by transparent and clear reporting of predictor and outcome variables and their characteristics.

Risk prediction tools for pressure injury occurrence: an umbrella review of systematic reviews reporting model development and validation methods

BACKGROUND

Pressure injuries (PIs) place a substantial burden on healthcare systems worldwide. Risk stratification of those who are at risk of developing PIs allows preventive interventions to be focused on patients who are at the highest risk. The considerable number of risk assessment scales and prediction models available underscores the need for a thorough evaluation of their development, validation, and clinical utility. Our objectives were to identify and describe available risk prediction tools for PI occurrence, their content and the development and validation methods used.

METHODS

The umbrella review was conducted according to Cochrane guidance. MEDLINE, Embase, CINAHL, EPISTEMONIKOS, Google Scholar, and reference lists were searched to identify relevant systematic reviews. The risk of bias was assessed using adapted AMSTAR-2 criteria. Results were described narratively. All included reviews contributed to building a comprehensive list of risk prediction tools.

RESULTS

We identified 32 eligible systematic reviews only seven of which described the development and validation of risk prediction tools for PI. Nineteen reviews assessed the prognostic accuracy of the tools and 11 assessed clinical effectiveness. Of the seven reviews reporting model development and validation, six included only machine learning models. Two reviews included external validations of models, although only one review reported any details on external validation methods or results. This was also the only review to report measures of both discrimination and calibration. Five reviews presented measures of discrimination, such as the area under the curve (AUC), sensitivities, specificities, F1 scores, and G-means. For the four reviews that assessed the risk of bias assessment using the PROBAST tool, all models but one were found to be at high or unclear risk of bias.

CONCLUSIONS

Available tools do not meet current standards for the development or reporting of risk prediction models. The majority of tools have not been externally validated. Standardised and rigorous approaches to risk prediction model development and validation are needed.

TRIAL REGISTRATION

The protocol was registered on the Open Science Framework ( https://osf.io/tepyk ).

Machine Learning and Metabolomics Predict Mesenchymal Stem Cell Osteogenic Differentiation in 2D and 3D Cultures

Stem cells have been widely used to produce artificial bone grafts. Nonetheless, the variability in the degree of stem cell differentiation is an inherent drawback of artificial graft development and requires robust evaluation tools that can certify the quality of stem cell-based products and avoid source-tissue-related and patient-specific variability in outcomes. Omics analyses have been utilised for the evaluation of stem cell attributes in all stages of stem cell biomanufacturing. Herein, metabolomics in combination with machine learning was utilised for the benchmarking of osteogenic differentiation quality in 2D and 3D cultures. Metabolomics analysis was performed with the use of gas chromatography-mass spectrometry (GC-MS). A set of 11 metabolites was used to train an XGboost model which achieved excellent performance in distinguishing between differentiated and undifferentiated umbilical cord blood mesenchymal stem cells (UCB MSCs). The model was benchmarked against samples not present in the training set, being able to efficiently capture osteogenesis in 3D UCB MSC cultures with an area under the curve (AUC) of 82.6%. On the contrary, the model did not capture any differentiation in Wharton's Jelly MSC samples, which are well-known underperformers in osteogenic differentiation (AUC of 56.2%). Mineralisation was significantly correlated with the levels of fumarate, glycerol, and myo-inositol, the four metabolites found most important for model performance (R2 = 0.89, R2 = 0.94, and R2 = 0.96, and p = 0.016, p = 0.0059, and p = 0.0022, respectively). In conclusion, our results indicate that metabolomics in combination with machine learning can be used for the development of reliable potency assays for the evaluation of Advanced Therapy Medicinal Products.

The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study

OBJECTIVES

The development of clinical prediction models is often impeded by the occurrence of missing values in the predictors. Various methods for imputing missing values before modeling have been proposed. Some of them are based on variants of multiple imputations by chained equations, while others are based on single imputation. These methods may include elements of flexible modeling or machine learning algorithms, and for some of them user-friendly software packages are available. The aim of this study was to investigate by simulation if some of these methods consistently outperform others in performance measures of clinical prediction models.

STUDY DESIGN AND SETTING

We simulated development and validation cohorts by mimicking observed distributions of predictors and outcome variable of a real data set. In the development cohorts, missing predictor values were created in 36 scenarios defined by the missingness mechanism and proportion of noncomplete cases. We applied three imputation algorithms that were available in R software (R Foundation for Statistical Computing, Vienna, Austria): mice, aregImpute, and missForest. These algorithms differed in their use of linear or flexible models, or random forests, the way of sampling from the predictive posterior distribution, and the generation of a single or multiple imputed data set. For multiple imputation, we also investigated the impact of the number of imputations. Logistic regression models were fitted with the simulated development cohorts before (full data analysis) and after missing value generation (complete case analysis), and with the imputed data. Prognostic model performance was measured by the scaled Brier score, c-statistic, calibration intercept and slope, and by the mean absolute prediction error evaluated in validation cohorts without missing values. Performance of full data analysis was considered as ideal.

RESULTS

None of the imputation methods achieved the model's predictive accuracy that would be obtained in case of no missingness. In general, complete case analysis yielded the worst performance, and deviation from ideal performance increased with increasing percentage of missingness and decreasing sample size. Across all scenarios and performance measures, aregImpute and mice, both with 100 imputations, resulted in highest predictive accuracy. Surprisingly, aregImpute outperformed full data analysis in achieving calibration slopes very close to one across all scenarios and outcome models. The increase of mice's performance with 100 compared to five imputations was only marginal. The differences between the imputation methods decreased with increasing sample sizes and decreasing proportion of noncomplete cases.

CONCLUSION

In our simulation study, model calibration was more affected by the choice of the imputation method than model discrimination. While differences in model performance after using imputation methods were generally small, multiple imputation methods as mice and aregImpute that can handle linear or nonlinear associations between predictors and outcome are an attractive and reliable choice in most situations.

A beginner's guide to supervised analysis for mass cytometry data in cancer biology

Mass cytometry enables deep profiling of biological samples at single-cell resolution. This technology is more than relevant in cancer research due to high cellular heterogeneity and complexity. Downstream analysis of high-dimensional datasets increasingly relies on machine learning (ML) to extract clinically relevant information, including supervised algorithms for classification and regression purposes. In cancer research, they are used to develop predictive models that will guide clinical decision making. However, the development of supervised algorithms faces major challenges, such as sufficient validation, before being translated into the clinics. In this work, we provide a framework for the analysis of mass cytometry data with a specific focus on supervised algorithms and practical examples of their applications. We also raise awareness on key issues regarding good practices for researchers curious to implement supervised ML on their mass cytometry data. Finally, we discuss the challenges of supervised ML application to cancer research.

Establishment and validation of apnea risk prediction models in preterm infants: a retrospective case control study

BACKGROUND

Apnea is common in preterm infants and can be accompanied with severe hypoxic damage. Early assessment of apnea risk can impact the prognosis of preterm infants. We constructed a prediction model to assess apnea risk in premature infants for identifying high-risk groups.

METHODS

A total of 162 and 324 preterm infants with and without apnea who were admitted to the neonatal intensive care unit of Xiamen University between January 2018 and December 2021 were selected as the case and control groups, respectively. Demographic characteristics, laboratory indicators, complications of the patients, pregnancy-related factors, and perinatal risk factors of the mother were collected retrospectively. The participants were randomly divided into modeling (n = 388) and validation (n = 98) sets in an 8:2 ratio. Least Absolute Shrinkage and Selection Operator (LASSO) and multivariate logistic regression analyses were used to independently filter variables from the modeling set and build a model. A nomogram was used to visualize models. The calibration and clinical utility of the model was evaluated using consistency index, receiver operating characteristic (ROC) curve, calibration curve, and decision curve, and the model was verified using the validation set.

RESULTS

Results of LASSO combined with multivariate logistic regression analysis showed that gestational age at birth, birth length, Apgar score, and neonatal respiratory distress syndrome were predictors of apnea development in preterm infants. The model was presented as a nomogram and the Hosmer-Lemeshow goodness of fit test showed a good model fit (χ2=5.192, df=8, P=0.737), with Nagelkerke R2 of 0.410 and C-index of 0.831. The area under the ROC curve and 95% CI were 0.831 (0.787-0.874) and 0.829 (0.722-0.935), respectively. Delong's test comparing the AUC of the two data sets showed no significant difference (P=0.976). The calibration curve showed good agreement between the predicted and actual observations. The decision curve results showed that the threshold probability range of the model was 0.07-1.00, the net benefit was high, and the constructed clinical prediction model had clinical utility.

CONCLUSIONS

Our risk prediction model based on gestational age, birth length, Apgar score 10 min post-birth, and neonatal respiratory distress syndrome was validated in many aspects and had good predictive efficacy and clinical utility.

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

BACKGROUND

Risk prediction models are routinely used to assist in clinical decision making. A small sample size for model development can compromise model performance when the model is applied to new patients. For binary outcomes, the calibration slope (CS) and the mean absolute prediction error (MAPE) are two key measures on which sample size calculations for the development of risk models have been based. CS quantifies the degree of model overfitting while MAPE assesses the accuracy of individual predictions.

METHODS

Recently, two formulae were proposed to calculate the sample size required, given anticipated features of the development data such as the outcome prevalence and c-statistic, to ensure that the expectation of the CS and MAPE (over repeated samples) in models fitted using MLE will meet prespecified target values. In this article, we use a simulation study to evaluate the performance of these formulae.

RESULTS

We found that both formulae work reasonably well when the anticipated model strength is not too high (c-statistic < 0.8), regardless of the outcome prevalence. However, for higher model strengths the CS formula underestimates the sample size substantially. For example, for c-statistic = 0.85 and 0.9, the sample size needed to be increased by at least 50% and 100%, respectively, to meet the target expected CS. On the other hand, the MAPE formula tends to overestimate the sample size for high model strengths. These conclusions were more pronounced for higher prevalence than for lower prevalence. Similar results were drawn when the outcome was time to event with censoring. Given these findings, we propose a simulation-based approach, implemented in the new R package 'samplesizedev', to correctly estimate the sample size even for high model strengths. The software can also calculate the variability in CS and MAPE, thus allowing for assessment of model stability.

CONCLUSIONS

The calibration and MAPE formulae suggest sample sizes that are generally appropriate for use when the model strength is not too high. However, they tend to be biased for higher model strengths, which are not uncommon in clinical risk prediction studies. On those occasions, our proposed adjustments to the sample size calculations will be relevant.

CT-based pancreatic radiomics predicts secondary loss of response to infliximab in biologically naïve patients with Crohn's disease

OBJECTIVES

Predicting secondary loss of response (SLR) to infliximab (IFX) is paramount for tailoring personalized management regimens. Concurrent pancreatic manifestations in patients with Crohn's disease (CD) may correlate with SLR to anti-tumor necrosis factor treatment. This work aimed to evaluate the potential of pancreatic radiomics to predict SLR to IFX in biologic-naive individuals with CD.

METHODS

Three models were developed by logistic regression analyses to identify high-risk subgroup prone to SLR. The area under the curve (AUC), calibration curve, decision curve analysis (DCA), and integrated discrimination improvement (IDI) were applied for the verification of model performance. A quantitative nomogram was proposed based on the optimal prediction model, and its reliability was substantiated by 10-fold cross-validation.

RESULTS

In total, 184 CD patients were enrolled in the period January 2016 to February 2022. The clinical model incorporated age of onset, disease duration, disease location, and disease behavior, whereas the radiomics model consisted of five texture features. These clinical parameters and the radiomics score calculated by selected texture features were applied to build the combined model. Compared to other two models, combined model achieved favorable, significantly improved discrimination power (AUCcombined vs clinical 0.851 vs 0.694, p = 0.02; AUCcombined vs radiomics 0.851 vs 0.740, p = 0.04) and superior clinical usefulness, which was further converted into reliable nomogram with an accuracy of 0.860 and AUC of 0.872.

CONCLUSIONS

The first proposed pancreatic-related nomogram represents a credible, noninvasive predictive instrument to assist clinicians in accurately identifying SLR and non-SLR in CD patients.

CRITICAL RELEVANCE STATEMENT

This study first built a visual nomogram incorporating pancreatic texture features and clinical factors, which could facilitate clinicians to make personalized treatment decisions and optimize cost-effectiveness ratio for patients with CD.

KEY POINTS

• The first proposed pancreatic-related model predicts secondary loss of response for infliximab in Crohn's disease. • The model achieved satisfactory predictive accuracy, calibration ability, and clinical value. • The model-based nomogram has the potential to identify long-term failure in advance and tailor personalized management regimens.

Analysis of Risk Factors and Construction of a Predictive Model for Readmission in Patients with Coronary Slow Flow Phenomenon

Background

Coronary slow flow phenomenon (CSFP) is a phenomenon in which distal vascular perfusion is delayed on angiography, but coronary arteries are not significantly narrowed and there is no other organic cardiac disease. Patients with CSFP may be repeatedly readmitted to the hospital because of chest pain or other symptoms of precordial discomfort, and there is a risk of adverse events. In order to investigate the risk factors affecting the readmission of CSFP patients, a prediction model was constructed with the aim of identifying patients at risk of readmission at an early stage and providing a reference for further clinical intervention.

Methods

In this study, we collected clinical data from 397 CSFP patients between June 2021 and January 2023 in Xinjiang Medical University Hospital. Telephone follow-up clarified whether the patients were readmitted to the hospital. A predictive model for readmission of CSFP patients was constructed using multifactorial logistic regression. Nomogram was used to visualize the model and bootstrap was used to internally validate the model. ROC, DCA and Calibration curve were plotted to evaluate the calibration and discriminative ability of the column line graphs, respectively. Calibration and resolution of the column line graphs, respectively.

Results

A total of 34 of 397 CSFP patients experienced readmission. Smoking history, creatine kinase isoenzyme-MB, total cholesterol, and left ventricular ejection fraction were the predictors of readmission in patients with CSFP. The area under the curve of the Nomogram model was 0.87, which indicated that the model had good predictive ability and differentiation, and the DCA and Calibration curves also indicated that the model had good consistency and was clinically useful.

Conclusion

A readmission prediction model for patients with CSFP may facilitate early identification of patients at potential risk for readmission and timely interventional therapy to improve patient prognosis.