Impact of missing data in evaluating artificial neural networks trained on complete data

Predicting 90-day prognosis for patients with stroke: a machine learning approach

BACKGROUND

Stroke is a significant global health burden and ranks as the second leading cause of death worldwide.

OBJECTIVE

This study aims to develop and evaluate a machine learning-based predictive tool for forecasting the 90-day prognosis of stroke patients after discharge as measured by the modified Rankin Score.

METHODS

The study utilized data from a large national multiethnic stroke registry comprising 15,859 adult patients diagnosed with ischemic or hemorrhagic stroke. Of these, 7,452 patients satisfied the study's inclusion criteria. Feature selection was performed using the correlation and permutation importance methods. Six classifiers, including Random Forest (RF), Classification and Regression Tree, Linear Discriminant Analysis, Support Vector Machine, and k-Nearest Neighbors, were employed for prediction.

RESULTS

The RF model demonstrated superior performance, achieving the highest accuracy (0.823) and excellent discrimination power (AUC 0.893). Notably, stroke type, hospital acquired infections, admission location, and hospital length of stay emerged as the top-ranked predictors.

CONCLUSION

The RF model shows promise in predicting stroke prognosis, enabling personalized care plans and enhanced preventive measures for stroke patients. Prospective validation is essential to assess its real-world clinical performance and ensure successful implementation across diverse healthcare settings.

Predicting Major Adverse Cardiovascular Events Following Carotid Endarterectomy Using Machine Learning

Background Carotid endarterectomy (CEA) is a major vascular operation for stroke prevention that carries significant perioperative risks; however, outcome prediction tools remain limited. The authors developed machine learning algorithms to predict outcomes following CEA. Methods and Results The National Surgical Quality Improvement Program targeted vascular database was used to identify patients who underwent CEA between 2011 and 2021. Input features included 36 preoperative demographic/clinical variables. The primary outcome was 30-day major adverse cardiovascular events (composite of stroke, myocardial infarction, or death). The data were split into training (70%) and test (30%) sets. Using 10-fold cross-validation, 6 machine learning models were trained using preoperative features. The primary metric for evaluating model performance was area under the receiver operating characteristic curve. Model robustness was evaluated with calibration plot and Brier score. Overall, 38 853 patients underwent CEA during the study period. Thirty-day major adverse cardiovascular events occurred in 1683 (4.3%) patients. The best performing prediction model was XGBoost, achieving an area under the receiver operating characteristic curve of 0.91 (95% CI, 0.90-0.92). In comparison, logistic regression had an area under the receiver operating characteristic curve of 0.62 (95% CI, 0.60-0.64), and existing tools in the literature demonstrate area under the receiver operating characteristic curve values ranging from 0.58 to 0.74. The calibration plot showed good agreement between predicted and observed event probabilities with a Brier score of 0.02. The strongest predictive feature in our algorithm was carotid symptom status. Conclusions The machine learning models accurately predicted 30-day outcomes following CEA using preoperative data and performed better than existing tools. They have potential for important utility in guiding risk-mitigation strategies to improve outcomes for patients being considered for CEA.

Evaluating the Impact of Health Care Data Completeness for Deep Generative Models

BACKGROUND

 Deep generative models (DGMs) present a promising avenue for generating realistic, synthetic data to augment existing health care datasets. However, exactly how the completeness of the original dataset affects the quality of the generated synthetic data is unclear.

OBJECTIVES

 In this paper, we investigate the effect of data completeness on samples generated by the most common DGM paradigms.

METHODS

 We create both cross-sectional and panel datasets with varying missingness and subset rates and train generative adversarial networks, variational autoencoders, and autoregressive models (Transformers) on these datasets. We then compare the distributions of generated data with original training data to measure similarity.

RESULTS

 We find that increased incompleteness is directly correlated with increased dissimilarity between original and generated samples produced through DGMs.

CONCLUSIONS

 Care must be taken when using DGMs to generate synthetic data as data completeness issues can affect the quality of generated data in both panel and cross-sectional datasets.

Artificial Neural Network vs. Pharmacometric Model for Population Prediction of Plasma Concentration in Real-World Data: A Case Study on Valproic Acid

We compared the predictive performance of an artificial neural network to traditional pharmacometric modeling for population prediction of plasma concentrations of valproate in real-world data. We included individuals aged 65 years or older with epilepsy who redeemed their first prescription of valproate after the diagnosis of epilepsy and had at least one valproate plasma concentration measured. A long short-term memory neural network (LSTM) was developed using the training data set to fit the LSTM and the test data set to validate the model. Predictions from the LSTM were compared with those obtained from the population predictions from a pharmacometric model by Birnbaum et al. which had the best predictive performance for population predictions of valproate concentrations in Danish databases. We used the cutoff of ± 20 mg/L of prediction error to define good predictions. A total of 1,252 individuals were included in the study. The LSTM fitted using the training data set had poor predictive performance in the test data set, but better than that of the pharmacometric model. The proportion of individuals with at least one predicted concentration within ± 20 mg/L of observed concentration was largest in case of the LSTM (64.4%, 95% confidence interval (CI): 58.4-70.2%) compared with the pharmacometric model by Birnbaum et al. (49.8%, 95% CI: 47.0-52.6%). LSTM shows better predictive performance to predict valproate plasma concentrations compared with a traditional pharmacometric model in the investigated setting with real-world data in older patients with epilepsy where information on exact timepoints for both dosing and plasma concentration measurement are missing.

Integrity of clinical information in radiology reports documenting pulmonary nodules

OBJECTIVE

Quantify the integrity, measured as completeness and concordance with a thoracic radiologist, of documenting pulmonary nodule characteristics in CT reports and assess impact on making follow-up recommendations.

MATERIALS AND METHODS

This Institutional Review Board-approved, retrospective cohort study was performed at an academic medical center. Natural language processing was performed on radiology reports of CT scans of chest, abdomen, or spine completed in 2016 to assess presence of pulmonary nodules, excluding patients with lung cancer, of which 300 reports were randomly sampled to form the study cohort. Documentation of nodule characteristics were manually extracted from reports by 2 authors with 20% overlap. CT images corresponding to 60 randomly selected reports were further reviewed by a thoracic radiologist to record nodule characteristics. Documentation completeness for all characteristics were reported in percentage and compared using χ2 analysis. Concordance with a thoracic radiologist was reported as percentage agreement; impact on making follow-up recommendations was assessed using kappa.

RESULTS

Documentation completeness for pulmonary nodule characteristics differed across variables (range = 2%-90%, P < .001). Concordance with a thoracic radiologist was 75% for documenting nodule laterality and 29% for size. Follow-up recommendations were in agreement in 67% and 49% of reports when there was lack of completeness and concordance in documenting nodule size, respectively.

DISCUSSION

Essential pulmonary nodule characteristics were under-reported, potentially impacting recommendations for pulmonary nodule follow-up.

CONCLUSION

Lack of documentation of pulmonary nodule characteristics in radiology reports is common, with potential for compromising patient care and clinical decision support tools.

Development and testing of an artificial intelligence tool for predicting end-stage kidney disease in patients with immunoglobulin A nephropathy

We have developed an artificial neural network prediction model for end-stage kidney disease (ESKD) in patients with primary immunoglobulin A nephropathy (IgAN) using a retrospective cohort of 948 patients with IgAN. Our tool is based on a two-step procedure of a classifier model that predicts ESKD, and a regression model that predicts development of ESKD over time. The classifier model showed a performance value of 0.82 (area under the receiver operating characteristic curve) in patients with a follow-up of five years, which improved to 0.89 at the ten-year follow-up. Both models had a higher recall rate, which indicated the practicality of the tool. The regression model showed a mean absolute error of 1.78 years and a root mean square error of 2.15 years. Testing in an independent cohort of 167patients with IgAN found successful results for 91% of the patients. Comparison of our system with other mathematical models showed the highest discriminant Harrell C index at five- and ten-years follow-up (81% and 86%, respectively), paralleling the lowest Akaike information criterion values (355.01 and 269.56, respectively). Moreover, our system was the best calibrated model indicating that the predicted and observed outcome probabilities did not significantly differ. Finally, the dynamic discrimination indexes of our artificial neural network, expressed as the weighted average of time-dependent areas under the curve calculated at one and two years, were 0.80 and 0.79, respectively. Similar results were observed over a 25-year follow-up period. Thus, our tool identified individuals who were at a high risk of developing ESKD due to IgAN and predicted the time-to-event endpoint. Accurate prediction is an important step toward introduction of a therapeutic strategy for improving clinical outcomes.

The role of basal 18F-FDG PET/CT maximum standard uptake value and maximum standard uptake change in predicting pathological response in breast cancer patients receiving neoadjuvant chemotherapy

OBJECTIVE

The aim of this study was to determine the role of 18F-FDG PET/CT in predicting pathological response among patients diagnosed with local or locally advanced breast cancer and receiving neoadjuvant chemotherapy (NAC).

METHODS

Basal SUVmax value were analyzed in 212 patients and 142 of these patients had posttreatment SUVmax value. Overall pathological complete response (pCRC) was defined as no evidence of residual invasive cancer in breast (pCRB) and axilla (pCRA). Basal SUVmax value of the breast (SUVmaxBI) and axilla (SUVmaxAI) and change in SUVmax of the breast (ΔSUVmaxB) and axilla (ΔSUVmaxA) were measured. The optimal cutoff value of SUVmax and ΔSUVmax were determined by receiver operating characteristic curve analysis.

RESULTS

The number of patients with pCRB was 85 (40.1%), pCRA was 76 (42.5%) and pCRC was 70 (33%). In the artificial neural network-based analysis the ΔSUVmaxB (100%) was the most important variable for predicting pCRB. ΔSUVmaxA (100%) was the most important variable in estimation of pCRA. When pCRC was evaluated, the highest relation was found with ΔSUVmaxB. When the ΔSUVmaxB cutoff value for pCRB and pCRC accepted as ≤-87.9%, its sensitivity was 82.3 and 82.4%, and specificity was 72.5% and 65.9%, respectively (P < 0.001 and P < 0.001, respectively). When the ΔSUVmaxA cutoff value for pCRA and pCRC accepted as ≤-86.6%, its sensitivity was 94.3% and 97.6%, and specificity was 31.3% and 28.2%, respectively (P = 0.017 and P = 0.024, respectively).

CONCLUSION

Albeit varies according to the molecular subtypes of the breast cancer during NAC, ΔSUVmax value seems to be the most strong factor associated with pCR.

Methods Used in Computer-Aided Diagnosis for Breast Cancer Detection Using Mammograms: A Review

According to the American Cancer Society's forecasts for 2019, there will be about 268,600 new cases in the United States with invasive breast cancer in women, about 62,930 new noninvasive cases, and about 41,760 death cases from breast cancer. As a result, there is a high demand for breast imaging specialists as indicated in a recent report for the Institute of Medicine and National Research Council. One way to meet this demand is through developing Computer-Aided Diagnosis (CAD) systems for breast cancer detection and diagnosis using mammograms. This study aims to review recent advancements and developments in CAD systems for breast cancer detection and diagnosis using mammograms and to give an overview of the methods used in its steps starting from preprocessing and enhancement step and ending in classification step. The current level of performance for the CAD systems is encouraging but not enough to make CAD systems standalone detection and diagnose clinical systems. Unless the performance of CAD systems enhanced dramatically from its current level by enhancing the existing methods, exploiting new promising methods in pattern recognition like data augmentation in deep learning and exploiting the advances in computational power of computers, CAD systems will continue to be a second opinion clinical procedure.

Automatic Estimation of Volumetric Breast Density Using Artificial Neural Network-Based Calibration of Full-Field Digital Mammography: Feasibility on Japanese Women With and Without Breast Cancer

Breast cancer is the most common invasive cancer among women and its incidence is increasing. Risk assessment is valuable and recent methods are incorporating novel biomarkers such as mammographic density. Artificial neural networks (ANN) are adaptive algorithms capable of performing pattern-to-pattern learning and are well suited for medical applications. They are potentially useful for calibrating full-field digital mammography (FFDM) for quantitative analysis. This study uses ANN modeling to estimate volumetric breast density (VBD) from FFDM on Japanese women with and without breast cancer. ANN calibration of VBD was performed using phantom data for one FFDM system. Mammograms of 46 Japanese women diagnosed with invasive carcinoma and 53 with negative findings were analyzed using ANN models learned. ANN-estimated VBD was validated against phantom data, compared intra-patient, with qualitative composition scoring, with MRI VBD, and inter-patient with classical risk factors of breast cancer as well as cancer status. Phantom validations reached an R ² of 0.993. Intra-patient validations ranged from R ² of 0.789 with VBD to 0.908 with breast volume. ANN VBD agreed well with BI-RADS scoring and MRI VBD with R ² ranging from 0.665 with VBD to 0.852 with breast volume. VBD was significantly higher in women with cancer. Associations with age, BMI, menopause, and cancer status previously reported were also confirmed. ANN modeling appears to produce reasonable measures of mammographic density validated with phantoms, with existing measures of breast density, and with classical biomarkers of breast cancer. FFDM VBD is significantly higher in Japanese women with cancer.

Missing data imputation on the 5-year survival prediction of breast cancer patients with unknown discrete values

Breast cancer is the most frequently diagnosed cancer in women. Using historical patient information stored in clinical datasets, data mining and machine learning approaches can be applied to predict the survival of breast cancer patients. A common drawback is the absence of information, i.e., missing data, in certain clinical trials. However, most standard prediction methods are not able to handle incomplete samples and, then, missing data imputation is a widely applied approach for solving this inconvenience. Therefore, and taking into account the characteristics of each breast cancer dataset, it is required to perform a detailed analysis to determine the most appropriate imputation and prediction methods in each clinical environment. This research work analyzes a real breast cancer dataset from Institute Portuguese of Oncology of Porto with a high percentage of unknown categorical information (most clinical data of the patients are incomplete), which is a challenge in terms of complexity. Four scenarios are evaluated: (I) 5-year survival prediction without imputation and 5-year survival prediction from cleaned dataset with (II) Mode imputation, (III) Expectation-Maximization imputation and (IV) K-Nearest Neighbors imputation. Prediction models for breast cancer survivability are constructed using four different methods: K-Nearest Neighbors, Classification Trees, Logistic Regression and Support Vector Machines. Experiments are performed in a nested ten-fold cross-validation procedure and, according to the obtained results, the best results are provided by the K-Nearest Neighbors algorithm: more than 81% of accuracy and more than 0.78 of area under the Receiver Operator Characteristic curve, which constitutes very good results in this complex scenario.

Artificial neural networks in mammography interpretation and diagnostic decision making

Screening mammography is the most effective means for early detection of breast cancer. Although general rules for discriminating malignant and benign lesions exist, radiologists are unable to perfectly detect and classify all lesions as malignant and benign, for many reasons which include, but are not limited to, overlap of features that distinguish malignancy, difficulty in estimating disease risk, and variability in recommended management. When predictive variables are numerous and interact, ad hoc decision making strategies based on experience and memory may lead to systematic errors and variability in practice. The integration of computer models to help radiologists increase the accuracy of mammography examinations in diagnostic decision making has gained increasing attention in the last two decades. In this study, we provide an overview of one of the most commonly used models, artificial neural networks (ANNs), in mammography interpretation and diagnostic decision making and discuss important features in mammography interpretation. We conclude by discussing several common limitations of existing research on ANN-based detection and diagnostic models and provide possible future research directions.

Treating metastatic disease: Which survival model is best suited for the clinic?

BACKGROUND

To avoid complications associated with under- or overtreatment of patients with skeletal metastases, doctors need accurate survival estimates. Unfortunately, prognostic models for patients with skeletal metastases of the extremities are lacking, and physician-based estimates are generally inaccurate.

QUESTIONS/PURPOSES

We developed three types of prognostic models and compared them using calibration plots, receiver operating characteristic (ROC) curves, and decision curve analysis to determine which one is best suited for clinical use.

METHODS

A training set consisted of 189 patients who underwent surgery for skeletal metastases. We created models designed to predict 3- and 12-month survival using three methods: an Artificial Neural Network (ANN), a Bayesian Belief Network (BBN), and logistic regression. We then performed crossvalidation and compared the models in three ways: calibration plots plotting predicted against actual risk; area under the ROC curve (AUC) to discriminate the probability that a patient who died has a higher predicted probability of death compared to a patient who did not die; and decision curve analysis to quantify the clinical consequences of over- or undertreatment.

RESULTS

All models appeared to be well calibrated, with the exception of the BBN, which underestimated 3-month survival at lower probability estimates. The ANN models had the highest discrimination, with an AUC of 0.89 and 0.93, respectively, for the 3- and 12-month models. Decision analysis revealed all models could be used clinically, but the ANN models consistently resulted in the highest net benefit, outperforming the BBN and logistic regression models.

CONCLUSIONS

Our observations suggest use of the ANN model to aid decisions about surgery would lead to better patient outcomes than other alternative approaches to decision making.

LEVEL OF EVIDENCE

Level II, prognostic study. See Instructions for Authors for a complete description of levels of evidence.

Selection-Fusion Approach for Classification of Datasets with Missing Values

This paper proposes a new approach based on missing value pattern discovery for classifying incomplete data. This approach is particularly designed for classification of datasets with a small number of samples and a high percentage of missing values where available missing value treatment approaches do not usually work well. Based on the pattern of the missing values, the proposed approach finds subsets of samples for which most of the features are available and trains a classifier for each subset. Then, it combines the outputs of the classifiers. Subset selection is translated into a clustering problem, allowing derivation of a mathematical framework for it. A trade off is established between the computational complexity (number of subsets) and the accuracy of the overall classifier. To deal with this trade off, a numerical criterion is proposed for the prediction of the overall performance. The proposed method is applied to seven datasets from the popular University of California, Irvine data mining archive and an epilepsy dataset from Henry Ford Hospital, Detroit, Michigan (total of eight datasets). Experimental results show that classification accuracy of the proposed method is superior to those of the widely used multiple imputations method and four other methods. They also show that the level of superiority depends on the pattern and percentage of missing values.

The prediction of breast cancer biopsy outcomes using two CAD approaches that both emphasize an intelligible decision process

Mammography is the most effective method for breast cancer screening available today. However, the low positive predictive value of breast biopsy resulting from mammogram interpretation leads to approximately 70% unnecessary biopsies with benign outcomes. To reduce the high number of unnecessary breast biopsies, several computer-aided diagnosis (CAD) systems have been proposed in the last several years. These systems help physicians in their decision to perform a breast biopsy on a suspicious lesion seen in a mammogram or to perform a short term follow-up examination instead. We present two novel CAD approaches that both emphasize an intelligible decision process to predict breast biopsy outcomes from BI-RADS findings. An intelligible reasoning process is an important requirement for the acceptance of CAD systems by physicians. The first approach induces a global model based on decison-tree learning. The second approach is based on case-based reasoning and applies an entropic similarity measure. We have evaluated the performance of both CAD approaches on two large publicly available mammography reference databases using receiver operating characteristic (ROC) analysis, bootstrap sampling, and the ANOVA statistical significance test. Both approaches outperform the diagnosis decisions of the physicians. Hence, both systems have the potential to reduce the number of unnecessary breast biopsies in clinical practice. A comparison of the performance of the proposed decision tree and CBR approaches with a state of the art approach based on artificial neural networks (ANN) shows that the CBR approach performs slightly better than the ANN approach, which in turn results in slightly better performance than the decision-tree approach. The differences are statistically significant (p value < 0.001). On 2100 masses extracted from the DDSM database, the CRB approach for example resulted in an area under the ROC curve of A(z) = 0.89 +/- 0.01, the decision-tree approach in A(z) = 0.87 +/- 0.01, and the ANN approach in A(z) = 0.88 +/- 0.01.

Modeling the role of genetic factors in characterizing extra-intestinal manifestations in Crohn's disease patients: does this improve outcome predictions?

OBJECTIVE

To evaluate to what extent an inefficient statistical model affects the study of genetic factors in extra-intestinal manifestations of Crohn's disease (CD) and how clinical predictions can be improved using more adequate techniques.

MATERIALS

Extra-intestinal manifestations were studied in 152 CD patients. Three sets of variables were considered: (1) disease characteristics--presentation, behavior, location; (2) generic risk factors--age, gender, smoke and familiarity; and (3) genetic polymorphisms of the NOD2, CD14, TNF, IL12B, and IL1RN genes, whose involvement in CD is known or suspected.

METHODS

Six statistical classifiers and data mining models were applied: (1) logistic regression as a benchmark; (2) generalized additive model; (3) projection pursuit regression; (4) linear discriminant analysis, (5) quadratic discriminant analysis; (6) artificial neural networks one-layer feed forward. Models were selected using the Akaike Information criterion and their accuracy was compared with several indexes.

RESULTS

Extra-intestinal manifestations occurred in 75 patients. The model with clinical variables only selected familiarity, gender, presentation, and behavior as significantly associated with extra-intestinal manifestations, whereas when the genetic factors were also included familiarity was no longer significant, being replaced by the NOD2, TNF, and IL12B single nucleotide polymorphisms. The projection pursuit regression performed best in predicting individual outcomes (Kappa statistics 0.078 [SE 0.09] without and 0.108 [SE 0.075] with genetic information). One-layer artificial neural networks did not show any particular improvement in terms of model accuracy over nonlinear techniques.

CONCLUSIONS

The correct identification of factors associated with extra-intestinal symptoms in CD, in particular the genetic ones, is highly dependent on the model chosen for the analysis. By using the most sophisticated statistical models, the accuracy of prediction can be strengthened by 10-64%, compared with linear regression.

Would artificial neural networks implemented in clinical wards help nephrologists in predicting epoetin responsiveness?

Background

Due to its strong intra- and inter-individual variability, predicting the ideal erythropoietin dose is a difficult task. The aim of this study was to re-evaluate the impact of the main parameters known to influence the responsiveness to epoetin beta and to test the performance of artificial neural networks (ANNs) in predicting the dose required to reach the haemoglobin target and the monthly dose adjustments.

Methods

We did a secondary analysis of the survey on Anaemia Management in dialysis patients in Switzerland; a prospective, non-randomized observational study, enrolling 340 patients of 26 centres and in order to have additional information about erythropoietin responsiveness, we included a further 92 patients from the Renal Services of the Ente Ospedaliero Cantonale, Bellinzona, Switzerland. The performance of ANNs in predicting the epoetin dose was compared with that of linear regressions and of nephrologists in charge of the patients.

Results

For a specificity of 50%, the sensitivity of ANNs compared with linear regressions in predicting the erythropoietin dose to reach the haemoglobin target was 78 vs. 44% (P < 0.001). The ANN built to predict the monthly adaptations in erythropoietin dose, compared with the nephrologists' opinion, allowed to detect 48 vs. 25% (P < 0.05) of the patients treated with an insufficient dose with a specificity of 92 vs. 83% (P < 0.05).

Conclusion

In predicting the erythropoietin dose required for an individual patient and the monthly dose adjustments ANNs are superior to nephrologists' opinion. Thus, ANN may be a useful and promising tool that could be implemented in clinical wards to help nephrologists in prescribing erythropoietin.