A systematic review of machine learning models for predicting outcomes of stroke with structured data

Developing machine learning models to predict multi-class functional outcomes and death three months after stroke in Sweden

Globally, stroke is the third-leading cause of mortality and disability combined, and one of the costliest diseases in society. More accurate predictions of stroke outcomes can guide healthcare organizations in allocating appropriate resources to improve care and reduce both the economic and social burden of the disease. We aim to develop and evaluate the performance and explainability of three supervised machine learning models and the traditional multinomial logistic regression (mLR) in predicting functional dependence and death three months after stroke, using routinely-collected data. This prognostic study included adult patients, registered in the Swedish Stroke Registry (Riksstroke) from 2015 to 2020. Riksstroke contains information on stroke care and outcomes among patients treated in hospitals in Sweden. Prognostic factors (features) included demographic characteristics, pre-stroke functional status, cardiovascular risk factors, medications, acute care, stroke type, and severity. The outcome was measured using the modified Rankin Scale at three months after stroke (a scale of 0-2 indicates independent, 3-5 dependent, and 6 dead). Outcome prediction models included support vector machines, artificial neural networks (ANN), eXtreme Gradient Boosting (XGBoost), and mLR. The models were trained and evaluated on 75% and 25% of the dataset, respectively. Model predictions were explained using SHAP values. The study included 102,135 patients (85.8% ischemic stroke, 53.3% male, mean age 75.8 years, and median NIHSS of 3). All models demonstrated similar overall accuracy (69%-70%). The ANN and XGBoost models performed significantly better than the mLR in classifying dependence with F1-scores of 0.603 (95% CI; 0.594-0.611) and 0.577 (95% CI; 0.568-0.586), versus 0.544 (95% CI; 0.545-0.563) for the mLR model. The factors that contributed most to the predictions were expectedly similar in the models, based on clinical knowledge. Our ANN and XGBoost models showed a modest improvement in prediction performance and explainability compared to mLR using routinely-collected data. Their improved ability to predict functional dependence may be of particular importance for the planning and organization of acute stroke care and rehabilitation.

Predicting ischemic stroke patients' prognosis changes using machine learning in a nationwide stroke registry

Accurately predicting the prognosis of ischemic stroke patients after discharge is crucial for physicians to plan for long-term health care. Although previous studies have demonstrated that machine learning (ML) shows reasonably accurate stroke outcome predictions with limited datasets, to identify specific clinical features associated with prognosis changes after stroke that could aid physicians and patients in devising improved recovery care plans have been challenging. This study aimed to overcome these gaps by utilizing a large national stroke registry database to assess various prediction models that estimate how patients' prognosis changes over time with associated clinical factors. To properly evaluate the best predictive approaches currently available and avoid prejudice, this study employed three different prognosis prediction models including a statistical logistic regression model, commonly used clinical-based scores, and a latest high-performance ML-based XGBoost model. The study revealed that the XGBoost model outperformed other two traditional models, achieving an AUROC of 0.929 in predicting the prognosis changes of stroke patients followed for 3 months. In addition, the XGBoost model maintained remarkably high precision even when using only selected 20 most relevant clinical features compared to full clinical datasets used in the study. These selected features closely correlated with significant changes in clinical outcomes for stroke patients and showed to be effective for predicting prognosis changes after discharge, allowing physicians to make optimal decisions regarding their patients' recovery.

Machine learning algorithms for the prognostication of abdominal aortic aneurysm progression: a systematic review

INTRODUCTION

Abdominal aortic aneurysm (AAA), often characterized by an abdominal aortic diameter over 3.0 cm, is managed through screening, surveillance, and surgical intervention. AAA growth can be heterogeneous and rupture carries a high mortality rate, with size and certain risk factors influencing rupture risk. Research is ongoing to accurately predict individual AAA growth rates for personalized management. Machine learning, a subset of artificial intelligence, has shown promise in various medical fields, including endoleak detection post-EVAR. However, its application for predicting AAA growth remains insufficiently explored, thus necessitating further investigation. Subsequently, this paper aims to summarize the current status of machine learning in predicting AAA growth.

EVIDENCE ACQUISITION

A systematic database search of Embase, MEDLINE, Cochrane, PubMed and Google Scholar from inception till December 2022 was conducted of original articles that discussed the use of machine learning in predicting AAA growth using the aforementioned databases.

EVIDENCE SYNTHESIS

Overall, 2742 articles were extracted, of which seven retrospective studies involving 410 patients were included using a predetermined criteria. Six out of seven studies applied a supervised learning approach for their machine learning (ML) models, with considerable diversity observed within specific ML models. The majority of the studies concluded that machine learning models perform better in predicting AAA growth in comparison to reference models. All studies focused on predicting AAA growth over specified durations. Maximal luminal diameter was the most frequently used indicator, with alternative predictors being AAA volume, ILT (intraluminal thrombus) and flow-medicated diameter (FMD).

CONCLUSIONS

The nascent field of applying machine learning (ML) for Abdominal Aortic Aneurysm (AAA) expansion prediction exhibits potential to enhance predictive accuracy across diverse parameters. Future studies must emphasize evidencing clinical utility in a healthcare system context, thereby ensuring patient outcome improvement. This will necessitate addressing key ethical implications in establishing prospective studies related to this topic and collaboration among pivotal stakeholders within the AI field.

Middle meningeal artery embolization for chronic subdural hematoma: A single-center experience and predictive modeling of outcomes

BACKGROUND

Remarkable interest is rising around middle meningeal artery embolization (MMAE) as an emerging alternative therapy for chronic subdural hematoma (cSDH). The study aims to highlight a large center experience and the variables associated with treatment failure and build experimental machine learning (ML) models for outcome prediction.

MATERIAL AND METHODS

A 2-year experience in MMAE for managing patients with chronic subdural hematoma was analyzed. Descriptive statistical analysis was conducted using imaging and clinical features of the patients and cSDH, which were subsequently used to build predictive models for the procedure outcome. The modeling evaluation metrics were the area under the ROC curve and F1-score.

RESULTS

A total of 100 cSDH of 76 patients who underwent MMAE were included with an average follow-up of 6 months. The intervention had a per procedure success rate of 92%. Thrombocytopenia had a highly significant association with treatment failure. Two patients suffered a complication related to the procedure. The best performing machine learning models in predicting MMAE failure achieved an ROC-AUC of 70%, and an F1-score of 67%, including all patients with or without surgical intervention prior to embolization, and an ROC-AUC of 82% and an F1-score of 69% when only patients who underwent upfront MMAE were included.

CONCLUSION

MMAE is a safe and minimally invasive procedure with great potential in transforming the management of cSDH and reducing the risk of surgical complications in selected patients. An ML approach with larger sample size might help better predict outcomes and highlight important predictors following MMAE in patients with cSDH.

Explainable multimodal prediction of treatment-resistance in patients with depression leveraging brain morphometry and natural language processing

Although 20 % of patients with depression receiving treatment do not achieve remission, predicting treatment-resistant depression (TRD) remains challenging. In this study, we aimed to develop an explainable multimodal prediction model for TRD using structured electronic medical record data, brain morphometry, and natural language processing. In total, 247 patients with a new depressive episode were included. TRD-predictive models were developed based on the combination of following parameters: selected tabular dataset features, independent components-map weightings from brain T1-weighted magnetic resonance imaging (MRI), and topic probabilities from clinical notes. All models applied the extreme gradient boosting (XGBoost) algorithm via five-fold cross-validation. The model using all data sources showed the highest area under the receiver operating characteristic of 0.794, followed by models that used combined brain MRI and structured data, brain MRI and clinical notes, clinical notes and structured data, brain MRI only, structured data only, and clinical notes only (0.770, 0.762, 0.728, 0.703, 0.684, and 0.569, respectively). Classifications of TRD were driven by several predictors, such as previous exposure to antidepressants and antihypertensive medications, sensorimotor network, default mode network, and somatic symptoms. Our findings suggest that a combination of clinical data with neuroimaging and natural language processing variables improves the prediction of TRD.

Development and internal validation of machine learning-based models and external validation of existing risk scores for outcome prediction in patients with ischaemic stroke

Aims

We developed new machine learning (ML) models and externally validated existing statistical models [ischaemic stroke predictive risk score (iScore) and totalled health risks in vascular events (THRIVE) scores] for predicting the composite of recurrent stroke or all-cause mortality at 90 days and at 3 years after hospitalization for first acute ischaemic stroke (AIS).

Methods and results

In adults hospitalized with AIS from January 2005 to November 2016, with follow-up until November 2019, we developed three ML models [random forest (RF), support vector machine (SVM), and extreme gradient boosting (XGBOOST)] and externally validated the iScore and THRIVE scores for predicting the composite outcomes after AIS hospitalization, using data from 721 patients and 90 potential predictor variables. At 90 days and 3 years, 11 and 34% of patients, respectively, reached the composite outcome. For the 90-day prediction, the area under the receiver operating characteristic curve (AUC) was 0.779 for RF, 0.771 for SVM, 0.772 for XGBOOST, 0.720 for iScore, and 0.664 for THRIVE. For 3-year prediction, the AUC was 0.743 for RF, 0.777 for SVM, 0.773 for XGBOOST, 0.710 for iScore, and 0.675 for THRIVE.

Conclusion

The study provided three ML-based predictive models that achieved good discrimination and clinical usefulness in outcome prediction after AIS and broadened the application of the iScore and THRIVE scoring system for long-term outcome prediction. Our findings warrant comparative analyses of ML and existing statistical method-based risk prediction tools for outcome prediction after AIS in new data sets.

Systematic reviews of machine learning in healthcare: a literature review

INTRODUCTION

The increasing availability of data and computing power has made machine learning (ML) a viable approach to faster, more efficient healthcare delivery.

METHODS

A systematic literature review (SLR) of published SLRs evaluating ML applications in healthcare settings published between1 January 2010 and 27 March 2023 was conducted.

RESULTS

In total 220 SLRs covering 10,462 ML algorithms were reviewed. The main application of AI in medicine related to the clinical prediction and disease prognosis in oncology and neurology with the use of imaging data. Accuracy, specificity, and sensitivity were provided in 56%, 28%, and 25% SLRs respectively. Internal and external validation was reported in 53% and less than 1% of the cases respectively. The most common modeling approach was neural networks (2,454 ML algorithms), followed by support vector machine and random forest/decision trees (1,578 and 1,522 ML algorithms, respectively).

EXPERT OPINION

The review indicated considerable reporting gaps in terms of the ML's performance, both internal and external validation. Greater accessibility to healthcare data for developers can ensure the faster adoption of ML algorithms into clinical practice.

Machine Learning Models for Predicting Stroke Mortality in Malaysia: An Application and Comparative Analysis

Background Stroke is a significant public health concern characterized by increasing mortality and morbidity. Accurate long-term outcome prediction for acute stroke patients, particularly stroke mortality, is vital for clinical decision-making and prognostic management. This study aimed to develop and compare various prognostic models for stroke mortality prediction. Methods In a retrospective cohort study from January 2016 to December 2021, we collected data from patients diagnosed with acute stroke from five selected hospitals. Data contained variables on demographics, comorbidities, and interventions retrieved from medical records. The cohort comprised 950 patients with 20 features. Outcomes (censored vs. death) were determined by linking data with the Malaysian National Mortality Registry. We employed three common survival modeling approaches, the Cox proportional hazard regression (Cox), support vector machine (SVM), and random survival forest (RSF), while enhancing the Cox model with Elastic Net (Cox-EN) for feature selection. Models were compared using the concordance index (C-index), time-dependent area under the curve (AUC), and discrimination index (D-index), with calibration assessed by the Brier score. Results The support vector machine (SVM) model excelled among the four, with three-month, one-year, and three-year time-dependent AUC values of 0.842, 0.846, and 0.791; a D-index of 5.31 (95% CI: 3.86, 7.30); and a C-index of 0.803 (95% CI: 0.758, 0.847). All models exhibited robust calibration, with three-month, one-year, and three-year Brier scores ranging from 0.103 to 0.220, all below 0.25. Conclusion The support vector machine (SVM) model demonstrated superior discriminative performance, suggesting its efficacy in developing prognostic models for stroke mortality. This study enhances stroke mortality prediction and supports clinical decision-making, emphasizing the utility of the support vector machine method.

Developing and externally validating a machine learning risk prediction model for 30-day mortality after stroke using national stroke registers in the UK and Sweden

OBJECTIVES

We aimed to develop and externally validate a generalisable risk prediction model for 30-day stroke mortality suitable for supporting quality improvement analytics in stroke care using large nationwide stroke registers in the UK and Sweden.

DESIGN

Registry-based cohort study.

SETTING

Stroke registries including the Sentinel Stroke National Audit Programme (SSNAP) in England, Wales and Northern Ireland (2013-2019) and the national Swedish stroke register (Riksstroke 2015-2020).

PARTICIPANTS AND METHODS

Data from SSNAP were used for developing and temporally validating the model, and data from Riksstroke were used for external validation. Models were developed with the variables available in both registries using logistic regression (LR), LR with elastic net and interaction terms and eXtreme Gradient Boosting (XGBoost). Performances were evaluated with discrimination, calibration and decision curves.

OUTCOME MEASURES

The primary outcome was all-cause 30-day in-hospital mortality after stroke.

RESULTS

In total, 488 497 patients who had a stroke with 12.4% 30-day in-hospital mortality were used for developing and temporally validating the model in the UK. A total of 128 360 patients who had a stroke with 10.8% 30-day in-hospital mortality and 13.1% all mortality were used for external validation in Sweden. In the SSNAP temporal validation set, the final XGBoost model achieved the highest area under the receiver operating characteristic curve (AUC) (0.852 (95% CI 0.848 to 0.855)) and was well calibrated. The performances on the external validation in Riksstroke were as good and achieved AUC at 0.861 (95% CI 0.858 to 0.865) for in-hospital mortality. For Riksstroke, the models slightly overestimated the risk for in-hospital mortality, while they were better calibrated at the risk for all mortality.

CONCLUSION

The risk prediction model was accurate and externally validated using high quality registry data. This is potentially suitable to be deployed as part of quality improvement analytics in stroke care to enable the fair comparison of stroke mortality outcomes across hospitals and health systems across countries.

Machine learning prediction models for clinical management of blood-borne viral infections: a systematic review of current applications and future impact

BACKGROUND

Machine learning (ML) prediction models to support clinical management of blood-borne viral infections are becoming increasingly abundant in medical literature, with a number of competing models being developed for the same outcome or target population. However, evidence on the quality of these ML prediction models are limited.

OBJECTIVE

This study aimed to evaluate the development and quality of reporting of ML prediction models that could facilitate timely clinical management of blood-borne viral infections.

METHODS

We conducted narrative evidence synthesis following the synthesis without meta-analysis guidelines. We searched PubMed and Cochrane Central Register of Controlled Trials for all studies applying ML models for predicting clinical outcomes associated with hepatitis B virus (HBV), human immunodeficiency virus (HIV), or hepatitis C virus (HCV).

RESULTS

We found 33 unique ML prediction models aiming to support clinical decision making. Overall, 12 (36.4%) focused on HBV, 10 (30.3%) on HCV, 10 on HIV (30.3%) and two (6.1%) on co-infection. Among these, six (18.2%) addressed the diagnosis of infection, 16 (48.5%) the prognosis of infection, eight (24.2%) the prediction of treatment response, two (6.1%) progression through a cascade of care, and one (3.03%) focused on the choice of antiretroviral therapy (ART). Nineteen prediction models (57.6%) were developed using data from high-income countries. Evaluation of prediction models was limited to measures of performance. Detailed information on software code accessibility was often missing. Independent validation on new datasets and/or in other institutions was rarely done.

CONCLUSION

Promising approaches for ML prediction models in blood-borne viral infections were identified, but the lack of robust validation, interpretability/explainability, and poor quality of reporting hampered their clinical relevance. Our findings highlight important considerations that can inform standard reporting guidelines for ML prediction models in the future (e.g., TRIPOD-AI), and provides critical data to inform robust evaluation of the models.

Combining clinical and imaging data for predicting functional outcomes after acute ischemic stroke: an automated machine learning approach

This study aimed to develop and validate an automated machine learning (ML) system that predicts 3-month functional outcomes in acute ischemic stroke (AIS) patients by combining clinical and neuroimaging features. Functional outcomes were categorized as unfavorable (modified Rankin Scale ≥ 3) or not. A clinical model employing optimal clinical features (Model_A), a convolutional neural network model incorporating imaging data (Model_B), and an integrated model combining both imaging and clinical features (Model_C) were developed and tested to predict unfavorable outcomes. The developed models were compared with each other and with traditional risk-scoring models. The dataset comprised 4147 patients from a multicenter stroke registry, with 1268 (30.6%) experiencing unfavorable outcomes. Age, initial NIHSS, and early neurologic deterioration were identified as the most important clinical features. The ML model prediction achieved an area under the curves of 0.757 (95% CI 0.726-0.789) for Model_A, 0.725 (95% CI 0.693-0.755) for Model_B, and 0.786 (95% CI 0.757-0.814) for Model_C in the test set. The integrated models outperformed traditional risk-scoring models by 0.21 (95% CI 0.16-0.25) for HIAT and 0.15 (95% CI 0.11-0.19) for THRIVE. In conclusion, the integrated ML system enhanced stroke outcome prediction by combining imaging data and clinical features, outperforming traditional risk-scoring models.

A systematic review of clinical health conditions predicted by machine learning diagnostic and prognostic models trained or validated using real-world primary health care data

With the advances in technology and data science, machine learning (ML) is being rapidly adopted by the health care sector. However, there is a lack of literature addressing the health conditions targeted by the ML prediction models within primary health care (PHC) to date. To fill this gap in knowledge, we conducted a systematic review following the PRISMA guidelines to identify health conditions targeted by ML in PHC. We searched the Cochrane Library, Web of Science, PubMed, Elsevier, BioRxiv, Association of Computing Machinery (ACM), and IEEE Xplore databases for studies published from January 1990 to January 2022. We included primary studies addressing ML diagnostic or prognostic predictive models that were supplied completely or partially by real-world PHC data. Studies selection, data extraction, and risk of bias assessment using the prediction model study risk of bias assessment tool were performed by two investigators. Health conditions were categorized according to international classification of diseases (ICD-10). Extracted data were analyzed quantitatively. We identified 106 studies investigating 42 health conditions. These studies included 207 ML prediction models supplied by the PHC data of 24.2 million participants from 19 countries. We found that 92.4% of the studies were retrospective and 77.3% of the studies reported diagnostic predictive ML models. A majority (76.4%) of all the studies were for models' development without conducting external validation. Risk of bias assessment revealed that 90.8% of the studies were of high or unclear risk of bias. The most frequently reported health conditions were diabetes mellitus (19.8%) and Alzheimer's disease (11.3%). Our study provides a summary on the presently available ML prediction models within PHC. We draw the attention of digital health policy makers, ML models developer, and health care professionals for more future interdisciplinary research collaboration in this regard.

Machine Learning Application: A Bibliometric Analysis From a Half-Century of Research on Stroke

The quick advancement of digital technology through artificial intelligence has made it possible to deploy machine learning to predict stroke outcomes. Our aim is to examine the trend of machine learning applications in stroke-related research over the past 50 years. We used search terms stroke and machine learning to search for English versions of original and review articles and conference proceedings published over the past 50 years in Scopus and Web of Science databases. The Biblioshiny web application was utilized for the analysis. The trend of publication and prominent authors and journals were analyzed and identified. The collaborative network between countries was mapped, and a thematic map was used to monitor the authors' trending keywords. In total, 10,535 publications authored by 44,990 authors from 2,212 sources were retrieved. Two distinct clusters of collaborative network nodes were observed, with the United States serving as a connecting node. Three terms - deep learning, algorithms, and neural networks - are observed in the early stages of the emerging theme. Overall, international research collaborations, the establishment of global research initiatives, the development of computational science, and the availability of big data have facilitated the pervasive use of machine learning techniques in stroke research.

Improvement of predictive accuracies of functional outcomes after subacute stroke inpatient rehabilitation by machine learning models

OBJECTIVES

Stepwise linear regression (SLR) is the most common approach to predicting activities of daily living at discharge with the Functional Independence Measure (FIM) in stroke patients, but noisy nonlinear clinical data decrease the predictive accuracies of SLR. Machine learning is gaining attention in the medical field for such nonlinear data. Previous studies reported that machine learning models, regression tree (RT), ensemble learning (EL), artificial neural networks (ANNs), support vector regression (SVR), and Gaussian process regression (GPR), are robust to such data and increase predictive accuracies. This study aimed to compare the predictive accuracies of SLR and these machine learning models for FIM scores in stroke patients.

METHODS

Subacute stroke patients (N = 1,046) who underwent inpatient rehabilitation participated in this study. Only patients' background characteristics and FIM scores at admission were used to build each predictive model of SLR, RT, EL, ANN, SVR, and GPR with 10-fold cross-validation. The coefficient of determination (R2) and root mean square error (RMSE) values were compared between the actual and predicted discharge FIM scores and FIM gain.

RESULTS

Machine learning models (R2 of RT = 0.75, EL = 0.78, ANN = 0.81, SVR = 0.80, GPR = 0.81) outperformed SLR (0.70) to predict discharge FIM motor scores. The predictive accuracies of machine learning methods for FIM total gain (R2 of RT = 0.48, EL = 0.51, ANN = 0.50, SVR = 0.51, GPR = 0.54) were also better than of SLR (0.22).

CONCLUSIONS

This study suggested that the machine learning models outperformed SLR for predicting FIM prognosis. The machine learning models used only patients' background characteristics and FIM scores at admission and more accurately predicted FIM gain than previous studies. ANN, SVR, and GPR outperformed RT and EL. GPR could have the best predictive accuracy for FIM prognosis.

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

OBJECTIVES

In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices.

RESULTS

We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion.

CONCLUSION

The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

Development and Validation of a Novel Score for Predicting Long-Term Mortality after an Acute Ischemic Stroke

BACKGROUND

Long-term mortality prediction can guide feasible discharge care plans and coordinate appropriate rehabilitation services. We aimed to develop and validate a prediction model to identify patients at risk of mortality after acute ischemic stroke (AIS).

METHODS

The primary outcome was all-cause mortality, and the secondary outcome was cardiovascular death. This study included 21,463 patients with AIS. Three risk prediction models were developed and evaluated: a penalized Cox model, a random survival forest model, and a DeepSurv model. A simplified risk scoring system, called the C-HAND (history of Cancer before admission, Heart rate, Age, eNIHSS, and Dyslipidemia) score, was created based on regression coefficients in the multivariate Cox model for both study outcomes.

RESULTS

All experimental models achieved a concordance index of 0.8, with no significant difference in predicting poststroke long-term mortality. The C-HAND score exhibited reasonable discriminative ability for both study outcomes, with concordance indices of 0.775 and 0.798.

CONCLUSIONS

Reliable prediction models for long-term poststroke mortality were developed using information routinely available to clinicians during hospitalization.

The use of machine learning and deep learning techniques to assess proprioceptive impairments of the upper limb after stroke

BACKGROUND

Robots can generate rich kinematic datasets that have the potential to provide far more insight into impairments than standard clinical ordinal scales. Determining how to define the presence or absence of impairment in individuals using kinematic data, however, can be challenging. Machine learning techniques offer a potential solution to this problem. In the present manuscript we examine proprioception in stroke survivors using a robotic arm position matching task. Proprioception is impaired in 50-60% of stroke survivors and has been associated with poorer motor recovery and longer lengths of hospital stay. We present a simple cut-off score technique for individual kinematic parameters and an overall task score to determine impairment. We then compare the ability of different machine learning (ML) techniques and the above-mentioned task score to correctly classify individuals with or without stroke based on kinematic data.

METHODS

Participants performed an Arm Position Matching (APM) task in an exoskeleton robot. The task produced 12 kinematic parameters that quantify multiple attributes of position sense. We first quantified impairment in individual parameters and an overall task score by determining if participants with stroke fell outside of the 95% cut-off score of control (normative) values. Then, we applied five machine learning algorithms (i.e., Logistic Regression, Decision Tree, Random Forest, Random Forest with Hyperparameters Tuning, and Support Vector Machine), and a deep learning algorithm (i.e., Deep Neural Network) to classify individual participants as to whether or not they had a stroke based only on kinematic parameters using a tenfold cross-validation approach.

RESULTS

We recruited 429 participants with neuroimaging-confirmed stroke (< 35 days post-stroke) and 465 healthy controls. Depending on the APM parameter, we observed that 10.9-48.4% of stroke participants were impaired, while 44% were impaired based on their overall task score. The mean performance metrics of machine learning and deep learning models were: accuracy 82.4%, precision 85.6%, recall 76.5%, and F1 score 80.6%. All machine learning and deep learning models displayed similar classification accuracy; however, the Random Forest model had the highest numerical accuracy (83%). Our models showed higher sensitivity and specificity (AUC = 0.89) in classifying individual participants than the overall task score (AUC = 0.85) based on their performance in the APM task. We also found that variability was the most important feature in classifying performance in the APM task.

CONCLUSION

Our ML models displayed similar classification performance. ML models were able to integrate more kinematic information and relationships between variables into decision making and displayed better classification performance than the overall task score. ML may help to provide insight into individual kinematic features that have previously been overlooked with respect to clinical importance.

Machine learning-based prediction of clinical outcomes after first-ever ischemic stroke

Background

Accurate prediction of clinical outcomes in individual patients following acute stroke is vital for healthcare providers to optimize treatment strategies and plan further patient care. Here, we use advanced machine learning (ML) techniques to systematically compare the prediction of functional recovery, cognitive function, depression, and mortality of first-ever ischemic stroke patients and to identify the leading prognostic factors.

Methods

We predicted clinical outcomes for 307 patients (151 females, 156 males; 68 ± 14 years) from the PROSpective Cohort with Incident Stroke Berlin study using 43 baseline features. Outcomes included modified Rankin Scale (mRS), Barthel Index (BI), Mini-Mental State Examination (MMSE), Modified Telephone Interview for Cognitive Status (TICS-M), Center for Epidemiologic Studies Depression Scale (CES-D) and survival. The ML models included a Support Vector Machine with a linear kernel and a radial basis function kernel as well as a Gradient Boosting Classifier based on repeated 5-fold nested cross-validation. The leading prognostic features were identified using Shapley additive explanations.

Results

The ML models achieved significant prediction performance for mRS at patient discharge and after 1 year, BI and MMSE at patient discharge, TICS-M after 1 and 3 years and CES-D after 1 year. Additionally, we showed that National Institutes of Health Stroke Scale (NIHSS) was the top predictor for most functional recovery outcomes as well as education for cognitive function and depression.

Conclusion

Our machine learning analysis successfully demonstrated the ability to predict clinical outcomes after first-ever ischemic stroke and identified the leading prognostic factors that contribute to this prediction.

Application of machine learning methods in predicting schizophrenia and bipolar disorders: A systematic review

Background and Aim

Schizophrenia and bipolar disorder (BD) are critical and high-risk inherited mental disorders with debilitating symptoms. Worldwide, 3% of the population suffers from these disorders. The mortality rate of these patients is higher compared to other people. Current procedures cannot effectively diagnose these disorders because it takes an average of 10 years from the onset of the first symptoms to the definitive diagnosis of the disease. Machine learning (ML) techniques are used to meet this need. This study aimed to summarize information on the use of ML techniques for predicting schizophrenia and BD to help early and timely diagnosis of the disease.

Methods

A systematic literature search included articles published until January 19, 2020 in 3 databases. Two reviewers independently assessed original papers to determine eligibility for inclusion in this review. PRISMA guidelines were followed to conduct the study, and the Prediction Model Risk of Bias Assessment Tool (PROBAST) to assess included papers.

Results

In this review, 1243 papers were retrieved through database searches, of which 15 papers were included based on full-text assessment. ML techniques were used to predict schizophrenia and BDs. The main algorithms applied were support vector machine (SVM) (10 studies), random forests (RF) (5 studies), and gradient boosting (GB) (3 studies). Input and output characteristics were very diverse and have been kept to enable future research. RFs algorithms demonstrated significantly higher accuracy and sensitivity than SVM and GB. GB demonstrated significantly higher specificity than SVM and RF. We found no significant difference between RF and SVM in terms of specificity.

Conclusion

ML can precisely predict results and assist in making clinical decisions-concerning schizophrenia and BD. RF often performed better than other algorithms in supervised learning tasks. This study identified gaps in the literature and opportunities for future psychological ML research.

Reproducibility of artificial intelligence models in computed tomography of the head: a quantitative analysis

When developing artificial intelligence (AI) software for applications in radiology, the underlying research must be transferable to other real-world problems. To verify to what degree this is true, we reviewed research on AI algorithms for computed tomography of the head. A systematic review was conducted according to the preferred reporting items for systematic reviews and meta-analyses. We identified 83 articles and analyzed them in terms of transparency of data and code, pre-processing, type of algorithm, architecture, hyperparameter, performance measure, and balancing of dataset in relation to epidemiology. We also classified all articles by their main functionality (classification, detection, segmentation, prediction, triage, image reconstruction, image registration, fusion of imaging modalities). We found that only a minority of authors provided open source code (10.15%, n 0 7), making the replication of results difficult. Convolutional neural networks were predominantly used (32.61%, n = 15), whereas hyperparameters were less frequently reported (32.61%, n = 15). Data sets were mostly from single center sources (84.05%, n = 58), increasing the susceptibility of the models to bias, which increases the error rate of the models. The prevalence of brain lesions in the training (0.49 ± 0.30) and testing (0.45 ± 0.29) datasets differed from real-world epidemiology (0.21 ± 0.28), which may overestimate performances. This review highlights the need for open source code, external validation, and consideration of disease prevalence.

An Explainable Machine Learning Pipeline for Stroke Prediction on Imbalanced Data

Stroke is an acute neurological dysfunction attributed to a focal injury of the central nervous system due to reduced blood flow to the brain. Nowadays, stroke is a global threat associated with premature death and huge economic consequences. Hence, there is an urgency to model the effect of several risk factors on stroke occurrence, and artificial intelligence (AI) seems to be the appropriate tool. In the present study, we aimed to (i) develop reliable machine learning (ML) prediction models for stroke disease; (ii) cope with a typical severe class imbalance problem, which is posed due to the stroke patients’ class being significantly smaller than the healthy class; and (iii) interpret the model output for understanding the decision-making mechanism. The effectiveness of the proposed ML approach was investigated in a comparative analysis with six well-known classifiers with respect to metrics that are related to both generalization capability and prediction accuracy. The best overall false-negative rate was achieved by the Multi-Layer Perceptron (MLP) classifier (18.60%). Shapley Additive Explanations (SHAP) were employed to investigate the impact of the risk factors on the prediction output. The proposed AI method could lead to the creation of advanced and effective risk stratification strategies for each stroke patient, which would allow for timely diagnosis and the right treatments.

Artificial intelligence and its impact on the domains of universal health coverage, health emergencies and health promotion: An overview of systematic reviews

•

An overview of systematic reviews on the application of AI including 129 studies.

•

AI use is prominent in Universal Health Coverage, featuring image analysis in neoplasms.

•

Half of the reviews did not evaluate validation procedures nor reporting guidelines.

•

Risk of bias was only included un a third of the reviews.

•

There is not sufficient evidence to transfer AI to actual healthcare delivery.

Background

Artificial intelligence is fueling a new revolution in medicine and in the healthcare sector. Despite the growing evidence on the benefits of artificial intelligence there are several aspects that limit the measure of its impact in people’s health. It is necessary to assess the current status on the application of AI towards the improvement of people’s health in the domains defined by WHO’s Thirteenth General Programme of Work (GPW13) and the European Programme of Work (EPW), to inform about trends, gaps, opportunities, and challenges.

Objective

To perform a systematic overview of systematic reviews on the application of artificial intelligence in the people’s health domains as defined in the GPW13 and provide a comprehensive and updated map on the application specialties of artificial intelligence in terms of methodologies, algorithms, data sources, outcomes, predictors, performance, and methodological quality.

Methods

A systematic search in MEDLINE, EMBASE, Cochrane and IEEEXplore was conducted between January 2015 and June 2021 to collect systematic reviews using a combination of keywords related to the domains of universal health coverage, health emergencies protection, and better health and wellbeing as defined by the WHO’s PGW13 and EPW. Eligibility criteria was based on methodological quality and the inclusion of practical implementation of artificial intelligence. Records were classified and labeled using ICD-11 categories into the domains of the GPW13. Descriptors related to the area of implementation, type of modeling, data entities, outcomes and implementation on care delivery were extracted using a structured form and methodological aspects of the included reviews studies was assessed using the AMSTAR checklist.

Results

The search strategy resulted in the screening of 815 systematic reviews from which 203 were assessed for eligibility and 129 were included in the review. The most predominant domain for artificial intelligence applications was Universal Health Coverage (N = 98) followed by Health Emergencies (N = 16) and Better Health and Wellbeing (N = 15). Neoplasms area on Universal Health Coverage was the disease area featuring most of the applications (21.7 %, N = 28). The reviews featured analytics primarily over both public and private data sources (67.44 %, N = 87). The most used type of data was medical imaging (31.8 %, N = 41) and predictors based on regions of interest and clinical data. The most prominent subdomain of Artificial Intelligence was Machine Learning (43.4 %, N = 56), in which Support Vector Machine method was predominant (20.9 %, N = 27). Regarding the purpose, the application of Artificial Intelligence I is focused on the prediction of the diseases (36.4 %, N = 47). With respect to the validation, more than a half of the reviews (54.3 %, N = 70) did not report a validation procedure and, whenever available, the main performance indicator was the accuracy (28.7 %, N = 37). According to the methodological quality assessment, a third of the reviews (34.9 %, N = 45) implemented methods for analysis the risk of bias and the overall AMSTAR score below was 5 (4.01 ± 1.93) on all the included systematic reviews.

Conclusion

Artificial intelligence is being used for disease modelling, diagnose, classification and prediction in the three domains of GPW13. However, the evidence is often limited to laboratory and the level of adoption is largely unbalanced between ICD-11 categoriesand diseases. Data availability is a determinant factor on the developmental stage of artificial intelligence applications. Most of the reviewed studies show a poor methodological quality and are at high risk of bias, which limits the reproducibility of the results and the reliability of translating these applications to real clinical scenarios. The analyzed papers show results only in laboratory and testing scenarios and not in clinical trials nor case studies, limiting the supporting evidence to transfer artificial intelligence to actual care delivery.

A descriptive appraisal of quality of reporting in a cohort of machine learning studies in anesthesiology

BACKGROUND

The field of machine learning is being employed more and more in medicine. However, studies have shown that the quality of published studies frequently lacks completeness and adherence to published reporting guidelines. This assessment has not been done in the subspecialty of anesthesiology.

METHODS

We appraised the quality of reporting of a convenience sample of 67 peer-reviewed publications sourced from the scoping review by Hashimoto et al. Each publication was appraised on the presence of reporting elements (reporting compliance) selected from 4 peer-reviewed guidelines for reporting on machine learning studies. Results are described in several cross sections, including by section of manuscript (e.g. abstract, introduction, etc.), year of publication, impact factor of journal, and impact of publication.

RESULTS

On average, reporting compliance was 64% ± 13%. There was marked heterogeneity of reporting based on section of manuscript. There was a mild trend towards increased quality of reporting with increasing impact factor of journal of publication and increasing average number of citations per year since publication, and no trend regarding recency of publication.

CONCLUSION

The quality of reporting of machine learning studies in anesthesiology is on par with other fields, but can benefit from improvement, especially in presenting methodology, results, and discussion points, including interpretation of models and pitfalls therein. Clinicians in today's learning health systems will benefit from skills in appraisal of evidence. Several reporting guidelines have been released, and updates to mainstream guidelines are under development, which we hope will usher in improvement in reporting quality.

Risk of bias of prognostic models developed using machine learning: a systematic review in oncology

BACKGROUND

Prognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain.

METHODS

We conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately.

RESULTS

We included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation.

CONCLUSIONS

The quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models.

Machine learning predicts clinically significant health related quality of life improvement after sensorimotor rehabilitation interventions in chronic stroke

Health related quality of life (HRQOL) reflects individuals perceived of wellness in health domains and is often deteriorated after stroke. Precise prediction of HRQOL changes after rehabilitation interventions is critical for optimizing stroke rehabilitation efficiency and efficacy. Machine learning (ML) has become a promising outcome prediction approach because of its high accuracy and easiness to use. Incorporating ML models into rehabilitation practice may facilitate efficient and accurate clinical decision making. Therefore, this study aimed to determine if ML algorithms could accurately predict clinically significant HRQOL improvements after stroke sensorimotor rehabilitation interventions and identify important predictors. Five ML algorithms including the random forest (RF), k-nearest neighbors (KNN), artificial neural network, support vector machine and logistic regression were used. Datasets from 132 people with chronic stroke were included. The Stroke Impact Scale was used for assessing multi-dimensional and global self-perceived HRQOL. Potential predictors included personal characteristics and baseline cognitive/motor/sensory/functional/HRQOL attributes. Data were divided into training and test sets. Tenfold cross-validation procedure with the training data set was used for developing models. The test set was used for determining model performance. Results revealed that RF was effective at predicting multidimensional HRQOL (accuracy: 85%; area under the receiver operating characteristic curve, AUC-ROC: 0.86) and global perceived recovery (accuracy: 80%; AUC-ROC: 0.75), and KNN was effective at predicting global perceived recovery (accuracy: 82.5%; AUC-ROC: 0.76). Age/gender, baseline HRQOL, wrist/hand muscle function, arm movement efficiency and sensory function were identified as crucial predictors. Our study indicated that RF and KNN outperformed the other three models on predicting HRQOL recovery after sensorimotor rehabilitation in stroke patients and could be considered for future clinical application.

Risk prediction of 30-day mortality after stroke using machine learning: a nationwide registry-based cohort study

Backgrounds

We aimed to develop and validate machine learning (ML) models for 30-day stroke mortality for mortality risk stratification and as benchmarking models for quality improvement in stroke care.

Methods

Data from the UK Sentinel Stroke National Audit Program between 2013 to 2019 were used. Models were developed using XGBoost, Logistic Regression (LR), LR with elastic net with/without interaction terms using 80% randomly selected admissions from 2013 to 2018, validated on the 20% remaining admissions, and temporally validated on 2019 admissions. The models were developed with 30 variables. A reference model was developed using LR and 4 variables. Performances of all models was evaluated in terms of discrimination, calibration, reclassification, Brier scores and Decision-curves.

Results

In total, 488,497 stroke patients with a 12.3% 30-day mortality rate were included in the analysis. In 2019 temporal validation set, XGBoost model obtained the lowest Brier score (0.069 (95% CI: 0.068–0.071)) and the highest area under the ROC curve (AUC) (0.895 (95% CI: 0.891–0.900)) which outperformed LR reference model by 0.04 AUC (p < 0.001) and LR with elastic net and interaction term model by 0.003 AUC (p < 0.001). All models were perfectly calibrated for low (< 5%) and moderate risk groups (5–15%) and ≈1% underestimation for high-risk groups (> 15%). The XGBoost model reclassified 1648 (8.1%) low-risk cases by the LR reference model as being moderate or high-risk and gained the most net benefit in decision curve analysis.

Conclusions

All models with 30 variables are potentially useful as benchmarking models in stroke-care quality improvement with ML slightly outperforming others.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12883-022-02722-1.

Efficacy and Applications of Artificial Intelligence and Machine Learning Analyses in Total Joint Arthroplasty: A Call for Improved Reporting

BACKGROUND

There has been a considerable increase in total joint arthroplasty (TJA) research using machine learning (ML). Therefore, the purposes of this study were to synthesize the applications and efficacies of ML reported in the TJA literature, and to assess the methodological quality of these studies.

METHODS

PubMed, OVID/MEDLINE, and Cochrane libraries were queried in January 2021 for articles regarding the use of ML in TJA. Study demographics, topic, primary and secondary outcomes, ML model development and testing, and model presentation and validation were recorded. The TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines were used to assess the methodological quality.

RESULTS

Fifty-five studies were identified: 31 investigated clinical outcomes and resource utilization; 11, activity and motion surveillance; 10, imaging detection; and 3, natural language processing. For studies reporting the area under the receiver operating characteristic curve (AUC), the median AUC (and range) was 0.80 (0.60 to 0.97) among 26 clinical outcome studies, 0.99 (0.83 to 1.00) among 6 imaging-based studies, and 0.88 (0.76 to 0.98) among 3 activity and motion surveillance studies. Twelve studies compared ML to logistic regression, with 9 (75%) reporting that ML was superior. The average number of TRIPOD guidelines met was 11.5 (range: 5 to 18), with 38 (69%) meeting greater than half of the criteria. Presentation and explanation of the full model for individual predictions and assessments of model calibration were poorly reported (<30%).

CONCLUSIONS

The performance of ML models was good to excellent when applied to a wide variety of clinically relevant outcomes in TJA. However, reporting of certain key methodological and model presentation criteria was inadequate. Despite the recent surge in TJA literature utilizing ML, the lack of consistent adherence to reporting guidelines needs to be addressed to bridge the gap between model development and clinical implementation.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

BACKGROUND

Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology.

METHODS

We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models.

RESULTS

Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available.

CONCLUSIONS

The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

Hospital Length of Stay and 30-Day Mortality Prediction in Stroke: A Machine Learning Analysis of 17,000 ICU Admissions in Brazil

BACKGROUND

Hospital length of stay and mortality are associated with resource use and clinical severity, respectively, in patients admitted to the intensive care unit (ICU) with acute stroke. We proposed a structured data-driven methodology to develop length of stay and 30-day mortality prediction models in a large multicenter Brazilian ICU cohort.

METHODS

We analyzed data from 130 ICUs from 43 Brazilian hospitals. All consecutive adult patients admitted with stroke (ischemic or nontraumatic hemorrhagic) to the ICU from January 2011 to December 2020 were included. Demographic data, comorbidities, acute disease characteristics, organ support, and laboratory data were retrospectively analyzed by a data-driven methodology, which included seven different types of machine learning models applied to training and test sets of data. The best performing models, based on discrimination and calibration measures, are reported as the main results. Outcomes were hospital length of stay and 30-day in-hospital mortality.

RESULTS

Of 17,115 ICU admissions for stroke, 16,592 adult patients (13,258 ischemic and 3334 hemorrhagic) were analyzed; 4298 (26%) patients had a prolonged hospital length of stay (> 14 days), and 30-day mortality was 8% (n = 1392). Prolonged hospital length of stay was best predicted by the random forests model (Brier score = 0.17, area under the curve = 0.73, positive predictive value = 0.61, negative predictive value = 0.78). Mortality prediction also yielded the best discrimination and calibration through random forests (Brier score = 0.05, area under the curve = 0.90, positive predictive value = 0.66, negative predictive value = 0.94). Among the 20 strongest contributor variables in both models were (1) premorbid conditions (e.g., functional impairment), (2) multiple organ dysfunction parameters (e.g., hypotension, mechanical ventilation), and (3) acute neurological aspects of stroke (e.g., Glasgow coma scale score on admission, stroke type).

CONCLUSIONS

Hospital length of stay and 30-day mortality of patients admitted to the ICU with stroke were accurately predicted through machine learning methods, even in the absence of stroke-specific data, such as the National Institutes of Health Stroke Scale score or neuroimaging findings. The proposed methods using general intensive care databases may be used for resource use allocation planning and performance assessment of ICUs treating stroke. More detailed acute neurological and management data, as well as long-term functional outcomes, may improve the accuracy and applicability of future machine-learning-based prediction algorithms.

Multivariable prediction models for health care spending using machine learning: a protocol of a systematic review

BACKGROUND

With rising cost pressures on health care systems, machine-learning (ML)-based algorithms are increasingly used to predict health care costs. Despite their potential advantages, the successful implementation of these methods could be undermined by biases introduced in the design, conduct, or analysis of studies seeking to develop and/or validate ML models. The utility of such models may also be negatively affected by poor reporting of these studies. In this systematic review, we aim to evaluate the reporting quality, methodological characteristics, and risk of bias of ML-based prediction models for individual-level health care spending.

METHODS

We will systematically search PubMed and Embase to identify studies developing, updating, or validating ML-based models to predict an individual's health care spending for any medical condition, over any time period, and in any setting. We will exclude prediction models of aggregate-level health care spending, models used to infer causality, models using radiomics or speech parameters, models of non-clinically validated predictors (e.g., genomics), and cost-effectiveness analyses without predicting individual-level health care spending. We will extract data based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS), previously published research, and relevant recommendations. We will assess the adherence of ML-based studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement and examine the inclusion of transparency and reproducibility indicators (e.g. statements on data sharing). To assess the risk of bias, we will apply the Prediction model Risk Of Bias Assessment Tool (PROBAST). Findings will be stratified by study design, ML methods used, population characteristics, and medical field.

DISCUSSION

Our systematic review will appraise the quality, reporting, and risk of bias of ML-based models for individualized health care cost prediction. This review will provide an overview of the available models and give insights into the strengths and limitations of using ML methods for the prediction of health spending.

Implementation of prognostic machine learning algorithms in paediatric chronic respiratory conditions: a scoping review

Machine learning (ML) holds great potential for predicting clinical outcomes in heterogeneous chronic respiratory diseases (CRD) affecting children, where timely individualised treatments offer opportunities for health optimisation. This paper identifies rate-limiting steps in ML prediction model development that impair clinical translation and discusses regulatory, clinical and ethical considerations for ML implementation. A scoping review of ML prediction models in paediatric CRDs was undertaken using the PRISMA extension scoping review guidelines. From 1209 results, 25 articles published between 2013 and 2021 were evaluated for features of a good clinical prediction model using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines.

Most of the studies were in asthma (80%), with few in cystic fibrosis (12%), bronchiolitis (4%) and childhood wheeze (4%). There were inconsistencies in model reporting and studies were limited by a lack of validation, and absence of equations or code for replication. Clinician involvement during ML model development is essential and diversity, equity and inclusion should be assessed at each step of the ML pipeline to ensure algorithms do not promote or amplify health disparities among marginalised groups. As ML prediction studies become more frequent, it is important that models are rigorously developed using published guidelines and take account of regulatory frameworks which depend on model complexity, patient safety, accountability and liability.

The Allure of Big Data to Improve Stroke Outcomes: Review of Current Literature

PURPOSE OF REVIEW

To critically appraise literature on recent advances and methods using "big data" to evaluate stroke outcomes and associated factors.

RECENT FINDINGS

Recent big data studies provided new evidence on the incidence of stroke outcomes, and important emerging predictors of these outcomes. Main highlights included the identification of COVID-19 infection and exposure to a low-dose particulate matter as emerging predictors of mortality post-stroke. Demographic (age, sex) and geographical (rural vs. urban) disparities in outcomes were also identified. There was a surge in methodological (e.g., machine learning and validation) studies aimed at maximizing the efficiency of big data for improving the prediction of stroke outcomes. However, considerable delays remain between data generation and publication. Big data are driving rapid innovations in research of stroke outcomes, generating novel evidence for bridging practice gaps. Opportunity exists to harness big data to drive real-time improvements in stroke outcomes.

Development of Deep Learning Models for Predicting In-Hospital Mortality Using an Administrative Claims Database: Retrospective Cohort Study

BACKGROUND

Administrative claims databases have been used widely in studies because they have large sample sizes and are easily available. However, studies using administrative databases lack information on disease severity, so a risk adjustment method needs to be developed.

OBJECTIVE

We aimed to develop and validate deep learning-based prediction models for in-hospital mortality of acute care patients.

METHODS

The main model was developed using only administrative claims data (age, sex, diagnoses, and procedures on the day of admission). We also constructed disease-specific models for acute myocardial infarction, heart failure, stroke, and pneumonia using common severity indices for these diseases. Using the Japanese Diagnosis Procedure Combination data from July 2010 to March 2017, we identified 46,665,933 inpatients and divided them into derivation and validation cohorts in a ratio of 95:5. The main model was developed using a 9-layer deep neural network with 4 hidden dense layers that had 1000 nodes and were fully connected to adjacent layers. We evaluated model discrimination ability by an area under the receiver operating characteristic curve (AUC) and calibration ability by calibration plot.

RESULTS

Among the eligible patients, 2,005,035 (4.3%) died. Discrimination and calibration of the models were satisfactory. The AUC of the main model in the validation cohort was 0.954 (95% CI 0.954-0.955). The main model had higher discrimination ability than the disease-specific models.

CONCLUSIONS

Our deep learning-based model using diagnoses and procedures produced valid predictions of in-hospital mortality.

External Validation of a Neural Network Model in Aneurysmal Subarachnoid Hemorrhage: A Comparison With Conventional Logistic Regression Models

BACKGROUND

Interest in machine learning (ML)-based predictive modeling has led to the development of models predicting outcomes after aneurysmal subarachnoid hemorrhage (aSAH), including the Nijmegen acute subarachnoid hemorrhage calculator (Nutshell). Generalizability of such models to external data remains unclear.

OBJECTIVE

To externally validate the performance of the Nutshell tool while comparing it with the conventional Subarachnoid Hemorrhage International Trialists (SAHIT) models and to review the ML literature on outcome prediction after aSAH and aneurysm treatment.

METHODS

A prospectively maintained database of patients with aSAH presenting consecutively to our institution in the 2013 to 2018 period was used. The web-based Nutshell and SAHIT calculators were used to derive the risks of poor long-term (12-18 months) outcomes and 30-day mortality. Discrimination was evaluated using the area under the curve (AUC), and calibration was investigated using calibration plots. The literature on relevant ML models was surveyed for a synopsis.

RESULTS

In 269 patients with aSAH, the SAHIT models outperformed the Nutshell tool (AUC: 0.786 vs 0.689, P = .025) in predicting long-term functional outcomes. A logistic regression model of the Nutshell variables derived from our data achieved adequate discrimination (AUC = 0.759) of poor outcomes. The SAHIT models outperformed the Nutshell tool in predicting 30-day mortality (AUC: 0.810 vs 0.636, P < .001). Calibration properties were more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms.

CONCLUSION

The Nutshell tool demonstrated limited performance on external validation in comparison with the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.

Machine learning prediction models in orthopedic surgery: A systematic review in transparent reporting

Machine learning (ML) studies are becoming increasingly popular in orthopedics but lack a critically appraisal of their adherence to peer-reviewed guidelines. The objective of this review was to (1) evaluate quality and transparent reporting of ML prediction models in orthopedic surgery based on the transparent reporting of multivariable prediction models for individual prognosis or diagnosis (TRIPOD), and (2) assess risk of bias with the Prediction model Risk Of Bias ASsessment Tool. A systematic review was performed to identify all ML prediction studies published in orthopedic surgery through June 18th, 2020. After screening 7138 studies, 59 studies met the study criteria and were included. Two reviewers independently extracted data and discrepancies were resolved by discussion with at least two additional reviewers present. Across all studies, the overall median completeness for the TRIPOD checklist was 53% (interquartile range 47%-60%). The overall risk of bias was low in 44% (n = 26), high in 41% (n = 24), and unclear in 15% (n = 9). High overall risk of bias was driven by incomplete reporting of performance measures, inadequate handling of missing data, and use of small datasets with inadequate outcome numbers. Although the number of ML studies in orthopedic surgery is increasing rapidly, over 40% of the existing models are at high risk of bias. Furthermore, over half incompletely reported their methods and/or performance measures. Until these issues are adequately addressed to give patients and providers trust in ML models, a considerable gap remains between the development of ML prediction models and their implementation in orthopedic practice.

Predicting 1-Hour Thrombolysis Effect of r-tPA in Patients With Acute Ischemic Stroke Using Machine Learning Algorithm

Background: Thrombolysis with r-tPA is recommended for patients after acute ischemic stroke (AIS) within 4.5 h of symptom onset. However, only a few patients benefit from this therapeutic regimen. Thus, we aimed to develop an interpretable machine learning (ML)–based model to predict the thrombolysis effect of r-tPA at the super-early stage.

Methods: A total of 353 patients with AIS were divided into training and test data sets. We then used six ML algorithms and a recursive feature elimination (RFE) method to explore the relationship among the clinical variables along with the NIH stroke scale score 1 h after thrombolysis treatment. Shapley additive explanations and local interpretable model–agnostic explanation algorithms were applied to interpret the ML models and determine the importance of the selected features.

Results: Altogether, 353 patients with an average age of 63.0 (56.0–71.0) years were enrolled in the study. Of these patients, 156 showed a favorable thrombolysis effect and 197 showed an unfavorable effect. A total of 14 variables were enrolled in the modeling, and 6 ML algorithms were used to predict the thrombolysis effect. After RFE screening, seven variables under the gradient boosting decision tree (GBDT) model (area under the curve = 0.81, specificity = 0.61, sensitivity = 0.9, and F1 score = 0.79) demonstrated the best performance. Of the seven variables, activated partial thromboplastin clotting time (time), B-type natriuretic peptide, and fibrin degradation products were the three most important clinical characteristics that might influence r-tPA efficiency.

Conclusion: This study demonstrated that the GBDT model with the seven variables could better predict the early thrombolysis effect of r-tPA.

Development, validation and application of a machine learning model to estimate salt consumption in 54 countries

Global targets to reduce salt intake have been proposed but their monitoring is challenged by the lack of population-based data on salt consumption. We developed a machine learning (ML) model to predict salt consumption at the population level based on simple predictors and applied this model to national surveys in 54 countries. We used 21 surveys with spot urine samples for the ML model derivation and validation; we developed a supervised ML regression model based on: sex, age, weight, height, systolic and diastolic blood pressure. We applied the ML model to 54 new surveys to quantify the mean salt consumption in the population. The pooled dataset in which we developed the ML model included 49,776 people. Overall, there were no substantial differences between the observed and ML-predicted mean salt intake (p<0.001). The pooled dataset where we applied the ML model included 166,677 people; the predicted mean salt consumption ranged from 6.8 g/day (95% CI: 6.8-6.8 g/day) in Eritrea to 10.0 g/day (95% CI: 9.9-10.0 g/day) in American Samoa. The countries with the highest predicted mean salt intake were in Western Pacific. The lowest predicted intake was found in Africa. The country-specific predicted mean salt intake was within reasonable difference from the best available evidence. A ML model based on readily available predictors estimated daily salt consumption with good accuracy. This model could be used to predict mean salt consumption in the general population where urine samples are not available.

Psychosis Relapse Prediction Leveraging Electronic Health Records Data and Natural Language Processing Enrichment Methods

BACKGROUND

Identifying patients at a high risk of psychosis relapse is crucial for early interventions. A relevant psychiatric clinical context is often recorded in clinical notes; however, the utilization of unstructured data remains limited. This study aimed to develop psychosis-relapse prediction models using various types of clinical notes and structured data.

METHODS

Clinical data were extracted from the electronic health records of the Ajou University Medical Center in South Korea. The study population included patients with psychotic disorders, and outcome was psychosis relapse within 1 year. Using only structured data, we developed an initial prediction model, then three natural language processing (NLP)-enriched models using three types of clinical notes (psychological tests, admission notes, and initial nursing assessment) and one complete model. Latent Dirichlet Allocation was used to cluster the clinical context into similar topics. All models applied the least absolute shrinkage and selection operator logistic regression algorithm. We also performed an external validation using another hospital database.

RESULTS

A total of 330 patients were included, and 62 (18.8%) experienced psychosis relapse. Six predictors were used in the initial model and 10 additional topics from Latent Dirichlet Allocation processing were added in the enriched models. The model derived from all notes showed the highest value of the area under the receiver operating characteristic (AUROC = 0.946) in the internal validation, followed by models based on the psychological test notes, admission notes, initial nursing assessments, and structured data only (0.902, 0.855, 0.798, and 0.784, respectively). The external validation was performed using only the initial nursing assessment note, and the AUROC was 0.616.

CONCLUSIONS

We developed prediction models for psychosis relapse using the NLP-enrichment method. Models using clinical notes were more effective than models using only structured data, suggesting the importance of unstructured data in psychosis prediction.

Data-driven methods for dengue prediction and surveillance using real-world and Big Data: A systematic review

Background

Traditionally, dengue surveillance is based on case reporting to a central health agency. However, the delay between a case and its notification can limit the system responsiveness. Machine learning methods have been developed to reduce the reporting delays and to predict outbreaks, based on non-traditional and non-clinical data sources. The aim of this systematic review was to identify studies that used real-world data, Big Data and/or machine learning methods to monitor and predict dengue-related outcomes.

Methodology/Principal findings

We performed a search in PubMed, Scopus, Web of Science and grey literature between January 1, 2000 and August 31, 2020. The review (ID: CRD42020172472) focused on data-driven studies. Reviews, randomized control trials and descriptive studies were not included. Among the 119 studies included, 67% were published between 2016 and 2020, and 39% used at least one novel data stream. The aim of the included studies was to predict a dengue-related outcome (55%), assess the validity of data sources for dengue surveillance (23%), or both (22%). Most studies (60%) used a machine learning approach. Studies on dengue prediction compared different prediction models, or identified significant predictors among several covariates in a model. The most significant predictors were rainfall (43%), temperature (41%), and humidity (25%). The two models with the highest performances were Neural Networks and Decision Trees (52%), followed by Support Vector Machine (17%). We cannot rule out a selection bias in our study because of our two main limitations: we did not include preprints and could not obtain the opinion of other international experts.

Conclusions/Significance

Combining real-world data and Big Data with machine learning methods is a promising approach to improve dengue prediction and monitoring. Future studies should focus on how to better integrate all available data sources and methods to improve the response and dengue management by stakeholders.

Dengue is one of the most important arbovirus infections in the world and its public health, societal and economic burden is increasing. Although the majority of dengue cases are asymptomatic or mild, severe disease forms can lead to death. For this reason, early diagnosis and monitoring of dengue are crucial to decrease mortality. However, most endemic regions still rely on traditional monitoring methods, despite the growing availability of novel data sources and data-driven methods based on real-world data, Big Data, and machine learning algorithms. In this systematic review, we identified and analyzed studies that used these novel approaches for dengue monitoring and/or prediction. We found that novel data streams, such as Internet search engines and social media platforms, and machine learning methods can be successfully used to improve dengue management, but are still vastly ignored in real life. These approaches should be combined with traditional methods to help stakeholders better prepare for each outbreak and improve early responsiveness.

Potential applications and performance of machine learning techniques and algorithms in clinical practice: A systematic review

PURPOSE

The advent of clinically adapted machine learning algorithms can solve numerous problems ranging from disease diagnosis and prognosis to therapy recommendations. This systematic review examines the performance of machine learning (ML) algorithms and evaluates the progress made to date towards their implementation in clinical practice.

METHODS

Systematic searching of databases (PubMed, MEDLINE, Scopus, Google Scholar, Cochrane Library and WHO Covid-19 database) to identify original articles published between January 2011 and October 2021. Studies reporting ML techniques in clinical practice involving humans and ML algorithms with a performance metric were considered.

RESULTS

Of 873 unique articles identified, 36 studies were eligible for inclusion. The XGBoost (extreme gradient boosting) algorithm showed the highest potential for clinical applications (n = 7 studies); this was followed jointly by random forest algorithm, logistic regression, and the support vector machine, respectively (n = 5 studies). Prediction of outcomes (n = 33), in particular Inflammatory diseases (n = 7) received the most attention followed by cancer and neuropsychiatric disorders (n = 5 for each) and Covid-19 (n = 4). Thirty-three out of the thirty-six included studies passed more than 50% of the selected quality assessment criteria in the TRIPOD checklist. In contrast, none of the studies could achieve an ideal overall bias rating of 'low' based on the PROBAST checklist. In contrast, only three studies showed evidence of the deployment of ML algorithm(s) in clinical practice.

CONCLUSIONS

ML is potentially a reliable tool for clinical decision support. Although advocated widely in clinical practice, work is still in progress to validate clinically adapted ML algorithms. Improving quality standards, transparency, and interpretability of ML models will further lower the barriers to acceptability.

Systematic review of prediction models for postacute care destination decision-making

OBJECTIVE

This article reports a systematic review of studies containing development and validation of models predicting postacute care destination after adult inpatient hospitalization, summarizes clinical populations and variables, evaluates model performance, assesses risk of bias and applicability, and makes recommendations to reduce bias in future models.

MATERIALS AND METHODS

A systematic literature review was conducted following PRISMA guidelines and the Cochrane Prognosis Methods Group criteria. Online databases were searched in June 2020 to identify all published studies in this area. Data were extracted based on the CHARMS checklist, and studies were evaluated based on predictor variables, validation, performance in validation, risk of bias, and applicability using the Prediction Model Risk of Bias Assessment Tool (PROBAST) tool.

RESULTS

The final sample contained 28 articles with 35 models for evaluation. Models focused on surgical (22), medical (5), or both (8) populations. Eighteen models were internally validated, 10 were externally validated, and 7 models underwent both types. Model performance varied within and across populations. Most models used retrospective data, the median number of predictors was 8.5, and most models demonstrated risk of bias.

DISCUSSION AND CONCLUSION

Prediction modeling studies for postacute care destinations are becoming more prolific in the literature, but model development and validation strategies are inconsistent, and performance is variable. Most models are developed using regression, but machine learning methods are increasing in frequency. Future studies should ensure the rigorous variable selection and follow TRIPOD guidelines. Only 14% of the models have been tested or implemented beyond original studies, so translation into practice requires further investigation.

Machine learning methods applied to triage in emergency services: A systematic review

BACKGROUND

In emergency services is important to accurately assess and classify symptoms, which may be improved with the help of technology. One mechanism that could help and improve predictions from health records or patient flow is machine learning (ML).

AIM

To analyse the effectiveness of ML systems in triage for making predictions at the emergency department in comparison with other triage scales/scores.

METHODS

Following the PRISMA recommendations, a systematic review was conducted using CINAHL, Cochrane, Cuiden, Medline and Scopus databases with the search equation "Machine learning AND triage AND emergency".

RESULTS

Eleven studies were identified. The studies show that the use of ML methods consistently predict important outcomes like mortality, critical care outcomes and admission, and the need for hospitalization in comparison with scales like Emergency Severity Index or others. Among the ML models considered, XGBoost and Deep Neural Networks obtained the highest levels of prediction accuracy, while Logistic Regression performed obtained the worst values.

CONCLUSIONS

Machine learning methods can be a good instrument for helping triage process with the prediction of important emergency variables like mortality or the need for critical care or hospitalization.

Machine Learning Models Prognosticate Functional Outcomes Better than Clinical Scores in Spontaneous Intracerebral Haemorrhage

OBJECTIVE

This study aims to develop and compare the use of deep neural networks (DNN) and support vector machines (SVM) to clinical prognostic scores for prognosticating 30-day mortality and 90-day poor functional outcome (PFO) in spontaneous intracerebral haemorrhage (SICH).

MATERIALS AND METHODS

We conducted a retrospective cohort study of 297 SICH patients between December 2014 and May 2016. Clinical data was collected from electronic medical records using standardized data collection forms. The machine learning workflow included imputation of missing data, dimensionality reduction, imbalanced-class correction, and evaluation using cross-validation and comparison of accuracy against clinical prognostic scores.

RESULTS

32 (11%) patients had 30-day mortality while 177 (63%) patients had 90-day PFO. For prognosticating 30-day mortality, the class-balanced accuracies for DNN (0.875; 95% CI 0.800-0.950; McNemar's p-value 1.000) and SVM (0.848; 95% CI 0.767-0.930; McNemar's p-value 0.791) were comparable to that of the original ICH score (0.833; 95% CI 0.748-0.918). The c-statistics for DNN (0.895; DeLong's p-value 0.715), and SVM (0.900; DeLong's p-value 0.619), though greater than that of the original ICH score (0.862), were not significantly different. For prognosticating 90-day PFO, the class-balanced accuracies for DNN (0.853; 95% CI 0.772-0.934; McNemar's p-value 0.003) and SVM (0.860; 95% CI 0.781-0.939; McNemar's p-value 0.004) were better than that of the ICH-Grading Scale (0.706; 95% CI 0.600-0.812). The c-statistic for SVM (0.883; DeLong's p-value 0.022) was significantly greater than that of the ICH-Grading Scale (0.778), while the c-statistic for DNN was 0.864 (DeLong's p-value 0.055).

CONCLUSION

We showed that the SVM model performs significantly better than clinical prognostic scores in predicting 90-day PFO in SICH.

Hospital Facebook Reviews Analysis Using a Machine Learning Sentiment Analyzer and Quality Classifier

While experts have recognised the significance and necessity of social media integration in healthcare, no systematic method has been devised in Malaysia or Southeast Asia to include social media input into the hospital quality improvement process. The goal of this work is to explain how to develop a machine learning system for classifying Facebook reviews of public hospitals in Malaysia by using service quality (SERVQUAL) dimensions and sentiment analysis. We developed a Machine Learning Quality Classifier (MLQC) based on the SERVQUAL model and a Machine Learning Sentiment Analyzer (MLSA) by manually annotated multiple batches of randomly chosen reviews. Logistic regression (LR), naive Bayes (NB), support vector machine (SVM), and other methods were used to train the classifiers. The performance of each classifier was tested using 5-fold cross validation. For topic classification, the average F1-score was between 0.687 and 0.757 for all models. In a 5-fold cross validation of each SERVQUAL dimension and in sentiment analysis, SVM consistently outperformed other methods. The study demonstrates how to use supervised learning to automatically identify SERVQUAL domains and sentiments from patient experiences on a hospital's Facebook page. Malaysian healthcare providers can gather and assess data on patient care via the use of these content analysis technology to improve hospital quality of care.