The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies

Exploring Multimorbidity Patterns in older hospitalized Norwegian patients using Network Analysis modularity

BACKGROUND

Understanding Multimorbidity Patterns (MPs) is crucial for planning healthcare interventions, allocating resources, and improving patients' outcomes.

OBJECTIVE

We aim to demonstrate the use of Network Analysis (NA) to explore the MPs in hospitalized Norwegian older patients.

METHODS

We utilized data from the Norwegian Patient Registry (NPR) of all admissions between 2017 and 2019. The study population included patients ≥ 65 years old with two or more different conditions. Multimorbidity was defined as the co-occurrence of two or more associated chronic conditions. Chronic conditions were identified using the Chronic Condition Indicator Refined (CCIR) list. The association between chronic conditions was determined by calculating Relative Risk (RR) and Phi-correlation to detect pairs of conditions that co-occur beyond chance. A multimorbidity network was created, and MPs were detected using Louvain method for community detection. We suggested a clinical interpretation for these MPs.

RESULTS

A total of 539 chronic conditions were used to create a multimorbidity network revealing several MPs. These modules included patterns of vision and hearing disorders, cardiorenal syndrome, metabolic and cardiovascular disorders, respiratory disorders, endocrine and skin conditions, autoimmune and musculoskeletal disorders, as well as mental and behavioral disorders. Using NA centrality measures, we identified the most influential conditions in each module. An interactive network and sunburst graphs for each module are publicly available.

CONCLUSION

The study demonstrates the use of NA modularity detection in identifying MPs. The findings highlight the complex interaction of chronic conditions in the elderly and the potential of NA methodology in exploring these relationships.

Interpretable machine learning model for prediction functional cure in chronic hepatitis B patients receiving Peg-IFN therapy: A multi-center study

BACKGROUND

Functional cure is the ideal treatment goal for chronic hepatitis B (CHB) treatment. We developed and validated machine learning (ML) models to predict functional cure in CHB patients.

METHODS

This study retrospectively recruited 534 CHB patients who received Peg-IFN treatment to construct model and 269 patients for external validation. We analyzed three strategies: baseline, week 12, week 24. Seven ML models were constructed using selected variables by Boruta and least absolute shrinkage and selection operator regression algorithm, and performance metrics, including area under the curve (AUC), sensitivity, specificity, and F1 score were applied to determine the best model. We utilized SHapley Additive exPlanation to visualize and interpret the best model and built a website to conveniently predict functional cure of CHB.

RESULTS

A total of 272 participants were cured in our study. Compared to baseline and week 12 strategies, week 24 using Support Vector Machine (SVM) model can better predict functional cure of CHB, with reliable predictive performance (AUC = 0.981), calibration and clinical applicability in external validation cohort. Age, ALT ratio at week 12, HBsAg at week 24 and HBsAg ratio at week 24 were important features. In order to enhance clinical convenience and effectiveness of the constructed model, a web-based dynamic nomogram was created (Dynamic Nomogram (shinyapps.io)).

CONCLUSION

This study developed SVM model to predict functional cure in CHB patients treated with Peg-IFN. Furthermore, we also built a website that clinicians can individualized predict the efficacy of Peg-IFN therapy in CHB patients.

Systematic literature review on the application of explainable artificial intelligence in palliative care studies

BACKGROUND

As machine learning models become increasingly prevalent in palliative care, explainability has become a critical factor in their successful deployment in this sensitive field, where decisions can profoundly impact patient health and quality of life. To address these concerns, Explainable AI (XAI) aims to make complex AI models more understandable and trustworthy.

OBJECTIVE

This study aims to assess the current state of machine learning models in palliative care, specifically focusing on their compliance with the principles of XAI.

METHODS

A comprehensive literature search in four databases was conducted to identify articles on machine learning in palliative care studies published until May 2024, followed by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guideline. The Checklist for Assessment of Medical Artificial Intelligence was used to evaluate the quality of the studies.

RESULTS

Mortality and survival prediction were the primary focus areas in 15 (54%) of the included 28 studies. Regarding data explainability, 20 studies (71%) documented their data preprocessing methods. However, a notable concern is that 45% of the studies did not address handling missing data. Across these studies, 74 machine learning algorithms were employed. Complex models, including Random Forest, Support Vector Machines, Gradient Boosting Machines, and Deep Neural Networks, were predominantly used (64%) due to their high predictive power, achieving AUC values between 0.82 and 0.96. Post-hoc explanation techniques were applied in only 11 studies, using seven different XAI techniques, focusing on global explanations to enhance understanding of model behavior.

CONCLUSION

Given the critical role of AI-driven decisions in patient care, adopting XAI techniques is essential for fostering trust and usability. Although progress has been made, significant gaps persist. A main challenge remains the trade-off between model performance and interpretability, as highly accurate models often lack the transparency required to build trust in clinical settings. Additionally, complex models frequently provide inadequate explanations for their outputs, lack consistent documentation, and have limited XAI applications, reducing the interpretability of machine learning studies for clinicians and decision-makers.

Wet and dry cough classification using cough sound characteristics and machine learning: A systematic review

BACKGROUND

Distinguishing between productive (wet) and non-productive (dry) cough types is important for evaluating respiratory health, assisting in differential diagnosis, and monitoring disease progression. However, assessing cough type through the perception of cough sounds in clinical settings poses challenges due to its subjectivity. Employing objective cough sound analysis holds promise for aiding diagnostic assessments and guiding the management of respiratory conditions. This systematic review aims to assess and summarize the predictive capabilities of machine learning algorithms in analyzing cough sounds to determine cough type.

METHOD

A systematic search of the Scopus, Medline, and Embase databases conducted on March 8, 2025, yielded three studies that met the inclusion criteria. The quality assessment of these studies was conducted using the checklist for the assessment of medical artificial intelligence (ChAMAI).

RESULTS

The inter-rater agreement for annotating wet and dry coughs ranged from 0.22 to 0.81 across the three studies. Furthermore, these studies employed diverse inputs for their machine learning algorithms, including different cough sound features and time-frequency representations. The algorithms used ranged from conventional classifiers like logistic regression to neural networks. While the classification accuracy for identifying wet and dry coughs ranged from 78% to 87% across these studies, none of them assessed their algorithms through external validation.

CONCLUSION

The high variability in inter-rater agreement highlights the subjectivity in manually interpreting cough sounds and underscores the need for objective cough sound analysis methods. The predictive ability of cough-type classification algorithms shows promise in the small number of studies analyzed in this systematic review. However, more studies are needed, particularly those validating their models on independent and external datasets.

Comparing logistic regression and machine learning for obesity risk prediction: A systematic review and meta-analysis

BACKGROUND

Logistic regression (LR) has traditionally been the standard method used for predicting binary health outcomes; however, machine learning (ML) methods are increasingly popular.

OBJECTIVE

This study aimed to compare the performance of ML and LR for obesity risk prediction, identify how LR and ML were being compared, and identify the commonly used ML methods.

METHODS

We conducted comprehensive searches in PubMed, Scopus, Embase, IEEE Xplore, and Web of Science databases on 24th November 2023, with no restrictions on publication dates. Meta-analyses were performed to quantify the overall predictive performance of the methods using the area under the curve (AUC) for LR, AUC for the best performing ML, as well as the difference in the AUC between the two approaches as the effect measures.

RESULTS

We included 28 studies out of 913 abstracts screened. Accuracy and sensitivity were the most commonly used performance measures. More than half of the studies used AUC, with no calibration assessment conducted in any of the studies. Decision trees followed by boosting algorithms were the most commonly used ML methods. Seventy-five percent of the studies were at high risk of bias. There were 14 included studies in the meta-analysis. The pooled AUC for LR was 0.75 (95% CI 0.70 to 0.80) and the pooled AUC for ML was 0.76 (95% CI 0.70 to 0.82). The pooled difference in logit(AUC) between ML and LR was 0.13 (95% CI -0.11 to 0.37).

CONCLUSION

We conclude that there is no significant difference in the performance of ML and LR for obesity risk prediction. However, there is a need for improved quality of reporting of studies, the use of more performance measures particularly calibration, and to validate models in different populations.

Development and validation of a nomogram to predict the risk of in-hospital MACE for emergence NSTE-ACS: A retrospective multicenter study based on the Chinese population

PURPOSE

Our study aims to develop and validate an effective in-hospital major adverse cardiovascular events(MACE) prediction model for patients with emergency Non-ST elevation acute coronary syndrome(NSTE-ACS).

METHODS

We retrospectively collected NSTE-ACS patients in three tertiary hospitals in Chongqing. In-hospital MACE was the predicted outcome. Patients from one hospital were divided into training set and internal validation set according to the ratio of 7:3. Besides, 662 patients from two other tertiary hospitals were for external validation. Patient information including demographics, laboratory tests results and disease course records were for comprehensive analysis. Finally, LASSO were used to identify the predictors and develop the model. This model was subsequently visualized as a nomogram, followed by both internal and external validations.The receiver operating characteristic curve, calibration curve and clinical decision curve analysis were used to assess the model's discrimination, calibration and clinical applicability, respectively.

RESULTS

A total of 3,308 patients were included, 375 of whom developed in-hospital MACE. The LR model demonstrated that length of stay, neutrophils, myoglobin, NYHA, CCI, NT-proBNP, LVEF and respiratory failure were risk factors for in-hospital MACE in emergence NSTE-ACS patients. In the training set, the AUC was 0.860 (95%CI:0.831-0.889). In external validation,the AUC was 0.855(95%CI:0.808-0.902), and both the calibration curve and DCA in validation set also revealed stable predictive accuracy and clinical validity.Additionally,it is available to calculate the MACE risk online via the web page (https://cocozhou99.shinyapps.io/DynNomapp/).

CONCLUSION

The prediction model we constructed has good predictive performance and can help healthcare professionals accurately assess the risk of in-hospital MACE in emergence NSTE-ACS patients.

The promise and challenges of computer mouse trajectories in DMHIs - A feasibility study on pre-treatment dropout predictions

With the impetus of Digital Mental Health Interventions (DMHIs), complex data can be leveraged to improve and personalize mental health care. However, most approaches rely on a very limited number of often costly features. Computer mouse trajectories can be unobtrusively and cost-efficiently gathered and seamlessly integrated into current baseline processes. Empirical evidence suggests that mouse movements hold information on user motivation and attention, both valuable aspects otherwise difficult to measure at scale. Further, mouse trajectories can already be collected on pre-treatment questionnaires, making them a promising candidate for early predictions informing treatment allocation. Therefore, this paper discusses how to collect and process mouse trajectory data on questionnaires in DMHIs. Covering different complexity levels, we combine hand-crafted features with non-sequential machine learning models, as well as spatiotemporal raw mouse data with state-of-the-art sequential neural networks. The data processing pipeline for the latter includes task-specific pre-processing to convert the variable length trajectories into a single prediction per user. As a feasibility study, we collected mouse trajectory data from 183 patients filling out a pre-intervention depression questionnaire. While the hand-crafted features slightly improve baseline predictions, the spatiotemporal models underperform. However, considering our small data set size, we propose more research to investigate the potential value of this novel and promising data type and provide the necessary steps and open-source code to do so.

Stress monitoring using low-cost electroencephalogram devices: A systematic literature review

INTRODUCTION

The use of low-cost, consumer-grade wearable health monitoring devices has become increasingly prevalent in mental health research, including stress studies. While cortisol response magnitude remains the gold standard for stress assessment, an expanding body of research employs low-cost EEG devices as primary tools for recording biomarker data, often combined with wrist and ring-based wearables. However, the technical variability among low-cost EEG devices, particularly in sensor count and placement according to the 10-20 Electrode Placement System, poses challenges for reproducibility in study outcomes.

OBJECTIVE

This review aims to provide an overview of the growing application of low-cost EEG devices and machine learning techniques for assessing brain function, with a focus on stress detection. It also highlights the strengths and weaknesses of various machine learning methods commonly used in stress research, and evaluates the reproducibility of reported findings along with sensor count and placement importance.

METHODS

A comprehensive review was conducted of published studies utilizing EEG devices for stress detection and their associated machine learning approaches. Searches were performed across databases including Scopus, Google Scholar, ScienceDirect, Nature, and PubMed, yielding 69 relevant articles for analysis. The selected studies were synthesized into four thematic categories: stress assessment using EEG, low-cost EEG devices, datasets for EEG-based stress measurement, and machine learning techniques for EEG-based stress analysis. For machine learning-focused studies, validation and reproducibility methods were critically assessed. Study quality was evaluated and scored using the IJMEDI checklist.

RESULTS

The review identified several studies employing low-cost EEG devices to monitor brain activity during stress and relaxation phases, with many reporting high predictive accuracy using various machine learning validation techniques. However, only 54% of the studies included health screening prior to experimentation, and 58% were categorized as low-powered due to limited sample sizes. Additionally, few studies validated their results using an independent validation set or cortisol response as a correlating biomarker and there was a lack of consensus on data pre-processing and sensor placement as a key contributor to improving model generalization and accuracy.

CONCLUSION

Low-cost consumer-grade wearable devices, including EEG and wrist-based monitors, are increasingly utilized in stress-related research, offering promising avenues for non-invasive biomarker monitoring. However, significant gaps remain in standardizing EEG signal processing and sensor placement, both of which are critical for enhancing model generalization and accuracy. Furthermore, the limited use of independent validation sets and cortisol response as correlating biomarkers highlights the need for more robust validation methodologies. Future research should focus on addressing these limitations and establishing consensus on data pre-processing techniques and sensor configurations to improve the reliability and reproducibility of findings in this growing field.

Predictive value of machine learning for in-hospital mortality risk in acute myocardial infarction: A systematic review and meta-analysis

BACKGROUND

Machine learning (ML) models have been constructed to predict the risk of in-hospital mortality in patients with myocardial infarction (MI). Due to diverse ML models and modeling variables, along with the significant imbalance in data, the predictive accuracy of these models remains controversial.

OBJECTIVE

This study aimed to review the accuracy of ML in predicting in-hospital mortality risk in MI patients and to provide evidence-based advices for the development or updating of clinical tools.

METHODS

PubMed, Embase, Cochrane, and Web of Science databases were searched, up to June 4, 2024. PROBAST and ChAMAI checklist are utilized to assess the risk of bias in the included studies. Since the included studies constructed models based on severely unbalanced datasets, subgroup analyses were conducted by the type of dataset (balanced data, unbalanced data, model type).

RESULTS

This meta-analysis included 32 studies. In the validation set, the pooled C-index, sensitivity, and specificity of prediction models based on balanced data were 0.83 (95 % CI: 0.795-0.866), 0.81 (95 % CI: 0.79-0.84), and 0.82 (95 % CI: 0.78-0.86), respectively. In the validation set, the pooled C-index, sensitivity, and specificity of ML models based on imbalanced data were 0.815 (95 % CI: 0.789-0.842), 0.66 (95 % CI: 0.60-0.72), and 0.84 (95 % CI: 0.83-0.85), respectively.

CONCLUSIONS

ML models such as LR, SVM, and RF exhibit high sensitivity and specificity in predicting in-hospital mortality in MI patients. However, their sensitivity is not superior to well-established scoring tools. Mitigating the impact of imbalanced data on ML models remains challenging.

Performance of multimodal prediction models for intracerebral hemorrhage outcomes using real-world data

BACKGROUND

We aimed to develop and validate multimodal models integrating computed tomography (CT) images, text and tabular clinical data to predict poor functional outcomes and in-hospital mortality in patients with intracerebral hemorrhage (ICH). These models were designed to assist non-specialists in emergency settings with limited access to stroke specialists.

METHODS

A retrospective analysis of 527 patients with ICH admitted to a Japanese tertiary hospital between April 2019 and February 2022 was conducted. Deep learning techniques were used to extract features from three-dimensional CT images and unstructured data, which were then combined with tabular data to develop an L1-regularized logistic regression model to predict poor functional outcomes (modified Rankin scale score 3-6) and in-hospital mortality. The model's performance was evaluated by assessing discrimination metrics, calibration plots, and decision curve analysis (DCA) using temporal validation data.

RESULTS

The multimodal model utilizing both imaging and text data, such as medical interviews, exhibited the highest performance in predicting poor functional outcomes. In contrast, the model that combined imaging with tabular data, including physiological and laboratory results, demonstrated the best predictive performance for in-hospital mortality. These models exhibited high discriminative performance, with areas under the receiver operating curve (AUROCs) of 0.86 (95% CI: 0.79-0.92) and 0.91 (95% CI: 0.84-0.96) for poor functional outcomes and in-hospital mortality, respectively. Calibration was satisfactory for predicting poor functional outcomes, but requires refinement for mortality prediction. The models performed similar to or better than conventional risk scores, and DCA curves supported their clinical utility.

CONCLUSION

Multimodal prediction models have the potential to aid non-specialists in making informed decisions regarding ICH cases in emergency departments as part of clinical decision support systems. Enhancing real-world data infrastructure and improving model calibration are essential for successful implementation in clinical practice.

Pediatric Predictive Artificial Intelligence Implemented in Clinical Practice from 2010 to 2021: A Systematic Review

To review pediatric artificial intelligence (AI) implementation studies from 2010 to 2021 and analyze reported performance measures.We searched PubMed/Medline, Embase CINHAL, Cochrane Library CENTRAL, IEEE, and Web of Science with controlled vocabulary. Inclusion criteria: AI intervention in a pediatric clinical setting that learns from data (i.e., data-driven, as opposed to rule-based) and takes actions to make patient-specific recommendations; published between 01/2010 and 10/2021; must have agency (AI must provide guidance that affects clinical care, not merely running in the background). We extracted study characteristics, target users, implementation setting, time span, and performance measures.Of 126 articles reviewed as full text, 17 met inclusion criteria. Eight studies (47%) reported both clinical outcomes and process measures, six (35%) reported only process measures and two (12%) reported only clinical outcomes. Five studies (30%) reported no difference in clinical outcomes with AI, four (24%) reported improvement in clinical outcomes compared with controls, two (12%) reported positive effects on clinical outcomes with use of AI but had no formal comparison or controls, and one (6%) reported poor clinical outcomes with AI. Twelve studies (71%) reported improvement in process measures, while two (12%) reported no improvement. Five (30%) studies reported on at least 1 human performance measure.While there are many published pediatric AI models, the number of AI implementations is minimal with no standardized reporting of outcomes, care processes, or human performance measures. More comprehensive evaluations will help elucidate mechanisms of impact.

Machine Learning-Based Prediction Models for Healthcare Outcomes in Patients Participating in Cardiac Rehabilitation: A Systematic Review

PURPOSE

Cardiac rehabilitation (CR) has been proven to reduce mortality and morbidity in patients with cardiovascular disease. Machine learning (ML) techniques are increasingly used to predict healthcare outcomes in various fields of medicine including CR. This systemic review aims to perform critical appraisal of existing ML-based prognosis predictive model within CR and identify key research gaps in this area.

REVIEW METHODS

A systematic literature search was conducted in Scopus, PubMed, Web of Science, and Google Scholar from the inception of each database to January 28, 2024. The data extracted included clinical features, predicted outcomes, model development, and validation as well as model performance metrics. Included studies underwent quality assessments using the IJMEDI and Prediction Model Risk of Bias Assessment Tool checklist.

SUMMARY

A total of 22 ML-based clinical models from 7 studies across multiple phases of CR were included. Most models were developed using smaller patient cohorts from 41 to 227, with one exception involving 2280 patients. The prediction objectives ranged from patient intention to initiate CR to graduate from outpatient CR along with interval physiological and psychological progression in CR. The best-performing ML models reported area under the receiver operating characteristics curve between 0.82 and 0.91, with sensitivity from 0.77 to 0.95, indicating good prediction capabilities. However, none of them underwent calibration or external validation. Most studies raised concerns about bias. Readiness of these models for implementation into practice is questionable. External validation of existing models and development of new models with robust methodology based on larger populations and targeting diverse clinical outcomes in CR are needed.

Rigorous validation of machine learning in laboratory medicine: guidance toward quality improvement

The application of artificial intelligence (AI) in laboratory medicine will revolutionize predictive modeling using clinical laboratory information. Machine learning (ML), a sub-discipline of AI, involves fitting algorithms to datasets and is broadly used for data-driven predictive modeling in various disciplines. The majority of ML studies reported in systematic reviews lack key aspects of quality assurance. In clinical laboratory medicine, it is important to consider how differences in analytical methodologies, assay calibration, harmonization, pre-analytical errors, interferences, and physiological factors affecting measured analyte concentrations may also affect the downstream robustness and reliability of ML models. In this article, we address the need for quality improvement and proper validation of ML classification models, with the goal of bringing attention to key concepts pertinent to researchers, manuscript reviewers, and journal editors within the field of pathology and laboratory medicine. Several existing predictive modeling guidelines and recommendations can be readily adapted to the development of ML models in laboratory medicine. We summarize a basic overview of ML and key points from current guidelines including advantages and pitfalls of applied ML. In addition, we draw a parallel between validation of clinical assays and ML models in the context of current regulatory frameworks. The importance of classification performance metrics, model explainability, and data quality along with recommendations for strengthening journal submission requirements are also discussed. Although the focus of this article is on the application of ML in laboratory medicine, many of these concepts extend into other areas of medicine and biomedical science as well.

Individual risk and prognostic value prediction by interpretable machine learning for distant metastasis in neuroblastoma: A population-based study and an external validation

PURPOSE

Neuroblastoma (NB) is a childhood malignancy with a poor prognosis and a propensity for distant metastasis (DM). We aimed to establish machine learning (ML) based model to accurately predict risk of DM and prognosis of NB patients with DM.

METHODS

We analyzed NB patients from the Surveillance, Epidemiology, and End Results (SEER) database between 2000 and 2020. Univariate and multivariate logistic analysis were employed to select meaning variables. Recursive Feature Elimination (RFE) method based on 6 ML algorithms was utilized in feature selection. To construct predictive model, 13 ML algorithms were evaluated by area under the operating characteristic curve (AUC), accuracy, sensitivity, specificity, precision, cross-entropy, Brier scores, Balanced Accuracy and F-beta score. An optimal ML model was constructed to predict DM, and the predictive results were explained by SHapley Additive exPlanations (SHAP) framework. Meanwhile, 101 ML algorithm combinations were developed to select the best model with highest C-index to predict prognosis of NB patients with DM.

RESULTS

A total of 1,668 NB patients from SEER database was consecutively enrolled. We identified that tumor primary site, grade, surgery type, regional lymph nodes, radiotherapy and chemotherapy are significant risk factors for DM. CatBoost model was selected as the best prediction model, and AUC was 0.846 (95 %CI: [0.804,0.899]), 0.834 (95 %CI: [0.796,0.873]) and 0.813 (95 %CI: [0.776,0.852]) in training, internal test and external test sets, with 0.777 accuracy, 0.839 sensitivity, 0.72 specificity and 0.731 precision in training set. Grade, chemotherapy and radiotherapy had the greatest effects on DM according to SHAP results. For prognosis prediction, "RSF + GBM" algorithm was the best prognostic model with C-index of 0.656, 0.611 and 0.629 in training, internal test and external test sets.

CONCLUSIONS

Our ML models demonstrate excellent accuracy and reliability, offering more precise personalized metastasis diagnosis and prognostic prediction to NB patients.

Combining machine learning models and rule engines in clinical decision systems: Exploring optimal aggregation methods for vaccine hesitancy prediction

BACKGROUND

With the increasing application of artificial intelligence (AI) technologies in the healthcare sector and the emergence of new solutions, such as large language models, there is a growing need to combine medical knowledge, often expressed as clinical rules, with advances in machine learning (ML) offering higher prediction accuracy at the expense of decision-making transparency.

PURPOSE

This study investigates the efficacy of various aggregation methods combining the decisions of an AI model and a clinical rule-based (RB) engine in predicting vaccine hesitancy to maximize the effectiveness of patient incentive programs. This is the first study of parallel ensemble of rules and machine learning in clinical context proposing RB confidence-led fusion of ML and RB inference.

METHODS

A clinical decision system for predicting hesitation to vaccinate is developed based on a differentially private set of longitudinal health records of 974,000 US patients and clinical rules obtained from the present literature. Various approaches based on possibility theory have been explored to maximize classification accuracy, capture and hurdle rates while ensuring trustworthiness in clinical interventions.

RESULTS

Our findings reveal that the hybrid approach outperforms the individual models and RB systems when transparency and accuracy are critical. A RB confidence-led approach emerged as the most effective method. The aggregation of mismatched classes relies on RB results when the RB engine has high confidence (expressed as more than the median degree of membership to the vaccination hesitation output function) and on ML predictions when the RB engine exhibits lower confidence.

CONCLUSIONS

Implementing such an aggregation method preserves the accuracy and capture rates of a clinical decision system, while potentially improving acceptance among healthcare providers.

Time-dependent interpretable survival prediction model for second primary NSCLC patients

OBJECTIVE

Accurate predictive models for second primary non-small cell lung cancer (SP-NSCLC) are limited. This study aimed to develop and validate overall survival (OS) prediction models for SP-NSCLC patients using time-dependent interpretable survival machine learning algorithms.

METHODS

This study utilized data from the Surveillance, Epidemiology, and End Results (SEER) database, encompassing 8 and 12 registries, to extract data on patients aged 20-89 diagnosed with SP-NSCLC between 1988 and 2020. The dataset was divided into development, external temporal and spatial validation cohorts. Predictors included demographic, clinical, pathological and initial primary cancer-related features. Multiple survival machine learning algorithms were developed and validated, assessing model performance using C-index, time-dependent area under the receiver operating characteristic curve (time-AUC), and time-dependent Brier Score. The time-dependent interpretability analysis was employed to explore the time-dependent feature importance of key predictors.

RESULTS

The Blackboost model demonstrated excellent performance (C-index: 0.7517, and time-AUC: 0.8438), and good calibration (time-Brier Score of 0.0754). External validations and subgroup analyses demonstrated the robustness, generalizability, and fairness. Utilizing the optimal cutoff threshold, high-risk groups could be effectively identified. Surgery was the most critical predictor across the entire survival period. Combined stage (distant) and chemotherapy were the second most important predictors within 0 to 5 years, while age replaced from 5 to 20 years. Additionally, we developed an online visualization tool.

CONCLUSIONS

The Blackboost survival model achieved accurate, fair, and robust survival prediction for SP-NSCLC patients. Surgery, combined stage (distant), chemotherapy, and age contributed differently across various survival periods. The online visualization tool facilitated personalized survival predictions.

Predicting abnormal C-reactive protein level for improving utilization by deep neural network model

BACKGROUND

C-reactive protein (CRP) is an inflammatory biomarker frequently used in clinical practice. However, insufficient evidence-based ordering inevitably results in its overuse or underuse. This study aims to predict its normal and abnormal levels using the deep neural network (DNN) models, helping clinicians order this item more appropriately and intelligently.

METHODS

We considered complete blood count (CBC) parameters as feature vectors and 10 mg/L as a cutoff value for CRP. Several models, including linear support vector classification, logistic regression, decision trees, random forests, and DNN, were developed based on a dataset of 53834 medical records to predict binary output. We externally validated DNN models on independent 20723 samples through discrimination, calibration curve, and decision curve analysis.

RESULTS

DNN models has the best area under the receiver operating characteristic curves (AUC). Learning curves revealed that models' AUC, balanced accuracy, and F1 score do not significantly and continuously improve following increasing data volume. In internal validation, the AUC, balanced accuracy, and the F1 score of 10 models were 0.818 (0.95 CI: 0.812-0.824), 0.741 (0.95 CI: 0.736-0.747), and 0.649 (0.95 CI: 0.643-0.656), respectively. These metrics were 0.817 (0.95 CI: 0.816-0.817), 0.741 (0.95 CI: 0.740-0.742), and 0.641 (0.95 CI: 0.640-0.642), respectively, in external validation. AUC and balanced accuracy shown no significant difference (P-values were 0.106 and 0.339). CRP10-C2 model has the lowest Brier score of 0.154, AUC of 0.818, and calibration curve formula of y=1.001x-0.010, which was identified as a target model to deploy in the app.

CONCLUSIONS

DNN models obtained moderate performance, surpassing baseline indices in distinguishing binary CRP levels. They are good generalizations and well-calibrated. The CRP-C2 model can enhance CRP utilization by informing the orders appropriately and can contribute to inflammatory diagnostics in primary health care where CBC is available, but the CRP test is inaccessible.

Active learning for extracting rare adverse events from electronic health records: A study in pediatric cardiology

OBJECTIVE

Automate the extraction of adverse events from the text of electronic medical records of patients hospitalized for cardiac catheterization.

METHODS

We focused on events related to cardiac catheterization as defined by the NCDR-IMPACT registry. These events were extracted from the Necker Children's Hospital data warehouse. Electronic health records were pre-screened using regular expressions. The resulting datasets contained numerous false positives sentences that were annotated by a cardiologist using an active learning process. A deep learning text classifier was then trained on this active learning-annotated dataset to accurately identify patients who have suffered a serious adverse event.

RESULTS

The dataset included 2,980 patients. Regular expression based extraction of adverse events related to cardiac catheterization achieved a perfect recall. Due to the rarity of adverse events, the dataset obtained from this initial pre-screening step was imbalanced, containing a significant number of false positives. The active learning annotation enabled the acquisition of a representative dataset suitable for training a deep learning model. The deep learning text-classifier identified patients who underwent adverse events after cardiac catheterization with a recall of 0.78 and a specificity of 0.94.

CONCLUSION

Our model effectively identified patients who experienced adverse events related to cardiac catheterization using real clinical data. Enabled by an active learning annotation process, it shows promise for large language model applications in clinical research, especially for rare diseases with limited annotated databases. Our model's strength lies in its development by physicians for physicians, ensuring its relevance and applicability in clinical practice.

Complete Blood Count and Monocyte Distribution Width-Based Machine Learning Algorithms for Sepsis Detection: Multicentric Development and External Validation Study

BACKGROUND

Sepsis is an organ dysfunction caused by a dysregulated host response to infection. Early detection is fundamental to improving the patient outcome. Laboratory medicine can play a crucial role by providing biomarkers whose alteration can be detected before the onset of clinical signs and symptoms. In particular, the relevance of monocyte distribution width (MDW) as a sepsis biomarker has emerged in the previous decade. However, despite encouraging results, MDW has poor sensitivity and positive predictive value when compared to other biomarkers.

OBJECTIVE

This study aims to investigate the use of machine learning (ML) to overcome the limitations mentioned earlier by combining different parameters and therefore improving sepsis detection. However, making ML models function in clinical practice may be problematic, as their performance may suffer when deployed in contexts other than the research environment. In fact, even widely used commercially available models have been demonstrated to generalize poorly in out-of-distribution scenarios.

METHODS

In this multicentric study, we developed ML models whose intended use is the early detection of sepsis on the basis of MDW and complete blood count parameters. In total, data from 6 patient cohorts (encompassing 5344 patients) collected at 5 different Italian hospitals were used to train and externally validate ML models. The models were trained on a patient cohort encompassing patients enrolled at the emergency department, and it was externally validated on 5 different cohorts encompassing patients enrolled at both the emergency department and the intensive care unit. The cohorts were selected to exhibit a variety of data distribution shifts compared to the training set, including label, covariate, and missing data shifts, enabling a conservative validation of the developed models. To improve generalizability and robustness to different types of distribution shifts, the developed ML models combine traditional methodologies with advanced techniques inspired by controllable artificial intelligence (AI), namely cautious classification, which gives the ML models the ability to abstain from making predictions, and explainable AI, which provides health operators with useful information about the models' functioning.

RESULTS

The developed models achieved good performance on the internal validation (area under the receiver operating characteristic curve between 0.91 and 0.98), as well as consistent generalization performance across the external validation datasets (area under the receiver operating characteristic curve between 0.75 and 0.95), outperforming baseline biomarkers and state-of-the-art ML models for sepsis detection. Controllable AI techniques were further able to improve performance and were used to derive an interpretable set of diagnostic rules.

CONCLUSIONS

Our findings demonstrate how controllable AI approaches based on complete blood count and MDW may be used for the early detection of sepsis while also demonstrating how the proposed methodology can be used to develop ML models that are more resistant to different types of data distribution shifts.

Next-generation biomarkers for alcohol consumption and alcohol use disorder diagnosis, prognosis, and treatment: A critical review

This critical review summarizes the current state of omics-based biomarkers in the alcohol research field. We first provide definitions and background information on alcohol and alcohol use disorder (AUD), biomarkers, and "omic" technologies. We next summarize using (1) genetic information as risk/prognostic biomarkers for the onset of alcohol-related problems and the progression from regular drinking to problematic drinking (including AUD), (2) epigenetic information as diagnostic biomarkers for AUD and risk biomarkers for alcohol consumption, (3) transcriptomic information as diagnostic biomarkers for AUD, risk biomarkers for alcohol consumption, and (4) metabolomic information as diagnostic biomarkers for AUD, risk biomarkers for alcohol consumption, and predictive biomarkers for response to acamprosate in subjects with AUD. In the final section, the clinical implications of the findings are discussed, and recommendations are made for future research.

Machine learning models to further identify advantaged populations that can achieve functional cure of chronic hepatitis B virus infection after receiving Peg-IFN alpha treatment

OBJECTIVE

Functional cure is currently the highest goal of hepatitis B virus(HBV) treatment.Pegylated interferon(Peg-IFN) alpha is an important drug for this purpose,but even in the hepatitis B e antigen(HBeAg)-negative population,there is still a portion of the population respond poorly to it.Therefore,it is important to explore the influencing factors affecting the response rate of Peg-IFN alpha and establish a prediction model to further identify advantaged populations.

METHODS

We retrospectively analyzed 382 patients.297 patients were in the training set and 85 patients from another hospital were in the test set.The intersect features were extracted from all variables using the recursive feature elimination(RFE) algorithm, Boruta algorithm, and least absolute shrinkage and selection operator(LASSO) regression algorithm in the training dataset.Then,we employed six machine learning(ML) algorithms-Logistic Regression(LR),Random Forest(RF),Support Vector Machines(SVM),K Nearest Neighbors(KNN),Light Gradient Boosting Machine(LightGBM) and Extreme Gradient Boosting(XGBoost)-to develop the model.Internal 10-fold cross-validation helped determine the best-performing model,which was then tested externally.Model performance was assessed using metrics such as area under the curve(AUC) and other metrics.SHapley Additive exPlanations(SHAP) plots were used to interpret variable significance.

RESULTS

138/382(36.13 %) patients achieved functional cure.HBsAg at baseline,HBsAg decline at week12,non-alcoholic fatty liver disease(NAFLD) and age were identified as significant variables.RF performed the best,with AUC value of 0.988,and maintained good performance in test set.The SHapley Additive exPlanations(SHAP) plot highlighted HBsAg at baseline and HBsAg decline at week 12 are the top two predictors.The web-calculator was designed to predict functional cure more conveniently(https://www.xsmartanalysis.com/model/list/predict/model/html?mid = 17054&symbol = 317ad245Hx628ko3uW51).

CONCLUSION

We developed a prediction model,which can be used to not only accurately identifies advantageous populations with Peg-IFN alpha,but also determines whether to continue subsequent Peg-IFN alpha.

Eight quick tips for biologically and medically informed machine learning

Machine learning has become a powerful tool for computational analysis in the biomedical sciences, with its effectiveness significantly enhanced by integrating domain-specific knowledge. This integration has give rise to informed machine learning, in contrast to studies that lack domain knowledge and treat all variables equally (uninformed machine learning). While the application of informed machine learning to bioinformatics and health informatics datasets has become more seamless, the likelihood of errors has also increased. To address this drawback, we present eight guidelines outlining best practices for employing informed machine learning methods in biomedical sciences. These quick tips offer recommendations on various aspects of informed machine learning analysis, aiming to assist researchers in generating more robust, explainable, and dependable results. Even if we originally crafted these eight simple suggestions for novices, we believe they are deemed relevant for expert computational researchers as well.

Development and validation of interpretable machine learning models to predict glomerular filtration rate in chronic kidney disease Colombian patients

BACKGROUND

ML predictive models have shown their capability to improve risk prediction and assist medical decision-making, nevertheless, there is a lack of accuracy systems to early identify future rapid CKD progressors in Colombia and even in South America.

OBJECTIVE

The purpose of this study was to develop a series of interpretable machine learning models that predict GFR at 6-months, 9-months, and 12-months.

STUDY DESIGN AND SETTING

Over 29,000 CKD patients stage 1 to 3b (estimated GFR, <60 mL/min/1.73 m2) with an average of 3-year follow-up data were included. We used the machine learning extreme gradient boosting (XGBoost) to build three models to predict the next eGFR. Models were internally and externally validated. In addition, we included SHapley Additive exPlanation (SHAP) values to offer interpretable global and local prediction models.

RESULTS

All models showed a good performance in development and external validation. However, the 6-months XGBoost prediction model showed the best performance in internal (MAE average = 6.07; RSME = 78.87), and in external validation (MAE average = 6.45, RSME = 18.94). The top 3 most influential features that pushed the predicted eGFR value to lower values were the interpolated values for eGFR and creatinine, and eGFR at baseline.

CONCLUSION

In the current study we have developed and validated machine learning models to predict the next eGFR value at different intervals. Furthermore, we attempted to approach the need for prediction explanation by offering transparent predictions.

Machine learning-based prediction of sarcopenia in community-dwelling middle-aged and older adults: findings from the CHARLS

BACKGROUND

Sarcopenia is a prominent issue among aging populations and associated with poor health outcomes. This study aimed to examine the predictive value of questionnaire and biomarker data for sarcopenia, and to further develop a user-friendly calculator for community-dwelling middle-aged and older adults.

METHODS

We used two waves (2011 and 2013) of the China Health and Retirement Longitudinal Study (CHARLS) to predict sarcopenia, defined by the Asian Working Group for Sarcopenia 2019 criteria. We restricted the analytical sample to adults aged 45 or above (N = 2934). Five machine learning models were used to construct Q-based (only questionnaire variables), Bio-based (only biomarker variables), and combined (questionnaire plus biomarker variables) models. Area under the receiver operating characteristic curve (AUROC) was used for performance assessment. Temporal external validation was performed based on two datasets from CHARLS. Important predictors were identified by Shapley values and coefficients.

RESULTS

Extreme gradient boosting (XGBoost), considering both questionnaire and biomarker characteristics, emerged as the optimal model, and its AUROC was 0.759 (95% CI: 0.747-0.771) at a decision threshold of 0.20 on the test set. Models also performed well on the external datasets. We found that cognitive function was the most important predictor in both Q-based and combined models, and blood urea nitrogen was the most important predictor in the Bio-based model. Other key predictors included education, haematocrit, total cholesterol, drinking, number of chronic diseases, and instrumental activities of daily living score.

CONCLUSIONS

Our findings offer a potential for early screening and targeted prevention of sarcopenia among middle-aged and older adults in the community setting.

Accuracy of machine learning in diagnosing microsatellite instability in gastric cancer: A systematic review and meta-analysis

BACKGROUND

Significant challenges persist in the early identification of microsatellite instability (MSI) within current clinical practice. In recent years, with the growing utilization of machine learning (ML) in the diagnosis and management of gastric cancer (GC), numerous researchers have explored the effectiveness of ML methodologies in detecting MSI. Nevertheless, the predictive value of these approaches still lacks comprehensive evidence. Accordingly, this study was carried out to consolidate the accuracy of ML in the prompt detection of MSI in GC.

METHODS

PubMed, the Cochrane Library, the Web of Science, and Embase were retrieved up to March 20, 2024. The risk of bias in the encompassed studies was evaluated utilizing a risk assessment tool for predictive models. Models were then subjected to subgroup analysis based on the modeling variables.

RESULTS

A total of 12 studies, encompassing 11,912 patients with GC, satisfied the predefined inclusion criteria. ML models established in these studies were primarily based on pathological images, clinical features, and radiomics. The results suggested that in the validation sets, the pathological image-based models had a synthesized c-index of 0.86 [95 % CI (0.83-0.89)], with sensitivity and specificity being 0.86 [95 % CI (0.76-0.92)] and 0.83 [95 % CI (0.78-0.87)], respectively; radiomics feature-based models achieved respective values of 0.87 [95 % CI (0.81-0.92)], 0.77 [95 % CI (0.70-0.83)] and 0.81 [95 % CI (0.74-0.87)]; radiomics feature-based models + clinical feature-based models achieved respective values of 0.87 [95 % CI (0.81-0.93)], 0.78 [95 % CI (0.70-0.84)] and 0.79 [95 % CI (0.69-0.86)].

CONCLUSIONS

ML has demonstrated optimal performance in detecting MSI in GC and could serve as a prospective early adjunctive detection tool for MSI in GC. Future research should contemplate minimally invasive or non-invasive, readily collectible, and efficient predictors to augment the predictive accuracy of ML.

hvEEGNet: a novel deep learning model for high-fidelity EEG reconstruction

Introduction

Modeling multi-channel electroencephalographic (EEG) time-series is a challenging tasks, even for the most recent deep learning approaches. Particularly, in this work, we targeted our efforts to the high-fidelity reconstruction of this type of data, as this is of key relevance for several applications such as classification, anomaly detection, automatic labeling, and brain-computer interfaces.

Methods

We analyzed the most recent works finding that high-fidelity reconstruction is seriously challenged by the complex dynamics of the EEG signals and the large inter-subject variability. So far, previous works provided good results in either high-fidelity reconstruction of single-channel signals, or poor-quality reconstruction of multi-channel datasets. Therefore, in this paper, we present a novel deep learning model, called hvEEGNet, designed as a hierarchical variational autoencoder and trained with a new loss function. We tested it on the benchmark Dataset 2a (including 22-channel EEG data from 9 subjects).

Results

We show that it is able to reconstruct all EEG channels with high-fidelity, fastly (in a few tens of epochs), and with high consistency across different subjects. We also investigated the relationship between reconstruction fidelity and the training duration and, using hvEEGNet as an anomaly detector, we spotted some data in the benchmark dataset that are corrupted and never highlighted before.

Discussion

Thus, hvEEGNet could be very useful in several applications where automatic labeling of large EEG dataset is needed and time-consuming. At the same time, this work opens new fundamental research questions about (1) the effectiveness of deep learning models training (for EEG data) and (2) the need for a systematic characterization of the input EEG data to ensure robust modeling.

Machine learning to predict adverse drug events based on electronic health records: a systematic review and meta-analysis

OBJECTIVE

This systematic review aimed to provide a comprehensive overview of the application of machine learning (ML) in predicting multiple adverse drug events (ADEs) using electronic health record (EHR) data.

METHODS

Systematic searches were conducted using PubMed, Web of Science, Embase, and IEEE Xplore from database inception until 21 November 2023. Studies that developed ML models for predicting multiple ADEs based on EHR data were included.

RESULTS

Ten studies met the inclusion criteria. Twenty ML methods were reported, most commonly random forest (RF, n = 9), followed by AdaBoost (n = 4), eXtreme Gradient Boosting (n = 3), and support vector machine (n = 3). The mean area under the summary receiver operator characteristics curve (AUC) was 0.76 (95% confidence interval [CI] = 0.26-0.95). RF combined with resampling-based approaches achieved high AUCs (0.9448-0.9457). The common risk factors of ADEs included the length of hospital stay, number of prescribed drugs, and admission type. The pooled estimated AUC was 0.72 (95% CI = 0.68-0.75).

CONCLUSIONS

Future studies should adhere to more rigorous reporting standards and consider new ML methods to facilitate the application of ML models in clinical practice.

Machine learning for predicting duration of surgery and length of stay: A literature review on joint arthroplasty

INTRODUCTION

In recent years, different factors such as population aging have caused escalating demand for hip and knee arthroplasty straining already limited hospitals' resources. To address this challenge, focus is put on medical and operational efficiency improvements. This includes an increased use of machine learning (ML) to predict duration of surgery (DOS) and length of stay (LOS) for total knee and total hip arthroplasty, which can be utilized for optimizing resource allocation to satisfy medical and operational limitations. This paper explores the development and performance of ML models in predicting DOS and LOS.

METHODS

A systematic search of publications between 2010-2023 was conducted following PRISMA guidelines. Considering the inclusion and exclusion criteria, 28 out of 722 gathered papers from PubMed, Web of Science, and manual search were included in the study. Descriptive statistics was used to analyze the extracted data regarding data preprocessing, model development, and model performance assessment.

RESULTS

Most of the papers work on LOS as a binary variable. Patient's age was identified as the most frequently used and reported as important variable for predicting DOS and LOS. Investigations also illustrated that within the resulting 28 papers, more than 71% of models reached good to perfect performance based on the area under the receiver operating characteristic curve (AUC), where artificial neural networks and ensemble learning models had the biggest share among the best-performing models.

CONCLUSION

The utilization of ML models is increasing in the literature. The current performance level indicates that ML can potentially turn to powerful tools in predicting DOS and LOS for different purposes. Meanwhile, the literature is not matured yet in reporting real-life application. Future studies can focus on model specification and validation by considering empirical application.

Inflammatory bowel disease genomics, transcriptomics, proteomics and metagenomics meet artificial intelligence

Various extrinsic and intrinsic factors such as drug exposures, antibiotic treatments, smoking, lifestyle, genetics, immune responses, and the gut microbiome characterize ulcerative colitis and Crohn's disease, collectively called inflammatory bowel disease (IBD). All these factors contribute to the complexity and heterogeneity of the disease etiology and pathogenesis leading to major challenges for the scientific community in improving management, medical treatments, genetic risk, and exposome impact. Understanding the interaction(s) among these factors and their effects on the immune system in IBD patients has prompted advances in multi-omics research, the development of new tools as part of system biology, and more recently, artificial intelligence (AI) approaches. These innovative approaches, supported by the availability of big data and large volumes of digital medical datasets, hold promise in better understanding the natural histories, predictors of disease development, severity, complications and treatment outcomes in complex diseases, providing decision support to doctors, and promising to bring us closer to the realization of the "precision medicine" paradigm. This review aims to provide an overview of current IBD omics based on both individual (genomics, transcriptomics, proteomics, metagenomics) and multi-omics levels, highlighting how AI can facilitate the integration of heterogeneous data to summarize our current understanding of the disease and to identify current gaps in knowledge to inform upcoming research in this field.

Moving Toward Meaningful Evaluations of Monitoring in e-Mental Health Based on the Case of a Web-Based Grief Service for Older Mourners: Mixed Methods Study

Background

Artificial intelligence (AI) tools hold much promise for mental health care by increasing the scalability and accessibility of care. However, current development and evaluation practices of AI tools limit their meaningfulness for health care contexts and therefore also the practical usefulness of such tools for professionals and clients alike.

Objective

The aim of this study is to demonstrate the evaluation of an AI monitoring tool that detects the need for more intensive care in a web-based grief intervention for older mourners who have lost their spouse, with the goal of moving toward meaningful evaluation of AI tools in e-mental health.

Methods

We leveraged the insights from three evaluation approaches: (1) the F1-score evaluated the tool's capacity to classify user monitoring parameters as either in need of more intensive support or recommendable to continue using the web-based grief intervention as is; (2) we used linear regression to assess the predictive value of users' monitoring parameters for clinical changes in grief, depression, and loneliness over the course of a 10-week intervention; and (3) we collected qualitative experience data from e-coaches (N=4) who incorporated the monitoring in their weekly email guidance during the 10-week intervention.

Results

Based on n=174 binary recommendation decisions, the F1-score of the monitoring tool was 0.91. Due to minimal change in depression and loneliness scores after the 10-week intervention, only 1 linear regression was conducted. The difference score in grief before and after the intervention was included as a dependent variable. Participants' (N=21) mean score on the self-report monitoring and the estimated slope of individually fitted growth curves and its standard error (ie, participants' response pattern to the monitoring questions) were used as predictors. Only the mean monitoring score exhibited predictive value for the observed change in grief (R2=1.19, SE 0.33; t16=3.58, P=.002). The e-coaches appreciated the monitoring tool as an opportunity to confirm their initial impression about intervention participants, personalize their email guidance, and detect when participants' mental health deteriorated during the intervention.

Conclusions

The monitoring tool evaluated in this paper identified a need for more intensive support reasonably well in a nonclinical sample of older mourners, had some predictive value for the change in grief symptoms during a 10-week intervention, and was appreciated as an additional source of mental health information by e-coaches who supported mourners during the intervention. Each evaluation approach in this paper came with its own set of limitations, including (1) skewed class distributions in prediction tasks based on real-life health data and (2) choosing meaningful statistical analyses based on clinical trial designs that are not targeted at evaluating AI tools. However, combining multiple evaluation methods facilitates drawing meaningful conclusions about the clinical value of AI monitoring tools for their intended mental health context.

Comparative analysis of BERT-based and generative large language models for detecting suicidal ideation: a performance evaluation study

Artificial intelligence can detect suicidal ideation manifestations in texts. Studies demonstrate that BERT-based models achieve better performance in text classification problems. Large language models (LLMs) answer free-text queries without being specifically trained. This work aims to compare the performance of three variations of BERT models and LLMs (Google Bard, Microsoft Bing/GPT-4, and OpenAI ChatGPT-3.5) for identifying suicidal ideation from nonclinical texts written in Brazilian Portuguese. A dataset labeled by psychologists consisted of 2,691 sentences without suicidal ideation and 1,097 with suicidal ideation, of which 100 sentences were selected for testing. We applied data preprocessing techniques, hyperparameter optimization, and hold-out cross-validation for training and testing BERT models. When evaluating LLMs, we used zero-shot prompting engineering. Each test sentence was labeled if it contained suicidal ideation, according to the chatbot's response. Bing/GPT-4 achieved the best performance, with 98% across all metrics. Fine-tuned BERT models outperformed the other LLMs: BERTimbau-Large performed the best with a 96% accuracy, followed by BERTimbau-Base with 94%, and BERT-Multilingual with 87%. Bard performed the worst with 62% accuracy, whereas ChatGPT-3.5 achieved 81%. The high recall capacity of the models suggests a low misclassification rate of at-risk patients, which is crucial to prevent missed interventions by professionals. However, despite their potential in supporting suicidal ideation detection, these models have not been validated in a patient monitoring clinical setting. Therefore, caution is advised when using the evaluated models as tools to assist healthcare professionals in detecting suicidal ideation.

AI-Based Noninvasive Blood Glucose Monitoring: Scoping Review

BACKGROUND

Current blood glucose monitoring (BGM) methods are often invasive and require repetitive pricking of a finger to obtain blood samples, predisposing individuals to pain, discomfort, and infection. Noninvasive blood glucose monitoring (NIBGM) is ideal for minimizing discomfort, reducing the risk of infection, and increasing convenience.

OBJECTIVE

This review aimed to map the use cases of artificial intelligence (AI) in NIBGM.

METHODS

A systematic scoping review was conducted according to the Arksey O'Malley five-step framework. Eight electronic databases (CINAHL, Embase, PubMed, Web of Science, Scopus, The Cochrane-Central Library, ACM Digital Library, and IEEE Xplore) were searched from inception until February 8, 2023. Study selection was conducted by 2 independent reviewers, descriptive analysis was conducted, and findings were presented narratively. Study characteristics (author, country, type of publication, study design, population characteristics, mean age, types of noninvasive techniques used, and application, as well as characteristics of the BGM systems) were extracted independently and cross-checked by 2 investigators. Methodological quality appraisal was conducted using the Checklist for assessment of medical AI.

RESULTS

A total of 33 papers were included, representing studies from Asia, the United States, Europe, the Middle East, and Africa published between 2005 and 2023. Most studies used optical techniques (n=19, 58%) to estimate blood glucose levels (n=27, 82%). Others used electrochemical sensors (n=4), imaging (n=2), mixed techniques (n=2), and tissue impedance (n=1). Accuracy ranged from 35.56% to 94.23% and Clarke error grid (A+B) ranged from 86.91% to 100%. The most popular machine learning algorithm used was random forest (n=10) and the most popular deep learning model was the artificial neural network (n=6). The mean overall checklist for assessment of medical AI score on the included papers was 33.5 (SD 3.09), suggesting an average of medium quality. The studies reviewed demonstrate that some AI techniques can accurately predict glucose levels from noninvasive sources while enhancing comfort and ease of use for patients. However, the overall range of accuracy was wide due to the heterogeneity of models and input data.

CONCLUSIONS

Efforts are needed to standardize and regulate the use of AI technologies in BGM, as well as develop consensus guidelines and protocols to ensure the quality and safety of AI-assisted monitoring systems. The use of AI for NIBGM is a promising area of research that has the potential to revolutionize diabetes management.

Predicting adverse drug event using machine learning based on electronic health records: a systematic review and meta-analysis

Introduction

Adverse drug events (ADEs) pose a significant challenge in current clinical practice. Machine learning (ML) has been increasingly used to predict specific ADEs using electronic health record (EHR) data. This systematic review provides a comprehensive overview of the application of ML in predicting specific ADEs based on EHR data.

Methods

A systematic search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted to identify relevant articles published from the inception to 20 May 2024. Studies that developed ML models for predicting specific ADEs or ADEs associated with particular drugs were included using EHR data.

Results

A total of 59 studies met the inclusion criteria, covering 15 drugs and 15 ADEs. In total, 38 machine learning algorithms were reported, with random forest (RF) being the most frequently used, followed by support vector machine (SVM), eXtreme gradient boosting (XGBoost), decision tree (DT), and light gradient boosting machine (LightGBM). The performance of the ML models was generally strong, with an average area under the curve (AUC) of 76.68% ± 10.73, accuracy of 76.00% ± 11.26, precision of 60.13% ± 24.81, sensitivity of 62.35% ± 20.19, specificity of 75.13% ± 16.60, and an F1 score of 52.60% ± 21.10. The combined sensitivity, specificity, diagnostic odds ratio (DOR), and AUC from the summary receiver operating characteristic (SROC) curve using a random effects model were 0.65 (95% CI: 0.65-0.66), 0.89 (95% CI: 0.89-0.90), 12.11 (95% CI: 8.17-17.95), and 0.8069, respectively. The risk factors associated with different drugs and ADEs varied.

Discussion

Future research should focus on improving standardization, conducting multicenter studies that incorporate diverse data types, and evaluating the impact of artificial intelligence predictive models in real-world clinical settings.

Systematic Review Registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42024565842, identifier CRD42024565842.

Exploring the Trade-Off between generalist and specialized Models: A center-based comparative analysis for glioblastoma segmentation

INTRODUCTION

Inherent variations between inter-center data can undermine the robustness of segmentation models when applied at a specific center (dataset shift). We investigated whether specialized center-specific models are more effective compared to generalist models based on multi-center data, and how center-specific data could enhance the performance of generalist models within a particular center using a fine-tuning transfer learning approach. For this purpose, we studied the dataset shift at center level and conducted a comparative analysis to assess the impact of data source on glioblastoma segmentation models.

METHODS & MATERIALS

The three key components of dataset shift were studied: prior probability shift-variations in tumor size or tissue distribution among centers; covariate shift-inter-center MRI alterations; and concept shift-different criteria for tumor segmentation. BraTS 2021 dataset was used, which includes 1251 cases from 23 centers. Thereafter, 155 deep-learning models were developed and compared, including 1) generalist models trained with multi-center data, 2) specialized models using only center-specific data, and 3) fine-tuned generalist models using center-specific data.

RESULTS

The three key components of dataset shift were characterized. The amount of covariate shift was substantial, indicating large variations in MR imaging between different centers. Glioblastoma segmentation models tend to perform best when using data from the application center. Generalist models, trained with over 700 samples, achieved a median Dice score of 88.98%. Specialized models surpassed this with 200 cases, while fine-tuned models outperformed with 50 cases.

CONCLUSIONS

The influence of dataset shift on model performance is evident. Fine-tuned and specialized models, utilizing data from the evaluated center, outperform generalist models, which rely on data from other centers. These approaches could encourage medical centers to develop customized models for their local use, enhancing the accuracy and reliability of glioblastoma segmentation in a context where dataset shift is inevitable.

Examining individual and contextual predictors of disability in Chinese older adults: A machine learning approach

BACKGROUND

There is a large gap of understanding the determinants of disability, especially the contextual characteristics. Therefore, this study aimed to examine the important predictors of disability in Chinese older adults based on the social ecological framework.

METHODS

We used the China Health and Retirement Longitudinal Study to examine predictors of disability, defined as self-report of any difficulty for six activity of daily living items. We restricted analytical sample to older adults aged 65 or above (N=1816). We considered 44 predictors, including personal-, behavioral-, interpersonal-, community-, and policy-level characteristics. The built-in variable importance measure (VIM) of random forest and SHapley Additive exPlanations (SHAP) were applied to assess key predictors of disability. A multilevel logit regression was further used to examine the associations of individual and contextual characteristics with disability.

RESULTS

The mean age of study sample was 72.62 years old (standard deviation: 5.77). During a 2-year of follow-up, 518 (28.52 %) of them developed into disability. Walking speed, age, and peak expiratory flow were the top important predictors in both VIM and SHAP. Contextual characteristics such as humidity, PM2.5, temperature, normalized difference vegetation index, and landscape also showed promise in predicting disability. Multilevel logit regression showed that people with male gender, arthritis, vision impairment, unable to finish semi tandem, no social activity, lower grip strength, and higher waist circumference, had much higher risk of disability.

CONCLUSION

Disability prevention strategies should specifically focus on multilevel factors such as individual and contextual characteristics, although the latter is warranted to be verified in future studies.

Core Concepts in Pharmacoepidemiology: Principled Use of Artificial Intelligence and Machine Learning in Pharmacoepidemiology and Healthcare Research

Artificial intelligence (AI) and machine learning (ML) are important tools across many fields of health and medical research. Pharmacoepidemiologists can bring essential methodological rigor and study design expertise to the design and use of these technologies within healthcare settings. AI/ML-based tools also play a role in pharmacoepidemiology research, as we may apply them to answer our own research questions, take responsibility for evaluating medical devices with AI/ML components, or participate in interdisciplinary research to create new AI/ML algorithms. While epidemiologic expertise is essential to deploying AI/ML responsibly and ethically, the rapid advancement of these technologies in the past decade has resulted in a knowledge gap for many in the field. This article provides a brief overview of core AI/ML concepts, followed by a discussion of potential applications of AI/ML in pharmacoepidemiology research, and closes with a review of important concepts across application areas, including interpretability and fairness. This review is intended to provide an accessible, practical overview of AI/ML for pharmacoepidemiology research, with references to further, more detailed resources on fundamental topics.

Ability of clinical data to predict readmission in Child and Adolescent Mental Health Services

This study addresses the challenge of predicting readmissions in Child and Adolescent Mental Health Services (CAMHS) by analyzing the predictability of readmissions over short, medium, and long term periods. Using health records spanning 35 years, which included 22,643 patients and 30,938 episodes of care, we focused on the episode of care as a central unit, defined as a referral-discharge cycle that incorporates assessments and interventions. Data pre-processing involved handling missing values, normalizing, and transforming data, while resolving issues related to overlapping episodes and correcting registration errors where possible. Readmission prediction was inferred from electronic health records (EHR), as this variable was not directly recorded. A binary classifier distinguished between readmitted and non-readmitted patients, followed by a multi-class classifier to categorize readmissions based on timeframes: short (within 6 months), medium (6 months - 2 years), and long (more than 2 years). Several predictive models were evaluated based on metrics like AUC, F1-score, precision, and recall, and the K-prototype algorithm was employed to explore similarities between episodes through clustering. The optimal binary classifier (Oversampled Gradient Boosting) achieved an AUC of 0.7005, while the multi-class classifier (Oversampled Random Forest) reached an AUC of 0.6368. The K-prototype resulted in three clusters as optimal (SI: 0.256, CI: 4473.64). Despite identifying relationships between care intensity, case complexity, and readmission risk, generalizing these findings proved difficult, partly because clinicians often avoid discharging patients likely to be readmitted. Overall, while this dataset offers insights into patient care and service patterns, predicting readmissions remains challenging, suggesting a need for improved analytical models that consider patient development, disease progression, and intervention effects.

Machine Learning for Treatment Management Prediction in Laryngeal Fractures

OBJECTIVES

Laryngeal fractures are rare but potentially life-threatening traumas. Complications, such as airway obstruction and disrupted laryngeal anatomy, associate with significant morbidity. Early identification of at-risk patients and optimal management remain crucial for improved outcomes. Recently, machine learning (ML) has attained great attention as a unique and novel technique for evaluating complex nonlinear relationships between multiple observations to create a predictive model with greater accuracy. This study aimed to demonstrate the potential of ML in predicting airway and surgical management of laryngeal fracture patients and identify key contributing parameters for the predictive performance of the ML models.

METHODS

The ML models were developed using a patient series managed at the Helsinki University Hospital during 2005-2019. The developed models were further evaluated independently using a different cohort collected from the same institution between 1995 and 2004.

RESULTS

The ML showed a weighted area under curve (AUC) of 0.93 and accuracy of 0.86 following training for airway management. Likewise, for treatment approach, weighted AUC was 0.85 and accuracy 0.78. Injury type, Schaefer-Fuhrman grade (SF gr), age at incident, cause of injury, and fracture of the cricoid, in decreasing order of significance, were the most prominent features for the model's predictive performance for airway management. Similarly, our model identified SF gr, fracture of the cricoid, injury type, age at incident, and cause of injury as the most significant predictors for surgical treatment approach.

CONCLUSIONS

The proposed prediction of management approach by an ML technique can provide accurate predictions and thus aid clinicians in administering early and personalized interventions. The model may serve as a supporting tool in recognizing at-risk patients and in timely decision-making. Further independent external validation is warranted for model generalizability.

Meta-research on reporting guidelines for artificial intelligence: are authors and reviewers encouraged enough in radiology, nuclear medicine, and medical imaging journals?

PURPOSE

To determine how radiology, nuclear medicine, and medical imaging journals encourage and mandate the use of reporting guidelines for artificial intelligence (AI) in their author and reviewer instructions.

METHODS

The primary source of journal information and associated citation data used was the Journal Citation Reports (June 2023 release for 2022 citation data; Clarivate Analytics, UK). The first- and second-quartile journals indexed in the Science Citation Index Expanded and the Emerging Sources Citation Index were included. The author and reviewer instructions were evaluated by two independent readers, followed by an additional reader for consensus, with the assistance of automatic annotation. Encouragement and submission requirements were systematically analyzed. The reporting guidelines were grouped as AI-specific, related to modeling, and unrelated to modeling.

RESULTS

Out of 102 journals, 98 were included in this study, and all of them had author instructions. Only five journals (5%) encouraged the authors to follow AI-specific reporting guidelines. Among these, three required a filled-out checklist. Reviewer instructions were found in 16 journals (16%), among which one journal (6%) encouraged the reviewers to follow AI-specific reporting guidelines without submission requirements. The proportions of author and reviewer encouragement for AI-specific reporting guidelines were statistically significantly lower compared with those for other types of guidelines (P < 0.05 for all).

CONCLUSION

The findings indicate that AI-specific guidelines are not commonly encouraged and mandated (i.e., requiring a filled-out checklist) by these journals, compared with guidelines related to modeling and unrelated to modeling, leaving vast space for improvement. This meta-research study hopes to contribute to the awareness of the imaging community for AI reporting guidelines and ignite large-scale group efforts by all stakeholders, making AI research less wasteful.

CLINICAL SIGNIFICANCE

This meta-research highlights the need for improved encouragement of AI-specific guidelines in radiology, nuclear medicine, and medical imaging journals. This can potentially foster greater awareness among the AI community and motivate various stakeholders to collaborate to promote more efficient and responsible AI research reporting practices.

Ten quick tips for clinical electroencephalographic (EEG) data acquisition and signal processing

Electroencephalography (EEG) is a medical engineering technique aimed at recording the electric activity of the human brain. Brain signals derived from an EEG device can be processed and analyzed through computers by using digital signal processing, computational statistics, and machine learning techniques, that can lead to scientifically-relevant results and outcomes about how the brain works. In the last decades, the spread of EEG devices and the higher availability of EEG data, of computational resources, and of software packages for electroencephalography analysis has made EEG signal processing easier and faster to perform for any researcher worldwide. This increased ease to carry out computational analyses of EEG data, however, has made it easier to make mistakes, as well. And these mistakes, if unnoticed or treated wrongly, can in turn lead to wrong results or misleading outcomes, with worrisome consequences for patients and for the advancements of the knowledge about human brain. To tackle this problem, we present here our ten quick tips to perform electroencephalography signal processing analyses avoiding common mistakes: a short list of guidelines designed for beginners on what to do, how to do it, and what not to do when analyzing EEG data with a computer. We believe that following our quick recommendations can lead to better, more reliable and more robust results and outcome in clinical neuroscientific research.

Predictive analysis on the factors associated with birth Outcomes: A machine learning perspective

BACKGROUND

Recent studies reveal that around 1.9 million stillbirths occur annually worldwide, with Sub-Saharan Africa having among the highest cases. Some Sub-Saharan African countries, including Ghana, failed to meet Millennium Development Goal 5 (MDG5) by 2015 and may struggle to meet Sustainable Development Goal 3 (SDG3) despite maternal healthcare interventions. Concerns arise about Ghana's ability to achieve the World Health Organization's neonatal mortality goal of 12 per 1000 live births by 2030. This study aims to identify key factors influencing childbirth outcomes and create a predictive method for high-risk pregnancies.

METHODS

We compared four machine learning classifiers (Extreme Gradient Boosting, Random Forest, Logistic Regression, and Artificial Neural Network) in predicting childbirth outcomes using data from a tertiary health facility in Ghana. To address class imbalance, we employed the Synthetic Minority Over-sampling Technique (SMOTE).

RESULTS

Our findings show that fetal heartbeat, gestation age at birth are the most influential factors on birth outcome (stillbirth or live birth), while there is no significant association with maternal age, number of babies, and type of delivery method. Among the machine learning models considered, Random Forest emerged as the optimal model achieving an accuracy, F1-score, and AUC values of approximately 0.98, 0.99, and 0.90 respectively.

CONCLUSION

Our study identifies key factors affecting childbirth outcomes and highlights the potential of machine learning for early high-risk pregnancy detection in clinical settings. These findings are crucial for Ghana and other Sub-Saharan African countries striving to meet maternal and neonatal healthcare goals. Further research and policy initiatives can use these results to improve healthcare in the region and work toward the World Health Organization's objectives by 2030.

Analyzing Wav2Vec 1.0 Embeddings for Cross-Database Parkinson's Disease Detection and Speech Features Extraction

Advancements in deep learning speech representations have facilitated the effective use of extensive unlabeled speech datasets for Parkinson's disease (PD) modeling with minimal annotated data. This study employs the non-fine-tuned wav2vec 1.0 architecture to develop machine learning models for PD speech diagnosis tasks, such as cross-database classification and regression to predict demographic and articulation characteristics. The primary aim is to analyze overlapping components within the embeddings on both classification and regression tasks, investigating whether latent speech representations in PD are shared across models, particularly for related tasks. Firstly, evaluation using three multi-language PD datasets showed that wav2vec accurately detected PD based on speech, outperforming feature extraction using mel-frequency cepstral coefficients in the proposed cross-database classification scenarios. In cross-database scenarios using Italian and English-read texts, wav2vec demonstrated performance comparable to intra-dataset evaluations. We also compared our cross-database findings against those of other related studies. Secondly, wav2vec proved effective in regression, modeling various quantitative speech characteristics related to articulation and aging. Ultimately, subsequent analysis of important features examined the presence of significant overlaps between classification and regression models. The feature importance experiments discovered shared features across trained models, with increased sharing for related tasks, further suggesting that wav2vec contributes to improved generalizability. The study proposes wav2vec embeddings as a next promising step toward a speech-based universal model to assist in the evaluation of PD.

Automated Interpretation of Lung Sounds by Deep Learning in Children With Asthma: Scoping Review and Strengths, Weaknesses, Opportunities, and Threats Analysis

BACKGROUND

The interpretation of lung sounds plays a crucial role in the appropriate diagnosis and management of pediatric asthma. Applying artificial intelligence (AI) to this task has the potential to better standardize assessment and may even improve its predictive potential.

OBJECTIVE

This study aims to objectively review the literature on AI-assisted lung auscultation for pediatric asthma and provide a balanced assessment of its strengths, weaknesses, opportunities, and threats.

METHODS

A scoping review on AI-assisted lung sound analysis in children with asthma was conducted across 4 major scientific databases (PubMed, MEDLINE Ovid, Embase, and Web of Science), supplemented by a gray literature search on Google Scholar, to identify relevant studies published from January 1, 2000, until May 23, 2023. The search strategy incorporated a combination of keywords related to AI, pulmonary auscultation, children, and asthma. The quality of eligible studies was assessed using the ChAMAI (Checklist for the Assessment of Medical Artificial Intelligence).

RESULTS

The search identified 7 relevant studies out of 82 (9%) to be included through an academic literature search, while 11 of 250 (4.4%) studies from the gray literature search were considered but not included in the subsequent review and quality assessment. All had poor to medium ChAMAI scores, mostly due to the absence of external validation. Identified strengths were improved predictive accuracy of AI to allow for prompt and early diagnosis, personalized management strategies, and remote monitoring capabilities. Weaknesses were the heterogeneity between studies and the lack of standardization in data collection and interpretation. Opportunities were the potential of coordinated surveillance, growing data sets, and new ways of collaboratively learning from distributed data. Threats were both generic for the field of medical AI (loss of interpretability) but also specific to the use case, as clinicians might lose the skill of auscultation.

CONCLUSIONS

To achieve the opportunities of automated lung auscultation, there is a need to address weaknesses and threats with large-scale coordinated data collection in globally representative populations and leveraging new approaches to collaborative learning.

Evaluating Inflammatory Bowel Disease-Related Quality of Life Using an Interpretable Machine Learning Approach: A Multicenter Study in China

Purpose

Impaired quality of life (QOL) is common in patients with inflammatory bowel disease (IBD). A tool to more quickly identify IBD patients at high risk of impaired QOL improves opportunities for earlier intervention and improves long-term prognosis. The purpose of this study was to use a machine learning (ML) approach to develop risk stratification models for evaluating IBD-related QOL impairments.

Patients and Methods

An online questionnaire was used to collect clinical data on 2478 IBD patients from 42 hospitals distributed across 22 provinces in China from September 2021 to May 2022. Eight ML models used to predict the risk of IBD-related QOL impairments were developed and validated. Model performance was evaluated using a set of indexes and the best ML model was explained using a Local Interpretable Model-Agnostic Explanations (LIME) algorithm.

Results

The support vector machine (SVM) classifier algorithm-based model outperformed other ML models with an area under the receiver operating characteristic curve (AUC) and an accuracy of 0.80 and 0.71, respectively. The feature importance calculated by the SVM classifier algorithm revealed that glucocorticoid use, anxiety, abdominal pain, sleep disorders, and more severe disease contributed to a higher risk of impaired QOL, while longer disease course and the use of biological agents and immunosuppressants were associated with a lower risk.

Conclusion

An ML approach for assessing IBD-related QOL impairments is feasible and effective. This mechanism is a promising tool for gastroenterologists to identify IBD patients at high risk of impaired QOL.

Self-reported checklists and quality scoring tools in radiomics: a meta-research

OBJECTIVE

To evaluate the use of reporting checklists and quality scoring tools for self-reporting purposes in radiomics literature.

METHODS

Literature search was conducted in PubMed (date, April 23, 2023). The radiomics literature was sampled at random after a sample size calculation with a priori power analysis. A systematic assessment for self-reporting, including the use of documentation such as completed checklists or quality scoring tools, was conducted in original research papers. These eligible papers underwent independent evaluation by a panel of nine readers, with three readers assigned to each paper. Automatic annotation was used to assist in this process. Then, a detailed item-by-item confirmation analysis was carried out on papers with checklist documentation, with independent evaluation of two readers.

RESULTS

The sample size calculation yielded 117 papers. Most of the included papers were retrospective (94%; 110/117), single-center (68%; 80/117), based on their private data (89%; 104/117), and lacked external validation (79%; 93/117). Only seven papers (6%) had at least one self-reported document (Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), or Checklist for Artificial Intelligence in Medical Imaging (CLAIM)), with a statistically significant binomial test (p < 0.001). Median rate of confirmed items for all three documents was 81% (interquartile range, 6). For quality scoring tools, documented scores were higher than suggested scores, with a mean difference of - 7.2 (standard deviation, 6.8).

CONCLUSION

Radiomic publications often lack self-reported checklists or quality scoring tools. Even when such documents are provided, it is essential to be cautious, as the accuracy of the reported items or scores may be questionable.

CLINICAL RELEVANCE STATEMENT

Current state of radiomic literature reveals a notable absence of self-reporting with documentation and inaccurate reporting practices. This critical observation may serve as a catalyst for motivating the radiomics community to adopt and utilize such tools appropriately, thereby fostering rigor, transparency, and reproducibility of their research, moving the field forward.

KEY POINTS

• In radiomics literature, there has been a notable absence of self-reporting with documentation. • Even if such documents are provided, it is critical to exercise caution because the accuracy of the reported items or scores may be questionable. • Radiomics community needs to be motivated to adopt and appropriately utilize the reporting checklists and quality scoring tools.

Deep learning-based platform performs high detection sensitivity of intracranial aneurysms in 3D brain TOF-MRA: An external clinical validation study

PURPOSE

To evaluate the diagnostic efficacy of a developed artificial intelligence (AI) platform incorporating deep learning algorithms for the automated detection of intracranial aneurysms in time-of-flight (TOF) magnetic resonance angiography (MRA).

METHOD

This retrospective study encompassed 3D TOF MRA images acquired between January 2023 and June 2023, aiming to validate the presence of intracranial aneurysms via our developed AI platform. The manual segmentation results by experienced neuroradiologists served as the "gold standard". Following annotation of MRA images by neuroradiologists using InferScholar software, the AI platform conducted automatic segmentation of intracranial aneurysms. Various metrics including accuracy (ACC), balanced ACC, area under the curve (AUC), sensitivity (SE), specificity (SP), F1 score, Brier Score, and Net Benefit were utilized to evaluate the generalization of AI platform. Comparison of intracranial aneurysm identification performance was conducted between the AI platform and six radiologists with experience ranging from 3 to 12 years in interpreting MR images. Additionally, a comparative analysis was carried out between radiologists' detection performance based on independent visual diagnosis and AI-assisted diagnosis. Subgroup analyses were also performed based on the size and location of the aneurysms to explore factors impacting aneurysm detectability.

RESULTS

510 patients were enrolled including 215 patients (42.16 %) with intracranial aneurysms and 295 patients (57.84 %) without aneurysms. Compared with six radiologists, the AI platform showed competitive discrimination power (AUC, 0.96), acceptable calibration (Brier Score loss, 0.08), and clinical utility (Net Benefit, 86.96 %). The AI platform demonstrated superior performance in detecting aneurysms with an overall SE, SP, ACC, balanced ACC, and F1 score of 91.63 %, 92.20 %, 91.96 %, 91.92 %, and 90.57 % respectively, outperforming the detectability of the two resident radiologists. For subgroup analysis based on aneurysm size and location, we observed that the SE of the AI platform for identifying tiny (diameter＜3mm), small (3 mm ≤ diameter＜5mm), medium (5 mm ≤ diameter＜7mm) and large aneurysms (diameter ≥ 7 mm) was 87.80 %, 93.14 %, 95.45 %, and 100 %, respectively. Furthermore, the SE for detecting aneurysms in the anterior circulation was higher than that in the posterior circulation. Utilizing the AI assistance, six radiologists (i.e., two residents, two attendings and two professors) achieved statistically significant improvements in mean SE (residents: 71.40 % vs. 88.37 %; attendings: 82.79 % vs. 93.26 %; professors: 90.07 % vs. 97.44 %; P < 0.05) and ACC (residents: 85.29 % vs. 94.12 %; attendings: 91.76 % vs. 97.06 %; professors: 95.29 % vs. 98.82 %; P < 0.05) while no statistically significant change was observed in SP. Overall, radiologists' mean SE increased by 11.40 %, mean SP increased by 1.86 %, and mean ACC increased by 5.88 %, mean balanced ACC promoted by 6.63 %, mean F1 score grew by 7.89 %, and Net Benefit rose by 12.52 %, with a concurrent decrease in mean Brier score declined by 0.06.

CONCLUSIONS

The deep learning algorithms implemented in the AI platform effectively detected intracranial aneurysms on TOF-MRA and notably enhanced the diagnostic capabilities of radiologists. This indicates that the AI-based auxiliary diagnosis model can provide dependable and precise prediction to improve the diagnostic capacity of radiologists.

Exploring Pattern of Relapse in Pediatric Patients with Acute Lymphocytic Leukemia and Acute Myeloid Leukemia Undergoing Stem Cell Transplant Using Machine Learning Methods

Background. Leukemic relapse remains the primary cause of treatment failure and death after allogeneic hematopoietic stem cell transplant. Changes in post-transplant donor chimerism have been identified as a predictor of relapse. A better predictive model of relapse incorporating donor chimerism has the potential to improve leukemia-free survival by allowing earlier initiation of post-transplant treatment on individual patients. We explored the use of machine learning, a suite of analytical methods focusing on pattern recognition, to improve post-transplant relapse prediction. Methods. Using a cohort of 63 pediatric patients with acute lymphocytic leukemia (ALL) and 46 patients with acute myeloid leukemia (AML) who underwent stem cell transplant at a single institution, we built predictive models of leukemic relapse with both pre-transplant and post-transplant patient variables (specifically lineage-specific chimerism) using the random forest classifier. Local Interpretable Model-Agnostic Explanations, an interpretable machine learning tool was used to confirm our random forest classification result. Results. Our analysis showed that a random forest model using these hyperparameter values achieved 85% accuracy, 85% sensitivity, 89% specificity for ALL, while for AML 81% accuracy, 75% sensitivity, and 100% specificity at predicting relapses within 24 months post-HSCT in cross validation. The Local Interpretable Model-Agnostic Explanations tool was able to confirm many variables that the random forest classifier identified as important for the relapse prediction. Conclusions. Machine learning methods can reveal the interaction of different risk factors of post-transplant leukemic relapse and robust predictions can be obtained even with a modest clinical dataset. The random forest classifier distinguished different important predictive factors between ALL and AML in our relapse models, consistent with previous knowledge, lending increased confidence to adopting machine learning prediction to clinical management.

Good machine learning practices: Learnings from the modern pharmaceutical discovery enterprise

Machine Learning (ML) and Artificial Intelligence (AI) have become an integral part of the drug discovery and development value chain. Many teams in the pharmaceutical industry nevertheless report the challenges associated with the timely, cost effective and meaningful delivery of ML and AI powered solutions for their scientists. We sought to better understand what these challenges were and how to overcome them by performing an industry wide assessment of the practices in AI and Machine Learning. Here we report results of the systematic business analysis of the personas in the modern pharmaceutical discovery enterprise in relation to their work with the AI and ML technologies. We identify 23 common business problems that individuals in these roles face when they encounter AI and ML technologies at work, and describe best practices (Good Machine Learning Practices) that address these issues.

International Variation in Severe Exacerbation Rates in Patients With Severe Asthma

BACKGROUND

Exacerbation frequency strongly influences treatment choices in patients with severe asthma.

RESEARCH QUESTION

What is the extent of the variability of exacerbation rate across countries and its implications in disease management?

STUDY DESIGN AND METHODS

We retrieved data from the International Severe Asthma Registry, an international observational cohort of patients with a clinical diagnosis of severe asthma. We identified patients aged ≥ 18 years who did not initiate any biologics prior to baseline visit. A severe exacerbation was defined as the use of oral corticosteroids for ≥ 3 days or asthma-related hospitalization/ED visit. A series of negative binomial models were applied to estimate country-specific severe exacerbation rates during 365 days of follow-up, starting from a naive model with country as the only variable to an adjusted model with country as a random-effect term and patient and disease characteristics as independent variables.

RESULTS

The final sample included 7,510 patients from 17 countries (56% from the United States), contributing to 1,939 severe exacerbations (0.27/person-year). There was large between-country variation in observed severe exacerbation rate (minimum, 0.04 [Argentina]; maximum, 0.88 [Saudi Arabia]; interquartile range, 0.13-0.54), which remained substantial after adjusting for patient characteristics and sampling variability (interquartile range, 0.16-0.39).

INTERPRETATION

Individuals with similar patient characteristics but coming from different jurisdictions have varied severe exacerbation risks, even after controlling for patient and disease characteristics. This suggests unknown patient factors or system-level variations at play. Disease management guidelines should recognize such between-country variability. Risk prediction models that are calibrated for each jurisdiction will be needed to optimize treatment strategies.

Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support systems in pediatric outpatient clinics: an observational study

Artificial intelligence (AI) decision support systems in pediatric healthcare have a complex application background. As an AI decision support system (AI-DSS) can be costly, once applied, it is crucial to focus on its performance, interpret its success, and then monitor and update it to ensure ongoing success consistently. Therefore, a set of evaluation indicators was explicitly developed for AI-DSS in pediatric healthcare, enabling continuous and systematic performance monitoring. The study unfolded in two stages. The first stage encompassed establishing the evaluation indicator set through a literature review, a focus group interview, and expert consultation using the Delphi method. In the second stage, weight analysis was conducted. Subjective weights were calculated based on expert opinions through analytic hierarchy process, while objective weights were determined using the entropy weight method. Subsequently, subject and object weights were synthesized to form the combined weight. In the two rounds of expert consultation, the authority coefficients were 0.834 and 0.846, Kendall's coordination coefficient was 0.135 in Round 1 and 0.312 in Round 2. The final evaluation indicator set has three first-class indicators, fifteen second-class indicators, and forty-seven third-class indicators. Indicator I-1(Organizational performance) carries the highest weight, followed by Indicator I-2(Societal performance) and Indicator I-3(User experience performance) in the objective and combined weights. Conversely, 'Societal performance' holds the most weight among the subjective weights, followed by 'Organizational performance' and 'User experience performance'. In this study, a comprehensive and specialized set of evaluation indicators for the AI-DSS in the pediatric outpatient clinic was established, and then implemented. Continuous evaluation still requires long-term data collection to optimize the weight proportions of the established indicators.