Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

Privacy-preserving federated machine learning on FAIR health data: A real-world application

Objective

This paper introduces a privacy-preserving federated machine learning (ML) architecture built upon Findable, Accessible, Interoperable, and Reusable (FAIR) health data. It aims to devise an architecture for executing classification algorithms in a federated manner, enabling collaborative model-building among health data owners without sharing their datasets.

Materials and methods

Utilizing an agent-based architecture, a privacy-preserving federated ML algorithm was developed to create a global predictive model from various local models. This involved formally defining the algorithm in two steps: data preparation and federated model training on FAIR health data and constructing the architecture with multiple components facilitating algorithm execution. The solution was validated by five healthcare organizations using their specific health datasets.

Results

Five organizations transformed their datasets into Health Level 7 Fast Healthcare Interoperability Resources via a common FAIRification workflow and software set, thereby generating FAIR datasets. Each organization deployed a Federated ML Agent within its secure network, connected to a cloud-based Federated ML Manager. System testing was conducted on a use case aiming to predict 30-day readmission risk for chronic obstructive pulmonary disease patients and the federated model achieved an accuracy rate of 87%.

Discussion

The paper demonstrated a practical application of privacy-preserving federated ML among five distinct healthcare entities, highlighting the value of FAIR health data in machine learning when utilized in a federated manner that ensures privacy protection without sharing data.

Conclusion

This solution effectively leverages FAIR datasets from multiple healthcare organizations for federated ML while safeguarding sensitive health datasets, meeting legislative privacy and security requirements.

Prediction Models for Intravenous Immunoglobulin Non-Responders of Kawasaki Disease Using Machine Learning

BACKGROUND AND OBJECTIVE: Intravenous immunoglobulin (IVIG) is a prominent therapeutic agent for Kawasaki disease (KD) that significantly reduces the incidence of coronary artery anomalies. Various methodologies, including machine learning, have been employed to develop IVIG non-responder prediction models; however, their validation and reproducibility remain unverified. This study aimed to develop a predictive scoring system for identifying IVIG nonresponders and rigorously test the accuracy and reliability of this system. METHODS: The study included an exposure group of 228 IVIG non-responders and a control group of 997 IVIG responders. Subsequently, a predictive machine learning model was constructed. The Shizuoka score, including variables such as the "initial treatment date" (cutoff: < 4 days), sodium level (cutoff: < 133 mEq/L), total bilirubin level (cutoff: ≥ 0.5 mg/dL), and neutrophil-to-lymphocyte ratio (cutoff: ≥ 2.6), was established. Patients meeting two or more of these criteria were grouped as high-risk IVIG non-responders. Using the Shizuoka score to stratify IVIG responders, propensity score matching was used to analyze 85 patients each for IVIG and IVIG-added prednisolone treatment in the high-risk group. In the IVIG plus prednisolone group, the IVIG non-responder count significantly decreased (p < 0.001), with an odds ratio of 0.192 (95% confidence interval 0.078-0.441). CONCLUSIONS: Intravenous immunoglobulin non-responders were predicted using machine learning models and validated using propensity score matching. The initiation of initial IVIG-added prednisolone treatment in the high-risk group identified by the Shizuoka score, crafted using machine learning models, appears useful for predicting IVIG non-responders.

Prediction models for amputation after diabetic foot: systematic review and critical appraisal

BACKGROUND

Numerous studies have developed or validated prediction models aimed at estimating the likelihood of amputation in diabetic foot (DF) patients. However, the quality and applicability of these models in clinical practice and future research remain uncertain. This study conducts a systematic review and assessment of the risk of bias and applicability of amputation prediction models among individuals with DF.

METHODS

A comprehensive search was conducted across multiple databases, including PubMed, Web of Science, EBSCO CINAHL Plus, Embase, Cochrane Library, China National Knowledge Infrastructure (CNKI), Wanfang, Chinese Biomedical Literature Database (CBM), and Weipu (VIP) from their inception to December 24, 2023. Two investigators independently screened the literature and extracted data using the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies. The Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist was employed to evaluate both the risk of bias and applicability.

RESULTS

A total of 20 studies were included in this analysis, comprising 17 development studies and three validation studies, encompassing 20 prediction models and 11 classification systems. The incidence of amputation in patients with DF ranged from 5.9 to 58.5%. Machine learning-based methods were employed in more than half of the studies. The reported area under the curve (AUC) varied from 0.560 to 0.939. Independent predictors consistently identified by multivariate models included age, gender, HbA1c, hemoglobin, white blood cell count, low-density lipoprotein cholesterol, diabetes duration, and Wagner's Classification. All studies were found to exhibit a high risk of bias, primarily attributed to inadequate handling of outcome events and missing data, lack of model performance assessment, and overfitting.

CONCLUSIONS

The assessment using PROBAST revealed a notable risk of bias in the existing prediction models for amputation in patients with DF. It is imperative for future studies to concentrate on enhancing the robustness of current prediction models or constructing new models with stringent methodologies.

Prognostic models for patients suffering a heart failure with a preserved ejection fraction: a systematic review

The purpose of this study was to systematically review the development, performance, and applicability of prognostic models developed for predicting poor events in patients with heart failure with preserved ejection fraction (HFpEF). Databases including Embase, PubMed, Web of Science Core Collection, the Cochrane Library, China National Knowledge Infrastructure, Wan Fang, Wei Pu, and China Biological Medicine were queried from their respective dates of inception to 1 June 2023, to examine multivariate models for prognostic prediction in HFpEF. Both forward and backward citations of all studies were included in our analysis. Two researchers individually used the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist to extract data and assess the quality of the models using the Predictive Mode Bias Risk Assessment Tool (PROBAST). Among the 6897 studies screened, 16 studies derived and/or validated a total of 39 prognostic models. The sample size ranges for model development, internal validation, and external validation are 119 to 5988, 152 to 1000, and 30 to 5957, respectively. The most frequently employed modelling technique was Cox proportional hazards regression. Six studies (37.50%) conducted internal validation of models; bootstrap and k-fold cross-validation were the commonly used methods for internal validation of models. Ten of these models (25.64%) were validated externally, with reported the c-statistic in the external validation set ranging from 0.70 to 0.96, while the remaining models await external validation. The MEDIA echo score and I-PRESERVE-sudden cardiac death prediction mode have been externally validated using multiple cohorts, and the results consistently show good predictive performance. The most frequently used predictors identified among the models were age, n-terminal pro-brain natriuretic peptide, ejection fraction, albumin, and hospital stay in the last 5 months owing to heart failure. All study predictor domains and outcome domains were at low risk of bias, high or unclear risk of bias of all prognostic models due to underreporting in the area of analysis. All studies did not evaluate the clinical utility of the prognostic models. Predictive models for predicting prognostic outcomes in patients with HFpEF showed good discriminatory ability but their utility and generalization remain uncertain due to the risk of bias, differences in predictors between models, and the lack of clinical application studies. Future studies should improve the methodological quality of model development and conduct external validation of models.

Applications and Performance of Machine Learning Algorithms in Emergency Medical Services: A Scoping Review

OBJECTIVE

The aim of this study was to summarize the literature on the applications of machine learning (ML) and their performance in Emergency Medical Services (EMS).

METHODS

Four relevant electronic databases were searched (from inception through January 2024) for all original studies that employed EMS-guided ML algorithms to enhance the clinical and operational performance of EMS. Two reviewers screened the retrieved studies and extracted relevant data from the included studies. The characteristics of included studies, employed ML algorithms, and their performance were quantitively described across primary domains and subdomains.

RESULTS

This review included a total of 164 studies published from 2005 through 2024. Of those, 125 were clinical domain focused and 39 were operational. The characteristics of ML algorithms such as sample size, number and type of input features, and performance varied between and within domains and subdomains of applications. Clinical applications of ML algorithms involved triage or diagnosis classification (n = 62), treatment prediction (n = 12), or clinical outcome prediction (n = 50), mainly for out-of-hospital cardiac arrest/OHCA (n = 62), cardiovascular diseases/CVDs (n = 19), and trauma (n = 24). The performance of these ML algorithms varied, with a median area under the receiver operating characteristic curve (AUC) of 85.6%, accuracy of 88.1%, sensitivity of 86.05%, and specificity of 86.5%. Within the operational studies, the operational task of most ML algorithms was ambulance allocation (n = 21), followed by ambulance detection (n = 5), ambulance deployment (n = 5), route optimization (n = 5), and quality assurance (n = 3). The performance of all operational ML algorithms varied and had a median AUC of 96.1%, accuracy of 90.0%, sensitivity of 94.4%, and specificity of 87.7%. Generally, neural network and ensemble algorithms, to some degree, out-performed other ML algorithms.

CONCLUSION

Triaging and managing different prehospital medical conditions and augmenting ambulance performance can be improved by ML algorithms. Future reports should focus on a specific clinical condition or operational task to improve the precision of the performance metrics of ML models.

Artificial intelligence in healthcare: why not apply the medico-legal method starting with the Collingridge dilemma?

Technology has greatly influenced and radically changed human life, from communication to creativity and from productivity to entertainment. The authors, starting from considerations concerning the implementation of new technologies with a strong impact on people's everyday lives, take up Collingridge's dilemma and relate it to the application of AI in healthcare. Collingridge's dilemma is an ethical and epistemological problem concerning the relationship between technology and society which involves two approaches. The proactive approach and socio-technological experimentation taken into account in the dilemma are discussed, the former taking health technology assessment (HTA) processes as a reference and the latter the AI studies conducted so far. As a possible prevention of the critical issues raised, the use of the medico-legal method is proposed, which classically lies between the prevention of possible adverse events and the reconstruction of how these occurred.The authors believe that this methodology, adopted as a European guideline in the medico-legal field for the assessment of medical liability, can be adapted to AI applied to the healthcare scenario and used for the assessment of liability issues. The topic deserves further investigation and will certainly be taken into consideration as a possible key to future scenarios.

Predictive value of machine learning for the severity of acute pancreatitis: A systematic review and meta-analysis

Background

Predicting the severity of acute pancreatitis (AP) early poses a challenge in clinical practice. While there are well-established clinical scoring tools, their actual predictive performance remains uncertain. Various studies have explored the application of machine-learning methods for early AP prediction. However, a more comprehensive evidence-based assessment is needed to determine their predictive accuracy. Hence, this systematic review and meta-analysis aimed to evaluate the predictive accuracy of machine learning in assessing the severity of AP.

Methods

PubMed, EMBASE, Cochrane Library, and Web of Science were systematically searched until December 5, 2023. The risk of bias in eligible studies was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Subgroup analyses, based on different machine learning types, were performed. Additionally, the predictive accuracy of mainstream scoring tools was summarized.

Results

This systematic review ultimately included 33 original studies. The pooled c-index in both the training and validation sets was 0.87 (95 % CI: 0.84-0.89) and 0.88 (95 % CI: 0.86-0.90), respectively. The sensitivity in the training set was 0.81 (95 % CI: 0.77-0.84), and in the validation set, it was 0.79 (95 % CI: 0.71-0.85). The specificity in the training set was 0.84 (95 % CI: 0.78-0.89), and in the validation set, it was 0.90 (95 % CI: 0.86-0.93). The primary model incorporated was logistic regression; however, its predictive accuracy was found to be inferior to that of neural networks, random forests, and xgboost. The pooled c-index of the APACHE II, BISAP, and Ranson were 0.74 (95 % CI: 0.68-0.80), 0.77 (95 % CI: 0.70-0.85), and 0.74 (95 % CI: 0.68-0.79), respectively.

Conclusions

Machine learning demonstrates excellent accuracy in predicting the severity of AP, providing a reference for updating or developing a straightforward clinical prediction tool.

Investigating Rhythmicity in App Usage to Predict Depressive Symptoms: Protocol for Personalized Framework Development and Validation Through a Countrywide Study

BACKGROUND

Understanding a student's depressive symptoms could facilitate significantly more precise diagnosis and treatment. However, few studies have focused on depressive symptom prediction through unobtrusive systems, and these studies are limited by small sample sizes, low performance, and the requirement for higher resources. In addition, research has not explored whether statistically significant rhythms based on different app usage behavioral markers (eg, app usage sessions) exist that could be useful in finding subtle differences to predict with higher accuracy like the models based on rhythms of physiological data.

OBJECTIVE

The main objective of this study is to explore whether there exist statistically significant rhythms in resource-insensitive app usage behavioral markers and predict depressive symptoms through these marker-based rhythmic features. Another objective of this study is to understand whether there is a potential link between rhythmic features and depressive symptoms.

METHODS

Through a countrywide study, we collected 2952 students' raw app usage behavioral data and responses to the 9 depressive symptoms in the 9-item Patient Health Questionnaire (PHQ-9). The behavioral data were retrieved through our developed app, which was previously used in our pilot studies in Bangladesh on different research problems. To explore whether there is a rhythm based on app usage data, we will conduct a zero-amplitude test. In addition, we will develop a cosinor model for each participant to extract rhythmic parameters (eg, acrophase). In addition, to obtain a comprehensive picture of the rhythms, we will explore nonparametric rhythmic features (eg, interdaily stability). Furthermore, we will conduct regression analysis to understand the association of rhythmic features with depressive symptoms. Finally, we will develop a personalized multitask learning (MTL) framework to predict symptoms through rhythmic features.

RESULTS

After applying inclusion criteria (eg, having app usage data of at least 2 days to explore rhythmicity), we kept the data of 2902 (98.31%) students for analysis, with 24.48 million app usage events, and 7 days' app usage of 2849 (98.17%) students. The students are from all 8 divisions of Bangladesh, both public and private universities (19 different universities and 52 different departments). We are analyzing the data and will publish the findings in a peer-reviewed publication.

CONCLUSIONS

Having an in-depth understanding of app usage rhythms and their connection with depressive symptoms through a countrywide study can significantly help health care professionals and researchers better understand depressed students and may create possibilities for using app usage-based rhythms for intervention. In addition, the MTL framework based on app usage rhythmic features may more accurately predict depressive symptoms due to the rhythms' capability to find subtle differences.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/51540.

Use of Artificial Intelligence in Improving Outcomes in Heart Disease: A Scientific Statement From the American Heart Association

A major focus of academia, industry, and global governmental agencies is to develop and apply artificial intelligence and other advanced analytical tools to transform health care delivery. The American Heart Association supports the creation of tools and services that would further the science and practice of precision medicine by enabling more precise approaches to cardiovascular and stroke research, prevention, and care of individuals and populations. Nevertheless, several challenges exist, and few artificial intelligence tools have been shown to improve cardiovascular and stroke care sufficiently to be widely adopted. This scientific statement outlines the current state of the art on the use of artificial intelligence algorithms and data science in the diagnosis, classification, and treatment of cardiovascular disease. It also sets out to advance this mission, focusing on how digital tools and, in particular, artificial intelligence may provide clinical and mechanistic insights, address bias in clinical studies, and facilitate education and implementation science to improve cardiovascular and stroke outcomes. Last, a key objective of this scientific statement is to further the field by identifying best practices, gaps, and challenges for interested stakeholders.

Improving management of febrile neutropenia in oncology patients: the role of artificial intelligence and machine learning

INTRODUCTION

Artificial intelligence (AI) and machine learning (ML) have the potential to revolutionize the management of febrile neutropenia (FN) and drive progress toward personalized medicine.

AREAS COVERED

In this review, we detail how the collection of a large number of high-quality data can be used to conduct precise mathematical studies with ML and AI. We explain the foundations of these techniques, covering the fundamentals of supervised and unsupervised learning, as well as the most important challenges, e.g. data quality, 'black box' model interpretation and overfitting. To conclude, we provide detailed examples of how AI and ML have been used to enhance predictions of chemotherapy-induced FN, detection of bloodstream infections (BSIs) and multidrug-resistant (MDR) bacteria, and anticipation of severe complications and mortality.

EXPERT OPINION

There is promising potential of implementing accurate AI and ML models whilst managing FN. However, their integration as viable clinical tools poses challenges, including technical and implementation barriers. Improving global accessibility, fostering interdisciplinary collaboration, and addressing ethical and security considerations are essential. By overcoming these challenges, we could transform personalized care for patients with FN.

Measurement and Mitigation of Bias in Artificial Intelligence: A Narrative Literature Review for Regulatory Science

Artificial intelligence (AI) is increasingly being used in decision making across various industries, including the public health arena. Bias in any decision-making process can significantly skew outcomes, and AI systems have been shown to exhibit biases at times. The potential for AI systems to perpetuate and even amplify biases is a growing concern. Bias, as used in this paper, refers to the tendency toward a particular characteristic or behavior, and thus, a biased AI system is one that shows biased associations entities. In this literature review, we examine the current state of research on AI bias, including its sources, as well as the methods for measuring, benchmarking, and mitigating it. We also examine the biases and methods of mitigation specifically relevant to the healthcare field and offer a perspective on bias measurement and mitigation in regulatory science decision making.

The reporting completeness and transparency of systematic reviews of prognostic prediction models for COVID-19 was poor: a methodological overview of systematic reviews

OBJECTIVES

To conduct a methodological overview of reviews to evaluate the reporting completeness and transparency of systematic reviews (SRs) of prognostic prediction models (PPMs) for COVID-19.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, Cochrane Database of Systematic Reviews, and Epistemonikos (epistemonikos.org) were searched for SRs of PPMs for COVID-19 until December 31, 2022. The risk of bias in systematic reviews tool was used to assess the risk of bias. The protocol for this overview was uploaded in the Open Science Framework (https://osf.io/7y94c).

RESULTS

Ten SRs were retrieved; none of them synthesized the results in a meta-analysis. For most of the studies, there was absence of a predefined protocol and missing information on study selection, data collection process, and reporting of primary studies and models included, while only one SR had its data publicly available. In addition, for the majority of the SRs, the overall risk of bias was judged as being high. The overall corrected covered area was 6.3% showing a small amount of overlapping among the SRs.

CONCLUSION

The reporting completeness and transparency of SRs of PPMs for COVID-19 was poor. Guidance is urgently required, with increased awareness and education of minimum reporting standards and quality criteria. Specific focus is needed in predefined protocol, information on study selection and data collection process, and in the reporting of findings to improve the quality of SRs of PPMs for COVID-19.

Prognostic Models in Nephrology: Where Do We Stand and Where Do We Go from Here? Mapping Out the Evidence in a Scoping Review

Prognostic models can strongly support individualized care provision and well-informed shared decision making. There has been an upsurge of prognostic research in the field of nephrology, but the uptake of prognostic models in clinical practice remains limited. Therefore, we map out the research field of prognostic models for kidney patients and provide directions on how to proceed from here. We performed a scoping review of studies developing, validating, or updating a prognostic model for patients with CKD. We searched all published models in PubMed and Embase and report predicted outcomes, methodological quality, and validation and/or updating efforts. We found 602 studies, of which 30.1% concerned CKD populations, 31.6% dialysis populations, and 38.4% kidney transplantation populations. The most frequently predicted outcomes were mortality ( n =129), kidney disease progression ( n =75), and kidney graft survival ( n =54). Most studies provided discrimination measures (80.4%), but much less showed calibration results (43.4%). Of the 415 development studies, 28.0% did not perform any validation and 57.6% performed only internal validation. Moreover, only 111 models (26.7%) were externally validated either in the development study itself or in an independent external validation study. Finally, in 45.8% of development studies no useable version of the model was reported. To conclude, many prognostic models have been developed for patients with CKD, mainly for outcomes related to kidney disease progression and patient/graft survival. To bridge the gap between prediction research and kidney patient care, patient-reported outcomes, methodological rigor, complete reporting of prognostic models, external validation, updating, and impact assessment urgently need more attention.

Risk prediction models for cardiovascular events in hemodialysis patients: A systematic review

OBJECTIVE

To perform a systematic review of risk prediction models for cardiovascular (CV) events in hemodialysis (HD) patients, and provide a reference for the application and optimization of related prediction models.

METHODS

PubMed, The Cochrane Library, Web of Science, and Embase databases were searched from inception to 1 February 2023. Two authors independently conducted the literature search, selection, and screening. The Prediction model Risk Of Bias Assessment Tool (PROBAST) was applied to evaluate the risk of bias and applicability of the included literature.

RESULTS

A total of nine studies containing 12 models were included, with performance measured by the area under the receiver operating characteristic curve (AUC) lying between 0.70 and 0.88. Age, diabetes mellitus (DM), C-reactive protein (CRP), and albumin (ALB) were the most commonly identified predictors of CV events in HD patients. While the included models demonstrated good applicability, there were still certain risks of bias, primarily related to inadequate handling of missing data and transformation of continuous variables, as well as a lack of model performance validation.

CONCLUSION

The included models showed good overall predictive performance and can assist healthcare professionals in the early identification of high-risk individuals for CV events in HD patients. In the future, the modeling methods should be improved, or the existing models should undergo external validation to provide better guidance for clinical practice.

A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare

Trustworthy medical AI requires transparency about the development and testing of underlying algorithms to identify biases and communicate potential risks of harm. Abundant guidance exists on how to achieve transparency for medical AI products, but it is unclear whether publicly available information adequately informs about their risks. To assess this, we retrieved public documentation on the 14 available CE-certified AI-based radiology products of the II b risk category in the EU from vendor websites, scientific publications, and the European EUDAMED database. Using a self-designed survey, we reported on their development, validation, ethical considerations, and deployment caveats, according to trustworthy AI guidelines. We scored each question with either 0, 0.5, or 1, to rate if the required information was "unavailable", "partially available," or "fully available." The transparency of each product was calculated relative to all 55 questions. Transparency scores ranged from 6.4% to 60.9%, with a median of 29.1%. Major transparency gaps included missing documentation on training data, ethical considerations, and limitations for deployment. Ethical aspects like consent, safety monitoring, and GDPR-compliance were rarely documented. Furthermore, deployment caveats for different demographics and medical settings were scarce. In conclusion, public documentation of authorized medical AI products in Europe lacks sufficient public transparency to inform about safety and risks. We call on lawmakers and regulators to establish legally mandated requirements for public and substantive transparency to fulfill the promise of trustworthy AI for health.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies

Objectives

To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.

Design

Systematic review and meta-analysis of external validation studies.

Data sources

Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023.

Eligibility criteria for selecting studies

All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed.

Results

47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125).

Conclusions

The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed.

Systematic review registration

PROSPERO CRD42022373182.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Advancing Diagnostics and Patient Care: The Role of Biomarkers in Radiology

The integration of biomarkers into medical practice has revolutionized the field of radiology, allowing for enhanced diagnostic accuracy, personalized treatment strategies, and improved patient care outcomes. This review offers radiologists a comprehensive understanding of the diverse applications of biomarkers in medicine. By elucidating the fundamental concepts, challenges, and recent advancements in biomarker utilization, it will serve as a bridge between the disciplines of radiology and epidemiology. Through an exploration of various biomarker types, such as imaging biomarkers, molecular biomarkers, and genetic markers, I outline their roles in disease detection, prognosis prediction, and therapeutic monitoring. I also discuss the significance of robust study designs, blinding, power and sample size calculations, performance metrics, and statistical methodologies in biomarker research. By fostering collaboration between radiologists, statisticians, and epidemiologists, I hope to accelerate the translation of biomarker discoveries into clinical practice, ultimately leading to improved patient care.

Role of machine learning in the management of epilepsy: a systematic review protocol

INTRODUCTION

Machine learning is a rapidly expanding field and is already incorporated into many aspects of medicine including diagnostics, prognostication and clinical decision-support tools. Epilepsy is a common and disabling neurological disorder, however, management remains challenging in many cases, despite expanding therapeutic options. We present a systematic review protocol to explore the role of machine learning in the management of epilepsy.

METHODS AND ANALYSIS

This protocol has been drafted with reference to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for Protocols. A literature search will be conducted in databases including MEDLINE, Embase, Scopus and Web of Science. A PRISMA flow chart will be constructed to summarise the study workflow. As the scope of this review is the clinical application of machine learning, the selection of papers will be focused on studies directly related to clinical decision-making in management of epilepsy, specifically the prediction of response to antiseizure medications, development of drug-resistant epilepsy, and epilepsy surgery and neuromodulation outcomes. Data will be extracted following the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist. Prediction model Risk Of Bias ASsessment Tool will be used for the quality assessment of the included studies. Syntheses of quantitative data will be presented in narrative format.

ETHICS AND DISSEMINATION

As this study is a systematic review which does not involve patients or animals, ethics approval is not required. The results of the systematic review will be submitted to peer-review journals for publication and presented in academic conferences.

PROSPERO REGISTRATION NUMBER

CRD42023442156.

Machine Learning Approaches for the Image-Based Identification of Surgical Wound Infections: Scoping Review

BACKGROUND

Surgical site infections (SSIs) occur frequently and impact patients and health care systems. Remote surveillance of surgical wounds is currently limited by the need for manual assessment by clinicians. Machine learning (ML)-based methods have recently been used to address various aspects of the postoperative wound healing process and may be used to improve the scalability and cost-effectiveness of remote surgical wound assessment.

OBJECTIVE

The objective of this review was to provide an overview of the ML methods that have been used to identify surgical wound infections from images.

METHODS

We conducted a scoping review of ML approaches for visual detection of SSIs following the JBI (Joanna Briggs Institute) methodology. Reports of participants in any postoperative context focusing on identification of surgical wound infections were included. Studies that did not address SSI identification, surgical wounds, or did not use image or video data were excluded. We searched MEDLINE, Embase, CINAHL, CENTRAL, Web of Science Core Collection, IEEE Xplore, Compendex, and arXiv for relevant studies in November 2022. The records retrieved were double screened for eligibility. A data extraction tool was used to chart the relevant data, which was described narratively and presented using tables. Employment of TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines was evaluated and PROBAST (Prediction Model Risk of Bias Assessment Tool) was used to assess risk of bias (RoB).

RESULTS

In total, 10 of the 715 unique records screened met the eligibility criteria. In these studies, the clinical contexts and surgical procedures were diverse. All papers developed diagnostic models, though none performed external validation. Both traditional ML and deep learning methods were used to identify SSIs from mostly color images, and the volume of images used ranged from under 50 to thousands. Further, 10 TRIPOD items were reported in at least 4 studies, though 15 items were reported in fewer than 4 studies. PROBAST assessment led to 9 studies being identified as having an overall high RoB, with 1 study having overall unclear RoB.

CONCLUSIONS

Research on the image-based identification of surgical wound infections using ML remains novel, and there is a need for standardized reporting. Limitations related to variability in image capture, model building, and data sources should be addressed in the future.

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Over the past two decades, advances in computational power and data availability combined with increased accessibility to pre-trained models have led to an exponential rise in machine learning (ML) publications. While ML may have the potential to transform healthcare, this sharp increase in ML research output without focus on methodological rigor and standard reporting guidelines has fueled a reproducibility crisis. In addition, the rapidly growing complexity of these models compromises their interpretability, which currently impedes their successful and widespread clinical adoption. In medicine, where failure of such models may have severe implications for patients' health, the high requirements for accuracy, robustness, and interpretability confront ML researchers with a unique set of challenges. In this review, we discuss the semantics of reproducibility and interpretability, as well as related issues and challenges, and outline possible solutions to counteracting the "black box". To foster reproducibility, standard reporting guidelines need to be further developed and data or code sharing encouraged. Editors and reviewers may equally play a critical role by establishing high methodological standards and thus preventing the dissemination of low-quality ML publications. To foster interpretable learning, the use of simpler models more suitable for medical data can inform the clinician how results are generated based on input data. Model-agnostic explanation tools, sensitivity analysis, and hidden layer representations constitute further promising approaches to increase interpretability. Balancing model performance and interpretability are important to ensure clinical applicability. We have now reached a critical moment for ML in medicine, where addressing these issues and implementing appropriate solutions will be vital for the future evolution of the field.

Artificial intelligence in fracture detection with different image modalities and data types: A systematic review and meta-analysis

Artificial Intelligence (AI), encompassing Machine Learning and Deep Learning, has increasingly been applied to fracture detection using diverse imaging modalities and data types. This systematic review and meta-analysis aimed to assess the efficacy of AI in detecting fractures through various imaging modalities and data types (image, tabular, or both) and to synthesize the existing evidence related to AI-based fracture detection. Peer-reviewed studies developing and validating AI for fracture detection were identified through searches in multiple electronic databases without time limitations. A hierarchical meta-analysis model was used to calculate pooled sensitivity and specificity. A diagnostic accuracy quality assessment was performed to evaluate bias and applicability. Of the 66 eligible studies, 54 identified fractures using imaging-related data, nine using tabular data, and three using both. Vertebral fractures were the most common outcome (n = 20), followed by hip fractures (n = 18). Hip fractures exhibited the highest pooled sensitivity (92%; 95% CI: 87-96, p< 0.01) and specificity (90%; 95% CI: 85-93, p< 0.01). Pooled sensitivity and specificity using image data (92%; 95% CI: 90-94, p< 0.01; and 91%; 95% CI: 88-93, p < 0.01) were higher than those using tabular data (81%; 95% CI: 77-85, p< 0.01; and 83%; 95% CI: 76-88, p < 0.01), respectively. Radiographs demonstrated the highest pooled sensitivity (94%; 95% CI: 90-96, p < 0.01) and specificity (92%; 95% CI: 89-94, p< 0.01). Patient selection and reference standards were major concerns in assessing diagnostic accuracy for bias and applicability. AI displays high diagnostic accuracy for various fracture outcomes, indicating potential utility in healthcare systems for fracture diagnosis. However, enhanced transparency in reporting and adherence to standardized guidelines are necessary to improve the clinical applicability of AI. Review Registration: PROSPERO (CRD42021240359).

BOO-ST and CBCEC: two novel hybrid machine learning methods aim to reduce the mortality of heart failure patients

Heart failure (HF) is a leading cause of mortality worldwide. Machine learning (ML) approaches have shown potential as an early detection tool for improving patient outcomes. Enhancing the effectiveness and clinical applicability of the ML model necessitates training an efficient classifier with a diverse set of high-quality datasets. Hence, we proposed two novel hybrid ML methods ((a) consisting of Boosting, SMOTE, and Tomek links (BOO-ST); (b) combining the best-performing conventional classifier with ensemble classifiers (CBCEC)) to serve as an efficient early warning system for HF mortality. The BOO-ST was introduced to tackle the challenge of class imbalance, while CBCEC was responsible for training the processed and selected features derived from the Feature Importance (FI) and Information Gain (IG) feature selection techniques. We also conducted an explicit and intuitive comprehension to explore the impact of potential characteristics correlating with the fatality cases of HF. The experimental results demonstrated the proposed classifier CBCEC showcases a significant accuracy of 93.67% in terms of providing the early forecasting of HF mortality. Therefore, we can reveal that our proposed aspects (BOO-ST and CBCEC) can be able to play a crucial role in preventing the death rate of HF and reducing stress in the healthcare sector.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

The Lifelong Impact of Artificial Intelligence and Clinical Prediction Models on Patients With Tetralogy of Fallot

Medical advancements in the diagnosis, surgical techniques, perioperative care, and continued care throughout childhood have transformed the outlook for individuals with tetralogy of Fallot (TOF), improving survival and shifting the perspective towards lifelong care. However, with a growing population of survivors, longstanding challenges have been accentuated, and new challenges have surfaced, necessitating a re-evaluation of TOF care. Availability of prenatal diagnostics, insufficient information from traditional imaging techniques, previously unforeseen medical complications, and debates surrounding optimal timing and indications for reintervention are among the emerging issues. To address these challenges, the integration of artificial intelligence and machine learning holds great promise as they have the potential to revolutionize patient management and positively impact lifelong outcomes for individuals with TOF. Innovative applications of artificial intelligence and machine learning have spanned across multiple domains of TOF care, including screening and diagnosis, automated image processing and interpretation, clinical risk stratification, and planning and performing cardiac interventions. By embracing these advancements and incorporating them into routine clinical practice, personalized medicine could be delivered, leading to the best possible outcomes for patients. In this review, we provide an overview of these evolving applications and emphasize the challenges, limitations, and future potential for integrating them into clinical care.

Machine learning-based prediction models for pressure injury: A systematic review and meta-analysis

Despite the fact that machine learning (ML) algorithms to construct predictive models for pressure injury development are widely reported, the performance of the model remains unknown. The goal of the review was to systematically appraise the performance of ML models in predicting pressure injury. PubMed, Embase, Cochrane Library, Web of Science, CINAHL, Grey literature and other databases were systematically searched. Original journal papers were included which met the inclusion criteria. The methodological quality was assessed independently by two reviewers using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Meta-analysis was performed with Metadisc software, with the area under the receiver operating characteristic curve, sensitivity and specificity as effect measures. Chi-squared and I2 tests were used to assess the heterogeneity. A total of 18 studies were included for the narrative review, and 14 of them were eligible for meta-analysis. The models achieved excellent pooled AUC of 0.94, sensitivity of 0.79 (95% CI [0.78-0.80]) and specificity of 0.87 (95% CI [0.88-0.87]). Meta-regressions did not provide evidence that model performance varied by data or model types. The present findings indicate that ML models show an outstanding performance in predicting pressure injury. However, good-quality studies should be conducted to verify our results and confirm the clinical value of ML in pressure injury development.

Performances of machine learning algorithms in discriminating sacroiliitis features on MRI: a systematic review

OBJECTIVES

Summarise the evidence of the performance of the machine learning algorithm in discriminating sacroiliitis features on MRI and compare it with the accuracy of human physicians.

METHODS

MEDLINE, EMBASE, CIHNAL, Web of Science, IEEE, American College of Rheumatology and European Alliance of Associations for Rheumatology abstract archives were searched for studies published between 2008 and 4 June 2023. Two authors independently screened and extracted the variables, and the results are presented using tables and forest plots.

RESULTS

Ten studies were selected from 2381. Over half of the studies used deep learning models, using Assessment of Spondyloarthritis International Society sacroiliitis criteria as the ground truth, and manually extracted the regions of interest. All studies reported the area under the curve as a performance index, ranging from 0.76 to 0.99. Sensitivity and specificity were the second-most commonly reported indices, with sensitivity ranging from 0.56 to 1.00 and specificity ranging from 0.67 to 1.00; these results are comparable to a radiologist's sensitivity of 0.67-1.00 and specificity of 0.78-1.00 in the same cohort. More than half of the studies showed a high risk of bias in the analysis domain of quality appraisal owing to the small sample size or overfitting issues.

CONCLUSION

The performance of machine learning algorithms in discriminating sacroiliitis features on MRI varied owing to the high heterogeneity between studies and the small sample sizes, overfitting, and under-reporting issues of individual studies. Further well-designed and transparent studies are required.

Prediction models using artificial intelligence and longitudinal data from electronic health records: a systematic methodological review

OBJECTIVE

To describe and appraise the use of artificial intelligence (AI) techniques that can cope with longitudinal data from electronic health records (EHRs) to predict health-related outcomes.

METHODS

This review included studies in any language that: EHR was at least one of the data sources, collected longitudinal data, used an AI technique capable of handling longitudinal data, and predicted any health-related outcomes. We searched MEDLINE, Scopus, Web of Science, and IEEE Xplorer from inception to January 3, 2022. Information on the dataset, prediction task, data preprocessing, feature selection, method, validation, performance, and implementation was extracted and summarized using descriptive statistics. Risk of bias and completeness of reporting were assessed using a short form of PROBAST and TRIPOD, respectively.

RESULTS

Eighty-one studies were included. Follow-up time and number of registers per patient varied greatly, and most predicted disease development or next event based on diagnoses and drug treatments. Architectures generally were based on Recurrent Neural Networks-like layers, though in recent years combining different layers or transformers has become more popular. About half of the included studies performed hyperparameter tuning and used attention mechanisms. Most performed a single train-test partition and could not correctly assess the variability of the model's performance. Reporting quality was poor, and a third of the studies were at high risk of bias.

CONCLUSIONS

AI models are increasingly using longitudinal data. However, the heterogeneity in reporting methodology and results, and the lack of public EHR datasets and code sharing, complicate the possibility of replication.

REGISTRATION

PROSPERO database (CRD42022331388).

Clinical Prediction Models for Hospital-Induced Delirium Using Structured and Unstructured Electronic Health Record Data: Protocol for a Development and Validation Study

BACKGROUND

Hospital-induced delirium is one of the most common and costly iatrogenic conditions, and its incidence is predicted to increase as the population of the United States ages. An academic and clinical interdisciplinary systems approach is needed to reduce the frequency and impact of hospital-induced delirium.

OBJECTIVE

The long-term goal of our research is to enhance the safety of hospitalized older adults by reducing iatrogenic conditions through an effective learning health system. In this study, we will develop models for predicting hospital-induced delirium. In order to accomplish this objective, we will create a computable phenotype for our outcome (hospital-induced delirium), design an expert-based traditional logistic regression model, leverage machine learning techniques to generate a model using structured data, and use machine learning and natural language processing to produce an integrated model with components from both structured data and text data.

METHODS

This study will explore text-based data, such as nursing notes, to improve the predictive capability of prognostic models for hospital-induced delirium. By using supervised and unsupervised text mining in addition to structured data, we will examine multiple types of information in electronic health record data to predict medical-surgical patient risk of developing delirium. Development and validation will be compliant to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

RESULTS

Work on this project will take place through March 2024. For this study, we will use data from approximately 332,230 encounters that occurred between January 2012 to May 2021. Findings from this project will be disseminated at scientific conferences and in peer-reviewed journals.

CONCLUSIONS

Success in this study will yield a durable, high-performing research-data infrastructure that will process, extract, and analyze clinical text data in near real time. This model has the potential to be integrated into the electronic health record and provide point-of-care decision support to prevent harm and improve quality of care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/48521.

Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model

Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.

Using Visual Patient to Show Vital Sign Predictions, a Computer-Based Mixed Quantitative and Qualitative Simulation Study

BACKGROUND

Machine learning can analyze vast amounts of data and make predictions for events in the future. Our group created machine learning models for vital sign predictions. To transport the information of these predictions without numbers and numerical values and make them easily usable for human caregivers, we aimed to integrate them into the Philips Visual-Patient-avatar, an avatar-based visualization of patient monitoring.

METHODS

We conducted a computer-based simulation study with 70 participants in 3 European university hospitals. We validated the vital sign prediction visualizations by testing their identification by anesthesiologists and intensivists. Each prediction visualization consisted of a condition (e.g., low blood pressure) and an urgency (a visual indication of the timespan in which the condition is expected to occur). To obtain qualitative user feedback, we also conducted standardized interviews and derived statements that participants later rated in an online survey.

RESULTS

The mixed logistic regression model showed 77.9% (95% CI 73.2-82.0%) correct identification of prediction visualizations (i.e., condition and urgency both correctly identified) and 93.8% (95% CI 93.7-93.8%) for conditions only (i.e., without considering urgencies). A total of 49 out of 70 participants completed the online survey. The online survey participants agreed that the prediction visualizations were fun to use (32/49, 65.3%), and that they could imagine working with them in the future (30/49, 61.2%). They also agreed that identifying the urgencies was difficult (32/49, 65.3%).

CONCLUSIONS

This study found that care providers correctly identified >90% of the conditions (i.e., without considering urgencies). The accuracy of identification decreased when considering urgencies in addition to conditions. Therefore, in future development of the technology, we will focus on either only displaying conditions (without urgencies) or improving the visualizations of urgency to enhance usability for human users.

Machine learning algorithms for predicting determinants of COVID-19 mortality in South Africa

Background

COVID-19 has strained healthcare resources, necessitating efficient prognostication to triage patients effectively. This study quantified COVID-19 risk factors and predicted COVID-19 intensive care unit (ICU) mortality in South Africa based on machine learning algorithms.

Methods

Data for this study were obtained from 392 COVID-19 ICU patients enrolled between 26 March 2020 and 10 February 2021. We used an artificial neural network (ANN) and random forest (RF) to predict mortality among ICU patients and a semi-parametric logistic regression with nine covariates, including a grouping variable based on K-means clustering. Further evaluation of the algorithms was performed using sensitivity, accuracy, specificity, and Cohen's K statistics.

Results

From the semi-parametric logistic regression and ANN variable importance, age, gender, cluster, presence of severe symptoms, being on the ventilator, and comorbidities of asthma significantly contributed to ICU death. In particular, the odds of mortality were six times higher among asthmatic patients than non-asthmatic patients. In univariable and multivariate regression, advanced age, PF1 and 2, FiO2, severe symptoms, asthma, oxygen saturation, and cluster 4 were strongly predictive of mortality. The RF model revealed that intubation status, age, cluster, diabetes, and hypertension were the top five significant predictors of mortality. The ANN performed well with an accuracy of 71%, a precision of 83%, an F1 score of 100%, Matthew's correlation coefficient (MCC) score of 100%, and a recall of 88%. In addition, Cohen's k-value of 0.75 verified the most extreme discriminative power of the ANN. In comparison, the RF model provided a 76% recall, an 87% precision, and a 65% MCC.

Conclusion

Based on the findings, we can conclude that both ANN and RF can predict COVID-19 mortality in the ICU with accuracy. The proposed models accurately predict the prognosis of COVID-19 patients after diagnosis. The models can be used to prioritize COVID-19 patients with a high mortality risk in resource-constrained ICUs.

External validation of models for predicting cumulative live birth over multiple complete cycles of IVF treatment

STUDY QUESTION

Can two prediction models developed using data from 1999 to 2009 accurately predict the cumulative probability of live birth per woman over multiple complete cycles of IVF in an updated UK cohort?

SUMMARY ANSWER

After being updated, the models were able to estimate individualized chances of cumulative live birth over multiple complete cycles of IVF with greater accuracy.

WHAT IS KNOWN ALREADY

The McLernon models were the first to predict cumulative live birth over multiple complete cycles of IVF. They were converted into an online calculator called OPIS (Outcome Prediction In Subfertility) which has 3000 users per month on average. A previous study externally validated the McLernon models using a Dutch prospective cohort containing data from 2011 to 2014. With changes in IVF practice over time, it is important that the McLernon models are externally validated on a more recent cohort of patients to ensure that predictions remain accurate.

STUDY DESIGN, SIZE, DURATION

A population-based cohort of 91 035 women undergoing IVF in the UK between January 2010 and December 2016 was used for external validation. Data on frozen embryo transfers associated with these complete IVF cycles conducted from 1 January 2017 to 31 December 2017 were also collected.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Data on IVF treatments were obtained from the Human Fertilisation and Embryology Authority (HFEA). The predictive performances of the McLernon models were evaluated in terms of discrimination and calibration. Discrimination was assessed using the c-statistic and calibration was assessed using calibration-in-the-large, calibration slope, and calibration plots. Where any model demonstrated poor calibration in the validation cohort, the models were updated using intercept recalibration, logistic recalibration, or model revision to improve model performance.

MAIN RESULTS AND THE ROLE OF CHANCE

Following exclusions, 91 035 women who underwent 144 734 complete cycles were included. The validation cohort had a similar distribution age profile to women in the development cohort. Live birth rates over all complete cycles of IVF per woman were higher in the validation cohort. After calibration assessment, both models required updating. The coefficients of the pre-treatment model were revised, and the updated model showed reasonable discrimination (c-statistic: 0.67, 95% CI: 0.66 to 0.68). After logistic recalibration, the post-treatment model showed good discrimination (c-statistic: 0.75, 95% CI: 0.74 to 0.76). As an example, in the updated pre-treatment model, a 32-year-old woman with 2 years of primary infertility has a 42% chance of having a live birth in the first complete ICSI cycle and a 77% chance over three complete cycles. In a couple with 2 years of primary male factor infertility where a 30-year-old woman has 15 oocytes collected in the first cycle, a single fresh blastocyst embryo transferred in the first cycle and spare embryos cryopreserved, the estimated chance of live birth provided by the post-treatment model is 46% in the first complete ICSI cycle and 81% over three complete cycles.

LIMITATIONS, REASONS FOR CAUTION

Two predictors from the original models, duration of infertility and previous pregnancy, which were not available in the recent HFEA dataset, were imputed using data from the older cohort used to develop the models. The HFEA dataset does not contain some other potentially important predictors, e.g. BMI, ethnicity, race, smoking and alcohol intake in women, as well as measures of ovarian reserve such as antral follicle count.

WIDER IMPLICATIONS OF THE FINDINGS

Both updated models show improved predictive ability and provide estimates which are more reflective of current practice and patient case mix. The updated OPIS tool can be used by clinicians to help shape couples' expectations by informing them of their individualized chances of live birth over a sequence of multiple complete cycles of IVF.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by an Elphinstone scholarship scheme at the University of Aberdeen and Aberdeen Fertility Centre, University of Aberdeen. S.B. has a commitment of research funding from Merck. D.J.M. and M.B.R. declare support for the present manuscript from Elphinstone scholarship scheme at the University of Aberdeen and Assisted Reproduction Unit at Aberdeen Fertility Centre, University of Aberdeen. D.J.M. declares grants received by University of Aberdeen from NHS Grampian, The Meikle Foundation, and Chief Scientist Office in the past 3 years. D.J.M. declares receiving an honorarium for lectures from Merck. D.J.M. is Associate Editor of Human Reproduction Open and Statistical Advisor for Reproductive BioMed Online. S.B. declares royalties from Cambridge University Press for a book. S.B. declares receiving an honorarium for lectures from Merck, Organon, Ferring, Obstetric and Gynaecological Society of Singapore, and Taiwanese Society for Reproductive Medicine. S.B. has received support from Merck, ESHRE, and Ferring for attending meetings as speaker and is on the METAFOR and CAPRE Trials Data Monitoring Committee.

TRIAL REGISTRATION NUMBER

N/A.

Development and validation of a prediction model based on machine learning algorithms for predicting the risk of heart failure in middle-aged and older US people with prediabetes or diabetes

BACKGROUND

The purpose of this study was to develop and validate a machine learning (ML) based prediction model for the risk of heart failure (HF) in patients with prediabetes or diabetes.

METHODS

We used 3527 subjects aged 40 years and older with a prior diagnosis of prediabetes or diabetes from the National Health and Nutrition Examination Survey (NHANES) from 2007 to 2018. The search for independent risk variables linked to HF was conducted using univariate and multivariable logistic regression analysis. The 3527 subjects were randomly divided into training set and validation set in a 7:3 ratio. Five ML models were built on the training set using five ML algorithms, including random forest (RF), and then validated on the validation set. Receiver operating characteristic (ROC) curves, calibration curves, and decision curve analysis and Bootstrap resampling method were used to measure the predictive performance of the five ML models.

RESULTS

Multivariate logistic regression analysis showed that age, poverty-to-income ratio, myocardial infarction condition, coronary heart disease condition, chest pain condition, and glucose-lowering medication use were independent predictors of HF. By comparing the performance of the five ML models, the RF model (AUC = 0.978) was the best prediction model.

CONCLUSIONS

The risk of HF in middle-aged and elderly patients with prediabetes or diabetes can be accurately predicted using ML models. The best prediction performance is presented by RF model, which can assist doctors in making clinical decisions.

Electronic health record data quality assessment and tools: a systematic review

OBJECTIVE

We extended a 2013 literature review on electronic health record (EHR) data quality assessment approaches and tools to determine recent improvements or changes in EHR data quality assessment methodologies.

MATERIALS AND METHODS

We completed a systematic review of PubMed articles from 2013 to April 2023 that discussed the quality assessment of EHR data. We screened and reviewed papers for the dimensions and methods defined in the original 2013 manuscript. We categorized papers as data quality outcomes of interest, tools, or opinion pieces. We abstracted and defined additional themes and methods though an iterative review process.

RESULTS

We included 103 papers in the review, of which 73 were data quality outcomes of interest papers, 22 were tools, and 8 were opinion pieces. The most common dimension of data quality assessed was completeness, followed by correctness, concordance, plausibility, and currency. We abstracted conformance and bias as 2 additional dimensions of data quality and structural agreement as an additional methodology.

DISCUSSION

There has been an increase in EHR data quality assessment publications since the original 2013 review. Consistent dimensions of EHR data quality continue to be assessed across applications. Despite consistent patterns of assessment, there still does not exist a standard approach for assessing EHR data quality.

CONCLUSION

Guidelines are needed for EHR data quality assessment to improve the efficiency, transparency, comparability, and interoperability of data quality assessment. These guidelines must be both scalable and flexible. Automation could be helpful in generalizing this process.

Artificial intelligence to empower diagnosis of myelodysplastic syndromes by multiparametric flow cytometry

The diagnosis of myelodysplastic syndromes (MDS) might be challenging and relies on the convergence of cytological, cytogenetic, and molecular factors. Multiparametric flow cytometry (MFC) helps diagnose MDS, especially when other features do not contribute to the decision-making process, but its usefulness remains underestimated, mostly due to a lack of standardization of cytometers. We present here an innovative model integrating artificial intelligence (AI) with MFC to improve the diagnosis and the classification of MDS. We develop a machine learning model through an elasticnet algorithm directed on a cohort of 191 patients, only based on flow cytometry parameters selected by the Boruta algorithm, to build a simple but reliable prediction score with five parameters. Our AI-assisted MDS prediction score greatly improves the sensitivity of the Ogata score while keeping an excellent specificity validated on an external cohort of 89 patients with an Area Under the Curve of 0.935. This model allows the diagnosis of both high- and low-risk MDS with 91.8% sensitivity and 92.5% specificity. Interestingly, it highlights a progressive evolution of the score from clonal hematopoiesis of indeterminate potential (CHIP) to highrisk MDS, suggesting a linear evolution between these different stages. By significantly decreasing the overall misclassification of 52% for patients with MDS and of 31.3% for those without MDS (P=0.02), our AI-assisted prediction score outperforms the Ogata score and positions itself as a reliable tool to help diagnose MDS.

AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges

Aim

Artificial intelligence (AI) and machine learning (ML) are important areas of computer science that have recently attracted attention for their application to medicine. However, as techniques continue to advance and become more complex, it is increasingly challenging for clinicians to stay abreast of the latest research. This overview aims to translate research concepts and potential concerns to healthcare professionals interested in applying AI and ML to resuscitation research but who are not experts in the field.

Main text

We present various research including prediction models using structured and unstructured data, exploring treatment heterogeneity, reinforcement learning, language processing, and large-scale language models. These studies potentially offer valuable insights for optimizing treatment strategies and clinical workflows. However, implementing AI and ML in clinical settings presents its own set of challenges. The availability of high-quality and reliable data is crucial for developing accurate ML models. A rigorous validation process and the integration of ML into clinical practice is essential for practical implementation. We furthermore highlight the potential risks associated with self-fulfilling prophecies and feedback loops, emphasizing the importance of transparency, interpretability, and trustworthiness in AI and ML models. These issues need to be addressed in order to establish reliable and trustworthy AI and ML models.

Conclusion

In this article, we overview concepts and examples of AI and ML research in the resuscitation field. Moving forward, appropriate understanding of ML and collaboration with relevant experts will be essential for researchers and clinicians to overcome the challenges and harness the full potential of AI and ML in resuscitation.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

BACKGROUND

Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome.

METHODS

We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size.

RESULTS

A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84).

CONCLUSIONS

Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

To predict the risk of chronic kidney disease (CKD) using Generalized Additive2 Models (GA2M)

OBJECTIVE

To train and test a model predicting chronic kidney disease (CKD) using the Generalized Additive2 Model (GA2M), and compare it with other models being obtained with traditional or machine learning approaches.

MATERIALS

We adopted the Health Search Database (HSD) which is a representative longitudinal database containing electronic healthcare records of approximately 2 million adults.

METHODS

We selected all patients aged 15 years or older being active in HSD between January 1, 2018 and December 31, 2020 with no prior diagnosis of CKD. The following models were trained and tested using 20 candidate determinants for incident CKD: logistic regression, Random Forest, Gradient Boosting Machines (GBMs), GAM, and GA2M. Their prediction performances were compared by calculating Area Under Curve (AUC) and Average Precision (AP).

RESULTS

Comparing the predictive performances of the 7 models, the AUC and AP for GBM and GA2M showed the highest values which were equal to 88.9%, 88.8% and 21.8%, 21.1%, respectively. These 2 models outperformed the others including logistic regression. In contrast to GBMs, GA2M kept the interpretability of variable combinations, including interactions and nonlinearities assessment.

DISCUSSION

Although GA2M is slightly less performant than light GBM, it is not "black-box" algorithm, so being simply interpretable using shape and heatmap functions. This evidence supports the fact machine learning techniques should be adopted in case of complex algorithms such as those predicting the risk of CKD.

CONCLUSION

The GA2M was reliably performant in predicting CKD in primary care. A related decision support system might be therefore implemented.

A Fast and Minimal System to Identify Depression Using Smartphones: Explainable Machine Learning-Based Approach

BACKGROUND

Existing robust, pervasive device-based systems developed in recent years to detect depression require data collected over a long period and may not be effective in cases where early detection is crucial. Additionally, due to the requirement of running systems in the background for prolonged periods, existing systems can be resource inefficient. As a result, these systems can be infeasible in low-resource settings.

OBJECTIVE

Our main objective was to develop a minimalistic system to identify depression using data retrieved in the fastest possible time. Another objective was to explain the machine learning (ML) models that were best for identifying depression.

METHODS

We developed a fast tool that retrieves the past 7 days' app usage data in 1 second (mean 0.31, SD 1.10 seconds). A total of 100 students from Bangladesh participated in our study, and our tool collected their app usage data and responses to the Patient Health Questionnaire-9. To identify depressed and nondepressed students, we developed a diverse set of ML models: linear, tree-based, and neural network-based models. We selected important features using the stable approach, along with 3 main types of feature selection (FS) approaches: filter, wrapper, and embedded methods. We developed and validated the models using the nested cross-validation method. Additionally, we explained the best ML models through the Shapley additive explanations (SHAP) method.

RESULTS

Leveraging only the app usage data retrieved in 1 second, our light gradient boosting machine model used the important features selected by the stable FS approach and correctly identified 82.4% (n=42) of depressed students (precision=75%, F1-score=78.5%). Moreover, after comprehensive exploration, we presented a parsimonious stacking model where around 5 features selected by the all-relevant FS approach Boruta were used in each iteration of validation and showed a maximum precision of 77.4% (balanced accuracy=77.9%). Feature importance analysis suggested app usage behavioral markers containing diurnal usage patterns as being more important than aggregated data-based markers. In addition, a SHAP analysis of our best models presented behavioral markers that were related to depression. For instance, students who were not depressed spent more time on education apps on weekdays, whereas those who were depressed used a higher number of photo and video apps and also had a higher deviation in using photo and video apps over the morning, afternoon, evening, and night time periods of the weekend.

CONCLUSIONS

Due to our system's fast and minimalistic nature, it may make a worthwhile contribution to identifying depression in underdeveloped and developing regions. In addition, our detailed discussion about the implication of our findings can facilitate the development of less resource-intensive systems to better understand students who are depressed and take steps for intervention.