Validation of a clinical decision tree to predict if a patient has a bacteraemia due to a β-lactamase producing organism

Machine learning for predicting antimicrobial resistance in critical and high-priority pathogens: A systematic review considering antimicrobial susceptibility tests in real-world healthcare settings

BACKGROUND

Antimicrobial resistance (AMR) poses a worldwide health threat; quick and accurate identification of AMR enhances patient outcomes and reduces inappropriate antibiotic usage. The objective of this systematic review is to evaluate the efficacy of machine learning (ML) approaches in predicting AMR in critical and high-priority pathogens (CHPP), considering antimicrobial susceptibility tests in real-world healthcare settings.

METHODS

The search methodology encompassed the examination of several databases, such as PubMed/MEDLINE, EMBASE, Web of Science, SCOPUS, and SCIELO. An extensive electronic database search was conducted from the inception of these databases until November 2024.

RESULTS

After completing the final step of the eligibility assessment, the systematic review ultimately included 21 papers. All included studies were cohort observational studies assessing 688,107 patients and 1,710,867 antimicrobial susceptibility tests. GBDT, Random Forest, and XGBoost were the top-performing ML models for predicting antibiotic resistance in CHPP infections. GBDT exhibited the highest AuROC values compared to Logistic Regression (LR), with a mean value of 0.80 (range 0.77-0.90) and 0.68 (range 0.50-0.83), respectively. Similarly, Random Forest generally showed better AuROC values compared to LR (mean value 0.75, range 0.58-0.98 versus mean value 0.71, range 0.61-0.83). However, some predictors selected by these algorithms align with those suggested by LR.

CONCLUSIONS

ML displays potential as a technology for predicting AMR, incorporating antimicrobial susceptibility tests in CHPP in real-world healthcare settings. However, limitations such as retrospective methodology for model development, nonstandard data processing, and lack of validation in randomized controlled trials must be considered before applying these models in clinical practice.

Machine learning and clinician predictions of antibiotic resistance in Enterobacterales bloodstream infections

BACKGROUND

Patients with Gram-negative bloodstream infections are at risk of serious adverse outcomes without active treatment, but identifying who has antimicrobial resistance (AMR) to target empirical treatment is challenging.

METHODS

We used XGBoost machine learning models to predict antimicrobial resistance to seven antibiotics in patients with Enterobacterales bloodstream infection. Models were trained using hospital and community data from Oxfordshire, UK, for patients with positive blood cultures between 01-January-2017 and 31-December-2021. Model performance was evaluated by comparing predictions to final microbiology results in test datasets from 01-January-2022 to 31-December-2023 and to clinicians' prescribing.

FINDINGS

4709 infection episodes were used for model training and evaluation; antibiotic resistance rates ranged from 7-67%. In held-out test data, resistance prediction performance was similar for the seven antibiotics (AUCs 0.680 [95%CI 0.641-0.720] to 0.737 [0.674-0.797]). Performance improved for most antibiotics when species identifications (available ∼24 h later) were included as model inputs (AUCs 0.723 [0.652-0.791] to 0.827 [0.797-0.857]). In patients treated with a beta-lactam, clinician prescribing led to 70% receiving an active beta-lactam: 44% were over-treated (broader spectrum treatment than needed), 26% optimally-treated (narrowest spectrum active agent), and 30% under-treated (inactive beta-lactam). Model predictions without species data could have led to 79% of patients receiving an active beta-lactam: 45% over-treated, 34% optimally-treated, and 21% under-treated.

CONCLUSIONS

Predicting AMR in bloodstream infections is challenging for both clinicians and models. Despite modest performance, machine learning models could still increase the proportion of patients receiving active empirical treatment by up to 9% over current clinical practice in an environment prioritising antimicrobial stewardship.

Retrospective validation study of a machine learning-based software for empirical and organism-targeted antibiotic therapy selection

Errors in antibiotic prescriptions are frequent, often resulting from the inadequate coverage of the infection-causative microorganism. The efficacy of iAST, a machine-learning-based software offering empirical and organism-targeted antibiotic recommendations, was assessed. The study was conducted in a 12-hospital Spanish institution. After model fine-tuning with 27,531 historical antibiograms, 325 consecutive patients with acute infections were selected for retrospective validation. The primary endpoint was comparing each of the top three of iAST's antibiotic recommendations' success rates (confirmed by antibiogram results) with the antibiotic prescribed by the physicians. Secondary endpoints included examining the same hypothesis within specific study population subgroups and assessing antibiotic stewardship by comparing the percentage of antibiotics recommended that belonged to different World Health Organization AWaRe groups within each arm of the study. All of iAST first three recommendations were non-inferior to doctor prescription in the primary endpoint analysis population as well as the secondary endpoint. The overall success rate of doctors' empirical treatment was 68.93%, while that of the first three iAST options was 91.06% (P < 0.001), 90.63% (P < 0.001), and 91.06% (P < 001), respectively. For organism-targeted therapy, the doctor's overall success rate was 84.16%, and that of the first three ranked iAST options was 97.83% (P < 0.001), 94.09% (P < 0.001), and 91.30% (P < 0.001), respectively. In empirical therapy, compared to physician prescriptions, iAST demonstrated a greater propensity to recommend access antibiotics, fewer watch antibiotics, and higher reserve antibiotics. In organism-targeted therapy, iAST advised a higher utilization of access antibiotics. The present study demonstrates iAST accuracy in predicting antibiotic susceptibility, showcasing its potential to promote effective antibiotic stewardship.

CLINICAL TRIALS

This study is registered with ClinicalTrials.gov as NCT06174519.

Machine learning to predict antimicrobial resistance: future applications in clinical practice?

INTRODUCTION

Machine learning (ML) is increasingly being used to predict antimicrobial resistance (AMR). This review aims to provide physicians with an overview of the literature on ML as a means of AMR prediction.

METHODS

References for this review were identified through searches of MEDLINE/PubMed, EMBASE, Google Scholar, ACM Digital Library, and IEEE Xplore Digital Library up to December 2023.

RESULTS

Thirty-six studies were included in this review. Thirty-two studies (32/36, 89 %) were based on hospital data and four (4/36, 11 %) on outpatient data. The vast majority of them were conducted in high-resource settings (33/36, 92 %). Twenty-four (24/36, 67 %) studies developed systems to predict drug resistance in infected patients, eight (8/36, 22 %) tested the performances of ML-assisted antibiotic prescription, two (2/36, 6 %) assessed ML performances in predicting colonization with carbapenem-resistant bacteria and, finally, two assessed national and international AMR trends. The most common inputs were demographic characteristics (25/36, 70 %), previous antibiotic susceptibility testing (19/36, 53 %) and prior antibiotic exposure (15/36, 42 %). Thirty-three (92 %) studies targeted prediction of Gram-negative bacteria (GNB) resistance as an output (92 %). The studies included showed moderate to high performances, with AUROC ranging from 0.56 to 0.93.

CONCLUSION

ML can potentially provide valuable assistance in AMR prediction. Although the literature on this topic is growing, future studies are needed to design, implement, and evaluate the use and impact of ML decision support systems.

Machine learning-based antibiotic resistance prediction models: An updated systematic review and meta-analysis

BACKGROUND

The widespread use of antibiotics has led to a gradual adaptation of bacteria to these drugs, diminishing the effectiveness of treatments.

OBJECTIVE

To comprehensively assess the research progress of antibiotic resistance prediction models based on machine learning (ML) algorithms, providing the latest quantitative analysis and methodological evaluation.

METHODS

Relevant literature was systematically retrieved from databases, including PubMed, Embase and the Cochrane Library, from inception up to December 2023. Studies meeting predefined criteria were selected for inclusion. The prediction model risk of bias assessment tool was employed for methodological quality assessment, and a random-effects model was utilised for meta-analysis.

RESULTS

The systematic review included a total of 22 studies with a combined sample size of 43,628; 10 studies were ultimately included in the meta-analysis. Commonly used ML algorithms included random forest, decision trees and neural networks. Frequently utilised predictive variables encompassed demographics, drug use history and underlying diseases. The overall sensitivity was 0.57 (95% CI: 0.42-0.70; p< 0.001; I2= 99.7%), the specificity was 0.95 (95% CI: 0.79-0.99; p< 0.001; I2 = 99.9%), the positive likelihood ratio was 10.7 (95% CI: 2.9-39.5), the negative likelihood ratio was 0.46 (95% CI: 0.34-0.61), the diagnostic odds ratio was 23 (95% CI: 7-81) and the area under the receiver operating characteristic curve was 0.78 (95% CI: 0.74-0.81; p< 0.001), indicating a good discriminative ability of ML models for antibiotic resistance. However, methodological assessment and funnel plots suggested a high risk of bias and publication bias in the included studies.

CONCLUSION

This meta-analysis provides a current and comprehensive evaluation of ML models for predicting antibiotic resistance, emphasising their potential application in clinical practice. Nevertheless, stringent research design and reporting are warranted to enhance the quality and credibility of future studies. Future research should focus on methodological innovation and incorporate more high-quality studies to further advance this field.

Using Machine Learning to Predict Antimicrobial Resistance-A Literature Review

Machine learning (ML) algorithms are increasingly applied in medical research and in healthcare, gradually improving clinical practice. Among various applications of these novel methods, their usage in the combat against antimicrobial resistance (AMR) is one of the most crucial areas of interest, as increasing resistance to antibiotics and management of difficult-to-treat multidrug-resistant infections are significant challenges for most countries worldwide, with life-threatening consequences. As antibiotic efficacy and treatment options decrease, the need for implementation of multimodal antibiotic stewardship programs is of utmost importance in order to restrict antibiotic misuse and prevent further aggravation of the AMR problem. Both supervised and unsupervised machine learning tools have been successfully used to predict early antibiotic resistance, and thus support clinicians in selecting appropriate therapy. In this paper, we reviewed the existing literature on machine learning and artificial intelligence (AI) in general in conjunction with antimicrobial resistance prediction. This is a narrative review, where we discuss the applications of ML methods in the field of AMR and their value as a complementary tool in the antibiotic stewardship practice, mainly from the clinician's point of view.

Machine learning in predicting antimicrobial resistance: a systematic review and meta-analysis

INTRODUCTION

Antimicrobial resistance (AMR) is a global health threat; rapid and timely identification of AMR improves patient prognosis and reduces inappropriate antibiotic use.

METHODS

Relevant literature in PubMed, Web of Science, Embase and Institute of Electrical and Electronics Engineers prior to 28 September 2021 was searched. Any study that deployed machine learning (ML) or a risk score as a tool to predict AMR was included in the final review; there were 25 studies that employed the ML algorithm to predict AMR.

RESULTS

Extended spectrum β-lactamases, methicillin-resistant Staphylococcus aureus (MRSA) and carbapenem resistance were the most common outcomes in studies with a specific resistance pattern. The most common algorithms in ML prediction were logistic regression (n = 14 studies), decision tree (n = 14) and random forest (n = 7). The area under the curve (AUC) range for ML prediction was 0.48-0.93. The pooled AUC for ML prediction was 0.82 (0.78-0.85). Compared with risk score, higher specificity [87% (82-91) vs. 37% (25-51)] was indicated for ML prediction, but not sensitivity [67% (62-72) vs. 73% (67-79)].

CONCLUSIONS

Machine learning might be a potential technology for AMR prediction; however, retrospective methodology for model development, nonstandard data processing and scarcity of validation in a randomised controlled trial or real-world study limit the application of these models in clinical practice.

Comparison of the performance of a clinical classification tree versus clinical gestalt in predicting sepsis with extended-spectrum beta-lactamase–producing gram-negative rods

A clinical decision tree was developed using point-of-care characteristics to identify patients with culture-proven sepsis due to extended-spectrum β-lactamase–producing Enterobacterales (ESBL-PE). We compared its performance with the clinical gestalt of emergency department (ED) clinicians and hospital-based clinicians. The developed tree outperformed ED-based clinicians but was comparable to inpatient-based clinicians.

An international prospective cohort study to validate two prediction rules for infections caused by 3rd-generation cephalosporin-resistant Enterobacterales

Background

The possibility of bloodstream infections caused by third-generation cephalosporin-resistant Enterobacterales (3GC-R-BSI) leads to a trade-off between empiric inappropriate treatment (IAT) and unnecessary carbapenem use (UCU). Accurately predicting 3GC-R-BSI could reduce IAT and UCU. We externally validate 2 previously derived prediction rules for community-onset (CO) and hospital-onset (HO) suspected bloodstream infections.

Methods

In 33 hospitals in 13 countries we prospectively enrolled 200 patients per hospital in whom blood cultures were obtained and intravenous antibiotics with coverage for Enterobacterales were empirically started. Cases were defined as 3GC-R-BSI or 3GC-R gram-negative infection (3GC-R-GNI) (analysis 2); all other outcomes served as a comparator. Model discrimination and calibration were assessed. Impact on carbapenem use was assessed at several cutoff points.

Results

4650 CO infection episodes were included and the prevalence of 3GC-R-BSI was 2.1% (n = 97). IAT occurred in 69 of 97 (71.1%) 3GC-R-BSI and UCU in 398 of 4553 non–3GC-R-BSI patients (8.7%). Model calibration was good, and the AUC was .79 (95% CI, .75–.83) for 3GC-R-BSI. The prediction rule potentially reduced IAT to 62% (60/97) while keeping UCU comparable at 8.4% or could reduce UCU to 6.3% (287/4553) while keeping IAT equal. IAT and UCU in all 3GC-R-GNIs (analysis 2) improved at similar percentages. 1683 HO infection episodes were included and the prevalence of 3GC-R-BSI was 4.9% (n = 83). Here model calibration was insufficient.

Conclusions

A prediction rule for CO 3GC-R infection was validated in an international cohort and could improve empirical antibiotic use. Validation of the HO rule yielded suboptimal performance.