Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool

Electronic clinical decision support system guided blood culture stewardship in emergency departments: response to the national blood culture media shortage

CDSS-guided stewardship in six EDs during national culture bottles shortage was associated with significant reduction in median daily blood culture utilization per 1,000 ED visits from 141.5 (IQR:127.6-155.3) to 77.9 (IQR:68.3-86.3) and increased diagnostic yield from 6.2%(IQR:4.7%-7.6%) to 8.8%(IQR:6.1%-11.5%), without impacting length of stay or mortality.

A comparison of blood culture diagnostic stewardship across three emergency departments in a healthcare network

BACKGROUND

Blood culture (BCx) diagnostic stewardship is essential for reducing unnecessary treatments, minimizing false-positive results, and improving patient outcomes and hospital resource utilization. The objective of this study was to compare the effectiveness of diagnostic stewardship interventions on BCx utilization in three emergency departments (ED).

METHODS

We used a quasi-experimental pre-/post-intervention study to compare BCx rates (BCx/100 ED visits) between December 1, 2020, and February 29, 2024 before and after the implementation of a BCx algorithm and electronic health record (EHR) modifications at one large academic ED and level 1 trauma center, and two EDs at academic-affiliated community hospitals. A sample of visits with a BCx order were audited in one academic ED and one academic-affiliated ED, and summary data on indication appropriateness were provided to respective leadership. In the academic ED, there was weekly provider led audit and feedback on 3478 ED visits. In one academic-affiliated ED, one pharmacist reviewed five visits weekly (100 total) for appropriateness. The second academic-affiliated ED served as a control and did not receive any feedback on BCx utilization. Each ED's BCx rates were analyzed using interrupted time series models. Incidence rate ratios (IRR) compared BCx rates before and after the interventions.

RESULTS

A total of 211,950 BCxs over 572,776 ED visits were included in the analysis. The academic ED saw a 25 % decrease in BCx rate with IRR 0.80 (95 % CI 0.74, 0.86, p-value 0.01). The first academic-affiliated ED experienced a 0.8 % decrease in BCx rate with IRR 1.1 (95 % CI 1.01, 1.19, p-value 0.02). No change was observed in the second academic-affiliated ED.

CONCLUSIONS

Decreased BCx rates occurred only after direct audit and feedback and EHR modifications. Both the academic ED and first academic-affiliated ED saw a drift back towards pre-intervention BCx rates after the intervention.

Can we rely on artificial intelligence to guide antimicrobial therapy? A systematic literature review

Background

Artificial intelligence (AI) has the potential to enhance clinical decision-making, including in infectious diseases. By improving antimicrobial resistance prediction and optimizing antibiotic prescriptions, these technologies may support treatment strategies and address critical gaps in healthcare. This study evaluates the effectiveness of AI in guiding appropriate antibiotic prescriptions for infectious diseases through a systematic literature review.

Methods

We conducted a systematic review of studies evaluating AI (machine learning or large language models) used for guidance on prescribing appropriate antibiotics in infectious disease cases. Searches were performed in PubMed, CINAHL, Embase, Scopus, Web of Science, and Google Scholar for articles published up to October 25, 2024. Inclusion criteria focused on studies assessing the performance of AI in clinical practice, with outcomes related to antimicrobial management and decision-making.

Results

Seventeen studies used machine learning as part of clinical decision support systems (CDSS). They improved prediction of antimicrobial resistance and optimized antimicrobial use. Six studies focused on large language models to guide antimicrobial therapy; they had higher prescribing error rates, patient safety risks, and needed precise prompts to ensure accurate responses.

Conclusions

AI, particularly machine learning integrated into CDSS, holds promise in enhancing clinical decision-making and improving antimicrobial management. However, large language models currently lack the reliability required for complex clinical applications. The indispensable role of infectious disease specialists remains critical for ensuring accurate, personalized, and safe treatment strategies. Rigorous validation and regular updates are essential before the successful integration of AI into clinical practice.

Artificial Intelligence in Bacterial Infections Control: A Scoping Review

Background/Objectives: Artificial intelligence has made significant strides in healthcare, contributing to diagnosing, treating, monitoring, preventing, and testing various diseases. Despite its broad adoption, clinical consensus on AI's role in infection control remains uncertain. This scoping review aims to understand the characteristics of AI applications in bacterial infection control. Results: This review examines the characteristics of AI applications in bacterial infection control, analyzing 54 eligible studies across 5 thematic scopes. The search from 3 databases yielded a total of 1165 articles, only 54 articles met the eligibility criteria and were extracted and analyzed. Five thematic scopes were synthesized from the extracted data; countries, aim, type of AI, advantages, and limitations of AI applications in bacterial infection prevention and control. The majority of articles were reported from high-income countries, mainly by the USA. The most common aims are pathogen identification and infection risk assessment. The most common AI used in infection control is machine learning. The commonest reported advantage is predictive modeling and risk assessment, and the commonest disadvantage is generalizability of the models. Methods: This scoping review was developed according to Arksey and O'Malley frameworks. A comprehensive search across PubMed, Embase, and Web of Science was conducted using broad search terms, with no restrictions. Publications focusing on AI in infection control and prevention were included. Citations were managed via EndNote, with initial title and abstract screening by two authors. Data underwent comprehensive narrative mapping and categorization, followed by the construction of thematic scopes. Conclusions: Artificial intelligence applications in infection control need to be strengthened for low-income countries. More efforts should be dedicated to investing in models that have proven their effectiveness in infection control, to maximize their utilization and tackle challenges.

Impact of an Algorithm to Triage Patients Discharged From the Emergency Department With Blood Cultures Positive for Staphylococcus aureus or Coagulase-Negative Staphylococcus

Objectives

Blood cultures obtained in the emergency department (ED) may become positive after discharge. Healthcare professionals must determine if these results represent true infection or a likely contaminant. An institutional algorithm was developed to assist with healthcare professional response to positive blood cultures for S. aureus and coagulase-negative staphylococci (CoNS) in these situations.

Methods

We conducted a single system, multisite cohort study comparing before and after implementation of an ED decision-making algorithm from November 2022 to December 2023. Adults were included if they were discharged from the ED before blood cultures became positive for Staphylococcus species. The primary outcome was the difference in rates of patients called back to the ED pre- and postalgorithm implementation. Secondary endpoints evaluated algorithm adherence and safety.

Results

A total of 253 patients, 188 pre- and 65 postimplementation, were enrolled. There was a 7.3% reduction in patients called back to the ED after algorithm implementation (95% CI [-21.1 to 6.3], P = .3). Algorithm adherence after implementation was 84.6% with a difference in actual and algorithm-based callback rates of 4.6%. After algorithm implementation, no patients deemed to have a contaminant experienced an infectious-related safety event.

Conclusions

This time-saving algorithm was well received by our ED professionals and served as a helpful tool in safely and effectively triaging patients who had positive blood cultures for Staphylococcus species after discharge to determine who should be called back for further evaluation. There was a nonstatistically significant but clinically meaningful reduction in callback rates. Postimplementation algorithm adherence was high, and the majority of callback decisions were appropriate.

Twenty-Five Years of Evolution and Hurdles in Electronic Health Records and Interoperability in Medical Research: Comprehensive Review

BACKGROUND

Electronic health records (EHRs) facilitate the accessibility and sharing of patient data among various health care providers, contributing to more coordinated and efficient care.

OBJECTIVE

This study aimed to summarize the evolution of secondary use of EHRs and their interoperability in medical research over the past 25 years.

METHODS

We conducted an extensive literature search in the PubMed, Scopus, and Web of Science databases using the keywords Electronic health record and Electronic medical record in the title or abstract and Medical research in all fields from 2000 to 2024. Specific terms were applied to different time periods.

RESULTS

The review yielded 2212 studies, all of which were then screened and processed in a structured manner. Of these 2212 studies, 2102 (93.03%) were included in the review analysis, of which 1079 (51.33%) studies were from 2000 to 2009, 582 (27.69%) were from 2010 to 2019, 251 (11.94%) were from 2020 to 2023, and 190 (9.04%) were from 2024.

CONCLUSIONS

The evolution of EHRs marks an important milestone in health care's journey toward integrating technology and medicine. From early documentation practices to the sophisticated use of artificial intelligence and big data analytics today, EHRs have become central to improving patient care, enhancing public health surveillance, and advancing medical research.

A Comprehensive Evaluation of Emergency Department Utilization by Patients With Cirrhosis

INTRODUCTION

Emergency department (ED)-based care is required for cirrhosis management, yet the burden of cirrhosis-related ED healthcare utilization is understudied. We aimed to describe ED utilization within a statewide health system and compare the outcomes of high ED use (HEDU) vs non-HEDU in individuals with cirrhosis.

METHODS

We retrospectively reviewed charts of adults with cirrhosis who presented to any of 16 EDs within the Indiana University Health system in 2021. Patient characteristics, features of the initial ED visit, subsequent 90-day healthcare use, and 360-day outcomes were collected. Multivariable logistic regression models were used to identify predictors HEDU status which was defined as ≥2 ED visits within 90 days after the index ED visit.

RESULTS

There were 2,124 eligible patients (mean age 61.3 years, 53% male, and 91% White). Major etiologies of cirrhosis were alcohol (38%), metabolic dysfunction-associated steatohepatitis (27%), and viral hepatitis (21%). Cirrhosis was newly diagnosed in the ED visit for 18.4%. Most common reasons for ED visits were abdominal pain (21%), shortness of breath (19%), and ascites/volume overload (16%). Of the initial ED visits, 20% (n = 424) were potentially avoidable. The overall 90-day mortality was 16%. Within 90 days, there were 366 HEDU (20%). Notable variables independently associated with HEDU were model for end-stage liver disease-sodium (adjusted odds ratio [aOR] 1.044, 95% confidence interval [CI] 1.005-1.085), prior ED encounter (aOR 1.520, 95% CI 1.136-2.034), and avoidable initial ED visit (aOR 1.938, 95% CI 1.014-3.703).

DISCUSSION

Abdominal pain, shortness of breath, and ascites/fluid overload are the common presenting reasons for ED visits for patients with cirrhosis. Patients with cirrhosis presenting to the ED experience a 90-day mortality rate of 16%, and among those who initially visited the ED, 20% were HEDU. We identified several variables independently associated with HEDU. Our observations pave the way for developing interventions to optimize the care of patients with cirrhosis presenting to the ED and to lower repeated ED visits.

Assessing sepsis-induced immunosuppression to predict positive blood cultures

Introduction

Bacteremia is a life-threatening condition that can progress to sepsis and septic shock, leading to significant mortality in the emergency department (ED). The standard diagnostic method, blood culture, is time-consuming and prone to false positives and false negatives. Although not widely accepted, several clinical and artificial intelligence-based algorithms have been recently developed to predict bacteremia. However, these strategies require further identification of new variables to improve their diagnostic accuracy. This study proposes a novel strategy to predict positive blood cultures by assessing sepsis-induced immunosuppression status through endotoxin tolerance assessment.

Methods

Optimal assay conditions have been explored and tested in sepsis-suspected patients meeting the Sepsis-3 criteria. Blood samples were collected at ED admission, and endotoxin (lipopolysaccharide, LPS) challenge was performed to evaluate the innate immune response through cytokine profiling.

Results

Clinical variables, immune cell population biomarkers, and cytokine levels (tumor necrosis factor [TNFα], IL-1β, IL-6, IL-8, and IL-10) were measured. Patients with positive blood cultures exhibited significantly lower TNFα production after LPS challenge than did those with negative blood cultures. The study also included a validation cohort to confirm that the response was consistent.

Discussion

The results of this study highlight the innate immune system immunosuppression state as a critical parameter for sepsis diagnosis. Notably, the present study identified a reduction in monocyte populations and specific cytokine profiles as potential predictive markers. This study showed that the LPS challenge can be used to effectively distinguish between patients with bloodstream infection leading to sepsis and those whose blood cultures are negative, providinga rapid and reliable diagnostic tool to predict positive blood cultures. The potential applicability of these findings could enhance clinical practice in terms of the accuracy and promptness of sepsis diagnosis in the ED, improving patient outcomes through timely and appropriate treatment.

Leveraging explainable artificial intelligence for early prediction of bloodstream infections using historical electronic health records

Bloodstream infections (BSIs) are a severe public health threat due to their rapid progression into critical conditions like sepsis. This study presents a novel eXplainable Artificial Intelligence (XAI) framework to predict BSIs using historical electronic health records (EHRs). Leveraging a dataset from St. Olavs Hospital in Trondheim, Norway, encompassing 35,591 patients, the framework integrates demographic, laboratory, and comprehensive medical history data to classify patients into high-risk and low-risk BSI groups. By avoiding reliance on real-time clinical data, our model allows for enhanced scalability across various healthcare settings, including resource-limited environments. The XAI framework significantly outperformed traditional models, particularly with tree-based algorithms, demonstrating superior specificity and sensitivity in BSI prediction. This approach promises to optimize resource allocation and potentially reduce healthcare costs while providing interpretability for clinical decision-making, making it a valuable tool in hospital systems for early intervention and improved patient outcomes.

Secondary Bacterial Infections in Critically Ill COVID-19 Patients Admitted in the Intensive Care Unit of a Tertiary Hospital in Romania

Background: The outbreak of the COVID-19 pandemic caught healthcare systems in many countries unprepared. Shortages of personnel, medicines, disinfectants, and intensive care unit (ICU) capacities, combined with inadvertent use of antibiotics and emergence of drug-resistant secondary infections, led to a surge in COVID-19-related mortality. Objective: We aimed to evaluate the prevalence of secondary bacterial infections and the associated antibiotic resistance in a temporary established ICU dedicated to COVID-19 patients. We also assessed the utility of clinical and routine laboratory data as predictors of secondary infections and mortality in these patients. Methods: We examined the medical records of 243 patients admitted to the COVID-19 Medical Support Unit of Târgu Mures, Romania, between 1 August 2020 and 31 January 2021. Results: Among the 243 patients admitted to the COVID-19 Medical Support Unit of Târgu Mures between 1 August 2020 and 31 January 2021, 59 (24.3%) presented secondary infections. Acinetobacter baumannii and Klebsiella pneumoniae were the most frequent isolates (31.1% and 18.9%, respectively), most of them multidrug resistant. Chronic obstructive pulmonary disease had a higher prevalence in patients who developed secondary infections (p = 0.012). Secondary infections were associated with longer stay in the ICU and with higher mortality (p = 0.006 and p = 0.038, respectively). Conclusions: Early identification of secondary infections and proper use of antibiotics are necessary to limit the spread of multidrug-resistant microorganisms in COVID-19 patients admitted in the ICU.

Effectiveness of multidisciplinary interventions to improve blood culture efficiency and optimize antimicrobial utilization

Background

The low positive rate of blood cultures often leads to downstream consequences. We present a summary of multidisciplinary interventions implemented by a tertiary referral hospital to improve blood culture efficiency and optimize antimicrobial usage.

Methods

We evaluated the knowledge, attitude, and practice (KAP) of healthcare workers in a tertiary care hospital before and after intervention using a questionnaire. A multidisciplinary team was formed to implement the intervention, defining roles, standardizing procedures, continually improving education and feedback, and establishing incentive mechanisms. Regular quality control assessments are conducted on the responsible departments.

Results

Following the intervention, the median submission time for blood culture specimens was reduced from 2.2 h to 1.3 h (p < 0.001). Additionally, the intervention group showed significant (p < 0.05) increases in rates of positivity (9.9% vs. 8.6%), correct timing (98.7% vs. 89.6%), correct processing (98.1% vs. 92.3%), reduced contamination rates (0.9% vs. 1.4%), and disqualification rates (1.3% vs. 1.7%). The delivery rate of therapeutic antibacterial increased (16.1% vs. 15.2%), and the consumption of restrictive grade antimicrobial also significantly increased (26.7% vs. 22.9%). The intervention measures led to a substantial improvement in awareness and compliance with KAP of blood culture collection in the hospital. Hospital-wide antimicrobial usage deceased by 10.7% after intervention.

Conclusion

A multidisciplinary collaborative model proves effective in improving blood culture efficiency and optimizing antimicrobial usage.

Procalcitonin for safe reduction of unnecessary blood cultures in the emergency department: Development and validation of a prediction model

OBJECTIVES

Blood cultures (BCs) are commonly ordered in emergency departments (EDs), while a minority yields a relevant pathogen. Diagnostic stewardship is needed to safely reduce unnecessary BCs. We aimed to develop and validate a bacteremia prediction model for ED patients, with specific focus on the benefit of incorporating procalcitonin.

METHODS

We included adult patients with suspected bacteremia from a Dutch ED for a one-year period. We defined 23 candidate predictors for a "full model", of which nine were used for an automatable "basic model". Variations of both models with C-reactive protein and procalcitonin were constructed using LASSO regression, with bootstrapping for internal validation. External validation was done in an independent cohort of patients with confirmed infection from 71 Spanish EDs. We assessed discriminative performance using the C-statistic and calibration with calibration curves. Clinical usefulness was evaluated by sensitivity, specificity, saved BCs, and Net Benefit.

RESULTS

Among 2111 patients in the derivation cohort (mean age 63 years, 46% male), 273 (13%) had bacteremia, versus 896 (20%) in the external cohort (n = 4436). Adding procalcitonin substantially improved performance for all models. The basic model with procalcitonin showed most promise, with a C-statistic of 0.87 (0.86-0.88) upon external validation. At a 5% risk threshold, it showed a sensitivity of 99% and could have saved 29% of BCs while only missing 10 out of 896 (1.1%) bacteremia patients.

CONCLUSIONS

Procalcitonin-based bacteremia prediction models can safely reduce unnecessary BCs at the ED. Further validation is needed across a broader range of healthcare settings.

Artificial intelligence and prescription of antibiotic therapy: present and future

INTRODUCTION

In the past few years, the use of artificial intelligence in healthcare has grown exponentially. Prescription of antibiotics is not exempt from its rapid diffusion, and various machine learning (ML) techniques, from logistic regression to deep neural networks and large language models, have been explored in the literature to support decisions regarding antibiotic prescription.

AREAS COVERED

In this narrative review, we discuss promises and challenges of the application of ML-based clinical decision support systems (ML-CDSSs) for antibiotic prescription. A search was conducted in PubMed up to April 2024.

EXPERT OPINION

Prescribing antibiotics is a complex process involving various dynamic phases. In each of these phases, the support of ML-CDSSs has shown the potential, and also the actual ability in some studies, to favorably impacting relevant clinical outcomes. Nonetheless, before widely exploiting this massive potential, there are still crucial challenges ahead that are being intensively investigated, pertaining to the transparency of training data, the definition of the sufficient degree of prediction explanations when predictions are obtained through black box models, and the legal and ethical framework for decision responsibility whenever an antibiotic prescription is supported by ML-CDSSs.

Blood culture algorithm implementation in emergency department patients as a diagnostic stewardship intervention

OBJECTIVE

Blood cultures (BCx) are important for selecting appropriate antibiotic treatment. Ordering BCx for conditions with a low probability of bacteremia has limited utility, thus improved guidance for ordering BCx is needed. Inpatient studies have implemented BCx algorithms, but no studies examine the intervention in an Emergency Department (ED) setting.

METHODS

We performed a quasi-experimental pre and postintervention study from January 12, 2020, to October 31, 2023, at a single academic adult ED and implemented a BCx algorithm. The primary outcome was the blood culture event rates (BCE per 100 ED admissions) pre and postintervention. Secondary outcomes included adverse event rates (30-day ED and hospital readmission and antibiotic days of therapy). Seven ED physicians and APP reviewed BCx for appropriateness, with monthly feedback provided to ED leadership and physicians.

RESULTS

After the BCx algorithm implementation, the BCE rate decreased from 12.17 BCE/100 ED admissions to 10.50 BCE/100 ED admissions. Of the 3,478 reviewed BCE, we adjudicated 2,153 BCE (62%) as appropriate, 653 (19%) as inappropriate, and 672 (19%) as uncertain. Adverse safety events were not statistically different pre and postintervention.

CONCLUSIONS

Implementation of an ED BCx algorithm demonstrated a reduction in BCE, without increased adverse safety events. Future studies should compare outcomes of BCx algorithm implementation in a community hospital ED without intensive chart review.

Appropriate use of blood cultures in the emergency department through machine learning (ABC): study protocol for a randomised controlled non-inferiority trial

INTRODUCTION

The liberal use of blood cultures in emergency departments (EDs) leads to low yields and high numbers of false-positive results. False-positive, contaminated cultures are associated with prolonged hospital stays, increased antibiotic usage and even higher hospital mortality rates. This trial aims to investigate whether a recently developed and validated machine learning model for predicting blood culture outcomes can safely and effectively guide clinicians in withholding unnecessary blood culture analysis.

METHODS AND ANALYSIS

A randomised controlled, non-inferiority trial comparing current practice with a machine learning-guided approach. The primary objective is to determine whether the machine learning based approach is non-inferior to standard practice based on 30-day mortality. Secondary outcomes include hospital length-of stay and hospital admission rates. Other outcomes include model performance and antibiotic usage. Participants will be recruited in the EDs of multiple hospitals in the Netherlands. A total of 7584 participants will be included.

ETHICS AND DISSEMINATION

Possible participants will receive verbal information and a paper information brochure regarding the trial. They will be given at least 1 hour consideration time before providing informed consent. Research results will be published in peer-reviewed journals. This study has been approved by the Amsterdam University Medical Centers' local medical ethics review committee (No 22.0567). The study will be conducted in concordance with the principles of the Declaration of Helsinki and in accordance with the Medical Research Involving Human Subjects Act, General Data Privacy Regulation and Medical Device Regulation.

TRIAL REGISTRATION NUMBER

NCT06163781.

Diagnostic stewardship in infectious diseases: a scoping review

Introduction. The term 'diagnostic stewardship' is relatively new, with a recent surge in its use within the literature. Despite its increasing popularity, a precise definition remains elusive. Various attempts have been made to define it, with some viewing it as an integral part of antimicrobial stewardship. The World Health Organization offers a broad definition, emphasizing the importance of timely, accurate diagnostics. However, inconsistencies in the use of this term still persist, necessitating further clarification.Gap Statement. There are currently inconsistencies in the definition of diagnostic stewardship used within the academic literature.Aim. This scoping review aims to categorize the use of diagnostic stewardship approaches and define this approach by identifying common characteristics and factors of its use within the literature.Methodology. This scoping review undertook a multi-database search from date of inception until October 2022. Any observational or experimental study where the authors define the intervention to be diagnostic stewardship from any clinical area was included. Screening of all papers was undertaken by a single reviewer with 10% verification by a second reviewer. Data extraction was undertaken by a single reviewer using a pre-piloted form. Given the wide variation in study design and intervention outcomes, a narrative synthesis approach was applied. Studies were clustered around common diagnostic stewardship interventions where appropriate.Results. After duplicate removal, a total of 1310 citations were identified, of which, after full-paper screening, 105 studies were included in this scoping review. The classification of an intervention as taking a diagnostic stewardship approach is a relatively recent development, with the first publication in this field dating back to 2017. The majority of research in this area has been conducted within the USA, with very few studies undertaken outside this region. Visual inspection of the citation map reveals that the current evidence base is interconnected, with frequent references to each other's work. The interventions commonly adopt a restrictive approach, utilizing hard and soft stops within the pre-analytical phase to restrict access to testing. Upon closer examination of the outcomes, it becomes evident that there is a predominant focus on reducing the number of tests rather than enhancing the current test protocol. This is further reflected in the limited number of studies that report on test performance (including protocol improvements, specificity and sensitivity).Conclusion. Diagnostic stewardship seems to have deviated from its intended course, morphing into a rather rudimentary instrument wielded not to enhance but to constrict the scope of testing. Despite the World Health Organization's advocacy for an ideology that promotes a more comprehensive approach to quality improvement, it may be more appropriate to consider alternative regional narratives when categorizing these types of quality improvement interventions.

A customised down-sampling machine learning approach for sepsis prediction

OBJECTIVE

Sepsis is a life-threatening condition in the ICU and requires treatment in time. Despite the accuracy of existing sepsis prediction models, insufficient focus on reducing alarms could worsen alarm fatigue and desensitisation in ICUs, potentially compromising patient safety. In this retrospective study, we aim to develop an accurate, robust, and readily deployable method in ICUs, only based on the vital signs and laboratory tests.

METHODS

Our method consists of a customised down-sampling process and a specific dynamic sliding window and XGBoost to offer sepsis prediction. The down-sampling process was applied to the retrospective data for training the XGBoost model. During the testing stage, the dynamic sliding window and the trained XGBoost were used to predict sepsis on the retrospective datasets, PhysioNet and FHC.

RESULTS

With the filtered data from PhysioNet, our method achieved 80.74% accuracy (77.90% sensitivity and 84.42% specificity) and 83.95% (84.82% sensitivity and 82.00% specificity) on the test set of PhysioNet-A and PhysioNet-B, respectively. The AUC score was 0.89 for both datasets. On the FHC dataset, our method achieved 92.38% accuracy (88.37% sensitivity and 95.16% specificity) and 0.98 AUC score on the test set of FHC.

CONCLUSION

Our results indicate that the down-sampling process and the dynamic sliding window with XGBoost brought robust and accurate performance to give sepsis prediction under various hospital settings. The localisation and robustness of our method can assist in sepsis diagnosis in different ICU settings.

Impact of Blood Culture Contamination on Antibiotic Use, Resource Utilization, and Clinical Outcomes: A Retrospective Cohort Study in Dutch and US Hospitals

Background

Blood culture contamination (BCC) has been associated with prolonged antibiotic use (AU) and increased health care utilization; however, this has not been widely reevaluated in the era of increased attention to antibiotic stewardship. We evaluated the impact of BCC on AU, resource utilization, and length of stay in Dutch and US patients.

Methods

This retrospective observational study examined adults admitted to 2 hospitals in the Netherlands and 5 hospitals in the United States undergoing ≥2 blood culture (BC) sets. Exclusion criteria included neutropenia, no hospital admission, or death within 48 hours of hospitalization. The impact of BCC on clinical outcomes-overall inpatient days of antibiotic therapy, test utilization, length of stay, and mortality-was determined via a multivariable regression model.

Results

An overall 22 927 patient admissions were evaluated: 650 (4.1%) and 339 (4.8%) with BCC and 11 437 (71.8%) and 4648 (66.3%) with negative BC results from the Netherlands and the United States, respectively. Dutch and US patients with BCC had a mean ± SE 1.74 ± 0.27 (P < .001) and 1.58 ± 0.45 (P < .001) more days of antibiotic therapy than patients with negative BC results. They also had 0.6 ± 0.1 (P < .001) more BCs drawn. Dutch but not US patients with BCC had longer hospital stays (3.36 days; P < .001). There was no difference in mortality between groups in either cohort. AU remained higher in US but not Dutch patients with BCC in a subanalysis limited to BC obtained within the first 24 hours of admission.

Conclusions

BCC remains associated with higher inpatient AU and health care utilization as compared with patients with negative BC results, although the impact on these outcomes differs by country.

Detecting changes in the performance of a clinical machine learning tool over time

BACKGROUND

Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models.

METHODS

A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time.

FINDINGS

Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%.

INTERPRETATION

Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool.

FUNDING

No funding to disclose.

Real-time artificial intelligence system for bacteremia prediction in adult febrile emergency department patients

BACKGROUND

Artificial intelligence (AI) holds significant potential to be a valuable tool in healthcare. However, its application for predicting bacteremia among adult febrile patients in the emergency department (ED) remains unclear. Therefore, we conducted a study to provide clarity on this issue.

METHODS

Adult febrile ED patients with blood cultures at Chi Mei Medical Center were divided into derivation (January 2017 to June 2019) and validation groups (July 2019 to December 2020). The derivation group was utilized to develop AI models using twenty-one feature variables and five algorithms to predict bacteremia. The performance of these models was compared with qSOFA score. The AI model with the highest area under the receiver operating characteristics curve (AUC) was chosen to implement the AI prediction system and tested on the validation group.

RESULTS

The study included 5,647 febrile patients. In the derivation group, there were 3,369 patients with a mean age of 61.4 years, and 50.7% were female, including 508 (13.8%) with bacteremia. The model with the best AUC was built using the random forest algorithm (0.761), followed by logistic regression (0.755). All five models demonstrated better AUC than the qSOFA score (0.560). The random forest model was adopted to build a real-time AI prediction system integrated into the hospital information system, and the AUC achieved 0.709 in the validation group.

CONCLUSION

The AI model shows promise to predict bacteremia in adult febrile ED patients; however, further external validation in different hospitals and populations is necessary to verify its effectiveness.

Comparison of Administrative versus Electronic Health Record-based Methods for Identifying Sepsis Hospitalizations

Rationale: Despite the importance of sepsis surveillance, no optimal approach for identifying sepsis hospitalizations exists. The Centers for Disease Control and Prevention Adult Sepsis Event Definition (CDC-ASE) is an electronic medical record-based algorithm that yields more stable estimates over time than diagnostic coding-based approaches but may still result in misclassification. Objectives: We sought to assess three approaches to identifying sepsis hospitalizations, including a modified CDC-ASE. Methods: This cross-sectional study included patients in the Veterans Affairs Ann Arbor Healthcare System admitted via the emergency department (February 2021 to February 2022) with at least one episode of acute organ dysfunction within 48 hours of emergency department presentation. Patients were assessed for community-onset sepsis using three methods: 1) explicit diagnosis codes, 2) the CDC-ASE, and 3) a modified CDC-ASE. The modified CDC-ASE required at least two systemic inflammatory response syndrome criteria instead of blood culture collection and had a more sensitive definition of respiratory dysfunction. Each method was compared with a reference standard of physician adjudication via medical record review. Patients were considered to have sepsis if they had at least one episode of acute organ dysfunction graded as "definitely" or "probably" infection related on physician review. Results: Of 821 eligible hospitalizations, 449 were selected for physician review. Of these, 98 (21.8%) were classified as sepsis by medical record review, 103 (22.9%) by the CDC-ASE, 132 (29.4%) by the modified CDC-ASE, and 37 (8.2%) by diagnostic codes. Accuracy was similar across the three methods of interest (80.6% for the CDC-ASE, 79.6% for the modified CDC-ADE, and 84.2% for diagnostic codes), but sensitivity and specificity varied. The CDC-ASE algorithm had sensitivity of 58.2% (95% confidence interval [CI], 47.2-68.1%) and specificity of 86.9% (95% CI, 82.9-90.2%). The modified CDC-ASE algorithm had greater sensitivity (69.4% [95% CI, 59.3-78.3%]) but lower specificity (81.8% [95% CI, 77.3-85.7%]). Diagnostic codes had lower sensitivity (32.7% [95% CI, 23.5-42.9%]) but greater specificity (98.6% [95% CI, 96.7-99.55%]). Conclusions: There are several approaches to identifying sepsis hospitalizations for surveillance that have acceptable accuracy. These approaches yield varying sensitivity and specificity, so investigators should carefully consider the test characteristics of each method before determining an appropriate method for their intended use.

Machine learning pipeline for blood culture outcome prediction using Sysmex XN-2000 blood sample results in Western Australia

BACKGROUND

Bloodstream infections (BSIs) are a significant burden on the global population and represent a key area of focus in the hospital environment. Blood culture (BC) testing is the standard diagnostic test utilised to confirm the presence of a BSI. However, current BC testing practices result in low positive yields and overuse of the diagnostic test. Diagnostic stewardship research regarding BC testing is increasing, and becoming more important to reduce unnecessary resource expenditure and antimicrobial use, especially as antimicrobial resistance continues to rise. This study aims to establish a machine learning (ML) pipeline for BC outcome prediction using data obtained from routinely analysed blood samples, including complete blood count (CBC), white blood cell differential (DIFF), and cell population data (CPD) produced by Sysmex XN-2000 analysers.

METHODS

ML models were trained using retrospective data produced between 2018 and 2019, from patients at Sir Charles Gairdner hospital, Nedlands, Western Australia, and processed at Pathwest Laboratory Medicine, Nedlands. Trained ML models were evaluated using stratified 10-fold cross validation.

RESULTS

Two ML models, an XGBoost model using CBC/DIFF/CPD features with boruta feature selection (BFS) , and a random forest model trained using CBC/DIFF features with BFS were selected for further validation after obtaining AUC scores of [Formula: see text] and [Formula: see text] respectively using stratified 10-fold cross validation. The XGBoost model obtained an AUC score of 0.76 on a internal validation set. The random forest model obtained AUC scores of 0.82 and 0.76 on internal and external validation datasets respectively.

CONCLUSIONS

We have demonstrated the utility of using an ML pipeline combined with CBC/DIFF, and CBC/DIFF/CPD feature spaces for BC outcome prediction. This builds on the growing body of research in the area of BC outcome prediction, and provides opportunity for further research.

Machine learning of cell population data, complete blood count, and differential count parameters for early prediction of bacteremia among adult patients with suspected bacterial infections and blood culture sampling in emergency departments

BACKGROUND

Bacteremia is a life-threatening complication of infectious diseases. Bacteremia can be predicted using machine learning (ML) models, but these models have not utilized cell population data (CPD).

METHODS

The derivation cohort from emergency department (ED) of China Medical University Hospital (CMUH) was used to develop the model and was prospectively validated in the same hospital. External validation was performed using cohorts from ED of Wei-Gong Memorial Hospital (WMH) and Tainan Municipal An-Nan Hospital (ANH). Adult patients who underwent complete blood count (CBC), differential count (DC), and blood culture tests were enrolled in the present study. The ML model was developed using CBC, DC, and CPD to predict bacteremia from positive blood cultures obtained within 4 h before or after the acquisition of CBC/DC blood samples.

RESULTS

This study included 20,636 patients from CMUH, 664 from WMH, and 1622 patients from ANH. Another 3143 patients were included in the prospective validation cohort of CMUH. The CatBoost model achieved an area under the receiver operating characteristic curve of 0.844 in the derivation cross-validation, 0.812 in the prospective validation, 0.844 in the WMH external validation, and 0.847 in the ANH external validation. The most valuable predictors of bacteremia in the CatBoost model were the mean conductivity of lymphocytes, nucleated red blood cell count, mean conductivity of monocytes, and neutrophil-to-lymphocyte ratio.

CONCLUSIONS

ML model that incorporated CBC, DC, and CPD showed excellent performance in predicting bacteremia among adult patients with suspected bacterial infections and blood culture sampling in emergency departments.

Embracing cohort heterogeneity in clinical machine learning development: a step toward generalizable models

This study is a simple illustration of the benefit of averaging over cohorts, rather than developing a prediction model from a single cohort. We show that models trained on data from multiple cohorts can perform significantly better in new settings than models based on the same amount of training data but from just a single cohort. Although this concept seems simple and obvious, no current prediction model development guidelines recommend such an approach.

Artificial Intelligence Can Guide Antibiotic Choice in Recurrent UTIs and Become an Important Aid to Improve Antimicrobial Stewardship

BACKGROUND

A correct approach to recurrent urinary tract infections (rUTIs) is an important pillar of antimicrobial stewardship. We aim to define an Artificial Neural Network (ANN) for predicting the clinical efficacy of the empiric antimicrobial treatment in women with rUTIs.

METHODS

We extracted clinical and microbiological data from 1043 women. We trained an ANN on 725 patients and validated it on 318.

RESULTS

The ANN showed a sensitivity of 87.8% and specificity of 97.3% in predicting the clinical efficacy of empirical therapy. The previous use of fluoroquinolones (HR = 4.23; p = 0.008) and cephalosporins (HR = 2.81; p = 0.003) as well as the presence of Escherichia coli with resistance against cotrimoxazole (HR = 3.54; p = 0.001) have been identified as the most important variables affecting the ANN output decision predicting the fluoroquinolones-based therapy failure. A previous isolation of Escherichia coli with resistance against fosfomycin (HR = 2.67; p = 0.001) and amoxicillin-clavulanic acid (HR = 1.94; p = 0.001) seems to be the most influential variable affecting the output decision predicting the cephalosporins- and cotrimoxazole-based therapy failure. The previously mentioned Escherichia coli with resistance against cotrimoxazole (HR = 2.35; p < 0.001) and amoxicillin-clavulanic acid (HR = 3.41; p = 0.007) seems to be the most influential variable affecting the output decision predicting the fosfomycin-based therapy failure.

CONCLUSIONS

ANNs seem to be an interesting tool to guide the antimicrobial choice in the management of rUTIs at the point of care.