Predicting pulmonary embolism among hospitalized patients with machine learning algorithms

Performance of Plug-In Augmented ChatGPT and Its Ability to Quantify Uncertainty: Simulation Study on the German Medical Board Examination

Background

The GPT-4 is a large language model (LLM) trained and fine-tuned on an extensive dataset. After the public release of its predecessor in November 2022, the use of LLMs has seen a significant spike in interest, and a multitude of potential use cases have been proposed. In parallel, however, important limitations have been outlined. Particularly, current LLMs encounter limitations, especially in symbolic representation and accessing contemporary data. The recent version of GPT-4, alongside newly released plugin features, has been introduced to mitigate some of these limitations.

Objective

Before this background, this work aims to investigate the performance of GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins using pretranslated English text on the German medical board examination. Recognizing the critical importance of quantifying uncertainty for LLM applications in medicine, we furthermore assess this ability and develop a new metric termed "confidence accuracy" to evaluate it.

Methods

We used GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins and translation to answer questions from the German medical board examination. Additionally, we conducted an analysis to assess how the models justify their answers, the accuracy of their responses, and the error structure of their answers. Bootstrapping and CIs were used to evaluate the statistical significance of our findings.

Results

This study demonstrated that available GPT models, as LLM examples, exceeded the minimum competency threshold established by the German medical board for medical students to obtain board certification to practice medicine. Moreover, the models could assess the uncertainty in their responses, albeit exhibiting overconfidence. Additionally, this work unraveled certain justification and reasoning structures that emerge when GPT generates answers.

Conclusions

The high performance of GPTs in answering medical questions positions it well for applications in academia and, potentially, clinical practice. Its capability to quantify uncertainty in answers suggests it could be a valuable artificial intelligence agent within the clinical decision-making loop. Nevertheless, significant challenges must be addressed before artificial intelligence agents can be robustly and safely implemented in the medical domain.

Development and validation of a deep learning-enhanced prediction model for the likelihood of pulmonary embolism

Background

Pulmonary embolism (PE) is a common and potentially fatal condition. Timely and accurate risk assessment in patients with acute deep vein thrombosis (DVT) is crucial. This study aims to develop a deep learning-based, precise, and efficient PE risk prediction model (PE-Mind) to overcome the limitations of current clinical tools and provide a more targeted risk evaluation solution.

Methods

We analyzed clinical data from patients by first simplifying and organizing the collected features. From these, 37 key clinical features were selected based on their importance. These features were categorized and analyzed to identify potential relationships. Our prediction model uses a convolutional neural network (CNN), enhanced with three custom-designed modules for better performance. To validate its effectiveness, we compared this model with five commonly used prediction models.

Results

PE-Mind demonstrated the highest accuracy and reliability, achieving 0.7826 accuracy and an area under the receiver operating characteristic curve of 0.8641 on the prospective test set, surpassing other models. Based on this, we have also developed a Web server, PulmoRiskAI, for real-time clinician operation.

Conclusion

The PE-Mind model improves prediction accuracy and reliability for assessing PE risk in acute DVT patients. Its convolutional architecture and residual modules substantially enhance predictive performance.

Development of a predictive nomogram for early identification of pulmonary embolism in hospitalized patients: a retrospective cohort study

BACKGROUND

Hospitalized patients often present with complex clinical conditions, but there is a lack of effective tools to assess their risk of pulmonary embolism (PE). Therefore, our study aimed to develop a nomogram model for better predicting PE in hospitalized populations.

METHODS

Data from hospitalized patients (aged ≥ 15 years) who underwent computed tomography pulmonary angiography (CTPA) to confirm PE and non-PE were collected from December 2013 to April 2023. Univariate and multivariate stepwise logistic regression analyses were conducted to identify independent predictors of PE, followed by the construction of a predictive nomogram and internal validation. The efficiency and clinical utility of the nomogram model were assessed using receiver operating characteristic (ROC) curve, decision curve analysis (DCA), and clinical impact curve (CIC).

RESULTS

The study included 313 PE and 339 non-PE hospitalized patients. Male gender, dyspnea or shortness of breath, interstitial lung disease, lower limb deep vein thrombosis, elevated fibrin degradation product (FDP), pulmonary arterial hypertension, and tricuspid regurgitation were identified as independent risk factors. The AUC of the predictive nomogram model was 0.956 (95% CI: 0.939-0.974), demonstrating superior performance compared with the simplified Wells score of 0.698 (95% CI: 0.654-0.741) and the modified Geneva score of 0.758 (95% CI: 0.717-0.799).

CONCLUSION

Our study demonstrated that challenges remain in the accuracy of the Wells score and revised Geneva score in assessing PE in hospitalized patients. Fortunately, the nomogram we developed has shown a favorable ability to discriminate PE cases, providing high reference value for clinical practice. However, given that this was a single-center study, we plan to expand efforts to collect data from additional centers to further validate our model.

Early Detection of Pulmonary Embolism in a General Patient Population Immediately Upon Hospital Admission Using Machine Learning to Identify New, Unidentified Risk Factors: Model Development Study

BACKGROUND

Under- or late identification of pulmonary embolism (PE)-a thrombosis of 1 or more pulmonary arteries that seriously threatens patients' lives-is a major challenge confronting modern medicine.

OBJECTIVE

We aimed to establish accurate and informative machine learning (ML) models to identify patients at high risk for PE as they are admitted to the hospital, before their initial clinical checkup, by using only the information in their medical records.

METHODS

We collected demographics, comorbidities, and medications data for 2568 patients with PE and 52,598 control patients. We focused on data available prior to emergency department admission, as these are the most universally accessible data. We trained an ML random forest algorithm to detect PE at the earliest possible time during a patient's hospitalization-at the time of his or her admission. We developed and applied 2 ML-based methods specifically to address the data imbalance between PE and non-PE patients, which causes misdiagnosis of PE.

RESULTS

The resulting models predicted PE based on age, sex, BMI, past clinical PE events, chronic lung disease, past thrombotic events, and usage of anticoagulants, obtaining an 80% geometric mean value for the PE and non-PE classification accuracies. Although on hospital admission only 4% (1942/46,639) of the patients had a diagnosis of PE, we identified 2 clustering schemes comprising subgroups with more than 61% (705/1120 in clustering scheme 1; 427/701 and 340/549 in clustering scheme 2) positive patients for PE. One subgroup in the first clustering scheme included 36% (705/1942) of all patients with PE who were characterized by a definite past PE diagnosis, a 6-fold higher prevalence of deep vein thrombosis, and a 3-fold higher prevalence of pneumonia, compared with patients of the other subgroups in this scheme. In the second clustering scheme, 2 subgroups (1 of only men and 1 of only women) included patients who all had a past PE diagnosis and a relatively high prevalence of pneumonia, and a third subgroup included only those patients with a past diagnosis of pneumonia.

CONCLUSIONS

This study established an ML tool for early diagnosis of PE almost immediately upon hospital admission. Despite the highly imbalanced scenario undermining accurate PE prediction and using information available only from the patient's medical history, our models were both accurate and informative, enabling the identification of patients already at high risk for PE upon hospital admission, even before the initial clinical checkup was performed. The fact that we did not restrict our patients to those at high risk for PE according to previously published scales (eg, Wells or revised Genova scores) enabled us to accurately assess the application of ML on raw medical data and identify new, previously unidentified risk factors for PE, such as previous pulmonary disease, in general populations.

Availability in ECMO Reduces the Failure to Rescue in Patients With Pulmonary Embolism After Major Surgery: A Nationwide Analysis of 2.4 Million Cases

Objective

Postoperative pulmonary embolism (PE) is a rare but potentially life-threatening complication, which can be treated with extracorporeal membrane oxygenation (ECMO) therapy, a novel therapy option for acute cardiorespiratory failure. We postulate that hospitals with ECMO availability have more experienced staff, technical capabilities, and expertise in treating cardiorespiratory failure.

Design

A retrospective analysis of surgical procedures in Germany between 2012 and 2019 was performed using hospital billing data. High-risk surgical procedures for postoperative PE were analyzed according to the availability of and expertise in ECMO therapy and its effect on outcome, regardless of whether ECMO was used in patients with PE.

Methods

Descriptive, univariate, and multivariate analyses were applied to identify possible associations and correct for confounding factors (complications, complication management, and mortality).

Results

A total of 13,976,606 surgical procedures were analyzed, of which 2,407,805 were defined as high-risk surgeries. The overall failure to rescue (FtR) rate was 24.4% and increased significantly with patient age, as well as type of surgery. The availability of and experience in ECMO therapy (defined as at least 20 ECMO applications per year; ECMO centers) are associated with a significantly reduced FtR in patients with PE after high-risk surgical procedures. In a multivariate analysis, the odds ratio (OR) for FtR after postoperative PE was significantly lower in ECMO centers (OR, 0.75 [0.70-0.81], P < 0.001).

Conclusions

The availability of and expertise in ECMO therapy lead to a significantly reduced FtR rate of postoperative PE. This improved outcome is independent of the use of ECMO in these patients.

Machine Learning Approach with Harmonized Multinational Datasets for Enhanced Prediction of Hypothyroidism in Patients with Type 2 Diabetes

Type 2 diabetes (T2D) is a global health concern with increasing prevalence. Comorbid hypothyroidism (HT) exacerbates kidney, cardiac, neurological and other complications of T2D; these risks can be mitigated pharmacologically upon detecting HT. The current HT standard of care (SOC) screening in T2D is infrequent, delaying HT diagnosis and treatment. We present a first-to-date machine learning algorithm (MLA) clinical decision tool to classify patients as low vs. high risk for developing HT comorbid with T2D; the MLA was developed using readily available patient data from harmonized multinational datasets. The MLA was trained on data from NIH All of US (AoU) and UK Biobank (UKBB) (Combined dataset) and achieved a high negative predictive value (NPV) of 0.989 and an AUROC of 0.762 in the Combined dataset, exceeding AUROCs for the models trained on AoU or UKBB alone (0.666 and 0.622, respectively), indicating that increasing dataset diversity for MLA training improves performance. This high-NPV automated tool can supplement SOC screening and rule out T2D patients with low HT risk, allowing for the prioritization of lab-based testing for at-risk patients. Conversely, an MLA output that designates a patient to be at risk of developing HT allows for tailored clinical management and thereby promotes improved patient outcomes.

Establishment of machine learning-based tool for early detection of pulmonary embolism

BACKGROUND AND OBJECTIVES

Pulmonary embolism (PE) is a complex disease with high mortality and morbidity rate, leading to increasing society burden. However, current diagnosis is solely based on symptoms and laboratory data despite its complex pathology, which easily leads to misdiagnosis and missed diagnosis by inexperienced doctors. Especially, CT pulmonary angiography, the gold standard method, is not widely available. In this study, we aim to establish a rapid and accurate screening model for pulmonary embolism using machine learning technology. Importantly, data required for disease prediction are easily accessed, including routine laboratory data and medical record information of patients.

METHODS

We extracted features from patients' routine laboratory results and medical records, including blood routine, biochemical group, blood coagulation routine and other test results, as well as symptoms and medical history information. Samples with a feature loss rate greater than 0.8 were deleted from the original database. Data from 4723 cases were retained, 231 of which were positive for pulmonary embolism. 50 features were retained through the positive and negative statistical hypothesis testing which was used to build the predictive model. In order to avoid identification as majority-class samples caused by the imbalance of sample proportion, we used the method of Synthetic Minority Oversampling Technique (SMOTE) to increase the amount of information on minority samples. Five typical machine learning algorithms were used to model the screening of pulmonary embolism, including Support Vector Machines, Logistic Regression, Random Forest, XGBoost, and Back Propagation Neural Networks. To evaluate model performance, sensitivity, specificity and AUC curve were analyzed as the main evaluation indicators. Furthermore, a baseline model was established using the characteristics of the pulmonary embolism guidelines as a comparison model.

RESULTS

We found that XGBoost showed better performance compared to other models, with the highest sensitivity and specificity (0.99 and 0.99, respectively). Moreover, it showed significant improvement in performance compared to the baseline model (sensitivity and specificity were 0.76 and 0.76 respectively). More important, our model showed low missed diagnosis rate (0.46) and high AUC value (0.992). Finally, the calculation time of our model is only about 0.05 s to obtain the possibility of pulmonary embolism.

CONCLUSIONS

In this study, five machine learning classification models were established to assess the likelihood of patients suffering from pulmonary embolism, and the XGBoost model most significantly improved the precision, sensitivity, and AUC for pulmonary embolism screening. Collectively, we have established an AI-based model to accurately predict pulmonary embolism at early stage.

Advancing prognostic precision in pulmonary embolism: A clinical and laboratory-based artificial intelligence approach for enhanced early mortality risk stratification

BACKGROUND

Acute pulmonary embolism (PE) is a critical medical emergency that necessitates prompt identification and intervention. Accurate prognostication of early mortality is vital for recognizing patients at elevated risk for unfavourable outcomes and administering suitable therapy. Machine learning (ML) algorithms hold promise for enhancing the precision of early mortality prediction in PE patients.

OBJECTIVE

To devise an ML algorithm for early mortality prediction in PE patients by employing clinical and laboratory variables.

METHODS

This study utilized diverse oversampling techniques to improve the performance of various machine learning models including ANN, SVM, DT, RF, and AdaBoost for early mortality prediction. Appropriate oversampling methods were chosen for each model based on algorithm characteristics and dataset properties. Predictor variables included four lab tests, eight physiological time series indicators, and two general descriptors. Evaluation used metrics like accuracy, F1_score, precision, recall, Area Under the Curve (AUC) and Receiver Operating Characteristic (ROC) curves, providing a comprehensive view of models' predictive abilities.

RESULTS

The findings indicated that the RF model with random oversampling exhibited superior performance among the five models assessed, achieving elevated accuracy and precision alongside high recall for predicting the death class. The oversampling approaches effectively equalized the sample distribution among the classes and enhanced the models' performance.

CONCLUSIONS

The suggested ML technique can efficiently prognosticate mortality in patients afflicted with acute PE. The RF model with random oversampling can aid healthcare professionals in making well-informed decisions regarding the treatment of patients with acute PE. The study underscores the significance of oversampling methods in managing imbalanced data and emphasizes the potential of ML algorithms in refining early mortality prediction for PE patients.

Role of artificial intelligence in oncologic emergencies: a narrative review

Oncologic emergencies are a wide spectrum of oncologic conditions caused directly by malignancies or their treatment. Oncologic emergencies may be classified according to the underlying physiopathology in metabolic, hematologic, and structural conditions. In the latter, radiologists have a pivotal role, through an accurate diagnosis useful to provide optimal patient care. Structural conditions may involve the central nervous system, thorax, or abdomen, and emergency radiologists have to know the characteristics imaging findings of each one of them. The number of oncologic emergencies is growing due to the increased incidence of malignancies in the general population and also to the improved survival of these patients thanks to the advances in cancer treatment. Artificial intelligence (AI) could be a solution to assist emergency radiologists with this rapidly increasing workload. To our knowledge, AI applications in the setting of the oncologic emergency are mostly underexplored, probably due to the relatively low number of oncologic emergencies and the difficulty in training algorithms. However, cancer emergencies are defined by the cause and not by a specific pattern of radiological symptoms and signs. Therefore, it can be expected that AI algorithms developed for the detection of these emergencies in the non-oncological field can be transferred to the clinical setting of oncologic emergency. In this review, a craniocaudal approach was followed and central nervous system, thoracic, and abdominal oncologic emergencies have been addressed regarding the AI applications reported in literature. Among the central nervous system emergencies, AI applications have been reported for brain herniation and spinal cord compression. In the thoracic district the addressed emergencies were pulmonary embolism, cardiac tamponade and pneumothorax. Pneumothorax was the most frequently described application for AI, to improve sensibility and to reduce the time-to-diagnosis. Finally, regarding abdominal emergencies, AI applications for abdominal hemorrhage, intestinal obstruction, intestinal perforation, and intestinal intussusception have been described.