Novel Model to Predict Gastrointestinal Bleeding During Left Ventricular Assist Device Support

BACKGROUND

Gastrointestinal bleeding (GIB) is a leading cause of morbidity during continuous-flow left ventricular assist device (CF-LVAD) support. GIB risk assessment could have important implications for candidate selection, informed consent, and postimplant therapeutic strategies. The aim of the study is to derive and validate a predictive model of GIB in CF-LVAD patients.

METHODS AND RESULTS

CF-LVAD recipients at the Utah Transplantation Affiliated Hospitals program between 2004 and 2017 were included. GIB associated with a decrease in hemoglobin ≥2 g/dL was the primary end point. A weighted score comprising preimplant variables independently associated with GIB was derived and internally validated. A total of 351 patients (median age, 59 years; 82% male) were included. After a median of 196 days, GIB occurred in 120 (34%) patients. Independent predictors of GIB included age >54 years, history of previous bleeding, coronary artery disease, chronic kidney disease, severe right ventricular dysfunction, mean pulmonary artery pressure <18 mm Hg, and fasting glucose >107 mg/dL. A weighted score termed Utah bleeding risk score, effectively stratified patients based on their probability of GIB: low (0-1 points) 4.8%, intermediate (2-4) 39.8%, and high risk (5-9) 83.8%. Discrimination was good in the development sample (c-index: 0.83) and after internal bootstrap validation (c-index: 0.74).

CONCLUSIONS

The novel Utah bleeding risk score is a simple tool that can provide personalized GIB risk estimates in CF-LVAD patients. This scoring system may assist clinicians and investigators in designing tailored risk-based strategies aimed at reducing the burden posed by GIB in the individual CF-LVAD patient and healthcare systems.

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients

IMPORTANCE

To improve patient safety, health care systems need reliable methods to detect adverse events in large patient populations. Events are often described in clinical notes, rather than structured data, which make them difficult to identify on a large scale.

OBJECTIVE

To develop and compare 2 natural language processing methods, a rules-based approach and a machine learning (ML) approach, for identifying bleeding events in clinical notes.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study used deidentified notes from the Medical Information Mart for Intensive Care, which spans 2001 to 2012. A training set of 990 notes and a test set of 660 notes were randomly selected. Physicians classified each note as present or absent for a clinically relevant bleeding event during the hospitalization. A bleeding dictionary was developed for the rules-based approach; bleeding mentions were then aggregated to arrive at a classification for each note. Three ML models (support vector machine, extra trees, and convolutional neural network) were developed and trained using the 990-note training set. Another instance of each ML model was also trained on a sample of 450 notes, with equal numbers of bleeding-present and bleeding-absent notes. The notes were represented using term frequency-inverse document frequency vectors and global vectors for word representation.

MAIN OUTCOMES AND MEASURES

The main outcomes were accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for each model. Following training, the models were tested on the test set and sensitivities were compared using a McNemar test.

RESULTS

The 990-note training set represented 769 patients (296 [38.5%] female; mean [SD] age, 67.42 [14.7] years). The 660-note test set represented 527 patients (211 [40.0%] female; mean [SD] age, 67.86 [14.7] years). Bleeding was present in 146 notes (22.1%). The extra trees down-sampled model and rules-based approaches were similarly sensitive (93.8% vs 91.1%; difference, 2.7%; 95% CI, -3.8% to 7.9%; P = .44). The positive predictive value for the extra trees model, however, was 48.6%. The rules-based model had the best performance overall, with 84.6% specificity, 62.7% positive predictive value, and 97.1% negative predictive value.

CONCLUSIONS AND RELEVANCE

Bleeding is a common complication in health care, and these results demonstrate an automated and scalable detection method. The rules-based natural language processing approach, compared with ML, had the best performance in identifying bleeding, with high sensitivity and negative predictive value.

Abstract 11: Development and Comparison of Two Natural Language Processing Methods for Identifying Bleeding Events in Clinical Text

Background: Learning healthcare systems need techniques that can accurately and automatically identify health outcomes in large populations. Outcomes are often described in clinical narration in the electronic medical record.

Objective: To develop and compare two natural language processing (NLP) approaches, rules-based (RB) and machine-learning (ML), for identifying bleeding events in clinical notes.

Methods: We used de-identified notes from the Medical Information Mart for Intensive Care. We randomly selected 990 notes for a training set and 660 notes for a test set. Physicians classified each note as present or absent for a clinically relevant bleeding event during the hospitalization. We developed a dictionary of target and modifier words for the RB approach. In RB, the computer “reads” the text and tags bleeding targets as present or absent based on the modifier words; the mentions are aggregated to arrive at a classification for the note. For the ML approach, each note was represented as a high-dimensional vector where each dimension corresponds to the frequency of a certain word. Similar notes (e.g. bleeding present notes) have similar vectors; the computer learns these patterns to predict the class for an unseen note. One RB and three ML models (support vector machine (SVM), extra trees (ET), convolutional neural network (CNN)) were trained using the full 990-note training set. Another instance of each ML model was also trained on a down-sampled (DS) set of 450 notes, with equal positive and negative notes. We ran the trained models on the 660-note test set and compared classification performance using McNemar’s test.

Results: The 660 note test set represented 527 unique patients, 40% female. Bleeding events were present in 21% of the notes. The ET-DS model was the most sensitive, followed by the RB approach (93.8% versus 91.1%, p=0.44). The PPV value for the ET-DS model, however, was <50%. The RB had the best performance overall, with 84.6% specificity, 62.7% positive predictive value, and 97.1% negative predictive value (NPV) for identifying clinically relevant bleeding.

Discussion: A RB NLP approach, compared to ML, has the best overall performance in independently identifying bleeding events among critically ill patients. The current models have high NPV, so could be used to reduce the chart review burden.