Natural Language Processing Performance for the Identification of Venous Thromboembolism in an Integrated Healthcare System

ClotCatcher: a novel natural language model to accurately adjudicate venous thromboembolism from radiology reports

INTRODUCTION

Accurate identification of venous thromboembolism (VTE) is critical to develop replicable epidemiological studies and rigorous predictions models. Traditionally, VTE studies have relied on international classification of diseases (ICD) codes which are inaccurate - leading to misclassification bias. Here, we developed ClotCatcher, a novel deep learning model that uses natural language processing to detect VTE from radiology reports.

METHODS

Radiology reports to detect VTE were obtained from patients admitted to Emory University Hospital (EUH) and Grady Memorial Hospital (GMH). Data augmentation was performed using the Google PEGASUS paraphraser. This data was then used to fine-tune ClotCatcher, a novel deep learning model. ClotCatcher was validated on both the EUH dataset alone and GMH dataset alone.

RESULTS

The dataset contained 1358 studies from EUH and 915 studies from GMH (n = 2273). The dataset contained 1506 ultrasound studies with 528 (35.1%) studies positive for VTE, and 767 CT studies with 91 (11.9%) positive for VTE. When validated on the EUH dataset, ClotCatcher performed best (AUC = 0.980) when trained on both EUH and GMH dataset without paraphrasing. When validated on the GMH dataset, ClotCatcher performed best (AUC = 0.995) when trained on both EUH and GMH dataset with paraphrasing.

CONCLUSION

ClotCatcher, a novel deep learning model with data augmentation rapidly and accurately adjudicated the presence of VTE from radiology reports. Applying ClotCatcher to large databases would allow for rapid and accurate adjudication of incident VTE. This would reduce misclassification bias and form the foundation for future studies to estimate individual risk for patient to develop incident VTE.

A survey on clinical natural language processing in the United Kingdom from 2007 to 2022

Much of the knowledge and information needed for enabling high-quality clinical research is stored in free-text format. Natural language processing (NLP) has been used to extract information from these sources at scale for several decades. This paper aims to present a comprehensive review of clinical NLP for the past 15 years in the UK to identify the community, depict its evolution, analyse methodologies and applications, and identify the main barriers. We collect a dataset of clinical NLP projects (n = 94; £ = 41.97 m) funded by UK funders or the European Union's funding programmes. Additionally, we extract details on 9 funders, 137 organisations, 139 persons and 431 research papers. Networks are created from timestamped data interlinking all entities, and network analysis is subsequently applied to generate insights. 431 publications are identified as part of a literature review, of which 107 are eligible for final analysis. Results show, not surprisingly, clinical NLP in the UK has increased substantially in the last 15 years: the total budget in the period of 2019-2022 was 80 times that of 2007-2010. However, the effort is required to deepen areas such as disease (sub-)phenotyping and broaden application domains. There is also a need to improve links between academia and industry and enable deployments in real-world settings for the realisation of clinical NLP's great potential in care delivery. The major barriers include research and development access to hospital data, lack of capable computational resources in the right places, the scarcity of labelled data and barriers to sharing of pretrained models.

Synoptic reporting accuracy for computed tomography pulmonary arteriography among patients suspected of pulmonary embolism

Background

Structured reporting is an efficient and replicable method of presenting diagnostic results that eliminates variability inherent in narrative descriptive reporting and may improve clinical decisions. Synoptic element reporting can generate discrete coded data that then may inform clinical decision support and trigger downstream actions in computerized electronic health records.

Objective

Limited evidence exists for use of synoptic reporting for computed tomography pulmonary arteriography (CTPA) among patients suspected of pulmonary embolism. We reported the accuracy of synoptic reporting for the outcome of pulmonary embolism among patients who presented to an integrated health care system with CTPA performed for suspected pulmonary embolism.

Methods

Structured radiology reports with embedded synoptic elements were implemented for all CTPA examinations on March 1, 2018. Four hundred CTPA reports between January 4, 2019 and July 30, 2020 (200 reports each for which synoptic reporting recorded the presence or absence of pulmonary embolism [PE]) were selected at random. One non-diagnostic study was excluded from analysis. We then assessed the accuracy of synoptic reporting compared with the gold standard of manual chart review.

Results

Synoptic reporting and manual review agreed in 99.2% of patients undergoing CTPA for suspected PE, agreed on the presence of PE in 196 of 199 (98.5%) cases, the absence of PE in 200 of 200 (100%) cases with a sensitivity of 87.6% (76.1-96.1) a specificity of 99.9% (99.7%-100%), a positive predictive value of 99.5% (98.1-100), and a negative predictive value of 98% (95.7%-99.5%).

Conclusion

The overall rate of agreement was 99.2%, but we observed an unacceptable false-negative rate for clinical reliance on synoptic element reporting in isolation from dictated reports.

Natural Language Processing for Identification of Venous Thromboembolism in a Clinical Decision Support System: Validation Study (Preprint)

BACKGROUND

It remains unknown whether capturing data from electronic health records (EHRs) using natural language processing (NLP) can improve venous thromboembolism (VTE) detection in different clinical settings.

OBJECTIVE

The aim of this study was to validate the NLP algorithm in a clinical decision support system for VTE risk assessment and integrated care (DeVTEcare) to identify VTEs from EHRs.

METHODS

All inpatients aged ≥18 years in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital from January 1 to December 31, 2021, were included as the validation cohort. The sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively), area under the receiver operating characteristic curve (AUC), and F1-scores along with their 95% CIs were used to analyze the performance of the NLP tool, with manual review of medical records as the reference standard for detecting deep vein thrombosis (DVT) and pulmonary embolism (PE). The primary end point was the performance of the NLP approach embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE among different hospital departments with different VTE risks. Subgroup analyses were performed among age, sex, and the study season.

RESULTS

Among 30,152 patients (median age 56 [IQR 41-67] years; 14,247/30,152, 47.3% females), the prevalence of VTE, PE, and DVT was 2.1% (626/30,152), 0.6% (177/30,152), and 1.8% (532/30,152), respectively. The sensitivity, specificity, LR+, LR-, AUC, and F1-score of NLP-facilitated VTE detection were 89.9% (95% CI 87.3%-92.2%), 99.8% (95% CI 99.8%-99.9%), 483 (95% CI 370-629), 0.10 (95% CI 0.08-0.13), 0.95 (95% CI 0.94-0.96), and 0.90 (95% CI 0.90-0.91), respectively. Among departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) were in the surgery department (all P<.001). Among low, intermediate, and high VTE risks in hospital departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively) and F1-score (0.97 vs 0.90 vs 0.90, respectively) as well as the lowest LR- (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC; all P<.001). Subgroup analysis of the age, sex, and season demonstrated consistently good performance of VTE detection with >87% sensitivity and specificity and >89% AUC and F1-score. The NLP algorithm performed better among patients aged ≤65 years than among those aged >65 years (F1-score 0.93 vs 0.89, respectively; P<.001).

CONCLUSIONS

The NLP algorithm in our DeVTEcare identified VTE well across different clinical settings, especially in patients in surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm can help to inform accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.

Biomarker derived risk scores predict venous thromboembolism and major bleeding among patients with COVID-19

Background

Venous thromboembolism (VTE) risk is increased in patients with COVID‐19 infection. Understanding which patients are likely to develop VTE may inform pharmacologic VTE prophylaxis decision making. The hospital‐associated venous thromboembolism–Intermountain Risk Score (HA‐VTE IMRS) and the hospital‐associated major bleeding–Intermountain Risk Score (HA‐MB IMRS) are risk scores predictive of VTE and bleeding that were derived from only patient age and data found in the complete blood count (CBC) and basic metabolic panel (BMP).

Objectives

We assessed the HA‐VTE IMRS and HA‐MB IMRS for predictiveness of 90‐day VTE and major bleeding, respectively, among patients diagnosed with COVID‐19, and further investigated if adding D‐dimer improved these predictions. We also reported 30‐day outcomes.

Patients/Methods

We identified 5047 sequential patients with a laboratory confirmed diagnosis of COVID‐19 and a CBC and BMP between 2 days before and 7 days following the diagnosis of COVID‐19 from March 12, 2020, to February 28, 2021. We calculated the HA‐VTE IMRS and the HA‐MB IMRS for all patients. We assessed the added predictiveness of D‐dimer obtained within 48 hours of the COVID test.

Results

The HA‐VTE IMRS yielded a c‐statistic of 0.70 for predicting 90‐day VTE and adding D‐dimer improved the c‐statistic to 0.764 with the corollary sensitivity/specificity/positive/negative predictive values of 49.4%/75.7%/6.7%/97.7% and 58.8%/76.2%/10.9%/97.4%, respectively. Among hospitalized and ambulatory patients separately, the HA‐VTE IMRS performed similarly. The HA‐MB IMRS predictiveness for 90‐day major bleeding yielded a c‐statistic of 0.64.

Conclusion

The HA‐VTE IMRS and HA‐MB IMRS predict 90‐ and 30‐day VTE and major bleeding among COVID‐19 patients. Adding D‐dimer improved the predictiveness of the HA‐VTE IMRS for VTE.

D-dimer Thresholds to Exclude Pulmonary Embolism among COVID-19 Patients in the Emergency Department: Derivation with Independent Validation

Objective

To derive and validate a D-dimer cutoff for ruling out pulmonary embolism (PE) in COVID-19 patients presenting to the emergency department (ED).

Methods

A retrospective cohort study was performed in an integrated healthcare system including 22 adult ED's between March 1, 2020, and January 31, 2021. Results were validated among patients enrolled in the RECOVER Registry, representing data from 154 ED's from 26 US states. Consecutive ED patients with laboratory confirmed COVID-19, a D-dimer performed within 48 h of ED arrival, and with objectively confirmed PE were compared to those without PE. After identifying a D-dimer threshold at which the 95% confidence lower bound of the negative predictive value for PE was higher than 98% in the derivation cohort, it was validated using RECOVER registry data.

Results

Among 3978 patients with a D-dimer result, 3583 with confirmed COVID-19 infection were included in the derivation cohort. Overall, PE incidence was 4.1% and a D-dimer cutoff of <2 μ/mL (2000 ng/mL) was associated with a NPV of 98.5% (95% CI = 98.0%−98.9%). In the validation cohort of 13,091 patients with a D-dimer, 7748 had confirmed COVID-19 infection, and the PE incidence was 1.14%. A D-dimer cutoff of <2 μ/mL was associated with a NPV of 99.5% (95% CI = 99.3%−99.7%).

Conclusion

A D-dimer cutoff of <2 μ/ml was associated with a high negative predictive value for PE among patients with COVID-19. However, the resultant sensitivity for PE result at that threshold without pre-test probability assessment would be considered clinically unsafe.

Post-discharge thrombosis and bleeding in medical patients: A novel risk score derived from ubiquitous biomarkers

BACKGROUND

Some hospitalized medical patients experience venous thromboembolism (VTE) following discharge. Prophylaxis extended beyond hospital discharge (extended duration thromboprophylaxis [EDT]) may reduce this risk. However, EDT is costly and can cause bleeding, so selecting appropriate patients is essential. We formerly reported the performance of a mortality risk prediction score (Intermountain Risk Score [IMRS]) that was minimally predictive of 90-day hospital-associated venous thromboembolism (HA-VTE) and major bleeding (HA-MB). We used the components of the IMRS to calculate de novo risk scores to predict 90-day HA-VTE (HA-VTE IMRS) and major bleeding (HA-MB IMRS).

METHODS

From 45 669 medical patients we randomly assigned 30 445 to derive the HA-VTE IMRS and the HA-MB IMRS. Backward stepwise regression and bootstrapping identified predictor covariates from the blood count and basic chemistry. These candidate variables were split into quintiles, and the referent quintile was that with the lowest event rate for HA-VTE and HA-MB; respectively. A clinically relevant rate of HA-VTE and HA-MB was used to inform outcome rates. Performance was assessed in the derivation set of 15 224 patients.

RESULTS

The HA-VTE IMRS and HA-MB IMRS area under the receiver operating curve (AUC) in the derivation set were 0.646, and 0.691, respectively. In the validation set, the HA-VTE IMRS and HA-MB IMRS AUCs were 0.60 and 0.643.

CONCLUSIONS

Risk scores derived from components of routine labs ubiquitous in clinical care identify patients that are at risk for 90-day postdischarge HA-VTE and major bleeding. This may identify a subset of patients with high HA-VTE risk and low HA-MB risk who may benefit from EDT.