Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

BACKGROUND

The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs' performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

OBJECTIVE

This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

METHODS

Healthily's SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

RESULTS

Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

CONCLUSIONS

We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Natural language processing to identify and characterize spondyloarthritis in clinical practice

OBJECTIVE

This study aims to use a novel technology based on natural language processing (NLP) to extract clinical information from electronic health records (EHRs) to characterise the clinical profile of patients diagnosed with spondyloarthritis (SpA) at a large-scale hospital.

METHODS

An observational, retrospective analysis was conducted on EHR data from all patients with SpA (including psoriatic arthritis (PsA)) at Hospital Universitario La Paz, between 2020 and 2022. Data were collected using Savana Manager, an NLP-based system, enabling the extraction of information from unstructured, free-text EHRs. Variables analysed included demographic data, SpA subtypes, comorbidities and treatments. The performance of the technology in detecting SpA clinical entities was evaluated through precision, recall and F-1 score metrics.

RESULTS

From a hospital population of 639 474 patients, 4337 (0.7%) patients had a diagnosis of SpA or their subtypes in their EHR. The population predominantly comprised men (55.3%) with a mean age of 50.9 years. Peripheral SpA (including PsA) was reported in 31.6%, axial SpA in 20.9%, both axial and peripheral SpA in 3.7%, while 43.7% of patients did not have the SpA subtype reported. Common comorbidities included hypertension (25.0%), dyslipidaemia (22.2%) and diabetes mellitus (15.5%). The use of conventional disease-modifying antirheumatic drugs (csDMARDs) and biological DMARDs (bDMARDs) was documented, with methotrexate (25.3% of patients) being the most used csDMARDs and adalimumab (10.6% of patients) the most used bDMARD. The NLP technology demonstrated high precision and recall, with all the assessed F-1 score values over 0.80, indicating reliable data extraction.

CONCLUSION

The application of NLP technology facilitated the characterisation of the SpA patient profile, including demographics, clinical features, comorbidities and treatments. This study supports the utility of NLP in enhancing the understanding of SpA and suggests its potential for improving patient management by extracting meaningful information from unstructured EHR data.

Chronic liver disease-associated severe thrombocytopenia in Spain: Results from a retrospective study using machine learning and natural language processing

BACKGROUND

Patients with chronic liver disease (CLD) often develop thrombocytopenia (TCP) as a complication. Severe TCP (platelet count<50×109/L) can increase morbidity and complicate CLD management, increasing bleeding risk during invasive procedures.

OBJECTIVES

To describe the real-world scenario of CLD-associated severe TCP patients' clinical characteristics. To evaluate the association between invasive procedures, prophylactic treatments, and bleeding events in this group of patients. To describe their need of medical resource use in Spain.

METHODS

This is a retrospective, multicenter study including patients who had confirmed diagnosis of CLD and severe TCP in four hospitals within the Spanish National Healthcare Network from January 2014 to December 2018. We analyzed the free-text information from Electronic Health Records (EHRs) of patients using Natural Language Processing (NLP), machine learning techniques, and SNOMED-CT terminology. Demographics, comorbidities, analytical parameters and characteristics of CLD were extracted at baseline and need for invasive procedures, prophylactic treatments, bleeding events and medical resources used in the follow up period. Frequency tables were generated for categorical variables, whereas continuous variables were described in summary tables as mean (SD) and median (Q1-Q3).

RESULTS

Out of 1,765,675 patients, 1787 had CLD and severe TCP; 65.2% were male with a mean age of 54.7 years old. Cirrhosis was detected in 46% (n=820) of patients and 9.1% (n=163) had hepatocellular carcinoma. Invasive procedures were needed in 85.6% of patients during the follow up period. Patients undergoing procedures compared to those patients without invasive procedures presented higher rates of bleeding events (33% vs 8%, p<0.0001) and higher number of bleedings. While prophylactic platelet transfusions were given to 25.6% of patients undergoing procedures, TPO receptor agonist use was only detected in 3.1% of them. Most patients (60.9%) required at least one hospital admission during the follow up and 14.4% of admissions were due to bleeding events with a hospital length of stay of 6 (3, 9) days.

CONCLUSIONS

NLP and machine learning are useful tools to describe real-world data in patients with CLD and severe TCP in Spain. Bleeding events are frequent in those patients who need invasive procedures, even receiving platelet transfusions as a prophylactic treatment, increasing the further use of medical resources. Because that, new prophylactic treatments that are not yet generalized, are needed.

Hyperthyroidism and cardiovascular disease: an association study using big data analytics

BACKGROUND

The cardiovascular (CV) system is profoundly affected by thyroid hormones. Both hypo- and hyperthyroidism can increase the risk of severe CV complications.

OBJECTIVE

To assess the association of hyperthyroidism with major CV risk factors (CVRFs) and CV diseases (CVDs) using a big data methodology with the Savana Manager platform.

MATERIAL AND METHODS

This was an observational and retrospective study. The data were obtained from the electronic medical records of the University Hospital Puerta de Hierro Majadahonda (Spain). Artificial intelligence techniques were used to extract the information from the electronic health records and Savana Manager 3.0 software was used for analysis.

RESULTS

Of a total of 540,939 patients studied (53.62% females; mean age 42.2 ± 8.7 years), 5504 patients (1.02%; 69.9% women) had a diagnosis of hyperthyroidism. Patients with this diagnosis had a significantly (p < 0.0001) higher frequency of CVRFs than that found in non-hyperthyroid subjects. The higher frequency of CVRFs in patients with hyperthyroidism was observed in both women and men and in patients younger and older than 65 years of age. The total frequency of CVDs was also significantly (p < 0.0001) higher in patients diagnosed with hyperthyroidism than that found in patients without this diagnosis. The highest odds ratio values obtained were 6.40 (4.27-9.61) for embolic stroke followed by 5.99 (5.62-6.38) for atrial fibrillation. The frequency of all CVDs evaluated in patients with a diagnosis of hyperthyroidism was significantly higher in both women and men, as well as in those younger and older than 65 years, compared to subjects without this diagnosis. A multivariate regression analysis showed that hyperthyroidism was significantly and independently associated with all the CVDs evaluated except for embolic stroke.

CONCLUSION

The data from this hospital cohort suggest that there is a significant association between the diagnosis of hyperthyroidism and the main CVRFs and CVDs in our population, regardless of the age and gender of the patients. Our study, in addition to confirming this association, provides useful information for understanding the applicability of artificial intelligence techniques to "real-world data and information".

Prevalence and clinical characteristics of patients with rheumatoid arthritis with interstitial lung disease using unstructured healthcare data and machine learning

OBJECTIVES

Real-world data regarding rheumatoid arthritis (RA) and its association with interstitial lung disease (ILD) is still scarce. This study aimed to estimate the prevalence of RA and ILD in patients with RA (RAILD) in Spain, and to compare clinical characteristics of patients with RA with and without ILD using natural language processing (NLP) on electronic health records (EHR).

METHODS

Observational case-control, retrospective and multicentre study based on the secondary use of unstructured clinical data from patients with adult RA and RAILD from nine hospitals between 2014 and 2019. NLP was used to extract unstructured clinical information from EHR and standardise it into a SNOMED-CT terminology. Prevalence of RA and RAILD were calculated, and a descriptive analysis was performed. Characteristics between patients with RAILD and RA patients without ILD (RAnonILD) were compared.

RESULTS

From a source population of 3 176 165 patients and 64 241 683 EHRs, 13 958 patients with RA were identified. Of those, 5.1% patients additionally had ILD (RAILD). The overall age-adjusted prevalence of RA and RAILD were 0.53% and 0.02%, respectively. The most common ILD subtype was usual interstitial pneumonia (29.3%). When comparing RAILD versus RAnonILD patients, RAILD patients were older and had more comorbidities, notably concerning infections (33.6% vs 16.5%, p<0.001), malignancies (15.9% vs 8.5%, p<0.001) and cardiovascular disease (25.8% vs 13.9%, p<0.001) than RAnonILD. RAILD patients also had higher inflammatory burden reflected in more pharmacological prescriptions and higher inflammatory parameters and presented a higher in-hospital mortality with a higher risk of death (HR 2.32; 95% CI 1.59 to 2.81, p<0.001).

CONCLUSIONS

We found an estimated age-adjusted prevalence of RA and RAILD by analysing real-world data through NLP. RAILD patients were more vulnerable at the time of inclusion with higher comorbidity and inflammatory burden than RAnonILD, which correlated with higher mortality.

Predictive model for a second hip fracture occurrence using natural language processing and machine learning on electronic health records

Hip fractures (HFx) are associated with a higher morbidity and mortality rates, leading to a significant reduction in life quality and in limitation of patient´s mobility. The present study aimed to obtain real-world evidence on the clinical characteristics of patients with an initial and a second hip fracture (HFx) and develop a predictive model for second HFx using artificial intelligence. Electronic health records from one hospital centre in Spain from January 2011 to December 2019 were analysed using EHRead® technology, based on natural language processing and machine learning. A total of 1,960 patients with HFx were finally included during the study period after meeting all inclusion and exclusion criteria. From this total, 1835 (93.6%) patients were included in the HFx subgroup, while 124 (6.4%) were admitted to the second HFx (2HFx) subgroup. The mean age of the participants was 84 years and 75.5% were female. Most of comorbidities were more frequently identified in the HFx group, including hypertension (72.0% vs. 67.2%), cognitive impairment (33.0% vs. 31.2%), diabetes mellitus (28.7% vs. 24.8%), heart failure (27.6% vs. 22.4%) and chronic kidney disease (26.9% vs. 16.0%). Based on clinical criteria, 26 features were selected as potential prediction factors. From there, 16 demographics and clinical characteristics such as comorbidities, medications, measures of disabilities for ambulation and type of refracture were selected for development of a competitive risk model. Specifically, those predictors with different associated risk ratios, sorted from higher to lower risk relevance were visual deficit, malnutrition, walking assistance, hypothyroidism, female sex, osteoporosis treatment, pertrochanteric fracture, dementia, age at index, osteoporosis, renal failure, stroke, COPD, heart disease, anaemia, and asthma. This model showed good performance (dependent AUC: 0.69; apparent performance: 0.75) and could help the identification of patients with higher risk of developing a second HFx, allowing preventive measures. This study expands the current available information of HFx patients in Spain and identifies factors that exhibit potential in predicting a second HFx among older patients.

Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance

INTRODUCTION

As part of routine safety surveillance, thousands of articles of potential interest are manually triaged for review by safety surveillance teams. This manual triage task is an interesting candidate for automation based on the abundance of process data available for training, the performance of natural language processing algorithms for this type of cognitive task, and the small number of safety signals that originate from literature review, resulting in its lower risk profile. However, deep learning algorithms introduce unique risks and the validation of such models for use in Good Pharmacovigilance Practice remains an open question.

OBJECTIVE

Qualifying an automated, deep learning approach to literature surveillance for use at AstraZeneca.

METHODS

The study is a prospective validation of a literature surveillance triage model, comparing its real-world performance with that of human surveillance teams working in parallel. The biggest risk in modifying this triage process is missing a safety signal (resulting in model false negatives) and hence model recall is the main evaluation metric considered.

RESULTS

The model demonstrates consistent global performance from training through testing, with recall rates comparable to that of existing surveillance teams. The model is accepted for use specifically for those products where non-inferiority to the manual process is rigorously demonstrated.

CONCLUSION

Characterizing model performance prospectively, under real-world conditions, allows us to thoroughly examine model consistency and failure modes, qualifying it for use in our surveillance processes. We also identify potential future improvements and recognize the opportunity for the community to collaborate on this shared task.

Using natural language processing to explore characteristics and management of patients with axial spondyloarthritis and psoriatic arthritis treated under real-world conditions in Spain: SpAINET study

Background

Spondyloarthritis (SpA) is a group of related but phenotypically distinct inflammatory disorders that include axial SpA (axSpA) and psoriatic arthritis (PsA). Information on the characteristics and management of these patients in the real world remains scarce.

Objectives

To explore the characteristics and management [disease activity assessment and treatment with secukinumab (SEC) or other biologic disease-modifying antirheumatic drugs (bDMARDs)] of axSpA and PsA patients using natural language processing (NLP) in Electronic Health Records (EHRs).

Design

National, multicenter, observational, and retrospective study.

Methods

We analyzed free-text and structured clinical information from EHR at three hospitals. All adult patients with axSpA, PsA or non-classified SpA from 2018 to 2021 with minimum follow-up of three months were included when starting SEC or other bDMARDs. Clinical variables were extracted using EHRead® technology based on Systemized Nomenclature of Medicine-Clinical Terms (SNOMED CT) terminology.

Results

Out of 887,735 patients, 758 were included, of which 328 had axSpA [58.5% male; mean (SD) age of 50.7 (12.7) years], 365 PsA [54.8% female, 53.9 (12.4) years], and 65 non-classified SpA. Mean (SD) time since diagnosis was 36.8 (61.0) and 24.1 (35.2) months for axSpA and PsA, respectively. Only 116 axSpA patients (35.3%) had available Ankylosing Spondylitis Disease Activity Score (ASDAS) or Bath Ankylosing Spondylitis Disease Activity Index (BASDAI) at bDMARD onset, of which 61 presented active disease. Disease Activity in PSoriatic Arthritis (DAPSA) or Disease Assessment Score - 28 joints (DAS-28) values at bDMARD onset were available for only 61 PsA (16.7%) patients, with 23 of them having active disease. The number of patients with available tender joint count or swollen joint count assessment was 68 (20.7%) and 59 (18%) for axSpA, and 115 (31.5%) and 119 (32.6%) for PsA, respectively. SEC was used in 63 (19.2%) axSpA patients and in 63 (17.3%) PsA patients.

Conclusion

Using NLP, the study showed that around one-third of axSpA and one-sixth of PsA patients have disease activity assessments with ASDAS/BASDAI or DAPSA/DAS-28, respectively, highlighting an area of improvement in these patients' management.

The Clinical Profile of Patients with COPD Is Conditioned by Age

In recent years, many studies have analyzed the importance of integrating time, or aging, into the equation that relates genetics and the environment to the development and origin of COPD. Under conditions of daily clinical practice, our study attempts to identify the differences in the clinical profile of patients with COPD according to age and the impact on the global burden of the disease. This study is non-interventional and observational, using artificial intelligence and data captured from electronic medical records. The study population included patients who were diagnosed with COPD between 2011 and 2021. A total of 73,901 patients had a diagnosis of COPD. The mean age was 73 years (95% CI: 72.9-73.1), and 56,763 were men (76.8%). We observed a specific prevalence of obesity, heart failure, depression, and hiatal hernia in women (p < 0.001), and ischemic heart disease and obstructive sleep apnea (OSA) in men (p < 0.001). In the analysis by age ranges, a progressive increase in cardiovascular risk factors was observed with age. In conclusion, in a real-life setting, COPD is a disease that primarily affects older subjects and frequently presents with comorbidities that are decisive in the evolutionary course of the disease.

Chronic Respiratory Diseases as a Risk Factor for Herpes Zoster Infection

INTRODUCTION

Herpes zoster (HZ) is a condition that results from the reactivation of the varicella zoster virus (VZV). Several diseases have been reported to increase the risk of developing HZ and postherpetic neuralgia (PHN). The objective of this study is to analyze the prevalence and risk factors for HZ and PHN in the most frequent chronic respiratory diseases, which are chronic obstructive pulmonary disease (COPD), asthma, lung cancer and obstructive sleep apnea (OSA).

METHODS

We conducted an observational, retrospective, non-interventional study between January 2012 and December 2020 based on data from the Castilla-La Mancha Regional Health System in Spain. We used the Savana Manager 3.0 artificial intelligence-enabled system to collect information from electronic medical records.

RESULTS

31765 subjects presented a diagnosis of HZ. Mean age was 64.5 years (95%CI 64.3-64.7), and 58.2% were women. The prevalence of HZ showed an increasing trend in patients over the age of 50. A risk analysis adjusted for sex and comorbidities in COPD, asthma, lung cancer and OSA presented a higher risk of developing HZ in the first three (OR 1.16 [95%CI 1.13-1.19], 1.67 [1.63-1.71], 1.68 [1.60-1.76], respectively), which further increased in all three when associated with comorbidities. Regarding postherpetic neuralgia, an increased risk was only observed related to COPD and lung cancer (OR 1.24 [95%CI 1.23-1.25], 1.14 [1.13-1.16], respectively), further increasing when associated with comorbidities.

CONCLUSIONS

In a standard clinical practice setting, the most prevalent respiratory diseases (asthma, COPD and lung cancer) are related to a higher risk of HZ and PHN. These data are fundamental to assess the potential impact of vaccination in this population.

Primary Hyperparathyroidism and Cardiovascular Disease: An Association Study Using Clinical Natural Language Processing Systems and Big Data Analytics

Primary hyperparathyroidism (PHPT) seems to be associated with different cardiovascular diseases (CVDs). We evaluated the association of PHPT with major CV risk factors (CVRFs) and CVDs by using artificial intelligence (AI) tools. An observational and retrospective study was conducted using data from the electronic health records (EHRs) of the Hospital Universitario Puerta de Hierro Majadahonda (Spain). Of a total of 699,157 patients over 18 years of age studied (54.7% females), 6515 patients (0.9%; 65.4% women; mean age 67.6 ± 15.9 years) had a diagnosis of PHPT. The overall frequencies of hypertension, dyslipidemia, diabetes mellitus, and smoking habit in the cohort of patients with PTHP were all significantly (p < 0.001) higher than those found in patients without a diagnosis of PTHP. The total frequency of stroke, ischemic heart disease, atrial fibrillation, deep vein thrombosis, and pulmonary embolism in the cohort of PHPT patients were significantly (p < 0.001) higher than that found in patients without the diagnosis of PHPT. A multivariate regression analysis showed that PHPT was significantly (p < 0.001) and independently associated with all the CVDs evaluated. Our data show that there is a significant association between the diagnosis of PHPT and the main CVRFs and CVDs in our hospital population.

Prevalence of cancer in patients with hypothyroidism: Analysis using big data tools

OBJECTIVE

To evaluate the frequency of different types of cancer in patients diagnosed with hypothyroidism using big data methodology on the Savana Manager platform.

METHODS

An observational, retrospective study was carried out using electronic medical record (EMR) data from the Hospital Universitario Puerta de Hierro Majadahonda (Madrid). Information from the EMRs was extracted using artificial intelligence techniques and analysed using the Savana Manager v3.0 software. Searches were performed using the term "hypothyroidism" and the terms corresponding to the tumours analysed.

RESULTS

Of a total population of 506,749 patients, 23,570 (4.7%) were diagnosed with hypothyroidism. Patients with this diagnosis had a significantly higher frequency of cancer than that found in non-hypothyroid subjects (OR 2.09, 95% confidence interval [CI] 2.01-2.17). This higher frequency was found both in women (OR 1.99, 95% CI 1.90-2.08) and in men (OR 2.83, 95% CI 2.63-3.05). However, this higher frequency of cancer was not observed in hypothyroid patients older than 60 years (OR 0.97, 95% CI 0.92-1.02). Although the frequency of most of the neoplasms studied individually was higher in the population with hypothyroidism, we observed that hypothyroid patients over 60 years of age had a significant decrease in the frequency of prostate, lung, colorectal, and liver cancer.

CONCLUSION

Data from this hospital cohort suggest that there is a significant association between the diagnosis of hypothyroidism and cancer. However, this association is less evident in hypothyroid patients older than 60 years.

Real-World Evidence on the Clinical Characteristics and Management of Patients with Chronic Lymphocytic Leukemia in Spain Using Natural Language Processing: The SRealCLL Study

The SRealCLL study aimed to obtain real-world evidence on the clinical characteristics and treatment patterns of patients with chronic lymphocytic leukemia (CLL) using natural language processing (NLP). Electronic health records (EHRs) from seven Spanish hospitals (January 2016-December 2018) were analyzed using EHRead® technology, based on NLP and machine learning. A total of 534 CLL patients were assessed. No treatment was detected in 270 (50.6%) patients (watch-and-wait, W&W). First-line (1L) treatment was identified in 230 (43.1%) patients and relapsed/refractory (2L) treatment was identified in 58 (10.9%). The median age ranged from 71 to 75 years, with a uniform male predominance (54.8-63.8%). The main comorbidities included hypertension (W&W: 35.6%; 1L: 38.3%; 2L: 39.7%), diabetes mellitus (W&W: 24.4%; 1L: 24.3%; 2L: 31%), cardiac arrhythmia (W&W: 16.7%; 1L: 17.8%; 2L: 17.2%), heart failure (W&W 16.3%, 1L 17.4%, 2L 17.2%), and dyslipidemia (W&W: 13.7%; 1L: 18.7%; 2L: 19.0%). The most common antineoplastic treatment was ibrutinib in 1L (64.8%) and 2L (62.1%), followed by bendamustine + rituximab (12.6%), obinutuzumab + chlorambucil (5.2%), rituximab + chlorambucil (4.8%), and idelalisib + rituximab (3.9%) in 1L and venetoclax (15.5%), idelalisib + rituximab (6.9%), bendamustine + rituximab (3.5%), and venetoclax + rituximab (3.5%) in 2L. This study expands the information available on patients with CLL in Spain, describing the diversity in patient characteristics and therapeutic approaches in clinical practice.

Impact of Advanced Age on the Incidence of Major Adverse Cardiovascular Events in Patients with Type 2 Diabetes Mellitus and Stable Coronary Artery Disease in a Real-World Setting in Spain

Patients with type 2 diabetes mellitus (T2DM) and coronary artery disease (CAD) without myocardial infarction (MI) or stroke are at high risk for major cardiovascular events (MACEs). We aimed to provide real-world data on age-related clinical characteristics, treatment management, and incidence of major cardiovascular outcomes in T2DM-CAD patients in Spain from 2014 to 2018. We used EHRead® technology, which is based on natural language processing and machine learning, to extract unstructured clinical information from electronic health records (EHRs) from 12 hospitals. Of the 4072 included patients, 30.9% were younger than 65 years (66.3% male), 34.2% were aged 65-75 years (66.4% male), and 34.8% were older than 75 years (54.3% male). These older patients were more likely to have hypertension (OR 2.85), angina (OR 1.64), heart valve disease (OR 2.13), or peripheral vascular disease (OR 2.38) than those aged <65 years (p < 0.001 for all comparisons). In general, they were also more likely to receive pharmacological and interventional treatments. Moreover, these patients had a significantly higher risk of MACEs (HR 1.29; p = 0.003) and ischemic stroke (HR 2.39; p < 0.001). In summary, patients with T2DM-CAD in routine clinical practice tend to be older, have more comorbidities, are more heavily treated, and have a higher risk of developing MACE than is commonly assumed from clinical trial data.

Development of a predictive model of venous thromboembolism recurrence in anticoagulated cancer patients using machine learning

INTRODUCTION

Patients with cancer and venous thromboembolism (VTE) show a high risk of VTE recurrence during anticoagulant treatment. This study aimed to develop a predictive model to assess the risk of VTE recurrence within 6 months at the moment of primary VTE diagnosis in these patients.

MATERIALS AND METHODS

Using the EHRead® technology, based on Natural Language Processing (NLP) and machine learning (ML), the unstructured data in electronic health records from 9 Spanish hospitals between 2014 and 2018 were extracted. Both clinically- and ML-driven feature selection were performed to identify predictors for VTE recurrence. Logistic regression (LR), decision tree (DT), and random forest (RF) algorithms were used to train different prediction models, which were subsequently validated in a hold-out data set.

RESULTS

A total of 16,407 anticoagulated cancer patients with diagnosis of VTE were identified (54.4 % male and median age 70). Deep vein thrombosis, pulmonary embolism and metastases were observed in 67.2 %, 26.6 %, and 47.7 % of the patients, respectively. During the study follow-up, 11.4 % of the patients developed a recurrent VTE, being more frequent in patients with lung cancer. Feature selection and model training based on ML identified primary pulmonary embolism, deep vein thrombosis, metastasis, adenocarcinoma, hemoglobin and serum creatinine levels, platelet and leukocyte count, family history of VTE, and patients' age as predictors of VTE recurrence within 6 months of VTE diagnosis. The LR model had an AUC-ROC (95 % CI) of 0.66 (0.61, 0.70), the DT of 0.69 (0.65, 0.72) and the RF of 0.68 (0.63, 0.72).

CONCLUSIONS

This is the first ML-based predictive model designed to predict 6-months VTE recurrence in patients with cancer. These results hold great potential to assist clinicians to identify the high-risk patients and improve their clinical management.

Validation of an Artificial Intelligence driven framework to automatically detect red flag symptoms in screening for rare diseases in electronic health records: hereditary transthyretin amyloidosis polyneuropathy as a key example

Rare life-threatening conditions, such as multisystemic hereditary transthyretin amyloidosis (ATTRv) polyneuropathy, are often underdiagnosed or diagnosed late in the disease course, although early diagnosis is crucial for treatment success. Red flag symptoms have been identified, but manual screening of multidisciplinary medical records on this set of symptoms is time-consuming. This study aimed to validate a Natural Language Processing (NLP) algorithm to perform such a search in an automated manner, in order to improve early diagnosis and treatment. A novel state-of-the-art NLP procedure was applied to extract red flag symptoms from patients' electronic medical records and to select patients at risk for ATTRv polyneuropathy for further clinical review. Accuracy of the algorithm was assessed through comparison with a manual standard on a random sample of 300 patients. Out of a retrospective sample of 1015 patients, the NLP algorithm yielded 128 patients with three or more red flag symptoms of which 69 patients were considered eligible for genetic testing after clinical review. High accuracy was found in the detection of red flag symptoms, with F1 scores between 0.88 and 0.98. A relative increase of 48.6% in genetic testing, to identify patients with a rare disease earlier, was demonstrated. An NLP algorithm, after clinical validation, offers a valid and accurate tool to detect red flag symptoms in medical records across multiple disciplines, supporting better screening for patients with rare diseases. This opens the door to further NLP applications, facilitating rapid diagnosis and early treatment of rare diseases.

Thyroid carcinoma in elderly people: Characterization using big data tools

OBJECTIVE

To determine the clinical-histological data associated comorbidities and the use of health resources of elderly patients with thyroid cancer.

METHODS

An observational, retrospective study was carried out using data from the electronic medical record (EMR) of the Hospital Universitario Puerta de Hierro Majadahonda (Madrid, Spain). The information was extracted using artificial intelligence techniques and analysed using the Savana Manager 3.0 software. We differentiated between younger people (0-59 years) and older people (60 or more years) and, within this latter group, between people of advanced age (60-74 years) and elderly people (75 or more years).

RESULTS

Of a total of 509,517 patients, 1781 (0.35%) were diagnosed with thyroid cancer. Compared to younger patients, older patients presented a lower proportion of papillary carcinoma (64.2% vs. 75.3%) as well as a higher proportion of follicular carcinoma (9.3% vs. 5.0%) and other histological types (26.5% vs. 19.7%; p < 0.001). Young people with thyroid cancer exhibited prevalences of risk factors and most of the cardiovascular diseases studied significantly higher than those found in the general population. Elderly patients, compared with those of advanced age, showed greater comorbidity. However, a trend towards a lower consumption of healthcare resources was observed when elderly patients were compared with those of advanced age.

CONCLUSION

The clinical characteristics, comorbidities and consumption of health resources of patients with thyroid cancer vary markedly with age. Elderly patients are characterized by a high burden of comorbidities that is not accompanied by a notable increase in their consumption of health resources.

Symptoms timeline and outcomes in amyotrophic lateral sclerosis using artificial intelligence

Amyotrophic lateral sclerosis (ALS) is a fatal, neurodegenerative motor neuron disease. Although an early diagnosis is crucial to provide adequate care and improve survival, patients with ALS experience a significant diagnostic delay. This study aimed to use real-world data to describe the clinical profile and timing between symptom onset, diagnosis, and relevant outcomes in ALS. Retrospective and multicenter study in 5 representative hospitals and Primary Care services in the SESCAM Healthcare Network (Castilla-La Mancha, Spain). Using Natural Language Processing (NLP), the clinical information in electronic health records of all patients with ALS was extracted between January 2014 and December 2018. From a source population of all individuals attended in the participating hospitals, 250 ALS patients were identified (61.6% male, mean age 64.7 years). Of these, 64% had spinal and 36% bulbar ALS. For most defining symptoms, including dyspnea, dysarthria, dysphagia and fasciculations, the overall diagnostic delay from symptom onset was 11 (6-18) months. Prior to diagnosis, only 38.8% of patients had visited the neurologist. In a median post-diagnosis follow-up of 25 months, 52% underwent gastrostomy, 64% non-invasive ventilation, 16.4% tracheostomy, and 87.6% riluzole treatment; these were more commonly reported (all Ps < 0.05) and showed greater probability of occurrence (all Ps < 0.03) in bulbar ALS. Our results highlight the diagnostic delay in ALS and revealed differences in the clinical characteristics and occurrence of major disease-specific events across ALS subtypes. NLP holds great promise for its application in the wider context of rare neurological diseases.

Clinical Characterization of Inpatients with Acute Conjunctivitis: A Retrospective Analysis by Natural Language Processing and Machine Learning

Background Acute bacterial conjunctivitis (ABC) is a relatively common medical condition caused by different pathogens. Although it rarely threatens vision, it is one of the most common conditions that cause red eyes and may be accompanied by discomfort and discharge. The study aimed to identify and characterize inpatients with ABC treated with topical antibiotics. Methods The EHRead® technology, based on natural language processing (NLP) and machine learning, was used to extract and analyze the clinical information in the electronic health records (EHRs) of antibiotic-treated patients with conjunctivitis and admitted to five hospitals in Spain between January 2014 and December 2018. Categorical variables were described by frequency, whereas numerical variables included the mean, standard deviation, median, and quartiles. Results From a source population of 2,071,812 adult patients who attended the participating hospitals in the study period, 11,110 patients diagnosed with acute conjunctivitis were identified. Six thousand five hundred eighty-three patients were treated with antibiotics, comprising the final study population. Microbiology was tested only on 12.1% of patients. Antibiotics, mainly tobramycin, and corticosteroids, mainly dexamethasone, were usually prescribed. NSAIDs were also used in about 50% of patients, always combined with antibiotics. Conclusions The present study provided a realistic representation of the hospital practice concerning managing patients with acute antibiotic-treated conjunctivitis. The diagnosis is usually based on the clinical ground, microbiology is rarely tested, few bacteria species are involved, and local antibiotics are frequently associated with corticosteroids and/or NSAIDs. Moreover, this study provided clinically relevant outcomes, based on new technology, that could be applied in clinical practice.

How can artificial intelligence optimize value-based contracting?

Efforts in the pharmaceutical market have been aimed at ensuring that the benefits obtained from the introduction of new therapies justify the associated costs. In recent years, drug payment models in healthcare have undergone a dramatic shift from focusing on volume (i.e., size of the target clinical population) to focusing on value (i.e., drug performance in real-world settings). In this context, value-based contracts (VBCs) were designed to align the payment of a drug to its clinical performance outside clinical trials by evaluating the effectiveness using real-word evidence (RWE). Despite their widespread implementation, different factors jeopardize the application of VBCs to most marketed drugs in a near future, including the need for easily measurable and relevant outcomes associated with clinical improvements, and access to a large patient population to assess said outcomes. Here, we argue that the extraction and analysis of massive amounts of RWE captured in patients' electronic health records (EHRs) will circumvent these issues and optimize negotiations in VBCs. Particularly, the use of Natural Language Processing (NLP) has proven successful in the analysis of structured and unstructured clinical information in EHRs in multicenter research studies. Thus, the application of NLP to analyze patient-centered information in EHRs in the context of innovative contracting can be utterly beneficial as it enables the real-time evaluation of treatment response and financial impact in real-world settings.

Major Adverse Cardiovascular Events in Coronary Type 2 Diabetic Patients: Identification of Associated Factors Using Electronic Health Records and Natural Language Processing

Patients with Type 2 diabetes mellitus (T2DM) and coronary artery disease (CAD) are at high risk of developing major adverse cardiovascular events (MACE). This is a multicenter, retrospective, and observational study performed in Spain aimed to characterize these patients in a real-world setting. Unstructured data from the Electronic Health Records were extracted by EHRead^®, a technology based on Natural Language Processing and machine learning. The association between new MACE and the variables of interest were investigated by univariable and multivariable analyses. From a source population of 2,184,662 patients, we identified 4072 adults diagnosed with T2DM and CAD (62.2% male, mean age 70 ± 11). The main comorbidities observed included arterial hypertension, hyperlipidemia, and obesity, with metformin and statins being the treatments most frequently prescribed. MACE development was associated with multivessel (Hazard Ratio (HR) = 2.49) and single coronary vessel disease (HR = 1.71), transient ischemic attack (HR = 2.01), heart failure (HR = 1.32), insulin treatment (HR = 1.40), and percutaneous coronary intervention (PCI) (HR = 2.27), whilst statins (HR = 0.73) were associated with a lower risk of MACE occurrence. In conclusion, we found six risk factors associated with the development of MACE which were related with cardiovascular diseases and T2DM severity, and treatment with statins was identified as a protective factor for new MACE in this study.

Natural Language Processing for Computer-Assisted Chart Review to Assess Documentation of Substance use and Psychopathology in Heart Failure Patients Awaiting Cardiac Resynchronization Therapy

CONTEXT

Advanced heart failure (HF) patients often experience distressing psychological symptoms, frequently meeting diagnostic criteria for psychological disorders, including anxiety, depression, and substance use disorder. Patients with device-based HF therapies have added risk for psychological disorders, with consequences for their physiological functioning, including adverse cardiac outcomes.

OBJECTIVES

This study used natural language processing (NLP) for computer-assisted chart review to assess documentation of mental health and substance use in HF patients awaiting cardiac resynchronization therapy (CRT), a device-based HF therapy.

METHODS

We applied NLP to clinical notes from electronic health records (EHR) of 965 consecutive patients, with 9821 total clinical notes, at two academic medical centers between 2004 and 2015. We developed and validated a keyword library capturing terms related to mental health and substance use, while balancing specificity and sensitivity.

RESULTS

Mean age was 71.6 years (SD = 11.8), 78% male, and 87% non-Hispanic White. Of the 544 patients (56.4%) with documentation of mental health history, 9.7% had their mental health assessed and 6.6% had a plan documented. Of the 773 patients (80.1%) with documentation of substance use history, 10 (1.0%) had an assessment, and 3 (0.3%) had a plan.

CONCLUSION

Despite clinical recommendations and standards of care, clinicians are under documenting assessments and plans prior to CRT. Future research should develop an algorithm to prompt clinicians to document this content. Such quality improvement efforts may ensure adherence to standards of care and clinical guidelines.

A glance into the future of diagnosis and treatment of spondyloarthritis

The last two decades have seen major developments in the field of spondyloarthritis (SpA), but there are still important unmet needs to address. In the future, we envisage important advances in the diagnosis and treatment of SpA. In the diagnosis of SpA, the use of online and social media tools will increase awareness of the disease and facilitate the referral of patients to rheumatology clinics. In addition, more specific diagnostic tests will be available, especially advanced imaging methods and new biomarkers. This will allow most patients to be diagnosed at an early stage of the disease. In the treatment of SpA, an increasing number of novel treatment targets can be expected, most of which will be directed against intracellular enzymes. We hope to see more strategy trials shaping treatment pathways in SpA and accommodating principals of precision medicine. Approved treatment options will be available for both axial and peripheral SpA. We also hope to intervene not only at the inflammation level but also at the level of underlying immunological processes that might be associated with a higher probability of long-standing remission if not a cure. Finally, artificial intelligence techniques will allow for the analysis of large-scale data to answer relevant research questions for the diagnosis and management of patients with SpA.

Real-life burden of hospitalizations due to COPD exacerbations in Spain

Patients with chronic obstructive pulmonary disease (COPD) often suffer episodes of exacerbation of symptoms (ECOPD) that may eventually require hospitalization due to several, often overlapping, causes. We aimed to analyse the characteristics of patients hospitalized because of ECOPD in a real-life setting using a big-data approach. The study population included all patients older than 40 years with a diagnosis of COPD (n=69.359; prevalence 3.72%) registered since January 1st, 2011, until March 1, 2020, in the database of the public healthcare service (SESCAM) of Castilla-La Mancha (Spain) (n=1.863.759 subjects). We used natural language processing (Savana Manager v3.0) to identify those who were hospitalized during this period for any cause, including ECOPD. During the study 26.453 COPD patients (38.1%) were hospitalized (at least once). Main diagnoses at discharge were respiratory infection (51%), heart failure (38%) or pneumonia (19%). ECOPD was the main diagnosis at discharge (or hospital death) in 8.331 of them (12.0% of the entire COPD population and 31.5% of those hospitalized). In-hospital ECOPD-related mortality rate was 3.1%. These patients were hospitalized 2.36 times per patient, with a mean hospital stay of 6.1 days. Heart failure (HF) was the most frequent comorbidity in patients hospitalized because of ECOPD (52.6%). This analysis shows that, in a real-life setting, ECOPD hospitalizations are prevalent, complex (particularly in relation to HF), repetitive and associated with significant in-hospital mortality.

Clinical characteristics and prognostic factors for Crohn's disease relapses using natural language processing and machine learning: a pilot study

BACKGROUND

The impact of relapses on disease burden in Crohn's disease (CD) warrants searching for predictive factors to anticipate relapses. This requires analysis of large datasets, including elusive free-text annotations from electronic health records. This study aims to describe clinical characteristics and treatment with biologics of CD patients and generate a data-driven predictive model for relapse using natural language processing (NLP) and machine learning (ML).

METHODS

We performed a multicenter, retrospective study using a previously validated corpus of CD patient data from eight hospitals of the Spanish National Healthcare Network from 1 January 2014 to 31 December 2018 using NLP. Predictive models were created with ML algorithms, namely, logistic regression, decision trees, and random forests.

RESULTS

CD phenotype, analyzed in 5938 CD patients, was predominantly inflammatory, and tobacco smoking appeared as a risk factor, confirming previous clinical studies. We also documented treatments, treatment switches, and time to discontinuation in biologics-treated CD patients. We found correlations between CD and patient family history of gastrointestinal neoplasms. Our predictive model ranked 25 000 variables for their potential as risk factors for CD relapse. Of highest relative importance were past relapses and patients' age, as well as leukocyte, hemoglobin, and fibrinogen levels.

CONCLUSION

Through NLP, we identified variables such as smoking as a risk factor and described treatment patterns with biologics in CD patients. CD relapse prediction highlighted the importance of patients' age and some biochemistry values, though it proved highly challenging and merits the assessment of risk factors for relapse in a clinical setting.

Assessment of medical management in Coronary Type 2 Diabetic patients with previous percutaneous coronary intervention in Spain: A retrospective analysis of electronic health records using Natural Language Processing

INTRODUCTION AND OBJECTIVES

Patients with type 2 diabetes (T2D) and stable coronary artery disease (CAD) previously revascularized with percutaneous coronary intervention (PCI) are at high risk of recurrent ischemic events. We aimed to provide real-world insights into the clinical characteristics and management of this clinical population, excluding patients with a history of myocardial infarction (MI) or stroke, using Natural Language Processing (NLP) technology.

METHODS

This is a multicenter, retrospective study based on the secondary use of 2014-2018 real-world data captured in the Electronic Health Records (EHRs) of 1,579 patients (0.72% of the T2D population analyzed; n = 217,632 patients) from 12 representative hospitals in Spain. To access the unstructured clinical information in EHRs, we used the EHRead® technology, based on NLP and machine learning. Major adverse cardiovascular events (MACE) were considered: MI, ischemic stroke, urgent coronary revascularization, and hospitalization due to unstable angina. The association between MACE rates and the variables included in this study was evaluated following univariate and multivariate approaches.

RESULTS

Most patients were male (72.13%), with a mean age of 70.5±10 years. Regarding T2D, most patients were non-insulin-dependent T2D (61.75%) with high prevalence of comorbidities. The median (Q1-Q3) duration of follow-up was 1.2 (0.3-4.5) years. Overall, 35.66% of patients suffered from at least one MACE during follow up. Using a Cox Proportional Hazards regression model analysis, several independent factors were associated with MACE during follow up: CAD duration (p < 0.001), COPD/Asthma (p = 0.021), heart valve disease (p = 0.031), multivessel disease (p = 0.005), insulin treatment (p < 0.001), statins treatment (p < 0.001), and clopidogrel treatment (p = 0.039).

CONCLUSIONS

Our results showed high rates of MACE in a large real-world series of PCI-revascularized patients with T2D and CAD with no history of MI or stroke. These data represent a potential opportunity to improve the clinical management of these patients.

Evaluation of Natural Language Processing for the Identification of Crohn Disease-Related Variables in Spanish Electronic Health Records: A Validation Study for the PREMONITION-CD Project

Background

The exploration of clinically relevant information in the free text of electronic health records (EHRs) holds the potential to positively impact clinical practice as well as knowledge regarding Crohn disease (CD), an inflammatory bowel disease that may affect any segment of the gastrointestinal tract. The EHRead technology, a clinical natural language processing (cNLP) system, was designed to detect and extract clinical information from narratives in the clinical notes contained in EHRs.

Objective

The aim of this study is to validate the performance of the EHRead technology in identifying information of patients with CD.

Methods

We used the EHRead technology to explore and extract CD-related clinical information from EHRs. To validate this tool, we compared the output of the EHRead technology with a manually curated gold standard to assess the quality of our cNLP system in detecting records containing any reference to CD and its related variables.

Results

The validation metrics for the main variable (CD) were a precision of 0.88, a recall of 0.98, and an F1 score of 0.93. Regarding the secondary variables, we obtained a precision of 0.91, a recall of 0.71, and an F1 score of 0.80 for CD flare, while for the variable vedolizumab (treatment), a precision, recall, and F1 score of 0.86, 0.94, and 0.90 were obtained, respectively.

Conclusions

This evaluation demonstrates the ability of the EHRead technology to identify patients with CD and their related variables from the free text of EHRs. To the best of our knowledge, this study is the first to use a cNLP system for the identification of CD in EHRs written in Spanish.

MedTAG: a portable and customizable annotation tool for biomedical documents

BACKGROUND

Semantic annotators and Natural Language Processing (NLP) methods for Named Entity Recognition and Linking (NER+L) require plenty of training and test data, especially in the biomedical domain. Despite the abundance of unstructured biomedical data, the lack of richly annotated biomedical datasets poses hindrances to the further development of NER+L algorithms for any effective secondary use. In addition, manual annotation of biomedical documents performed by physicians and experts is a costly and time-consuming task. To support, organize and speed up the annotation process, we introduce MedTAG, a collaborative biomedical annotation tool that is open-source, platform-independent, and free to use/distribute.

RESULTS

We present the main features of MedTAG and how it has been employed in the histopathology domain by physicians and experts to annotate more than seven thousand clinical reports manually. We compare MedTAG with a set of well-established biomedical annotation tools, including BioQRator, ezTag, MyMiner, and tagtog, comparing their pros and cons with those of MedTag. We highlight that MedTAG is one of the very few open-source tools provided with an open license and a straightforward installation procedure supporting cross-platform use.

CONCLUSIONS

MedTAG has been designed according to five requirements (i.e. available, distributable, installable, workable and schematic) defined in a recent extensive review of manual annotation tools. Moreover, MedTAG satisfies 20 over 22 criteria specified in the same study.