1
|
Grout R, Gupta R, Bryant R, Elmahgoub MA, Li Y, Irfanullah K, Patel RF, Fawkes J, Inness C. Predicting disease onset from electronic health records for population health management: a scalable and explainable Deep Learning approach. Front Artif Intell 2024; 6:1287541. [PMID: 38259826 PMCID: PMC10800619 DOI: 10.3389/frai.2023.1287541] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Accepted: 12/06/2023] [Indexed: 01/24/2024] Open
Abstract
Introduction The move from a reactive model of care which treats conditions when they arise to a proactive model which intervenes early to prevent adverse healthcare events will benefit from advances in the predictive capabilities of Artificial Intelligence and Machine Learning. This paper investigates the ability of a Deep Learning (DL) approach to predict future disease diagnosis from Electronic Health Records (EHR) for the purposes of Population Health Management. Methods In this study, embeddings were created using a Word2Vec algorithm from structured vocabulary commonly used in EHRs e.g., Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) codes. This study is based on longitudinal medical data from ~50 m patients in the USA. We introduced a novel method of including binned observation values into an embeddings model. We also included novel features associated with wider determinants of health. Patient records comprising these embeddings were then fed to a Bidirectional Gated Recurrent Unit (GRU) model to predict the likelihood of patients developing Type 2 Diabetes Mellitus, Chronic Obstructive Pulmonary Disorder (COPD), Hypertension or experiencing an Acute Myocardial Infarction (MI) in the next 3 years. SHapley Additive exPlanations (SHAP) values were calculated to achieve model explainability. Results Increasing the data scope to include binned observations and wider determinants of health was found to improve predictive performance. We achieved an area under the Receiver Operating Characteristic curve value of 0.92 for Diabetes prediction, 0.94 for COPD, 0.92 for Hypertension and 0.94 for MI. The SHAP values showed that the models had learned features known to be associated with these outcomes. Discussion The DL approach outlined in this study can identify clinically-relevant features from large-scale EHR data and use these to predict future disease outcomes. This study highlights the promise of DL solutions for identifying patients at future risk of disease and providing clinicians with the means to understand and evaluate the drivers of those predictions.
Collapse
Affiliation(s)
| | | | | | | | - Yijie Li
- Accenture, London, United Kingdom
| | | | | | - Jake Fawkes
- Department of Statistics, University of Oxford, Oxford, United Kingdom
| | | |
Collapse
|
2
|
Shao Y, Cheng Y, Nelson SJ, Kokkinos P, Zamrini EY, Ahmed A, Zeng-Treitler Q. Hybrid Value-Aware Transformer Architecture for Joint Learning from Longitudinal and Non-Longitudinal Clinical Data. J Pers Med 2023; 13:1070. [PMID: 37511683 PMCID: PMC10381142 DOI: 10.3390/jpm13071070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 06/26/2023] [Accepted: 06/27/2023] [Indexed: 07/30/2023] Open
Abstract
Transformer is the latest deep neural network (DNN) architecture for sequence data learning, which has revolutionized the field of natural language processing. This success has motivated researchers to explore its application in the healthcare domain. Despite the similarities between longitudinal clinical data and natural language data, clinical data presents unique complexities that make adapting Transformer to this domain challenging. To address this issue, we have designed a new Transformer-based DNN architecture, referred to as Hybrid Value-Aware Transformer (HVAT), which can jointly learn from longitudinal and non-longitudinal clinical data. HVAT is unique in the ability to learn from the numerical values associated with clinical codes/concepts such as labs, and in the use of a flexible longitudinal data representation called clinical tokens. We have also trained a prototype HVAT model on a case-control dataset, achieving high performance in predicting Alzheimer's disease and related dementias as the patient outcome. The results demonstrate the potential of HVAT for broader clinical data-learning tasks.
Collapse
Affiliation(s)
- Yijun Shao
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
| | - Yan Cheng
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
| | - Stuart J Nelson
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
| | - Peter Kokkinos
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
- Department of Kinesiology and Health, School of Arts and Sciences, Rutgers University, New Brunswick, NJ 08901, USA
| | - Edward Y Zamrini
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
- Department of Neurology, School of Medicine, University of Utah, Salt Lake City, UT 84112, USA
- Irvine Clinical Research, Irvine, CA 92614, USA
| | - Ali Ahmed
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
- Department of Medicine, School of Medicine, Georgetown University, Washington, DC 20057, USA
| | - Qing Zeng-Treitler
- Department of Clinical Research and Leadership, School of Medicine and Health Sciences, George Washington University, Washington, DC 20037, USA
- Washington DC VA Medical Center, Washington, DC 20422, USA
| |
Collapse
|
3
|
Li Z, Li R, Zhou Y, Rasmy L, Zhi D, Zhu P, Dono A, Jiang X, Xu H, Esquenazi Y, Zheng WJ. Prediction of Brain Metastases Development in Patients With Lung Cancer by Explainable Artificial Intelligence From Electronic Health Records. JCO Clin Cancer Inform 2023; 7:e2200141. [PMID: 37018650 PMCID: PMC10281421 DOI: 10.1200/cci.22.00141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2022] [Revised: 12/08/2022] [Accepted: 01/26/2023] [Indexed: 04/07/2023] Open
Abstract
PURPOSE Early detection of brain metastases (BMs) is critical for prompt treatment and optimal control of the disease. In this study, we seek to predict the risk of developing BM among patients diagnosed with lung cancer on the basis of electronic health record (EHR) data and to understand what factors are important for the model to predict BM development through explainable artificial intelligence approaches accurately. MATERIALS AND METHODS We trained a recurrent neural network model, REverse Time AttentIoN (RETAIN), to predict the risk of developing BM using structured EHR data. To interpret the model's decision process, we analyzed the attention weights in the RETAIN model and the SHAP values from a feature attribution method, Kernel SHAP, to identify the factors contributing to BM prediction. RESULTS We developed a high-quality cohort with 4,466 patients with BM from the Cerner Health Fact database, which contains over 70 million patients from more than 600 hospitals. RETAIN uses this data set to achieve the best area under the receiver operating characteristic curve at 0.825, a significant improvement over the baseline model. We also extended a feature attribution method, Kernel SHAP, to structured EHR data for model interpretation. Both RETAIN and Kernel SHAP can identify important features related to BM prediction. CONCLUSION To the best of our knowledge, this is the first study to predict BM using structured EHR data. We achieved decent prediction performance for BM prediction and identified factors highly relevant to BM development. The sensitivity analysis demonstrated that both RETAIN and Kernel SHAP could discriminate unrelated features and put more weight on the features important to BM. Our study explored the potential of applying explainable artificial intelligence for future clinical applications.
Collapse
Affiliation(s)
- Zhao Li
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Rongbin Li
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Yujia Zhou
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Laila Rasmy
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Degui Zhi
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Ping Zhu
- Vivian L. Smith Department of Neurosurgery, the University of Texas Health Science Center at Houston, Houston, TX
- Department of Epidemiology, Human Genetics, and Environmental Sciences, the University of Texas Health Science Center at Houston, Houston, TX
| | - Antonio Dono
- Vivian L. Smith Department of Neurosurgery, the University of Texas Health Science Center at Houston, Houston, TX
| | - Xiaoqian Jiang
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Hua Xu
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| | - Yoshua Esquenazi
- Vivian L. Smith Department of Neurosurgery, the University of Texas Health Science Center at Houston, Houston, TX
| | - W. Jim Zheng
- School of Biomedical Informatics, the University of Texas Health Science Center at Houston, Houston, TX
| |
Collapse
|
4
|
Shao Y, Cheng Y, Nelson SJ, Kokkinos P, Zamrini EY, Ahmed A, Zeng-Treitler Q. Hybrid Value-Aware Transformer Architecture for Joint Learning from Longitudinal and Non-Longitudinal Clinical Data. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.03.09.23287046. [PMID: 36993767 PMCID: PMC10055462 DOI: 10.1101/2023.03.09.23287046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Transformer is the latest deep neural network (DNN) architecture for sequence data learning that has revolutionized the field of natural language processing. This success has motivated researchers to explore its application in the healthcare domain. Despite the similarities between longitudinal clinical data and natural language data, clinical data presents unique complexities that make adapting Transformer to this domain challenging. To address this issue, we have designed a new Transformer-based DNN architecture, referred to as Hybrid Value-Aware Transformer (HVAT), which can jointly learn from longitudinal and non-longitudinal clinical data. HVAT is unique in the ability to learn from the numerical values associated with clinical codes/concepts such as labs, and also the use of a flexible longitudinal data representation called clinical tokens. We trained a prototype HVAT model on a case-control dataset, achieving high performance in predicting Alzheimer’s disease and related dementias as the patient outcome. The result demonstrates the potential of HVAT for broader clinical data learning tasks.
Collapse
Affiliation(s)
- Yijun Shao
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
| | - Yan Cheng
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
| | | | - Peter Kokkinos
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
- Rutgers University, New Brunswick, NJ
| | - Edward Y. Zamrini
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
- University of Utah, Salt Lake City, Utah
- Irvine Clinical Research, Irvine, CA
| | - Ali Ahmed
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
- Georgetown University, Washington, DC
| | - Qing Zeng-Treitler
- George Washington University, Washington, DC
- Washington DC VA Medical Center, Washington, DC
| |
Collapse
|
5
|
Xie F, Yuan H, Ning Y, Ong MEH, Feng M, Hsu W, Chakraborty B, Liu N. Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies. J Biomed Inform 2021; 126:103980. [PMID: 34974189 DOI: 10.1016/j.jbi.2021.103980] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 11/07/2021] [Accepted: 12/20/2021] [Indexed: 12/21/2022]
Abstract
OBJECTIVE Temporal electronic health records (EHRs) contain a wealth of information for secondary uses, such as clinical events prediction and chronic disease management. However, challenges exist for temporal data representation. We therefore sought to identify these challenges and evaluate novel methodologies for addressing them through a systematic examination of deep learning solutions. METHODS We searched five databases (PubMed, Embase, the Institute of Electrical and Electronics Engineers [IEEE] Xplore Digital Library, the Association for Computing Machinery [ACM] Digital Library, and Web of Science) complemented with hand-searching in several prestigious computer science conference proceedings. We sought articles that reported deep learning methodologies on temporal data representation in structured EHR data from January 1, 2010, to August 30, 2020. We summarized and analyzed the selected articles from three perspectives: nature of time series, methodology, and model implementation. RESULTS We included 98 articles related to temporal data representation using deep learning. Four major challenges were identified, including data irregularity, heterogeneity, sparsity, and model opacity. We then studied how deep learning techniques were applied to address these challenges. Finally, we discuss some open challenges arising from deep learning. CONCLUSION Temporal EHR data present several major challenges for clinical prediction modeling and data utilization. To some extent, current deep learning solutions can address these challenges. Future studies may consider designing comprehensive and integrated solutions. Moreover, researchers should incorporate clinical domain knowledge into study designs and enhance model interpretability to facilitate clinical implementation.
Collapse
Affiliation(s)
- Feng Xie
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Han Yuan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Yilin Ning
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore
| | - Mengling Feng
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore
| | - Wynne Hsu
- School of Computing, National University of Singapore, Singapore; Institute of Data Science, National University of Singapore, Singapore
| | - Bibhas Chakraborty
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore; Department of Statistics and Data Science, National University of Singapore, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, United States
| | - Nan Liu
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore; Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore; Institute of Data Science, National University of Singapore, Singapore; SingHealth AI Health Program, Singapore Health Services, Singapore.
| |
Collapse
|
6
|
Finch A, Crowell A, Chang YC, Parameshwarappa P, Martinez J, Horberg M. A comparison of attentional neural network architectures for modeling with electronic medical records. JAMIA Open 2021; 4:ooab064. [PMID: 34396057 PMCID: PMC8358476 DOI: 10.1093/jamiaopen/ooab064] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2021] [Revised: 07/08/2021] [Accepted: 07/16/2021] [Indexed: 11/14/2022] Open
Abstract
Objective Attention networks learn an intelligent weighted averaging mechanism over a series of entities, providing increases to both performance and interpretability. In this article, we propose a novel time-aware transformer-based network and compare it to another leading model with similar characteristics. We also decompose model performance along several critical axes and examine which features contribute most to our model's performance. Materials and methods Using data sets representing patient records obtained between 2017 and 2019 by the Kaiser Permanente Mid-Atlantic States medical system, we construct four attentional models with varying levels of complexity on two targets (patient mortality and hospitalization). We examine how incorporating transfer learning and demographic features contribute to model success. We also test the performance of a model proposed in recent medical modeling literature. We compare these models with out-of-sample data using the area under the receiver-operator characteristic (AUROC) curve and average precision as measures of performance. We also analyze the attentional weights assigned by these models to patient diagnoses. Results We found that our model significantly outperformed the alternative on a mortality prediction task (91.96% AUROC against 73.82% AUROC). Our model also outperformed on the hospitalization task, although the models were significantly more competitive in that space (82.41% AUROC against 80.33% AUROC). Furthermore, we found that demographic features and transfer learning features which are frequently omitted from new models proposed in the EMR modeling space contributed significantly to the success of our model. Discussion We proposed an original construction of deep learning electronic medical record models which achieved very strong performance. We found that our unique model construction outperformed on several tasks in comparison to a leading literature alternative, even when input data was held constant between them. We obtained further improvements by incorporating several methods that are frequently overlooked in new model proposals, suggesting that it will be useful to explore these options further in the future.
Collapse
Affiliation(s)
- Anthony Finch
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Alexander Crowell
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Yung-Chieh Chang
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | | | - Jose Martinez
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Michael Horberg
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA.,Kaiser Permanente Mid-Atlantic Permanente Research Institute, Rockville, Maryland, USA
| |
Collapse
|
7
|
Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit Med 2021; 4:86. [PMID: 34017034 PMCID: PMC8137882 DOI: 10.1038/s41746-021-00455-y] [Citation(s) in RCA: 151] [Impact Index Per Article: 50.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2020] [Accepted: 04/14/2021] [Indexed: 01/22/2023] Open
Abstract
Deep learning (DL)-based predictive models from electronic health records (EHRs) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required by these models to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pretraining of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. Inspired by BERT, we propose Med-BERT, which adapts the BERT framework originally developed for the text domain to the structured EHR domain. Med-BERT is a contextualized embedding model pretrained on a structured EHR dataset of 28,490,650 patients. Fine-tuning experiments showed that Med-BERT substantially improves the prediction accuracy, boosting the area under the receiver operating characteristics curve (AUC) by 1.21-6.14% in two disease prediction tasks from two clinical databases. In particular, pretrained Med-BERT obtains promising performances on tasks with small fine-tuning training sets and can boost the AUC by more than 20% or obtain an AUC as high as a model trained on a training set ten times larger, compared with deep learning models without Med-BERT. We believe that Med-BERT will benefit disease prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.
Collapse
Affiliation(s)
- Laila Rasmy
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA
| | | | - Ziqian Xie
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Cui Tao
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Degui Zhi
- School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA.
| |
Collapse
|
8
|
Finch A, Crowell A, Bhatia M, Parameshwarappa P, Chang YC, Martinez J, Horberg M. Exploiting hierarchy in medical concept embedding. JAMIA Open 2021; 4:ooab022. [PMID: 33748691 PMCID: PMC7962787 DOI: 10.1093/jamiaopen/ooab022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2020] [Revised: 02/02/2021] [Accepted: 02/26/2021] [Indexed: 11/13/2022] Open
Abstract
Objective To construct and publicly release a set of medical concept embeddings for codes following the ICD-10 coding standard which explicitly incorporate hierarchical information from medical codes into the embedding formulation. Materials and Methods We trained concept embeddings using several new extensions to the Word2Vec algorithm using a dataset of approximately 600,000 patients from a major integrated healthcare organization in the Mid-Atlantic US. Our concept embeddings included additional entities to account for the medical categories assigned to codes by the Clinical Classification Software Revised (CCSR) dataset. We compare these results to sets of publicly released pretrained embeddings and alternative training methodologies. Results We found that Word2Vec models which included hierarchical data outperformed ordinary Word2Vec alternatives on tasks which compared naïve clusters to canonical ones provided by CCSR. Our Skip-Gram model with both codes and categories achieved 61.4% normalized mutual information with canonical labels in comparison to 57.5% with traditional Skip-Gram. In models operating on two different outcomes, we found that including hierarchical embedding data improved classification performance 96.2% of the time. When controlling for all other variables, we found that co-training embeddings improved classification performance 66.7% of the time. We found that all models outperformed our competitive benchmarks. Discussion We found significant evidence that our proposed algorithms can express the hierarchical structure of medical codes more fully than ordinary Word2Vec models, and that this improvement carries forward into classification tasks. As part of this publication, we have released several sets of pretrained medical concept embeddings using the ICD-10 standard which significantly outperform other well-known pretrained vectors on our tested outcomes.
Collapse
Affiliation(s)
- Anthony Finch
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Alexander Crowell
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Mamta Bhatia
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA.,Kaiser Permanente Mid-Atlantic Permanente Research Institute, Rockville, Maryland, USA
| | | | - Yung-Chieh Chang
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Jose Martinez
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA
| | - Michael Horberg
- Kaiser Permanente Mid-Atlantic Permanente Medical Group, Rockville, Maryland, USA.,Kaiser Permanente Mid-Atlantic Permanente Research Institute, Rockville, Maryland, USA
| |
Collapse
|
9
|
Rasmy L, Tiryaki F, Zhou Y, Xiang Y, Tao C, Xu H, Zhi D. Representation of EHR data for predictive modeling: a comparison between UMLS and other terminologies. J Am Med Inform Assoc 2020; 27:1593-1599. [PMID: 32930711 PMCID: PMC7647355 DOI: 10.1093/jamia/ocaa180] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2020] [Accepted: 07/24/2020] [Indexed: 01/23/2023] Open
Abstract
OBJECTIVE Predictive disease modeling using electronic health record data is a growing field. Although clinical data in their raw form can be used directly for predictive modeling, it is a common practice to map data to standard terminologies to facilitate data aggregation and reuse. There is, however, a lack of systematic investigation of how different representations could affect the performance of predictive models, especially in the context of machine learning and deep learning. MATERIALS AND METHODS We projected the input diagnoses data in the Cerner HealthFacts database to Unified Medical Language System (UMLS) and 5 other terminologies, including CCS, CCSR, ICD-9, ICD-10, and PheWAS, and evaluated the prediction performances of these terminologies on 2 different tasks: the risk prediction of heart failure in diabetes patients and the risk prediction of pancreatic cancer. Two popular models were evaluated: logistic regression and a recurrent neural network. RESULTS For logistic regression, using UMLS delivered the optimal area under the receiver operating characteristics (AUROC) results in both dengue hemorrhagic fever (81.15%) and pancreatic cancer (80.53%) tasks. For recurrent neural network, UMLS worked best for pancreatic cancer prediction (AUROC 82.24%), second only (AUROC 85.55%) to PheWAS (AUROC 85.87%) for dengue hemorrhagic fever prediction. DISCUSSION/CONCLUSION In our experiments, terminologies with larger vocabularies and finer-grained representations were associated with better prediction performances. In particular, UMLS is consistently 1 of the best-performing ones. We believe that our work may help to inform better designs of predictive models, although further investigation is warranted.
Collapse
Affiliation(s)
- Laila Rasmy
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Firat Tiryaki
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Yujia Zhou
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Yang Xiang
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Cui Tao
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Hua Xu
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| | - Degui Zhi
- School of Biomedical Informatics University of Texas Health Science Center, Houston, Texas, USA
| |
Collapse
|
10
|
Lal A, Pinevich Y, Gajic O, Herasevich V, Pickering B. Artificial intelligence and computer simulation models in critical illness. World J Crit Care Med 2020; 9:13-19. [PMID: 32577412 PMCID: PMC7298588 DOI: 10.5492/wjccm.v9.i2.13] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/31/2019] [Revised: 04/21/2020] [Accepted: 05/12/2020] [Indexed: 02/06/2023] Open
Abstract
Widespread implementation of electronic health records has led to the increased use of artificial intelligence (AI) and computer modeling in clinical medicine. The early recognition and treatment of critical illness are central to good outcomes but are made difficult by, among other things, the complexity of the environment and the often non-specific nature of the clinical presentation. Increasingly, AI applications are being proposed as decision supports for busy or distracted clinicians, to address this challenge. Data driven "associative" AI models are built from retrospective data registries with missing data and imprecise timing. Associative AI models lack transparency, often ignore causal mechanisms, and, while potentially useful in improved prognostication, have thus far had limited clinical applicability. To be clinically useful, AI tools need to provide bedside clinicians with actionable knowledge. Explicitly addressing causal mechanisms not only increases validity and replicability of the model, but also adds transparency and helps gain trust from the bedside clinicians for real world use of AI models in teaching and patient care.
Collapse
Affiliation(s)
- Amos Lal
- Department of Medicine, Division of Pulmonary and Critical Care Medicine, Rochester, Mayo Clinic, MN 55905, United States
- Multidisciplinary Epidemiology and Translational Research in Intensive Care Group, Mayo Clinic, Rochester, MN 55905, United States
| | - Yuliya Pinevich
- Multidisciplinary Epidemiology and Translational Research in Intensive Care Group, Mayo Clinic, Rochester, MN 55905, United States
- Department of Anesthesiology and Perioperative Medicine, Division of Critical Care, Mayo Clinic, Rochester, MN 55905, United States
| | - Ognjen Gajic
- Department of Medicine, Division of Pulmonary and Critical Care Medicine, Rochester, Mayo Clinic, MN 55905, United States
- Multidisciplinary Epidemiology and Translational Research in Intensive Care Group, Mayo Clinic, Rochester, MN 55905, United States
| | - Vitaly Herasevich
- Multidisciplinary Epidemiology and Translational Research in Intensive Care Group, Mayo Clinic, Rochester, MN 55905, United States
- Department of Anesthesiology and Perioperative Medicine, Division of Critical Care, Mayo Clinic, Rochester, MN 55905, United States
| | - Brian Pickering
- Multidisciplinary Epidemiology and Translational Research in Intensive Care Group, Mayo Clinic, Rochester, MN 55905, United States
- Department of Anesthesiology and Perioperative Medicine, Division of Critical Care, Mayo Clinic, Rochester, MN 55905, United States
| |
Collapse
|
11
|
Chen Q, Lee K, Yan S, Kim S, Wei CH, Lu Z. BioConceptVec: Creating and evaluating literature-based biomedical concept embeddings on a large scale. PLoS Comput Biol 2020; 16:e1007617. [PMID: 32324731 PMCID: PMC7237030 DOI: 10.1371/journal.pcbi.1007617] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2019] [Revised: 05/19/2020] [Accepted: 12/19/2019] [Indexed: 12/14/2022] Open
Abstract
A massive number of biological entities, such as genes and mutations, are mentioned in the biomedical literature. The capturing of the semantic relatedness of biological entities is vital to many biological applications, such as protein-protein interaction prediction and literature-based discovery. Concept embeddings—which involve the learning of vector representations of concepts using machine learning models—have been employed to capture the semantics of concepts. To develop concept embeddings, named-entity recognition (NER) tools are first used to identify and normalize concepts from the literature, and then different machine learning models are used to train the embeddings. Despite multiple attempts, existing biomedical concept embeddings generally suffer from suboptimal NER tools, small-scale evaluation, and limited availability. In response, we employed high-performance machine learning-based NER tools for concept recognition and trained our concept embeddings, BioConceptVec, via four different machine learning models on ~30 million PubMed abstracts. BioConceptVec covers over 400,000 biomedical concepts mentioned in the literature and is of the largest among the publicly available biomedical concept embeddings to date. To evaluate the validity and utility of BioConceptVec, we respectively performed two intrinsic evaluations (identifying related concepts based on drug-gene and gene-gene interactions) and two extrinsic evaluations (protein-protein interaction prediction and drug-drug interaction extraction), collectively using over 25 million instances from nine independent datasets (17 million instances from six intrinsic evaluation tasks and 8 million instances from three extrinsic evaluation tasks), which is, by far, the most comprehensive to our best knowledge. The intrinsic evaluation results demonstrate that BioConceptVec consistently has, by a large margin, better performance than existing concept embeddings in identifying similar and related concepts. More importantly, the extrinsic evaluation results demonstrate that using BioConceptVec with advanced deep learning models can significantly improve performance in downstream bioinformatics studies and biomedical text-mining applications. Our BioConceptVec embeddings and benchmarking datasets are publicly available at https://github.com/ncbi-nlp/BioConceptVec. Capturing the semantics of related biological concepts, such as genes and mutations, is of significant importance to many research tasks in computational biology such as protein-protein interaction detection, gene-drug association prediction, and biomedical literature-based discovery. Here, we propose to leverage state-of-the-art text mining tools and machine learning models to learn the semantics via vector representations (aka. embeddings) of over 400,000 biological concepts mentioned in the entire PubMed abstracts. Our learned embeddings, namely BioConceptVec, can capture related concepts based on their surrounding contextual information in the literature, which is beyond exact term match or co-occurrence-based methods. BioConceptVec has been thoroughly evaluated in multiple bioinformatics tasks consisting of over 25 million instances from nine different biological datasets. The evaluation results demonstrate that BioConceptVec has better performance than existing methods in all tasks. Finally, BioConceptVec is made freely available to the research community and general public.
Collapse
Affiliation(s)
- Qingyu Chen
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| | - Kyubum Lee
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| | - Shankai Yan
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| | - Sun Kim
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| | - Chih-Hsuan Wei
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| | - Zhiyong Lu
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, Maryland, United States of America
| |
Collapse
|