1
|
Gugatschka M, Egger NM, Haspl K, Hortobagyi D, Jauk S, Feiner M, Kramer D. Clinical evaluation of a machine learning-based dysphagia risk prediction tool. Eur Arch Otorhinolaryngol 2024:10.1007/s00405-024-08678-x. [PMID: 38743079 DOI: 10.1007/s00405-024-08678-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Accepted: 04/12/2024] [Indexed: 05/16/2024]
Abstract
PURPOSE The rise of digitization promotes the development of screening and decision support tools. We sought to validate the results from a machine learning based dysphagia risk prediction tool with clinical evaluation. METHODS 149 inpatients in the ENT department were evaluated in real time by the risk prediction tool, as well as clinically over a 3-week period. Patients were classified by both as patients at risk/no risk. RESULTS The AUROC, reflecting the discrimination capability of the algorithm, was 0.97. The accuracy achieved 92.6% given an excellent specificity as well as sensitivity of 98% and 82.4% resp. Higher age, as well as male sex and the diagnosis of oropharyngeal malignancies were found more often in patients at risk of dysphagia. CONCLUSION The proposed dysphagia risk prediction tool proved to have an outstanding performance in discriminating risk from no risk patients in a prospective clinical setting. It is likely to be particularly useful in settings where there is a lower incidence of patients with dysphagia and less awareness among staff.
Collapse
Affiliation(s)
- Markus Gugatschka
- Department of Phoniatrics, ENT University Hospital Graz, Medical University Graz, Graz, Austria.
| | - Nina Maria Egger
- Department of Phoniatrics, ENT University Hospital Graz, Medical University Graz, Graz, Austria
| | - K Haspl
- Department of Phoniatrics, ENT University Hospital Graz, Medical University Graz, Graz, Austria
| | - David Hortobagyi
- Department of Phoniatrics, ENT University Hospital Graz, Medical University Graz, Graz, Austria
| | - Stefanie Jauk
- Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria
- PH Predicting Health GmbH, Graz, Austria
| | - Marlies Feiner
- Department of Phoniatrics, ENT University Hospital Graz, Medical University Graz, Graz, Austria
| | - Diether Kramer
- Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria
- PH Predicting Health GmbH, Graz, Austria
| |
Collapse
|
2
|
Sheehan KA, Shin S, Hall E, Mak DYF, Lapointe-Shaw L, Tang T, Marwaha S, Gandell D, Rawal S, Inouye S, Verma AA, Razak F. Characterizing medical patients with delirium: A cohort study comparing ICD-10 codes and a validated chart review method. PLoS One 2024; 19:e0302888. [PMID: 38739670 PMCID: PMC11090329 DOI: 10.1371/journal.pone.0302888] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 04/15/2024] [Indexed: 05/16/2024] Open
Abstract
BACKGROUND Delirium is a major cause of preventable mortality and morbidity in hospitalized adults, but accurately determining rates of delirium remains a challenge. OBJECTIVE To characterize and compare medical inpatients identified as having delirium using two common methods, administrative data and retrospective chart review. METHODS We conducted a retrospective study of 3881 randomly selected internal medicine hospital admissions from six acute care hospitals in Toronto and Mississauga, Ontario, Canada. Delirium status was determined using ICD-10-CA codes from hospital administrative data and through a previously validated chart review method. Baseline sociodemographic and clinical characteristics, processes of care and outcomes were compared across those without delirium in hospital and those with delirium as determined by administrative data and chart review. RESULTS Delirium was identified in 6.3% of admissions by ICD-10-CA codes compared to 25.7% by chart review. Using chart review as the reference standard, ICD-10-CA codes for delirium had sensitivity 24.1% (95%CI: 21.5-26.8%), specificity 99.8% (95%CI: 99.5-99.9%), positive predictive value 97.6% (95%CI: 94.6-98.9%), and negative predictive value 79.2% (95%CI: 78.6-79.7%). Age over 80, male gender, and Charlson comorbidity index greater than 2 were associated with misclassification of delirium. Inpatient mortality and median costs of care were greater in patients determined to have delirium by ICD-10-CA codes (5.8% greater mortality, 95% CI: 2.0-9.5 and $6824 greater cost, 95%CI: 4713-9264) and by chart review (11.9% greater mortality, 95%CI: 9.5-14.2% and $4967 greater cost, 95%CI: 4415-5701), compared to patients without delirium. CONCLUSIONS Administrative data are specific but highly insensitive, missing most cases of delirium in hospital. Mortality and costs of care were greater for both the delirium cases that were detected and missed by administrative data. Better methods of routinely measuring delirium in hospital are needed.
Collapse
Affiliation(s)
- Kathleen A. Sheehan
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
- Centre for Mental Health, University Health Network, Toronto, ON, Canada
| | - Saeha Shin
- St. Michael’s Hospital, Unity Health Network, Toronto, ON, Canada
| | - Elise Hall
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
- Department of Psychiatry, Unity Health Network, Toronto, ON, Canada
| | - Denise Y. F. Mak
- St. Michael’s Hospital, Unity Health Network, Toronto, ON, Canada
| | - Lauren Lapointe-Shaw
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, University Health Network, Toronto, ON, Canada
| | - Terence Tang
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Institute for Better Health, Trillium Health Partners, Mississauga, ON, Canada
| | - Seema Marwaha
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, Unity Health Network, Toronto, ON, Canada
| | - Dov Gandell
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, Sunnybrook Heatlh Sciences Centre, Toronto, ON, Canada
| | - Shail Rawal
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, University Health Network, Toronto, ON, Canada
| | - Sharon Inouye
- Aging Brain Center, Hebrew Senior Life, Boston, MA, United States of America
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, United States of America
| | - Amol A. Verma
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, Unity Health Network, Toronto, ON, Canada
| | - Fahad Razak
- Department of Medicine, University of Toronto, Toronto ON, Canada
- Department of Medicine, Unity Health Network, Toronto, ON, Canada
| |
Collapse
|
3
|
Haque MA, Gedara MLB, Nickel N, Turgeon M, Lix LM. The validity of electronic health data for measuring smoking status: a systematic review and meta-analysis. BMC Med Inform Decis Mak 2024; 24:33. [PMID: 38308231 PMCID: PMC10836023 DOI: 10.1186/s12911-024-02416-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 01/03/2024] [Indexed: 02/04/2024] Open
Abstract
BACKGROUND Smoking is a risk factor for many chronic diseases. Multiple smoking status ascertainment algorithms have been developed for population-based electronic health databases such as administrative databases and electronic medical records (EMRs). Evidence syntheses of algorithm validation studies have often focused on chronic diseases rather than risk factors. We conducted a systematic review and meta-analysis of smoking status ascertainment algorithms to describe the characteristics and validity of these algorithms. METHODS The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were followed. We searched articles published from 1990 to 2022 in EMBASE, MEDLINE, Scopus, and Web of Science with key terms such as validity, administrative data, electronic health records, smoking, and tobacco use. The extracted information, including article characteristics, algorithm characteristics, and validity measures, was descriptively analyzed. Sources of heterogeneity in validity measures were estimated using a meta-regression model. Risk of bias (ROB) in the reviewed articles was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. RESULTS The initial search yielded 2086 articles; 57 were selected for review and 116 algorithms were identified. Almost three-quarters (71.6%) of algorithms were based on EMR data. The algorithms were primarily constructed using diagnosis codes for smoking-related conditions, although prescription medication codes for smoking treatments were also adopted. About half of the algorithms were developed using machine-learning models. The pooled estimates of positive predictive value, sensitivity, and specificity were 0.843, 0.672, and 0.918 respectively. Algorithm sensitivity and specificity were highly variable and ranged from 3 to 100% and 36 to 100%, respectively. Model-based algorithms had significantly greater sensitivity (p = 0.006) than rule-based algorithms. Algorithms for EMR data had higher sensitivity than algorithms for administrative data (p = 0.001). The ROB was low in most of the articles (76.3%) that underwent the assessment. CONCLUSIONS Multiple algorithms using different data sources and methods have been proposed to ascertain smoking status in electronic health data. Many algorithms had low sensitivity and positive predictive value, but the data source influenced their validity. Algorithms based on machine-learning models for multiple linked data sources have improved validity.
Collapse
Affiliation(s)
- Md Ashiqul Haque
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada
| | | | - Nathan Nickel
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada
| | - Maxime Turgeon
- Department of Statistics, University of Manitoba, Winnipeg, MB, Canada
| | - Lisa M Lix
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada.
| |
Collapse
|
4
|
Yang T, Yang H, Liu Y, Liu X, Ding YJ, Li R, Mao AQ, Huang Y, Li XL, Zhang Y, Yu FX. Postoperative delirium prediction after cardiac surgery using machine learning models. Comput Biol Med 2024; 169:107818. [PMID: 38134752 DOI: 10.1016/j.compbiomed.2023.107818] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2023] [Revised: 11/03/2023] [Accepted: 12/03/2023] [Indexed: 12/24/2023]
Abstract
OBJECTIVE Postoperative delirium (POD) is a common postoperative complication in elderly patients, especially those undergoing cardiac surgery, which seriously affects the short- and long-term prognosis of patients. Early identification of risk factors for the development of POD can help improve the perioperative management of surgical patients. In the present study, five machine learning models were developed to predict patients at high risk of delirium after cardiac surgery and their performance was compared. METHODS A total of 367 patients who underwent cardiac surgery were retrospectively included in this study. Using single-factor analysis, 21 risk factors for POD were selected for inclusion in machine learning. The dataset was divided using 10-fold cross-validation for model training and testing. Five machine learning models (random forest (RF), support vector machine (SVM), radial based kernel neural network (RBFNN), K-nearest neighbour (KNN), and Kernel ridge regression (KRR)) were compared using area under the receiver operating characteristic curve (AUC-ROC), accuracy (ACC), sensitivity (SN), specificity (SPE), and Matthews coefficient (MCC). RESULTS Among 367 patients, 105 patients developed POD, the incidence of delirium was 28.6 %. Among the five ML models, RF had the best performance in ACC (87.99 %), SN (69.27 %), SPE (95.38 %), MCC (70.00 %) and AUC (0.9202), which was far superior to the other four models. CONCLUSION Delirium is common in patients after cardiac surgery. This analysis confirms the importance of the computational ML models in predicting the occurrence of delirium after cardiac surgery, especially the outstanding performance of the RF model, which has practical clinical applications for early identification of patients at risk of developing POD.
Collapse
Affiliation(s)
- Tan Yang
- Department of Cardiovascular Surgery, The Affiliated Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - Hai Yang
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - Yan Liu
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - Xiao Liu
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - Yi-Jie Ding
- Yangtze Delta Region Institute (Quzhou), University of Electronic Science and Technology of China, 324000 Quzhou, Zhejiang, China
| | - Run Li
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - An-Qiong Mao
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China
| | - Yue Huang
- Department of Anesthesiology, Zigong First People's Hospital, Zi Gong, 644099, Sichuan, China
| | - Xiao-Liang Li
- Department of Cardiothoracic Surgery, First Peoples Hospital of Neijiang, Nei Jiang, 641000, Sichuan, China
| | - Ying Zhang
- Department of Anesthesiology, The Affiliated Traditional Chinese Medicine Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China.
| | - Feng-Xu Yu
- Department of Cardiovascular Surgery, The Affiliated Hospital of Southwest Medical University, Luzhou, 646000, Sichuan, China.
| |
Collapse
|
5
|
Lee SH, Hur HJ, Kim SN, Ahn JH, Ro DH, Hong A, Park HY, Choe PG, Kim B, Park HY. Predicting delirium and the effects of medications in hospitalized COVID-19 patients using machine learning: A retrospective study within the Korean Multidisciplinary Cohort for Delirium Prevention (KoMCoDe). Digit Health 2024; 10:20552076231223811. [PMID: 38188862 PMCID: PMC10771056 DOI: 10.1177/20552076231223811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 12/14/2023] [Indexed: 01/09/2024] Open
Abstract
Objective Delirium is commonly reported from the inpatients with Coronavirus disease 2019 (COVID-19) infection. As delirium is closely associated with adverse clinical outcomes, prediction and prevention of delirium is critical. We developed a machine learning (ML) model to predict delirium in hospitalized patients with COVID-19 and to identify modifiable factors to prevent delirium. Methods The data set (n = 878) from four medical centers was constructed. Total of 78 predictors were included such as demographic characteristics, vital signs, laboratory results and medication, and the primary outcome was delirium occurrence during hospitalization. For analysis, the extreme gradient boosting (XGBoost) algorithm was applied, and the most influential factors were selected by recursive feature elimination. Among the indicators of performance for ML model, the area under the curve of the receiver operating characteristic (AUROC) curve was selected as the evaluation metric. Results Regarding the performance of developed delirium prediction model, the accuracy, precision, recall, F1 score, and the AUROC were calculated (0.944, 0.581, 0.421, 0.485, 0.873, respectively). The influential factors of delirium in this model included were mechanical ventilation, medication (antipsychotics, sedatives, ambroxol, piperacillin/tazobactam, acetaminophen, ceftriaxone, and propacetamol), and sodium ion concentration (all p < 0.05). Conclusions We developed and internally validated an ML model to predict delirium in COVID-19 inpatients. The model identified modifiable factors associated with the development of delirium and could be clinically useful for the prediction and prevention of delirium in COVID-19 inpatients.
Collapse
Affiliation(s)
- So Hee Lee
- Department of Psychiatry, National Medical Center, Seoul,
South Korea
| | - Hyun Jung Hur
- Department of Psychiatry, Seoul National University Bundang Hospital, Seongnam, South Korea
| | - Sung Nyun Kim
- Department of Psychiatry, Seoul Medical Center, Seoul, South Korea
| | - Jang Ho Ahn
- Seoul National University College of Medicine, Seoul, South Korea
| | - Du Hyun Ro
- Department of Orthopedic Surgery, Seoul National University Hospital, Seoul, South Korea
| | - Arum Hong
- Department of Psychiatry, Seoul National University Bundang Hospital, Seongnam, South Korea
| | - Hye Yoon Park
- Department of Psychiatry, Seoul National University Hospital, Seoul National University College of Medicine, Seongnam,
South Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul, South Korea
| | - Pyoeng Gyun Choe
- Department of Internal Medicine, Seoul National University Hospital, Seoul National University College of Medicine, Seongnam, South Korea
| | - Back Kim
- Seoul National University College of Medicine, Seoul, South Korea
| | - Hye Youn Park
- Department of Psychiatry, Seoul National University Bundang Hospital, Seongnam, South Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul, South Korea
| |
Collapse
|
6
|
Bergquist T, Schaffter T, Yan Y, Yu T, Prosser J, Gao J, Chen G, Charzewski Ł, Nawalany Z, Brugere I, Retkute R, Prusokas A, Prusokas A, Choi Y, Lee S, Choe J, Lee I, Kim S, Kang J, Mooney SD, Guinney J. Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine. J Am Med Inform Assoc 2023; 31:35-44. [PMID: 37604111 PMCID: PMC10746301 DOI: 10.1093/jamia/ocad159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Revised: 07/05/2023] [Accepted: 08/08/2023] [Indexed: 08/23/2023] Open
Abstract
OBJECTIVE Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. MATERIALS AND METHODS Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. RESULTS The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. DISCUSSION Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. CONCLUSION This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
Collapse
Affiliation(s)
- Timothy Bergquist
- Sage Bionetworks, Seattle, WA, United States
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States
| | | | - Yao Yan
- Sage Bionetworks, Seattle, WA, United States
- Molecular Engineering and Sciences Institute, University of Washington, Seattle, WA, United States
| | - Thomas Yu
- Sage Bionetworks, Seattle, WA, United States
| | - Justin Prosser
- Institute of Translational Health Sciences, University of Washington, Seattle, WA, United States
| | - Jifan Gao
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, United States
| | - Guanhua Chen
- Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, United States
| | - Łukasz Charzewski
- Proacta, Warsaw, Poland
- Division of Biophysics, University of Warsaw, Warsaw, Poland
| | | | - Ivan Brugere
- Department of Computer Science, University of Illinois at Chicago, Chicago, IL, United States
| | - Renata Retkute
- Department of Plant Sciences, University of Cambridge, Cambridge, United Kingdom
| | - Alidivinas Prusokas
- Plant and Molecular Sciences, School of Natural and Environmental Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom
| | - Augustinas Prusokas
- Department of Life Sciences, Imperial College London, London, United Kingdom
| | - Yonghwa Choi
- Department of Computer Science and Engineering, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Sanghoon Lee
- Department of Computer Science and Engineering, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Junseok Choe
- Department of Computer Science and Engineering, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Inggeol Lee
- Department of Interdisciplinary Program in Bioinformatics, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Sunkyu Kim
- Department of Computer Science and Engineering, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Jaewoo Kang
- Department of Computer Science and Engineering, College of Informatics, Korea University, Seoul, Republic of Korea
- Department of Interdisciplinary Program in Bioinformatics, College of Informatics, Korea University, Seoul, Republic of Korea
| | - Sean D Mooney
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States
| | - Justin Guinney
- Sage Bionetworks, Seattle, WA, United States
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States
| |
Collapse
|
7
|
Susanto AP, Lyell D, Widyantoro B, Berkovsky S, Magrabi F. Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review. J Am Med Inform Assoc 2023; 30:2050-2063. [PMID: 37647865 PMCID: PMC10654852 DOI: 10.1093/jamia/ocad180] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 08/01/2023] [Accepted: 08/23/2023] [Indexed: 09/01/2023] Open
Abstract
OBJECTIVE This study aims to summarize the research literature evaluating machine learning (ML)-based clinical decision support (CDS) systems in healthcare settings. MATERIALS AND METHODS We conducted a review in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Review). Four databases, including PubMed, Medline, Embase, and Scopus were searched for studies published from January 2016 to April 2021 evaluating the use of ML-based CDS in clinical settings. We extracted the study design, care setting, clinical task, CDS task, and ML method. The level of CDS autonomy was examined using a previously published 3-level classification based on the division of clinical tasks between the clinician and CDS; effects on decision-making, care delivery, and patient outcomes were summarized. RESULTS Thirty-two studies evaluating the use of ML-based CDS in clinical settings were identified. All were undertaken in developed countries and largely in secondary and tertiary care settings. The most common clinical tasks supported by ML-based CDS were image recognition and interpretation (n = 12) and risk assessment (n = 9). The majority of studies examined assistive CDS (n = 23) which required clinicians to confirm or approve CDS recommendations for risk assessment in sepsis and for interpreting cancerous lesions in colonoscopy. Effects on decision-making, care delivery, and patient outcomes were mixed. CONCLUSION ML-based CDS are being evaluated in many clinical areas. There remain many opportunities to apply and evaluate effects of ML-based CDS on decision-making, care delivery, and patient outcomes, particularly in resource-constrained settings.
Collapse
Affiliation(s)
- Anindya Pradipta Susanto
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
- Faculty of Medicine, Universitas Indonesia, Jakarta, DKI Jakarta 10430, Indonesia
| | - David Lyell
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| | - Bambang Widyantoro
- Faculty of Medicine, Universitas Indonesia, Jakarta, DKI Jakarta 10430, Indonesia
- National Cardiovascular Center Harapan Kita Hospital, Jakarta, DKI Jakarta 11420, Indonesia
| | - Shlomo Berkovsky
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| | - Farah Magrabi
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW 2109, Australia
| |
Collapse
|
8
|
Ser SE, Shear K, Snigurska UA, Prosperi M, Wu Y, Magoc T, Bjarnadottir RI, Lucero RJ. Clinical Prediction Models for Hospital-Induced Delirium Using Structured and Unstructured Electronic Health Record Data: Protocol for a Development and Validation Study. JMIR Res Protoc 2023; 12:e48521. [PMID: 37943599 PMCID: PMC10667972 DOI: 10.2196/48521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Revised: 09/01/2023] [Accepted: 09/05/2023] [Indexed: 11/10/2023] Open
Abstract
BACKGROUND Hospital-induced delirium is one of the most common and costly iatrogenic conditions, and its incidence is predicted to increase as the population of the United States ages. An academic and clinical interdisciplinary systems approach is needed to reduce the frequency and impact of hospital-induced delirium. OBJECTIVE The long-term goal of our research is to enhance the safety of hospitalized older adults by reducing iatrogenic conditions through an effective learning health system. In this study, we will develop models for predicting hospital-induced delirium. In order to accomplish this objective, we will create a computable phenotype for our outcome (hospital-induced delirium), design an expert-based traditional logistic regression model, leverage machine learning techniques to generate a model using structured data, and use machine learning and natural language processing to produce an integrated model with components from both structured data and text data. METHODS This study will explore text-based data, such as nursing notes, to improve the predictive capability of prognostic models for hospital-induced delirium. By using supervised and unsupervised text mining in addition to structured data, we will examine multiple types of information in electronic health record data to predict medical-surgical patient risk of developing delirium. Development and validation will be compliant to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. RESULTS Work on this project will take place through March 2024. For this study, we will use data from approximately 332,230 encounters that occurred between January 2012 to May 2021. Findings from this project will be disseminated at scientific conferences and in peer-reviewed journals. CONCLUSIONS Success in this study will yield a durable, high-performing research-data infrastructure that will process, extract, and analyze clinical text data in near real time. This model has the potential to be integrated into the electronic health record and provide point-of-care decision support to prevent harm and improve quality of care. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/48521.
Collapse
Affiliation(s)
- Sarah E Ser
- Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FL, United States
| | - Kristen Shear
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States
| | - Urszula A Snigurska
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States
| | - Mattia Prosperi
- Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FL, United States
| | - Yonghui Wu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, United States
| | - Tanja Magoc
- Integrated Data Repository Research Services, University of Florida, Gainesville, FL, United States
| | - Ragnhildur I Bjarnadottir
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States
| | - Robert J Lucero
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States
- School of Nursing, University of California Los Angeles, Los Angeles, CA, United States
| |
Collapse
|
9
|
Snigurska UA, Liu Y, Ser SE, Macieira TGR, Ansell M, Lindberg D, Prosperi M, Bjarnadottir RI, Lucero RJ. Risk of bias in prognostic models of hospital-induced delirium for medical-surgical units: A systematic review. PLoS One 2023; 18:e0285527. [PMID: 37590196 PMCID: PMC10434879 DOI: 10.1371/journal.pone.0285527] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Accepted: 04/25/2023] [Indexed: 08/19/2023] Open
Abstract
PURPOSE The purpose of this systematic review was to assess risk of bias in existing prognostic models of hospital-induced delirium for medical-surgical units. METHODS APA PsycInfo, CINAHL, MEDLINE, and Web of Science Core Collection were searched on July 8, 2022, to identify original studies which developed and validated prognostic models of hospital-induced delirium for adult patients who were hospitalized in medical-surgical units. The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies was used for data extraction. The Prediction Model Risk of Bias Assessment Tool was used to assess risk of bias. Risk of bias was assessed across four domains: participants, predictors, outcome, and analysis. RESULTS Thirteen studies were included in the qualitative synthesis, including ten model development and validation studies and three model validation only studies. The methods in all of the studies were rated to be at high overall risk of bias. The methods of statistical analysis were the greatest source of bias. External validity of models in the included studies was tested at low levels of transportability. CONCLUSIONS Our findings highlight the ongoing scientific challenge of developing a valid prognostic model of hospital-induced delirium for medical-surgical units to tailor preventive interventions to patients who are at high risk of this iatrogenic condition. With limited knowledge about generalizable prognosis of hospital-induced delirium in medical-surgical units, existing prognostic models should be used with caution when creating clinical practice policies. Future research protocols must include robust study designs which take into account the perspectives of clinicians to identify and validate risk factors of hospital-induced delirium for accurate and generalizable prognosis in medical-surgical units.
Collapse
Affiliation(s)
- Urszula A. Snigurska
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States of America
| | - Yiyang Liu
- Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FL, United States of America
| | - Sarah E. Ser
- Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FL, United States of America
| | - Tamara G. R. Macieira
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States of America
| | - Margaret Ansell
- Health Science Center Libraries, George A. Smathers Libraries, University of Florida, Gainesville, FL, United States of America
| | - David Lindberg
- Department of Statistics, College of Liberal Arts and Sciences, University of Florida, Gainesville, FL, United States of America
| | - Mattia Prosperi
- Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, FL, United States of America
| | - Ragnhildur I. Bjarnadottir
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States of America
| | - Robert J. Lucero
- Department of Family, Community, and Health Systems Science, College of Nursing, University of Florida, Gainesville, FL, United States of America
- School of Nursing, University of California Los Angeles, Los Angeles, CA, United States of America
| |
Collapse
|
10
|
Keller MS, Qureshi N, Albertson E, Pevnick J, Brandt N, Bui A, Sarkisian CA. Comparing risk prediction models aimed at predicting hospitalizations for adverse drug events in community dwelling older adults: a protocol paper. Res Sq 2023:rs.3.rs-2429369. [PMID: 36711695 PMCID: PMC9882666 DOI: 10.21203/rs.3.rs-2429369/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
Background The objective of this paper is to describe the creation, validation, and comparison of two risk prediction modeling approaches for community-dwelling older adults to identify individuals at highest risk for adverse drug event-related hospitalizations. One approach will use traditional statistical methods, the second will use a machine learning approach. Methods We will construct medication, clinical, health care utilization, and other variables known to be associated with adverse drug event-related hospitalizations. To create the cohort, we will include older adults (≥ 65 years of age) empaneled to a primary care physician within the Cedars-Sinai Health System primary care clinics with polypharmacy (≥ 5 medications) or at least 1 medication commonly implicated in ADEs (certain oral hypoglycemics, anti-coagulants, anti-platelets, and insulins). We will use a Fine-Gray Cox proportional hazards model for one risk modeling approach and DataRobot, a data science and analytics platform, to run and compare several widely used supervised machine learning algorithms, including Random Forest, Support Vector Machine, Extreme Gradient Boosting (XGBoost), Decision Tree, Naïve Bayes, and K-Nearest Neighbors. We will use a variety of metrics to compare model performance and to assess the risk of algorithmic bias. Discussion In conclusion, we hope to develop a pragmatic model that can be implemented in the primary care setting to risk stratify older adults to further optimize medication management.
Collapse
Affiliation(s)
| | | | | | | | | | - Alex Bui
- David Geffen School of Medicine: University of California Los Angeles David Geffen School of Medicine
| | - Catherine A Sarkisian
- David Geffen School of Medicine: University of California Los Angeles David Geffen School of Medicine
| |
Collapse
|
11
|
Jauk S, Kramer D, Veeranki SPK, Siml-Fraissler A, Lenz-Waldbauer A, Tax E, Leodolter W, Gugatschka M. Evaluation of a Machine Learning-Based Dysphagia Prediction Tool in Clinical Routine: A Prospective Observational Cohort Study. Dysphagia 2023;:1-9. [PMID: 36625964 DOI: 10.1007/s00455-022-10548-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Accepted: 12/28/2022] [Indexed: 01/11/2023]
Abstract
Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients' risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.
Collapse
|
12
|
Abraham J, Bartek B, Meng A, Ryan King C, Xue B, Lu C, Avidan MS. Integrating machine learning predictions for perioperative risk management: Towards an empirical design of a flexible-standardized risk assessment tool. J Biomed Inform 2023; 137:104270. [PMID: 36516944 DOI: 10.1016/j.jbi.2022.104270] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 12/02/2022] [Accepted: 12/07/2022] [Indexed: 12/14/2022]
Abstract
BACKGROUND Surgical patients are complex, vulnerable, and prone to postoperative complications that can potentially be mitigated with quality perioperative risk assessment and management. Several institutions have incorporated machine learning (ML) into their patient care to improve awareness and support clinician decision-making along the perioperative spectrum. Recent research suggests that ML risk prediction can support perioperative patient risk monitoring and management across several situations, including the operating room (OR) to intensive care unit (ICU) handoffs. OBJECTIVES Our study objectives were threefold: (1) evaluate whether ML-generated postoperative predictions are concordant with clinician-generated risk rankings for acute kidney injury, delirium, pneumonia, deep vein thrombosis, and pulmonary embolism, and establish their associated risk factors; (2) ascertain clinician end-user suggestions to improve adoption of ML-generated risks and their integration into the perioperative workflow; and (3) develop a user-friendly visualization format for a tool to display ML-generated risks and risk factors to support postoperative care planning, for example, within the context of OR-ICU handoffs. METHODS Graphical user interfaces for postoperative risk prediction models were assessed for end-user usability through cognitive walkthroughs and interviews with anesthesiologists, surgeons, certified registered nurse anesthetists, registered nurses, and critical care physicians. Thematic analysis relying on an explanation design framework was used to identify feedback and suggestions for improvement. RESULTS 17 clinicians participated in the evaluation. ML estimates of complication risks aligned with clinicians' independent rankings, and related displays were perceived as valuable for decision-making and care planning for postoperative care. During OR-ICU handoffs, the tool could speed up report preparation and remind clinicians to address patient-specific complications, thus providing more tailored care information. Suggestions for improvement centered on electronic tool delivery; methods to build trust in ML models; modifiable risks and risk mitigation strategies; and additional patient information based on individual preferences (e.g., surgical procedure). CONCLUSIONS ML estimates of postoperative complication risks can provide anticipatory guidance, potentially increasing the efficiency of care planning. We have offered an ML visualization framework for designing future ML-augmented tools and anticipate the development of tools that recommend specific actions to the user based on ML model output.
Collapse
Affiliation(s)
- Joanna Abraham
- Institute for Informatics, School of Medicine, Washington University in St Louis, MO, United States; Department of Anesthesiology, School of Medicine, Washington University in St Louis, MO, United States.
| | - Brian Bartek
- Institute for Informatics, School of Medicine, Washington University in St Louis, MO, United States
| | - Alicia Meng
- Department of Anesthesiology, School of Medicine, Washington University in St Louis, MO, United States
| | - Christopher Ryan King
- Department of Anesthesiology, School of Medicine, Washington University in St Louis, MO, United States
| | - Bing Xue
- Department of Electrical & Systems Engineering, McKelvey School of Engineering, Washington University in St Louis, MO, United States
| | - Chenyang Lu
- Department of Computer Science & Engineering, McKelvey School of Engineering, Washington University in St Louis, MO, United States
| | - Michael S Avidan
- Department of Anesthesiology, School of Medicine, Washington University in St Louis, MO, United States
| |
Collapse
|
13
|
Wang L, Zhang Y, Chignell M, Shan B, Sheehan KA, Razak F, Verma A. Boosting Delirium Identification Accuracy With Sentiment-Based Natural Language Processing: Mixed Methods Study. JMIR Med Inform 2022; 10:e38161. [PMID: 36538363 PMCID: PMC9812273 DOI: 10.2196/38161] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Revised: 08/22/2022] [Accepted: 09/19/2022] [Indexed: 01/07/2023] Open
Abstract
BACKGROUND Delirium is an acute neurocognitive disorder that affects up to half of older hospitalized medical patients and can lead to dementia, longer hospital stays, increased health costs, and death. Although delirium can be prevented and treated, it is difficult to identify and predict. OBJECTIVE This study aimed to improve machine learning models that retrospectively identify the presence of delirium during hospital stays (eg, to measure the effectiveness of delirium prevention interventions) by using the natural language processing (NLP) technique of sentiment analysis (in this case a feature that identifies sentiment toward, or away from, a delirium diagnosis). METHODS Using data from the General Medicine Inpatient Initiative, a Canadian hospital data and analytics network, a detailed manual review of medical records was conducted from nearly 4000 admissions at 6 Toronto area hospitals. Furthermore, 25.74% (994/3862) of the eligible hospital admissions were labeled as having delirium. Using the data set collected from this study, we developed machine learning models with, and without, the benefit of NLP methods applied to diagnostic imaging reports, and we asked the question "can NLP improve machine learning identification of delirium?" RESULTS Among the eligible 3862 hospital admissions, 994 (25.74%) admissions were labeled as having delirium. Identification and calibration of the models were satisfactory. The accuracy and area under the receiver operating characteristic curve of the main model with NLP in the independent testing data set were 0.807 and 0.930, respectively. The accuracy and area under the receiver operating characteristic curve of the main model without NLP in the independent testing data set were 0.811 and 0.869, respectively. Model performance was also found to be stable over the 5-year period used in the experiment, with identification for a likely future holdout test set being no worse than identification for retrospective holdout test sets. CONCLUSIONS Our machine learning model that included NLP (ie, sentiment analysis in medical image description text mining) produced valid identification of delirium with the sentiment analysis, providing significant additional benefit over the model without NLP.
Collapse
Affiliation(s)
- Lu Wang
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
- Department of Computer Science, Texas State University, San Marcos, TX, United States
| | - Yilun Zhang
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Mark Chignell
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Baizun Shan
- Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, Canada
| | - Kathleen A Sheehan
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Fahad Razak
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Faculty of Medicine & Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| | - Amol Verma
- GEMINI - The General Medicine Inpatient Initiative, Unity Health Toronto, Toronto, ON, Canada
- Faculty of Medicine & Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
14
|
Spiller TR, Tufan E, Petry H, Böttger S, Fuchs S, Duek O, Ben-Zion Z, Korem N, Harpaz-Rotem I, von Känel R, Ernst J. Delirium screening in an acute care setting with a machine learning classifier based on routinely collected nursing data: A model development study. J Psychiatr Res 2022; 156:194-199. [PMID: 36252349 DOI: 10.1016/j.jpsychires.2022.10.018] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 09/20/2022] [Accepted: 10/03/2022] [Indexed: 12/12/2022]
Abstract
Delirium screening in acute care settings is a resource intensive process with frequent deviations from screening protocols. A predictive model relying only on daily collected nursing data for delirium screening could expand the populations covered by such screening programs. Here, we present the results of the development and validation of a series of machine-learning based delirium prediction models. For this purpose, we used data of all patients 18 years or older which were hospitalized for more than a day between January 1, 2014, and December 31, 2018, at a single tertiary teaching hospital in Zurich, Switzerland. A total of 48,840 patients met inclusion criteria. 18,873 (38.6%) were excluded due to missing data. Mean age (SD) of the included 29,967 patients was 71.1 (12.2) years and 12,231 (40.8%) were women. Delirium was assessed with the Delirium Observation Scale (DOS) with a total score of 3 or greater indicating that a patient is at risk for delirium. Additional measures included structured data collected for nursing process planning and demographic characteristics. The performance of the machine learning models was assessed using the area under the receiver operating characteristic curve (AUC). The training set consisted of 21,147 patients (mean age 71.1 (12.1) years; 8,630 (40.8%) women|) including 233,024 observations with 16,167 (6.9%) positive DOS screens. The test set comprised 8,820 patients (median age 71.1 (12.4) years; 3,601 (40.8%) women) with 91,026 observations with 5,445 (6.0%) positive DOS screens. Overall, the gradient boosting machine model performed best with an AUC of 0.933 (95% CI, 0.929 - 0.936). In conclusion, machine learning models based only on structured nursing data can reliably predict patients at risk for delirium in an acute care setting. Prediction models, using existing data collection processes, could reduce the resources required for delirium screening procedures in clinical practice.
Collapse
Affiliation(s)
- Tobias R Spiller
- Department of Consultation-Liaison Psychiatry and Psychosomatic Medicine, University Hospital Zurich (USZ), Zurich, Switzerland; University of Zurich (UZH), Zurich, Switzerland; Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA; National Center for PTSD, Clinical Neurosciences Division, VA Connecticut Healthcare System, West Haven, CT, USA.
| | - Ege Tufan
- German Institute for Literature, Leipzig University, Leipzig, Germany
| | - Heidi Petry
- University of Zurich (UZH), Zurich, Switzerland; Center of Clinical Nursing Science, University Hospital Zurich (USZ), Zurich, Switzerland
| | - Sönke Böttger
- University of Zurich (UZH), Zurich, Switzerland; Department of Gastroenterology, University Hospital Zurich (USZ), Zurich, Switzerland
| | - Simon Fuchs
- Department of Consultation-Liaison Psychiatry and Psychosomatic Medicine, University Hospital Zurich (USZ), Zurich, Switzerland; University of Zurich (UZH), Zurich, Switzerland; Psychiatric University Hospital Zurich (PUK), Zurich, Switzerland
| | - Or Duek
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA; National Center for PTSD, Clinical Neurosciences Division, VA Connecticut Healthcare System, West Haven, CT, USA
| | - Ziv Ben-Zion
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA; National Center for PTSD, Clinical Neurosciences Division, VA Connecticut Healthcare System, West Haven, CT, USA
| | - Nachshon Korem
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA; National Center for PTSD, Clinical Neurosciences Division, VA Connecticut Healthcare System, West Haven, CT, USA
| | - Ilan Harpaz-Rotem
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, USA; National Center for PTSD, Clinical Neurosciences Division, VA Connecticut Healthcare System, West Haven, CT, USA; Northeast Program Evaluation Center, VA Connecticut Healthcare System, West Haven, USA; Department of Psychology, Yale University, New Haven, CT, USA
| | - Roland von Känel
- Department of Consultation-Liaison Psychiatry and Psychosomatic Medicine, University Hospital Zurich (USZ), Zurich, Switzerland; University of Zurich (UZH), Zurich, Switzerland
| | - Jutta Ernst
- University of Zurich (UZH), Zurich, Switzerland; Center of Clinical Nursing Science, University Hospital Zurich (USZ), Zurich, Switzerland
| |
Collapse
|
15
|
Liu S, Schlesinger JJ, McCoy AB, Reese TJ, Steitz B, Russo E, Koh B, Wright A. New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record. J Am Med Inform Assoc 2022; 30:120-131. [PMID: 36303456 PMCID: PMC9748586 DOI: 10.1093/jamia/ocac210] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 10/09/2022] [Accepted: 10/17/2022] [Indexed: 12/15/2022] Open
Abstract
OBJECTIVE To develop and test an accurate deep learning model for predicting new onset delirium in hospitalized adult patients. METHODS Using electronic health record (EHR) data extracted from a large academic medical center, we developed a model combining long short-term memory (LSTM) and machine learning to predict new onset delirium and compared its performance with machine-learning-only models (logistic regression, random forest, support vector machine, neural network, and LightGBM). The labels of models were confusion assessment method (CAM) assessments. We evaluated models on a hold-out dataset. We calculated Shapley additive explanations (SHAP) measures to gauge the feature impact on the model. RESULTS A total of 331 489 CAM assessments with 896 features from 34 035 patients were included. The LightGBM model achieved the best performance (AUC 0.927 [0.924, 0.929] and F1 0.626 [0.618, 0.634]) among the machine learning models. When combined with the LSTM model, the final model's performance improved significantly (P = .001) with AUC 0.952 [0.950, 0.955] and F1 0.759 [0.755, 0.765]. The precision value of the combined model improved from 0.497 to 0.751 with a fixed recall of 0.8. Using the mean absolute SHAP values, we identified the top 20 features, including age, heart rate, Richmond Agitation-Sedation Scale score, Morse fall risk score, pulse, respiratory rate, and level of care. CONCLUSION Leveraging LSTM to capture temporal trends and combining it with the LightGBM model can significantly improve the prediction of new onset delirium, providing an algorithmic basis for the subsequent development of clinical decision support tools for proactive delirium interventions.
Collapse
Affiliation(s)
- Siru Liu
- Corresponding Author: Siru Liu, PhD, Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave #1475, Nashville, TN 37212, USA;
| | - Joseph J Schlesinger
- Division of Critical Care Medicine, Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Thomas J Reese
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Bryan Steitz
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Elise Russo
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Brian Koh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
16
|
Moon KJ, Son CS, Lee JH, Park M. The development of a web-based app employing machine learning for delirium prevention in long-term care facilities in South Korea. BMC Med Inform Decis Mak 2022; 22:220. [PMID: 35978303 PMCID: PMC9383654 DOI: 10.1186/s12911-022-01966-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2021] [Accepted: 08/10/2022] [Indexed: 11/29/2022] Open
Abstract
Background Long-term care facilities (LCFs) in South Korea have limited knowledge of and capability to care for patients with delirium. They also often lack an electronic medical record system. These barriers hinder systematic approaches to delirium monitoring and intervention. Therefore, this study aims to develop a web-based app for delirium prevention in LCFs and analyse its feasibility and usability. Methods The app was developed based on the validity of the AI prediction model algorithm. A total of 173 participants were selected from LCFs to participate in a study to determine the predictive risk factors for delerium. The app was developed in five phases: (1) the identification of risk factors and preventive intervention strategies from a review of evidence-based literature, (2) the iterative design of the app and components of delirium prevention, (3) the development of a delirium prediction algorithm and cloud platform, (4) a pilot test and validation conducted with 33 patients living in a LCF, and (5) an evaluation of the usability and feasibility of the app, completed by nurses (Main users). Results A web-based app was developed to predict high risk of delirium and apply preventive interventions accordingly. Moreover, its validity, usability, and feasibility were confirmed after app development. By employing machine learning, the app can predict the degree of delirium risk and issue a warning alarm. Therefore, it can be used to support clinical decision-making, help initiate the assessment of delirium, and assist in applying preventive interventions. Conclusions This web-based app is evidence-based and can be easily mobilised to support care for patients with delirium in LCFs. This app can improve the recognition of delirium and predict the degree of delirium risk, thereby helping develop initiatives for delirium prevention and providing interventions. Moreover, this app can be extended to predict various risk factors of LCF and apply preventive interventions. Its use can ultimately improve patient safety and quality of care. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01966-8.
Collapse
Affiliation(s)
- Kyoung Ja Moon
- College of Nursing, Keimyung University, 1095 Dalgubeol-daero, Dalseo-gu, Daegu, 42601, South Korea.
| | - Chang-Sik Son
- Division of Intelligent Robots, Daegu Gyeongbuk Institute of Science and Technology (DGIST), 333, Techno jungang-daero, Hyeonpung-eup, Dalseong-gun, Daegu, South Korea
| | - Jong-Ha Lee
- College of Medicine, Keimyung University, 1095 Dalgubeol-daero, Dalseo-gu, Daegu, 42601, South Korea
| | - Mina Park
- College of Nursing, Keimyung University, 1095 Dalgubeol-daero, Dalseo-gu, Daegu, 42601, South Korea
| |
Collapse
|
17
|
Yoon HK, Yang HL, Jung CW, Lee HC. Artificial intelligence in perioperative medicine - a narrative review. Korean J Anesthesiol 2022; 75:202-215. [PMID: 35345305 PMCID: PMC9171545 DOI: 10.4097/kja.22157] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Accepted: 03/15/2022] [Indexed: 11/23/2022] Open
Abstract
Recent advancements in artificial intelligence (AI) techniques have enabled the development of accurate prediction models using clinical big data. AI models for perioperative risk stratification, intraoperative event prediction, biosignal analyses, and intensive care medicine have been developed in the field of perioperative medicine. Some of these models have been validated using external datasets and randomized controlled trials. Once these models are implemented in electronic health record systems or software medical devices, they could help anesthesiologists improve clinical outcomes by accurately predicting complications and suggesting optimal treatment strategies in real-time. This review provides an overview of the AI techniques used in perioperative medicine and a summary of the studies that have been published using these techniques. Understanding these techniques will aid in their appropriate application in clinical practice.
Collapse
Affiliation(s)
- Hyun-Kyu Yoon
- Department of Anesthesiology and Pain Medicine, Seoul National University Hospital, Seoul, Republic of Korea
| | - Hyun-Lim Yang
- Department of Anesthesiology and Pain Medicine, Seoul National University Hospital, Seoul, Republic of Korea.,Biomedical Research Institute, Seoul National University Hospital, Seoul, Republic of Korea
| | - Chul-Woo Jung
- Department of Anesthesiology and Pain Medicine, Seoul National University Hospital, Seoul, Republic of Korea.,Department of Anesthesiology and Pain Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Hyung-Chul Lee
- Department of Anesthesiology and Pain Medicine, Seoul National University Hospital, Seoul, Republic of Korea.,Department of Anesthesiology and Pain Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea
| |
Collapse
|
18
|
Sun H, Depraetere K, Meesseman L, Cabanillas Silva P, Szymanowsky R, Fliegenschmidt J, Hulde N, von Dossow V, Vanbiervliet M, De Baerdemaeker J, Roccaro-Waldmeyer DM, Stieg J, Domínguez Hidalgo M, Dahlweid FM. Evaluating live performance of machine learning based prediction models for different clinical risks: a study of live systems in different hospitals (Preprint). J Med Internet Res 2021; 24:e34295. [PMID: 35502887 PMCID: PMC9214618 DOI: 10.2196/34295] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2021] [Revised: 02/25/2022] [Accepted: 04/12/2022] [Indexed: 11/30/2022] Open
Abstract
Background Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. Objective The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. Methods We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. Results The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. Conclusions Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.
Collapse
Affiliation(s)
- Hong Sun
- Dedalus Healthcare, Antwerp, Belgium
| | | | | | | | | | - Janis Fliegenschmidt
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | - Nikolai Hulde
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | - Vera von Dossow
- Institute of Anesthesiology and Pain Therapy, Heart and Diabetes Centre North Rhine-Westphalia, University Hospital of Ruhr-University Bochum, Bad Oeynhausen, Germany
| | | | | | | | | | | | | |
Collapse
|
19
|
Son CS, Kang WS, Lee JH, Moon KJ. Machine Learning to Identify Psychomotor Behaviors of Delirium for Patients in Long-Term Care Facility. IEEE J Biomed Health Inform 2021; 26:1802-1814. [PMID: 34596563 DOI: 10.1109/jbhi.2021.3116967] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
This study aimed to develop accurate and explainable machine learning models for three psychomotor behaviors of delirium for hospitalized adult patients. A prospective pilot study was conducted with 33 participants admitted to a long-term care facility between August 10 and 25, 2020. During the pilot study, we collected 560 cases that included 33 clinical variables and the survey items from the short confusion assessment method (S-CAM), and developed a mobile-based application. Multiple machine learning algorithms, including four rule-mining algorithms (C4.5, CBA, MCAR, and LEM2) and four other statistical learning algorithms (LR, ANNs, SVMs with three kernel functions, and random forest), were validated by paired Wilcoxon signed-rank tests on both macro-averaged F1 and weighted average F1-measures during the 10-times stratified 2-fold cross-validation. The LEM2 algorithm achieved the best prediction performance (macro-averaged F1-measure of 49.35%; weighted average F1-measure of 96.55%), correctly identifying adult patients at delirium risk. In the pairwise comparison between predictive powers observed from independent models, the LEM2 model showed a medium or large effect size between 0.4925 and 0.8766 when compared with LR, ANN, SVM with RBF, and MCAR models. We have confirmed that acute consciousness in S-CAM assessment is closely associated with different predictors for screening three psychomotor behaviors of delirium: 1) education level, dementia type or its level, sleep disorder, dehydration, and infection in mixed-type delirium; 2) gender, education level, dementia type, dehydration, bedsores, and foley catheter in hyperactive delirium; and 3) pain, sleep disorder, and haloperidol use in hypoactive delirium.
Collapse
|
20
|
Ge W, Alabsi H, Jain A, Ye E, Sun H, Fernandes M, Magdamo C, Tesh RA, Collens SI, Newhouse A, Mvr Moura L, Zafar S, Hsu J, Akeju O, Robbins GK, Mukerji SS, Das S, Westover MB. Identifying patients with delirium based on unstructured clinical notes. (Preprint). JMIR Form Res 2021; 6:e33834. [PMID: 35749214 PMCID: PMC9270709 DOI: 10.2196/33834] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2021] [Revised: 01/22/2022] [Accepted: 02/10/2022] [Indexed: 11/23/2022] Open
Abstract
Background Delirium in hospitalized patients is a syndrome of acute brain dysfunction. Diagnostic (International Classification of Diseases [ICD]) codes are often used in studies using electronic health records (EHRs), but they are inaccurate. Objective We sought to develop a more accurate method using natural language processing (NLP) to detect delirium episodes on the basis of unstructured clinical notes. Methods We collected 1.5 million notes from >10,000 patients from among 9 hospitals. Seven experts iteratively labeled 200,471 sentences. Using these, we trained three NLP classifiers: Support Vector Machine, Recurrent Neural Networks, and Transformer. Testing was performed using an external data set. We also evaluated associations with delirium billing (ICD) codes, medications, orders for restraints and sitters, direct assessments (Confusion Assessment Method [CAM] scores), and in-hospital mortality. F1 scores, confusion matrices, and areas under the receiver operating characteristic curve (AUCs) were used to compare NLP models. We used the φ coefficient to measure associations with other delirium indicators. Results The transformer NLP performed best on the following parameters: micro F1=0.978, macro F1=0.918, positive AUC=0.984, and negative AUC=0.992. NLP detections exhibited higher correlations (φ) than ICD codes with deliriogenic medications (0.194 vs 0.073 for ICD codes), restraints and sitter orders (0.358 vs 0.177), mortality (0.216 vs 0.000), and CAM scores (0.256 vs –0.028). Conclusions Clinical notes are an attractive alternative to ICD codes for EHR delirium studies but require automated methods. Our NLP model detects delirium with high accuracy, similar to manual chart review. Our NLP approach can provide more accurate determination of delirium for large-scale EHR-based studies regarding delirium, quality improvement, and clinical trails.
Collapse
Affiliation(s)
- Wendong Ge
- Massachusetts General Hospital, Boston, MA, United States
| | - Haitham Alabsi
- Massachusetts General Hospital, Boston, MA, United States
| | - Aayushee Jain
- Massachusetts General Hospital, Boston, MA, United States
| | - Elissa Ye
- Massachusetts General Hospital, Boston, MA, United States
| | - Haoqi Sun
- Massachusetts General Hospital, Boston, MA, United States
| | | | - Colin Magdamo
- Massachusetts General Hospital, Boston, MA, United States
| | - Ryan A Tesh
- Massachusetts General Hospital, Boston, MA, United States
| | | | - Amy Newhouse
- Massachusetts General Hospital, Boston, MA, United States
| | | | - Sahar Zafar
- Massachusetts General Hospital, Boston, MA, United States
| | - John Hsu
- Massachusetts General Hospital, Boston, MA, United States
| | | | | | | | - Sudeshna Das
- Massachusetts General Hospital, Boston, MA, United States
| | | |
Collapse
|
21
|
Chowdhury M, Cervantes EG, Chan WY, Seitz DP. Use of Machine Learning and Artificial Intelligence Methods in Geriatric Mental Health Research Involving Electronic Health Record or Administrative Claims Data: A Systematic Review. Front Psychiatry 2021; 12:738466. [PMID: 34616322 PMCID: PMC8488098 DOI: 10.3389/fpsyt.2021.738466] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Accepted: 08/26/2021] [Indexed: 11/13/2022] Open
Abstract
Introduction: Electronic health records (EHR) and administrative healthcare data (AHD) are frequently used in geriatric mental health research to answer various health research questions. However, there is an increasing amount and complexity of data available that may lend itself to alternative analytic approaches using machine learning (ML) or artificial intelligence (AI) methods. We performed a systematic review of the current application of ML or AI approaches to the analysis of EHR and AHD in geriatric mental health. Methods: We searched MEDLINE, Embase, and PsycINFO to identify potential studies. We included all articles that used ML or AI methods on topics related to geriatric mental health utilizing EHR or AHD data. We assessed study quality either by Prediction model Risk OF Bias ASsessment Tool (PROBAST) or Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) checklist. Results: We initially identified 391 articles through an electronic database and reference search, and 21 articles met inclusion criteria. Among the selected studies, EHR was the most used data type, and the datasets were mainly structured. A variety of ML and AI methods were used, with prediction or classification being the main application of ML or AI with the random forest as the most common ML technique. Dementia was the most common mental health condition observed. The relative advantages of ML or AI techniques compared to biostatistical methods were generally not assessed. Only in three studies, low risk of bias (ROB) was observed according to all the PROBAST domains but in none according to QUADAS-2 domains. The quality of study reporting could be further improved. Conclusion: There are currently relatively few studies using ML and AI in geriatric mental health research using EHR and AHD methods, although this field is expanding. Aside from dementia, there are few studies of other geriatric mental health conditions. The lack of consistent information in the selected studies precludes precise comparisons between them. Improving the quality of reporting of ML and AI work in the future would help improve research in the field. Other courses of improvement include using common data models to collect/organize data, and common datasets for ML model validation.
Collapse
Affiliation(s)
- Mohammad Chowdhury
- Department of Psychiatry, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Eddie Gasca Cervantes
- Department of Electrical and Computer Engineering, Queen's University, Kingston, ON, Canada
| | - Wai-Yip Chan
- Department of Electrical and Computer Engineering, Queen's University, Kingston, ON, Canada
| | - Dallas P. Seitz
- Department of Psychiatry, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| |
Collapse
|
22
|
Yan C, Gao C, Zhang Z, Chen W, Malin BA, Ely EW, Patel MB, Chen Y. Predicting brain function status changes in critically ill patients via Machine learning. J Am Med Inform Assoc 2021; 28:2412-2422. [PMID: 34402496 DOI: 10.1093/jamia/ocab166] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Revised: 07/15/2021] [Accepted: 07/21/2021] [Indexed: 11/14/2022] Open
Abstract
OBJECTIVE In intensive care units (ICUs), a patient's brain function status can shift from a state of acute brain dysfunction (ABD) to one that is ABD-free and vice versa, which is challenging to forecast and, in turn, hampers the allocation of hospital resources. We aim to develop a machine learning model to predict next-day brain function status changes. MATERIALS AND METHODS Using multicenter prospective adult cohorts involving medical and surgical ICU patients from 2 civilian and 3 Veteran Affairs hospitals, we trained and externally validated a light gradient boosting machine to predict brain function status changes. We compared the performances of the boosting model against state-of-the-art models-an ABD predictive model and its variants. We applied Shapley additive explanations to identify influential factors to develop a compact model. RESULTS There were 1026 critically ill patients without evidence of prior major dementia, or structural brain diseases, from whom 12 295 daily transitions (ABD: 5847 days; ABD-free: 6448 days) were observed. The boosting model achieved an area under the receiver-operating characteristic curve (AUROC) of 0.824 (95% confidence interval [CI], 0.821-0.827), compared with the state-of-the-art models of 0.697 (95% CI, 0.693-0.701) with P < .001. Using 13 identified top influential factors, the compact model achieved 99.4% of the boosting model on AUROC. The boosting and the compact models demonstrated high generalizability in external validation by achieving an AUROC of 0.812 (95% CI, 0.812-0.813). CONCLUSION The inputs of the compact model are based on several simple questions that clinicians can quickly answer in practice, which demonstrates the model has direct prospective deployment potential into clinical practice, aiding in critical hospital resource allocation.
Collapse
Affiliation(s)
- Chao Yan
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA
| | - Cheng Gao
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Ziqi Zhang
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA
| | - Wencong Chen
- Department of Biostatistics, School of Medicine, Vanderbilt University, Nashville, Tennessee, USA.,Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Bradley A Malin
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA.,Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Biostatistics, School of Medicine, Vanderbilt University, Nashville, Tennessee, USA
| | - E Wesley Ely
- Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Geriatric Research and Education Clinical Center, Tennessee Valley Healthcare System, U.S. Department of Veteran Affairs, Nashville, Tennessee, USA
| | - Mayur B Patel
- Critical Illness, Brain Dysfunction, and Survivorship Center, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Geriatric Research and Education Clinical Center, Tennessee Valley Healthcare System, U.S. Department of Veteran Affairs, Nashville, Tennessee, USA.,Section of Surgical Sciences, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Neurosurgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Department of Hearing & Speech Sciences, Vanderbilt University Medical Center, Nashville, Tennessee, USA.,Division of Trauma, Surgical Critical Care, and Emergency General Surgery, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - You Chen
- Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, Tennessee, USA.,Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
23
|
Chua SJ, Wrigley S, Hair C, Sahathevan R. Prediction of delirium using data mining: A systematic review. J Clin Neurosci 2021; 91:288-298. [PMID: 34373042 DOI: 10.1016/j.jocn.2021.07.029] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Revised: 06/18/2021] [Accepted: 07/18/2021] [Indexed: 12/19/2022]
Abstract
Delirium remains a significant cause of morbidity, mortality and economic burden to society. "Big data" refers to data of significantly large volume, obtained from a variety of resources, which is created and processed at high velocity. We conducted a systematic review and meta-analysis exploring whether big data could predict the incidence of delirium of patients in the inpatient setting. Medline, Embase, the Cochrane Library, Web of Science, CINAHL, clinicaltrials.gov, who.int and IEEE Xplore were searched using MeSH terms "big data", "data mining", "delirium" and "confusion" up to 30th September 2019. We included both randomised and observational studies. The primary outcome of interest was development of delirium and the secondary outcomes of interest were type of statistical methods used, variables included in the mining algorithms and clinically important outcomes such as mortality and length of hospital stay. The quality of studies was graded using the CHARMs checklist. Six retrospective single centre observational studies were included (n = 178,091), of which 17, 574 participants developed delirium. Studies were of generally of low to moderate quality. The most commonly studied method was random forest, followed by support vector machine and artificial neural networks. The model with best performance for delirium prediction was random forest, with area under receiver operating curve (AUROC) ranging from 0.78 to 0.91. Sensitivity ranged from 0.59 to 0.81 and specificity ranged from 0.73 to 0.92. Our systematic review suggests that machine-learning techniques can be utilised to predict delirium.
Collapse
Affiliation(s)
- S J Chua
- Ballarat Health Services, Ballarat, Australia.
| | - S Wrigley
- Ballarat Health Services, Ballarat, Australia
| | - C Hair
- Ballarat Health Services, Ballarat, Australia
| | - R Sahathevan
- Ballarat Health Services, Ballarat, Australia; School of Medicine, Deakin University, Geelong, Australia; Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia
| |
Collapse
|
24
|
Ocagli H, Bottigliengo D, Lorenzoni G, Azzolina D, Acar AS, Sorgato S, Stivanello L, Degan M, Gregori D. A Machine Learning Approach for Investigating Delirium as a Multifactorial Syndrome. Int J Environ Res Public Health 2021; 18:ijerph18137105. [PMID: 34281037 PMCID: PMC8297073 DOI: 10.3390/ijerph18137105] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 06/11/2021] [Accepted: 06/14/2021] [Indexed: 12/12/2022]
Abstract
Delirium is a psycho-organic syndrome common in hospitalized patients, especially the elderly, and is associated with poor clinical outcomes. This study aims to identify the predictors that are mostly associated with the risk of delirium episodes using a machine learning technique (MLT). A random forest (RF) algorithm was used to evaluate the association between the subject’s characteristics and the 4AT (the 4 A’s test) score screening tool for delirium. RF algorithm was implemented using information based on demographic characteristics, comorbidities, drugs and procedures. Of the 78 patients enrolled in the study, 49 (63%) were at risk for delirium, 32 (41%) had at least one episode of delirium during the hospitalization (38% in orthopedics and 31% both in internal medicine and in the geriatric ward). The model explained 75.8% of the variability of the 4AT score with a root mean squared error of 3.29. Higher age, the presence of dementia, physical restraint, diabetes and a lower degree are the variables associated with an increase of the 4AT score. Random forest is a valid method for investigating the patients’ characteristics associated with delirium onset also in small case-series. The use of this model may allow for early detection of delirium onset to plan the proper adjustment in healthcare assistance.
Collapse
Affiliation(s)
- Honoria Ocagli
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Daniele Bottigliengo
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Giulia Lorenzoni
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
| | - Danila Azzolina
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
- Department of Medical Science, University of Ferrara, Via Fossato di Mortara 64B, 44121 Ferrara, Italy
| | - Aslihan S. Acar
- Department of Actuarial Sciences, Hacettepe University, Ankara 06800, Turkey;
| | - Silvia Sorgato
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Lucia Stivanello
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Mario Degan
- Health Professional Management Service (DPS) of the University Hospital of Padova, 35128 Padova, Italy; (S.S.); (L.S.); (M.D.)
| | - Dario Gregori
- Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Via Loredan 18, 35121 Padova, Italy; (H.O.); (D.B.); (G.L.); (D.A.)
- Correspondence: ; Tel.: +39-049-827-5384
| |
Collapse
|
25
|
Sun H, Depraetere K, Meesseman L, De Roo J, Vanbiervliet M, De Baerdemaeker J, Muys H, von Dossow V, Hulde N, Szymanowsky R. A scalable approach for developing clinical risk prediction applications in different hospitals. J Biomed Inform 2021; 118:103783. [DOI: 10.1016/j.jbi.2021.103783] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2020] [Revised: 04/07/2021] [Accepted: 04/08/2021] [Indexed: 12/19/2022]
|
26
|
Ocagli H, Azzolina D, Soltanmohammadi R, Aliyari R, Bottigliengo D, Acar AS, Stivanello L, Degan M, Baldi I, Lorenzoni G, Gregori D. Profiling Delirium Progression in Elderly Patients via Continuous-Time Markov Multi-State Transition Models. J Pers Med 2021; 11:445. [PMID: 34064001 DOI: 10.3390/jpm11060445] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2021] [Revised: 05/17/2021] [Accepted: 05/19/2021] [Indexed: 12/12/2022] Open
Abstract
Poor recognition of delirium among hospitalized elderlies is a typical challenge for health care professionals. Considering methodological insufficiency for assessing time-varying diseases, a continuous-time Markov multi-state transition model (CTMMTM) was used to investigate delirium evolution in elderly patients. This is a longitudinal observational study performed in September 2016 in an Italian hospital. Change of delirium states was modeled according to the 4AT score. A Cox model (CM) and a CTMMTM were used for identifying factors affecting delirium onset both with a two-state and three-state model. In this study, 78 patients were enrolled and evaluated for 5 days. Both the CM and the CTMMTM show that urine catheter (UC), aging, drugs, and invasive devices (ID) are risk factors for delirium onset. The CTMMTM model shows that transition from no-delirium/cognitive impairment to delirium was associated with aging (HR = 1.14; 95%CI, 1.05, 1.23) and neuroleptics (HR = 4.3; 1.57, 11.77), dopaminergic drugs (HR = 3.89; 1.2, 12.6), UC (HR = 2.92; 1.09, 7.79) and ID (HR = 1.67; 103, 2.71). These results are confirmed by the multivariable model. Aging, ID, antibiotics, drugs affecting the central nervous system, and absence of moving ability are identified as the significant predictors of delirium. Additionally, it seems that modeling with CTMMTM may show associations that are not directly detectable with the traditional CM.
Collapse
|
27
|
Affiliation(s)
- Stefanie Jauk
- Department of Information and Process Management, Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria.,Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
| |
Collapse
|
28
|
Rousseau JF, Tierney WM. Letter to the editor in response to "Risk prediction of delirium in hospitalized patients using machine learning: an implementation and prospective evaluation study". J Am Med Inform Assoc 2021; 28:664-665. [PMID: 33325514 DOI: 10.1093/jamia/ocaa285] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2020] [Accepted: 10/30/2020] [Indexed: 11/14/2022] Open
Affiliation(s)
- Justin F Rousseau
- Department of Population Health, Dell Medical School, The University of Texas at Austin, Austin, Texas, USA
- Department of Neurology, Dell Medical School, The University of Texas at Austin, Austin, Texas, USA
| | - William M Tierney
- Department of Population Health, Dell Medical School, The University of Texas at Austin, Austin, Texas, USA
| |
Collapse
|
29
|
Jauk S, Kramer D, Avian A, Berghold A, Leodolter W, Schulz S. Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study. J Med Syst 2021; 45:48. [PMID: 33646459 PMCID: PMC7921052 DOI: 10.1007/s10916-021-01727-6] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Accepted: 02/18/2021] [Indexed: 12/02/2022]
Abstract
Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed methods design to collect opinions and concerns from health care professionals including physicians and nurses who regularly used the application. The evaluation was framed by the Technology Acceptance Model assessing perceived ease of use, perceived usefulness, actual system use and output quality of the application. Questionnaire results from 47 nurses and physicians as well as qualitative results of four expert group meetings rated the overall usefulness of the delirium prediction positively. For healthcare professionals, the visualization and presented information was understandable, the application was easy to use and the additional information for delirium management was appreciated. The application did not increase their workload, but the actual system use was still low during the pilot study. Our study provides insights into the user acceptance of a machine learning-based application supporting delirium management in hospitals. In order to improve quality and safety in healthcare, computerized decision support should predict actionable events and be highly accepted by users.
Collapse
Affiliation(s)
- Stefanie Jauk
- Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Information and Process Management, Graz, Austria. .,Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Auenbruggerplatz 2, 8036, Graz, Austria.
| | - Diether Kramer
- Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Information and Process Management, Graz, Austria
| | - Alexander Avian
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Auenbruggerplatz 2, 8036, Graz, Austria
| | - Andrea Berghold
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Auenbruggerplatz 2, 8036, Graz, Austria
| | - Werner Leodolter
- Steiermärkische Krankenanstaltengesellschaft m.b.H. (KAGes), Information and Process Management, Graz, Austria
| | - Stefan Schulz
- Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Auenbruggerplatz 2, 8036, Graz, Austria
| |
Collapse
|
30
|
Shung DL. Advancing care for acute gastrointestinal bleeding using artificial intelligence. J Gastroenterol Hepatol 2021; 36:273-278. [PMID: 33624892 DOI: 10.1111/jgh.15372] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/30/2020] [Revised: 12/07/2020] [Accepted: 12/08/2020] [Indexed: 12/14/2022]
Abstract
The future of gastrointestinal bleeding will include the integration of machine learning algorithms to enhance clinician risk assessment and decision making. Machine learning algorithms have shown promise in outperforming existing clinical risk scores for both upper and lower gastrointestinal bleeding but have not been validated in any prospective clinical trials. The adoption of electronic health records provides an exciting opportunity to deploy risk prediction tools in real time and also to expand the data available to train predictive models. Machine learning algorithms can be used to identify patients with acute gastrointestinal bleeding using data extracted from the electronic health record. This can lead to an automated process to find patients with symptoms of acute gastrointestinal bleeding so that risk prediction tools can be then triggered to consistently provide decision support to the physician. Neural network models can be used to provide continuous risk predictions for patients who are at higher risk, which can be used to guide triage of patients to appropriate levels of care. Finally, the future will likely include neural network-based analysis of endoscopic stigmata of bleeding to help guide best practices for hemostasis during the endoscopic procedure. Machine learning will enhance the delivery of care at every level for patients with acute gastrointestinal bleeding through identifying very low risk patients for outpatient management, triaging high risk patients for higher levels of care, and guiding optimal intervention during endoscopy.
Collapse
|