1
|
Staiger RD, Mehra T, Haile SR, Domenghino A, Kümmerli C, Abbassi F, Kozbur D, Dutkowski P, Puhan MA, Clavien PA. Experts vs. machine - comparison of machine learning to expert-informed prediction of outcome after major liver surgery. HPB (Oxford) 2024; 26:674-681. [PMID: 38423890 DOI: 10.1016/j.hpb.2024.02.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Revised: 02/01/2024] [Accepted: 02/11/2024] [Indexed: 03/02/2024]
Abstract
BACKGROUND Machine learning (ML) has been successfully implemented for classification tasks (e.g., cancer diagnosis). ML performance for more challenging predictions is largely unexplored. This study's objective was to compare machine learning vs. expert-informed predictions for surgical outcome in patients undergoing major liver surgery. METHODS Single tertiary center data on preoperative parameters and postoperative complications for elective hepatic surgery patients were included (2008-2021). Expert-informed prediction models were established on 14 parameters identified by two expert liver surgeons to impact on postoperative outcome. ML models used all available preoperative patient variables (n = 62). Model performance was compared for predicting 3-month postoperative overall morbidity. Temporal validation and additional analysis in major liver resection patients were conducted. RESULTS 889 patients included. Expert-informed models showed low average bias (2-5 CCI points) with high over/underprediction. ML models performed similarly: average prediction 5-10 points higher than observed CCI values with high variability (95% CI -30 to 50). No performance improvement for major liver surgery patients. CONCLUSION No clinical relevance in the application of ML for predicting postoperative overall morbidity was found. Despite being a novel hype, ML has the potential for application in clinical practice. However, at this stage it does not replace established approaches of prediction modelling.
Collapse
Affiliation(s)
- Roxane D Staiger
- Department of Surgery & Transplantation, University Hospital Zurich, Zurich, Switzerland.
| | - Tarun Mehra
- Department of Medical Oncology and Hematology, University Hospital Zurich, Zurich, Switzerland
| | - Sarah R Haile
- Department of Epidemiology, Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland
| | - Anja Domenghino
- Department of Surgery & Transplantation, University Hospital Zurich, Zurich, Switzerland
| | | | - Fariba Abbassi
- Department of Surgery & Transplantation, University Hospital Zurich, Zurich, Switzerland
| | - Damian Kozbur
- Department of Economics, University of Zurich, Zurich, Switzerland
| | - Philipp Dutkowski
- Department of Surgery & Transplantation, University Hospital Zurich, Zurich, Switzerland
| | - Milo A Puhan
- Department of Epidemiology, Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland
| | - Pierre-Alain Clavien
- Department of Surgery & Transplantation, University Hospital Zurich, Zurich, Switzerland
| |
Collapse
|
2
|
Ingraham NE, Jones EK, King S, Dries J, Phillips M, Loftus T, Evans HL, Melton GB, Tignanelli CJ. Re-Aiming Equity Evaluation in Clinical Decision Support: A Scoping Review of Equity Assessments in Surgical Decision Support Systems. Ann Surg 2023; 277:359-364. [PMID: 35943199 PMCID: PMC9905217 DOI: 10.1097/sla.0000000000005661] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
OBJECTIVE We critically evaluated the surgical literature to explore the prevalence and describe how equity assessments occur when using clinical decision support systems. BACKGROUND Clinical decision support (CDS) systems are increasingly used to facilitate surgical care delivery. Despite formal recommendations to do so, equity evaluations are not routinely performed on CDS systems and underrepresented populations are at risk of harm and further health disparities. We explored surgical literature to determine frequency and rigor of CDS equity assessments and offer recommendations to improve CDS equity by appending existing frameworks. METHODS We performed a scoping review up to Augus 25, 2021 using PubMed and Google Scholar for the following search terms: clinical decision support, implementation, RE-AIM, Proctor, Proctor's framework, equity, trauma, surgery, surgical. We identified 1415 citations and 229 abstracts met criteria for review. A total of 84 underwent full review after 145 were excluded if they did not assess outcomes of an electronic CDS tool or have a surgical use case. RESULTS Only 6% (5/84) of surgical CDS systems reported equity analyses, suggesting that current methods for optimizing equity in surgical CDS are inadequate. We propose revising the RE-AIM framework to include an Equity element (RE 2 -AIM) specifying that CDS foundational analyses and algorithms are performed or trained on balanced datasets with sociodemographic characteristics that accurately represent the CDS target population and are assessed by sensitivity analyses focused on vulnerable subpopulations. CONCLUSION Current surgical CDS literature reports little with respect to equity. Revising the RE-AIM framework to include an Equity element (RE 2 -AIM) promotes the development and implementation of CDS systems that, at minimum, do not worsen healthcare disparities and possibly improve their generalizability.
Collapse
Affiliation(s)
- Nicholas E Ingraham
- Department of Medicine, University of Minnesota, Division of Pulmonary and Critical Care, Minneapolis, MN
| | - Emma K Jones
- Department of Surgery, University of Minnesota, Division of Acute Care Surgery, Minneapolis, MN
| | - Samantha King
- Department of Surgery, University of Minnesota, Division of Acute Care Surgery, Minneapolis, MN
| | - James Dries
- Department of Surgery, University of Minnesota, Division of Acute Care Surgery, Minneapolis, MN
| | - Michael Phillips
- Pediatric Surgery, University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Tyler Loftus
- Department of Surgery, University of Florida Health, Gainesville, FL
| | - Heather L Evans
- Department of Surgery, Medical University of South Carolina, Charleston, SC
| | - Genevieve B Melton
- Department of Surgery, University of Minnesota, Division of Acute Care Surgery, Minneapolis, MN
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN
| | - Christopher J Tignanelli
- Department of Surgery, University of Minnesota, Division of Acute Care Surgery, Minneapolis, MN
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN
| |
Collapse
|
3
|
Hill C, Versluijs Y, Furay E, Reese-White D, Holan C, Alexander J, Doggett S, Ring D, Buckley FP. Psychoemotional factors and their influence on the quality of life in patients with GERD. Surg Endosc 2020; 35:7219-7226. [PMID: 33237463 DOI: 10.1007/s00464-020-08145-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2020] [Accepted: 10/27/2020] [Indexed: 01/06/2023]
Abstract
Patient-reported outcomes (PROs) are integral to determining the success of foregut surgical interventions and psychoemotional factors have been hypothesized to impact the quality of life of patients. This study evaluates the correlation between PROs-specifically the Gastroesophageal Reflux Disease-Health-Related Quality of Life (GERD-HRQL) and the Laryngopharangeal Reflux Symptom Index (LPR-RSI)-and the recently validated Esophageal Hypervigilance Anxiety Scale (EHAS). We hypothesize that patients with higher EHAS scores have significantly elevated GERD-HRQL LPR-RSI compared to those with normal scores. EHAS has been developed and validated in chronic esophageal disorders, but clinical impact is unknown. In this retrospective study, 197 patients (38% men, average age 56 ± 16) completed the following surveys:(1) EHAS, (2) GERD-HRQL, and (3) LPR-RSI. All patients referred for surgical evaluation of GERD completed the surveys as part of their pre-operative workup and post-operative follow-up In bivariate analysis, EHAS correlated with both GERD-HRQL (r 0.53, P = <0.001) and LPR-RSI (r 0.36, P = 0.009). Accounting for potential confounding with sex and age in multivariable linear regression models, a higher GERD-HRQL score (β 0.38; 95% CI 0.29 to 0.48; P = <0.001; Semipartial R2 0.20) and a higher LPR-RSI score (β 0.21; 95% CI 0.13 to 0.29; P = <0.001; Semipartial R2 0.08) were independently associated with higher EHAS. The observed relationship between mental health and GERD symptom intensity is consistent with the biopsychosocial paradigm of illness. Future studies focused on post-surgical outcomes following the incorporation of EHAS into perioperative care is needed to evaluate its effectiveness as a clinical decision support tool in ARS.
Collapse
Affiliation(s)
- Charles Hill
- Department of Surgery and Perioperative Care, University of Texas at Austin, Austin, TX, USA.
| | - Yvonne Versluijs
- Department of Orthopaedic Surgery, University of Texas at Austin, Austin, TX, USA
| | - Elisa Furay
- Department of Surgery and Perioperative Care, University of Texas at Austin, Austin, TX, USA
| | | | - Cole Holan
- Dell Medical School, University of Texas at Austin, Austin, TX, USA
| | | | - Stephanie Doggett
- Department of Surgery and Perioperative Care, University of Texas at Austin, Austin, TX, USA
| | - David Ring
- Department of Orthopaedic Surgery, University of Texas at Austin, Austin, TX, USA
| | - F P Buckley
- Department of Surgery and Perioperative Care, University of Texas at Austin, Austin, TX, USA
| |
Collapse
|
4
|
Macheel C, Reicks P, Sybrant C, Evans C, Farhat J, West MA, Tignanelli CJ. Clinical Decision Support Intervention for Rib Fracture Treatment. J Am Coll Surg 2020; 231:249-256.e2. [DOI: 10.1016/j.jamcollsurg.2020.04.023] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2020] [Revised: 04/01/2020] [Accepted: 04/06/2020] [Indexed: 01/22/2023]
|
5
|
Beauchemin M, Murray MT, Sung L, Hershman DL, Weng C, Schnall R. Clinical decision support for therapeutic decision-making in cancer: A systematic review. Int J Med Inform 2019; 130:103940. [PMID: 31450082 PMCID: PMC7024607 DOI: 10.1016/j.ijmedinf.2019.07.019] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2019] [Revised: 07/05/2019] [Accepted: 07/31/2019] [Indexed: 12/28/2022]
Abstract
Cancer management, including supportive care, is complex and requires availability and synthesis of published and patient-specific data to make appropriate therapeutic decisions. Clinical decision support (CDS) may be an effective implementation strategy to support complex decision making although it is unclear whether it improves process outcomes, patient outcomes or both in cancer settings. We therefore conducted a systematic review to identify CDS that have been used to support therapeutic decision making in clinical cancer settings. Outcomes of interest included the effect of CDS on the process, such as clinician's decision making and effect on patient outcomes. Ten studies met inclusion criteria, with variability in the study design, setting, and intervention. Of the nine studies that measured process outcomes, five demonstrated significant improvement; and of the six that measured patient outcomes, four demonstrated significant improvement. All included studies utilized CDS that were informed by clinical practice guidelines. In conclusion, CDS to guide cancer therapeutic decision making is an understudied but promising area. Further research is needed.
Collapse
Affiliation(s)
- Melissa Beauchemin
- School of Nursing, Columbia University, New York, NY, 10032, United States; Herbert Irving Comprehensive Cancer Center, Columbia University, New York, NY 10032, United States.
| | - Meghan T Murray
- School of Nursing, Columbia University, New York, NY, 10032, United States
| | | | - Dawn L Hershman
- Herbert Irving Comprehensive Cancer Center, Columbia University, New York, NY 10032, United States
| | - Chunhua Weng
- Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States
| | - Rebecca Schnall
- School of Nursing, Columbia University, New York, NY, 10032, United States
| |
Collapse
|
6
|
Belard A, Buchman T, Forsberg J, Potter BK, Dente CJ, Kirk A, Elster E. Precision diagnosis: a view of the clinical decision support systems (CDSS) landscape through the lens of critical care. J Clin Monit Comput 2017; 31:261-271. [PMID: 26902081 DOI: 10.1007/s10877-016-9849-1] [Citation(s) in RCA: 42] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2015] [Accepted: 02/17/2016] [Indexed: 10/22/2022]
Abstract
Improving diagnosis and treatment depends on clinical monitoring and computing. Clinical decision support systems (CDSS) have been in existence for over 50 years. While the literature points to positive impacts on quality and patient safety, outcomes, and the avoidance of medical errors, technical and regulatory challenges continue to retard their rate of integration into clinical care processes and thus delay the refinement of diagnoses towards personalized care. We conducted a systematic review of pertinent articles in the MEDLINE, US Department of Health and Human Services, Agency for Health Research and Quality, and US Food and Drug Administration databases, using a Boolean approach to combine terms germane to the discussion (clinical decision support, tools, systems, critical care, trauma, outcome, cost savings, NSQIP, APACHE, SOFA, ICU, and diagnostics). References were selected on the basis of both temporal and thematic relevance, and subsequently aggregated around four distinct themes: the uses of CDSS in the critical and surgical care settings, clinical insertion challenges, utilization leading to cost-savings, and regulatory concerns. Precision diagnosis is the accurate and timely explanation of each patient's health problem and further requires communication of that explanation to patients and surrogate decision-makers. Both accuracy and timeliness are essential to critical care, yet computed decision support systems (CDSS) are scarce. The limitation arises from the technical complexity associated with integrating and filtering large data sets from diverse sources. Provider mistrust and resistance coupled with the absence of clear guidance from regulatory bodies further retard acceptance of CDSS. While challenges to develop and deploy CDSS are substantial, the clinical, quality, and economic impacts warrant the effort, especially in disciplines requiring complex decision-making, such as critical and surgical care. Improving diagnosis in health care requires accumulation, validation and transformation of data into actionable information. The aggregate of those processes-CDSS-is currently primitive. Despite technical and regulatory challenges, the apparent clinical and economic utilities of CDSS must lead to greater engagement. These tools play the key role in realizing the vision of a more 'personalized medicine', one characterized by individualized precision diagnosis rather than population-based risk-stratification.
Collapse
Affiliation(s)
- Arnaud Belard
- Uniformed Services University of the Health Sciences, Bethesda, MD, USA.
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA.
| | - Timothy Buchman
- Emory University and Grady Memorial Hospital, Atlanta, GA, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| | - Jonathan Forsberg
- Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- Naval Medical Research Center, Bethesda, MD, USA
- Walter Reed National Military Medical Center, Bethesda, MD, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| | - Benjamin K Potter
- Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- Walter Reed National Military Medical Center, Bethesda, MD, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| | - Christopher J Dente
- Emory University and Grady Memorial Hospital, Atlanta, GA, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| | - Allan Kirk
- Duke University, Durham, NC, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| | - Eric Elster
- Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- Walter Reed National Military Medical Center, Bethesda, MD, USA
- Surgical Critical Care Initiative (SC2i), Bethesda, MD, USA
| |
Collapse
|
7
|
Arsoniadis EG, Melton GB. Leveraging the electronic health record for research and quality improvement: Current strengths and future challenges. SEMINARS IN COLON AND RECTAL SURGERY 2016. [DOI: 10.1053/j.scrs.2016.01.009] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
8
|
A Smartphone-based Decision Support Tool Improves Test Performance Concerning Application of the Guidelines for Managing Regional Anesthesia in the Patient Receiving Antithrombotic or Thrombolytic Therapy. Anesthesiology 2016; 124:186-98. [PMID: 26513023 DOI: 10.1097/aln.0000000000000885] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND The American Society of Regional Anesthesia and Pain Medicine (ASRA) consensus statement on regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy is the standard for evaluation and management of these patients. The authors hypothesized that an electronic decision support tool (eDST) would improve test performance compared with native physician behavior concerning the application of this guideline. METHODS Anesthesiology trainees and faculty at 8 institutions participated in a prospective, randomized trial in which they completed a 20-question test involving clinical scenarios related to the ASRA guidelines. The eDST group completed the test using an iOS app programmed to contain decision logic and content of the ASRA guidelines. The control group completed the test by using any resource in addition to the app. A generalized linear mixed-effects model was used to examine the effect of the intervention. RESULTS After obtaining institutional review board's approval and informed consent, 259 participants were enrolled and randomized (eDST = 122; control = 137). The mean score was 92.4 ± 6.6% in the eDST group and 68.0 ± 15.8% in the control group (P < 0.001). eDST use increased the odds of selecting correct answers (7.8; 95% CI, 5.7 to 10.7). Most control group participants (63%) used some cognitive aid during the test, and they scored higher than those who tested from memory alone (76 ± 15% vs. 57 ± 18%, P < 0.001). There was no difference in time to completion of the test (P = 0.15) and no effect of training level (P = 0.56). CONCLUSIONS eDST use improved application of the ASRA guidelines compared with the native clinician behavior in a testing environment.
Collapse
|
9
|
Initial Experience of the American Society of Regional Anesthesia and Pain Medicine Coags Regional Smartphone Application. Reg Anesth Pain Med 2016; 41:334-8. [DOI: 10.1097/aap.0000000000000391] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
10
|
Colorectal cancer screening quality, cost and practice in an era of healthcare transformation. Dig Dis Sci 2015; 60:692-7. [PMID: 25492502 DOI: 10.1007/s10620-014-3447-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/10/2014] [Accepted: 11/15/2014] [Indexed: 01/16/2023]
Abstract
Colonoscopy for colorectal cancer screening is the largest single source of income for gastroenterologists in the USA. Despite its proven value in preventing colon cancer, it is being scrutinized by payers and regulators as a high-volume procedure with variable quality, cost and documentation. In a rapidly evolving era of reimbursement for added value (high quality, affordable cost) rather than for individual transactions/volume, gastroenterologists are under pressure to change our practices to respond to the demands of the marketplace. Excellent guidance is available through our professional societies.
Collapse
|