1
|
Gombolay GY, Silva A, Schrum M, Gopalan N, Hallman-Cooper J, Dutt M, Gombolay M. Effects of explainable artificial intelligence in neurology decision support. Ann Clin Transl Neurol 2024; 11:1224-1235. [PMID: 38581138 DOI: 10.1002/acn3.52036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Revised: 02/20/2024] [Accepted: 02/27/2024] [Indexed: 04/08/2024] Open
Abstract
OBJECTIVE Artificial intelligence (AI)-based decision support systems (DSS) are utilized in medicine but underlying decision-making processes are usually unknown. Explainable AI (xAI) techniques provide insight into DSS, but little is known on how to design xAI for clinicians. Here we investigate the impact of various xAI techniques on a clinician's interaction with an AI-based DSS in decision-making tasks as compared to a general population. METHODS We conducted a randomized, blinded study in which members of the Child Neurology Society and American Academy of Neurology were compared to a general population. Participants received recommendations from a DSS via a random assignment of an xAI intervention (decision tree, crowd sourced agreement, case-based reasoning, probability scores, counterfactual reasoning, feature importance, templated language, and no explanations). Primary outcomes included test performance and perceived explainability, trust, and social competence of the DSS. Secondary outcomes included compliance, understandability, and agreement per question. RESULTS We had 81 neurology participants with 284 in the general population. Decision trees were perceived as the more explainable by the medical versus general population (P < 0.01) and as more explainable than probability scores within the medical population (P < 0.001). Increasing neurology experience and perceived explainability degraded performance (P = 0.0214). Performance was not predicted by xAI method but by perceived explainability. INTERPRETATION xAI methods have different impacts on a medical versus general population; thus, xAI is not uniformly beneficial, and there is no one-size-fits-all approach. Further user-centered xAI research targeting clinicians and to develop personalized DSS for clinicians is needed.
Collapse
Affiliation(s)
- Grace Y Gombolay
- Department of Pediatrics, Division of Neurology, Children's Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
| | - Andrew Silva
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | | | - Jamika Hallman-Cooper
- Department of Pediatrics, Division of Neurology, Children's Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
| | - Monideep Dutt
- Department of Pediatrics, Division of Neurology, Children's Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
| | - Matthew Gombolay
- Department of Pediatrics, Division of Neurology, Children's Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, USA
| |
Collapse
|
2
|
Dodeja L, Tambwekar P, Hedlund-Botti E, Gombolay M. Towards the design of user-centric strategy recommendation systems for collaborative Human-AI tasks. Int J Hum Comput Stud 2024; 184:103216. [PMID: 38558883 PMCID: PMC10976429 DOI: 10.1016/j.ijhcs.2023.103216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Artificial Intelligence is being employed by humans to collaboratively solve complicated tasks for search and rescue, manufacturing, etc. Efficient teamwork can be achieved by understanding user preferences and recommending different strategies for solving the particular task to humans. Prior work has focused on personalization of recommendation systems for relatively well-understood tasks in the context of e-commerce or social networks. In this paper, we seek to understand the important factors to consider while designing user-centric strategy recommendation systems for decision-making. We conducted a human-subjects experiment (n=60) for measuring the preferences of users with different personality types towards different strategy recommendation systems. We conducted our experiment across four types of strategy recommendation modalities that have been established in prior work: (1) Single strategy recommendation, (2) Multiple similar recommendations, (3) Multiple diverse recommendations, (4) All possible strategies recommendations. While these strategy recommendation schemes have been explored independently in prior work, our study is novel in that we employ all of them simultaneously and in the context of strategy recommendations, to provide us an in-depth overview of the perception of different strategy recommendation systems. We found that certain personality traits, such as conscientiousness, notably impact the preference towards a particular type of system (𝑝 < 0.01). Finally, we report an interesting relationship between usability, alignment, and perceived intelligence wherein greater perceived alignment of recommendations with one's own preferences leads to higher perceived intelligence (𝑝 < 0.01) and higher usability (𝑝 < 0.01).
Collapse
Affiliation(s)
| | | | - Erin Hedlund-Botti
- School of Interactive Computing, Georgia Institute of Technology, Atlanta, 30332, GA, USA
| | - Matthew Gombolay
- School of Interactive Computing, Georgia Institute of Technology, Atlanta, 30332, GA, USA
| |
Collapse
|
3
|
Dias RD, Kennedy-Metz LR, Srey R, Rance G, Ebnali M, Arney D, Gombolay M, Zenati MA. Using Digital Biomarkers for Objective Assessment of Perfusionists' Workload and Acute Stress During Cardiac Surgery. Bioinform Biomed Eng (2023) 2023; 13919:443-454. [PMID: 37497240 PMCID: PMC10371197 DOI: 10.1007/978-3-031-34953-9_35] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/28/2023]
Abstract
The cardiac operating room (OR) is a high-risk, high-stakes environment inserted into a complex socio-technical healthcare system. During cardiopulmonary bypass (CPB), the most critical phase of cardiac surgery, the perfusionist has a crucial role within the interprofessional OR team, being responsible for optimizing patient perfusion while coordinating other tasks with the surgeon, anesthesiologist, and nurses. The aim of this study was to investigate objective digital biomarkers of perfusionists' workload and stress derived from heart rate variability (HRV) metrics captured via a wearable physiological sensor in a real cardiac OR. We explored the relationships between several HRV parameters and validated self-report measures of surgical task workload (SURG-TLX) and acute stress (STAI-SF), as well as surgical processes and outcome measures. We found that the frequency-domain HRV parameter HF relative power - FFT (%) presented the strongest association with task workload (correlation coefficient: -0.491, p-value: 0.003). We also found that the time-domain HRV parameter RMSSD (ms) presented the strongest correlation with perfusionists' acute stress (correlation coefficient: -0.489, p-value: 0.005). A few workload and stress biomarkers were also associated with bypass time and patient length of stay in the hospital. The findings from this study will inform future research regarding which HRV-based biomarkers are best suited for the development of cognitive support systems capable of monitoring surgical workload and stress in real time.
Collapse
Affiliation(s)
- Roger D Dias
- Harvard Medical School, Boston, MA, USA
- Department of Emergency Medicine, Mass General Brigham, Boston, MA, USA
| | | | - Rithy Srey
- Division of Cardiac Surgery, Veterans Affairs Boston Healthcare System, Boston, MA, USA
| | - Geoffrey Rance
- Department of Cardiac Surgery, Cape Cod Healthcare, Hyannis, MA, USA
| | - Mahdi Ebnali
- Harvard Medical School, Boston, MA, USA
- Department of Emergency Medicine, Mass General Brigham, Boston, MA, USA
| | - David Arney
- Harvard Medical School, Boston, MA, USA
- Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, MA, USA
| | | | - Marco A Zenati
- Harvard Medical School, Boston, MA, USA
- Division of Cardiac Surgery, Veterans Affairs Boston Healthcare System and Medical Robotics and Computer Assisted Surgery Lab, Boston, MA, USA
| |
Collapse
|
4
|
Zaidi Z, Martin D, Belles N, Zakharov V, Krishna A, Lee KM, Wagstaff P, Naik S, Sklar M, Choi S, Kakehi Y, Patil R, Mallemadugula D, Pesce F, Wilson P, Hom W, Diamond M, Zhao B, Moorman N, Paleja R, Chen L, Seraj E, Gombolay M. Athletic Mobile Manipulator System for Robotic Wheelchair Tennis. IEEE Robot Autom Lett 2023. [DOI: 10.1109/lra.2023.3249401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2023]
Affiliation(s)
| | | | | | | | | | - Kin Man Lee
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | - Sumedh Naik
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | - Sugju Choi
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | | | | | | | - Peter Wilson
- Georgia Institute of Technology, Atlanta, GA, USA
| | - Wendell Hom
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | - Bryan Zhao
- Georgia Institute of Technology, Atlanta, GA, USA
| | - Nina Moorman
- Georgia Institute of Technology, Atlanta, GA, USA
| | - Rohan Paleja
- Georgia Institute of Technology, Atlanta, GA, USA
| | - Letian Chen
- Georgia Institute of Technology, Atlanta, GA, USA
| | | | | |
Collapse
|
5
|
Dias RD, Kennedy-Metz LR, Yule SJ, Gombolay M, Zenati MA. Assessing Team Situational Awareness in the Operating Room via Computer Vision. IEEE Conf Cogn Comput Asp Situat Manag 2022; 2022:94-96. [PMID: 35994041 PMCID: PMC9386571 DOI: 10.1109/cogsima54611.2022.9830664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Situational awareness (SA) at both individual and team levels, plays a critical role in the operating room (OR). During the pre-incision time-out, the entire OR team comes together to deploy the surgical safety checklist (SSC). Worldwide, the implementation of the SSC has been shown to reduce intraoperative complications and mortality among surgical patients. In this study, we investigated the feasibility of applying computer vision analysis on surgical videos to extract team motion metrics that could differentiate teams with good SA from those with poor SA during the pre-incision time-out. We used a validated observation-based tool to assess SA, and a computer vision software to measure body position and motion patterns in the OR. Our findings showed that it is feasible to extract surgical team motion metrics captured via off-the-shelf OR cameras. Entropy as a measure of the level of team organization was able to distinguish surgical teams with good and poor SA. These findings corroborate existing studies showing that computer vision-based motion metrics have the potential to integrate traditional observation-based performance assessments in the OR.
Collapse
Affiliation(s)
- Roger D Dias
- Department of Emergency Medicine, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
| | - Lauren R Kennedy-Metz
- Department of Surgery, Harvard Medical School, VA Boston Healthcare System, West Roxbury, MA, USA
| | - Steven J Yule
- Department of Clinical Surgery, University of Edinburgh, Edinburgh, Scotland
| | - Matthew Gombolay
- College of Computing, Georgia Institute of Technology, Atlanta, GA, USA
| | - Marco A Zenati
- Department of Surgery, Harvard Medical School, VA Boston Healthcare System, West Roxbury, MA, USA
| |
Collapse
|
6
|
Silva A, Moorman N, Silva W, Zaidi Z, Gopalan N, Gombolay M. LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2021.3139667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
7
|
Dias RD, Zenati MA, Rance G, Srey R, Arney D, Chen L, Paleja R, Kennedy-Metz LR, Gombolay M. Using machine learning to predict perfusionists' critical decision-making during cardiac surgery. Comput Methods Biomech Biomed Eng Imaging Vis 2021; 10:308-312. [PMID: 35937956 PMCID: PMC9355042 DOI: 10.1080/21681163.2021.2002724] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2021] [Accepted: 11/02/2021] [Indexed: 06/15/2023]
Abstract
The cardiac surgery operating room is a high-risk and complex environment in which multiple experts work as a team to provide safe and excellent care to patients. During the cardiopulmonary bypass phase of cardiac surgery, critical decisions need to be made and the perfusionists play a crucial role in assessing available information and taking a certain course of action. In this paper, we report the findings of a simulation-based study using machine learning to build predictive models of perfusionists' decision-making during critical situations in the operating room (OR). Performing 30-fold cross-validation across 30 random seeds, our machine learning approach was able to achieve an accuracy of 78.2% (95% confidence interval: 77.8% to 78.6%) in predicting perfusionists' actions, having access to only 148 simulations. The findings from this study may inform future development of computerised clinical decision support tools to be embedded into the OR, improving patient safety and surgical outcomes.
Collapse
Affiliation(s)
- R. D. Dias
- Human Factors and Cognitive Engineering Lab, Stratus Center for Medical Simulation, Brigham and Women’s Hospital, Boston, MA, USA
- Department of Emergency Medicine, Harvard Medical School, Boston, Ma, USA
| | - M. A. Zenati
- Medical Robotics and Computer Assisted Surgery Lab, Division of Cardiac Surgery, Va Boston Healthcare System, Boston, Ma, USA
- Department of Surgery, Harvard Medical School, Boston, MA, USA
| | - G. Rance
- Medical Robotics and Computer Assisted Surgery Lab, Division of Cardiac Surgery, Va Boston Healthcare System, Boston, Ma, USA
- Department of Surgery, Harvard Medical School, Boston, MA, USA
| | - Rithy Srey
- Medical Robotics and Computer Assisted Surgery Lab, Division of Cardiac Surgery, Va Boston Healthcare System, Boston, Ma, USA
- Department of Surgery, Harvard Medical School, Boston, MA, USA
| | - D. Arney
- Medical Device Plug and Play Interoperability Program, Massachusetts General Hospital, Boston, Ma, USA
- Department of Anesthesia, Harvard Medical School, Boston, Ma, USA
| | - L. Chen
- College of Computing, Georgia Institute of Technology, Atlanta, GA, USA
| | - R. Paleja
- College of Computing, Georgia Institute of Technology, Atlanta, GA, USA
| | - L. R. Kennedy-Metz
- Medical Robotics and Computer Assisted Surgery Lab, Division of Cardiac Surgery, Va Boston Healthcare System, Boston, Ma, USA
- Department of Surgery, Harvard Medical School, Boston, MA, USA
| | - M. Gombolay
- College of Computing, Georgia Institute of Technology, Atlanta, GA, USA
| |
Collapse
|
8
|
|
9
|
Gombolay M, Jensen R, Stigile J, Golen T, Shah N, Son SH, Shah J. Human-Machine Collaborative Optimization via Apprenticeship Scheduling. J ARTIF INTELL RES 2018. [DOI: 10.1613/jair.1.11233] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the "single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes. We propose a new approach for capturing this decision-making process through counterfactual reasoning in pairwise comparisons. Our approach is model-free and does not require iterating through the state space. We demonstrate that this approach accurately learns multifaceted heuristics on a synthetic and real world data sets. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of schedule optimization. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates optimal solutions up to 9.5 times faster than a state-of-the-art optimization algorithm.
Collapse
|
10
|
Abstract
We conducted a study to investigate trust in and dependence upon robotic decision support among nurses and doctors on a labor and delivery floor. There is evidence that suggestions provided by embodied agents engender inappropriate degrees of trust and reliance among humans. This concern represents a critical barrier that must be addressed before fielding intelligent hospital service robots that take initiative to coordinate patient care. We conducted our experiment with nurses and physicians, and evaluated the subjects’ levels of trust in and dependence upon high- and low-quality recommendations issued by robotic versus computer-based decision support. The decision support, generated through action-driven learning from expert demonstration, produced high-quality recommendations that were accepted by nurses and physicians at a compliance rate of 90%. Rates of Type I and Type II errors were comparable between robotic and computer-based decision support. Furthermore, embodiment appeared to benefit performance, as indicated by a higher degree of appropriate dependence after the quality of recommendations changed over the course of the experiment. These results support the notion that a robotic assistant may be able to safely and effectively assist with patient care. Finally, we conducted a pilot demonstration in which a robot-assisted resource nurses on a labor and delivery floor at a tertiary care center.
Collapse
Affiliation(s)
| | - Xi Jessie Yang
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Bradley Hayes
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Nicole Seo
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Zixi Liu
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | | | - Tania Yu
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Neel Shah
- Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Toni Golen
- Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Julie Shah
- Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
11
|
Gombolay M, Bair A, Huang C, Shah J. Computational design of mixed-initiative human–robot teaming that considers human factors: situational awareness, workload, and workflow preferences. Int J Rob Res 2017. [DOI: 10.1177/0278364916688255] [Citation(s) in RCA: 75] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Affiliation(s)
| | - Anna Bair
- Massachusetts Institute of Technology, USA
| | | | - Julie Shah
- Massachusetts Institute of Technology, USA
| |
Collapse
|