1
|
Khan DZ, Newall N, Koh CH, Das A, Aapan S, Layard Horsfall H, Baldeweg SE, Bano S, Borg A, Chari A, Dorward NL, Elserius A, Giannis T, Jain A, Stoyanov D, Marcus HJ. Video-Based Performance Analysis in Pituitary Surgery - Part 2: Artificial Intelligence Assisted Surgical Coaching. World Neurosurg 2024:S1878-8750(24)01370-6. [PMID: 39127380 DOI: 10.1016/j.wneu.2024.07.219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2024] [Accepted: 07/31/2024] [Indexed: 08/12/2024]
Abstract
BACKGROUND Superior surgical skill improves surgical outcomes in endoscopic pituitary adenoma surgery. Video-based coaching programs, pioneered in professional sports, have shown promise in surgical training. In this study, we developed and assessed a video-based coaching program using artificial intelligence (AI) assistance. METHODS An AI-assisted video-based surgical coaching was implemented over 6 months with the pituitary surgery team. The program consisted of 1) monthly random video analysis and review; and 2) quarterly 2-hour educational meetings discussing these videos and learning points. Each video was annotated for surgical phases and steps using AI, which improved video interactivity and allowed the calculation of quantitative metrics. Primary outcomes were program feasibility, acceptability, and appropriateness. Surgical performance (via modified Objective Structured Assessment of Technical Skills) and early surgical outcomes were recorded for every case during the 6-month coaching period, and a preceding 6-month control period. Beta and logistic regression were used to assess the change in modified Objective Structured Assessment of Technical Skills scores and surgical outcomes after the coaching program implementation. RESULTS All participants highly rated the program's feasibility, acceptability, and appropriateness. During the coaching program, 63 endoscopic pituitary adenoma cases were included, with 41 in the control group. Surgical performance across all operative phases improved during the coaching period (P < 0.001), with a reduction in new postoperative anterior pituitary hormone deficit (P = 0.01). CONCLUSIONS We have developed a novel AI-assisted video surgical coaching program for endoscopic pituitary adenoma surgery - demonstrating its viability and impact on surgical performance. Early results also suggest improvement in patient outcomes. Future studies should be multicenter and longer term.
Collapse
Affiliation(s)
- Danyal Z Khan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK.
| | - Nicola Newall
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Chan Hee Koh
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Adrito Das
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Sanchit Aapan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Hugo Layard Horsfall
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Stephanie E Baldeweg
- Department of Diabetes & Endocrinology, University College London Hospitals NHS Foundation Trust, London, UK; Division of Medicine, Department of Experimental and Translational Medicine, Centre for Obesity and Metabolism, University College London, London, UK
| | - Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Anouk Borg
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Aswin Chari
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Neil L Dorward
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Anne Elserius
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Theofanis Giannis
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Abhiney Jain
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK; Digital Surgery Ltd, Medtronic, London, UK
| | - Hani J Marcus
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK; Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| |
Collapse
|
2
|
Howie EE, Harari R, Dias RD, Wigmore SJ, Skipworth RJE, Yule S. Feasibility of Wearable Sensors to Assess Cognitive Load During Clinical Performance: Lessons Learned and Blueprint for Success. J Surg Res 2024; 302:222-231. [PMID: 39106733 DOI: 10.1016/j.jss.2024.07.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Revised: 05/23/2024] [Accepted: 07/02/2024] [Indexed: 08/09/2024]
Abstract
INTRODUCTION Cognitive load (CogL) is increasingly recognized as an important resource underlying operative performance. Current innovations in surgery aim to develop objective performance metrics via physiological monitoring from wearable digital sensors. Surgeons have access to consumer technology that could measure CogL but need guidance regarding device selection and implementation. To realize the benefits of surgical performance improvement these methods must be feasible, incorporating human factors usability and design principles. This paper aims to evaluate the feasibility of using wearable sensors to assess CogL, identify the benefits and challenges of implementing devices, and develop guidance for surgeons planning to implement wearable devices in their research or practice. METHODS We examined the feasibility of wearable sensors from a series of empirical studies that measured aspects of clinical performance relating to CogL. Across four studies, 84 participants and five sensors were involved in the following clinical settings: (i) real intraoperative surgery; (ii) simulated laparoscopic surgery; and (iii) medical team performance outside the hospital. RESULTS Wearable devices worn on the wrist and chest were found to be comfortable. After a learning curve, electrodermal activity data were easily and reliably collected. Devices using photoplethysmography to determine heart rate variability were significantly limited by movement artifact. There was variable success with electroencephalography devices regarding connectivity, comfort, and usability. CONCLUSIONS It is feasible to use wearable sensors across various clinical settings, including surgery. There are some limitations, and their implementation is context and device dependent. To scale sensor use in clinical research, surgeons must embrace human factors principles to optimize wearability, usability, reliability, and data security.
Collapse
Affiliation(s)
- Emma E Howie
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland.
| | - Ryan Harari
- Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland; STRATUS Centre for Medical Simulation, Brigham & Women's Hospital, Boston, Massachusetts; Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts
| | - Roger D Dias
- Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland; STRATUS Centre for Medical Simulation, Brigham & Women's Hospital, Boston, Massachusetts; Department of Emergency Medicine, Harvard Medical School, Boston, Massachusetts
| | - Stephen J Wigmore
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland
| | - Richard J E Skipworth
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland
| | - Steven Yule
- Clinical Surgery, University of Edinburgh & Royal Infirmary of Edinburgh, Edinburgh, Scotland; Surgical Sabermetrics Laboratory, Usher Institute, University of Edinburgh, Edinburgh, Scotland.
| |
Collapse
|
3
|
Yanik E, Schwaitzberg S, De S. Deep Learning for Video-Based Assessment in Surgery. JAMA Surg 2024; 159:957-958. [PMID: 38837128 DOI: 10.1001/jamasurg.2024.1510] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/06/2024]
Abstract
This surgical innovation explains how applying deep neural networks could ensure the continued use of video-based assessment.
Collapse
Affiliation(s)
- Erim Yanik
- College of Engineering, Florida Agriculture and Mechanical University, Florida State University, Tallahassee
| | - Steven Schwaitzberg
- School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, New York
| | - Suvranu De
- College of Engineering, Florida Agriculture and Mechanical University, Florida State University, Tallahassee
| |
Collapse
|
4
|
Yanik E, Schwaitzberg S, Yang G, Intes X, Norfleet J, Hackett M, De S. One-shot skill assessment in high-stakes domains with limited data via meta learning. Comput Biol Med 2024; 174:108470. [PMID: 38636326 DOI: 10.1016/j.compbiomed.2024.108470] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 04/08/2024] [Accepted: 04/09/2024] [Indexed: 04/20/2024]
Abstract
Deep Learning (DL) has achieved robust competency assessment in various high-stakes fields. However, the applicability of DL models is often hampered by their substantial data requirements and confinement to specific training domains. This prevents them from transitioning to new tasks where data is scarce. Therefore, domain adaptation emerges as a critical element for the practical implementation of DL in real-world scenarios. Herein, we introduce A-VBANet, a novel meta-learning model capable of delivering domain-agnostic skill assessment via one-shot learning. Our methodology has been tested by assessing surgical skills on five laparoscopic and robotic simulators and real-life laparoscopic cholecystectomy. Our model successfully adapted with accuracies up to 99.5 % in one-shot and 99.9 % in few-shot settings for simulated tasks and 89.7 % for laparoscopic cholecystectomy. This study marks the first instance of a domain-agnostic methodology for skill assessment in critical fields setting a precedent for the broad application of DL across diverse real-life domains with limited data.
Collapse
Affiliation(s)
- Erim Yanik
- College of Engineering, Florida A&M University and the Florida State University, USA.
| | | | - Gene Yang
- School of Medicine and Biomedical Sciences, University at Buffalo, USA
| | - Xavier Intes
- Biomedical Engineering Department, Rensselaer Polytechnic Institute, USA
| | - Jack Norfleet
- U.S. Army Combat Capabilities Development Command Soldier Center STTC, USA
| | - Matthew Hackett
- U.S. Army Combat Capabilities Development Command Soldier Center STTC, USA
| | - Suvranu De
- College of Engineering, Florida A&M University and the Florida State University, USA
| |
Collapse
|
5
|
Hashimoto DA, Sambasastry SK, Singh V, Kurada S, Altieri M, Yoshida T, Madani A, Jogan M. A foundation for evaluating the surgical artificial intelligence literature. EUROPEAN JOURNAL OF SURGICAL ONCOLOGY 2024:108014. [PMID: 38360498 DOI: 10.1016/j.ejso.2024.108014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2023] [Revised: 01/06/2024] [Accepted: 02/09/2024] [Indexed: 02/17/2024]
Abstract
With increasing growth in applications of artificial intelligence (AI) in surgery, it has become essential for surgeons to gain a foundation of knowledge to critically appraise the scientific literature, commercial claims regarding products, and regulatory and legal frameworks that govern the development and use of AI. This guide offers surgeons a framework with which to evaluate manuscripts that incorporate the use of AI. It provides a glossary of common terms, an overview of prerequisite knowledge to maximize understanding of methodology, and recommendations on how to carefully consider each element of a manuscript to assess the quality of the data on which an algorithm was trained, the appropriateness of the methodological approach, the potential for reproducibility of the experiment, and the applicability to surgical practice, including considerations on generalizability and scalability.
Collapse
Affiliation(s)
- Daniel A Hashimoto
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA; Global Surgical AI Collaborative, Toronto, ON, USA.
| | - Sai Koushik Sambasastry
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Vivek Singh
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Sruthi Kurada
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Maria Altieri
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA; Global Surgical AI Collaborative, Toronto, ON, USA
| | - Takuto Yoshida
- Surgical AI Research Academy, Department of Surgery, University Health Network, Toronto, ON, USA
| | - Amin Madani
- Global Surgical AI Collaborative, Toronto, ON, USA; Surgical AI Research Academy, Department of Surgery, University Health Network, Toronto, ON, USA
| | - Matjaz Jogan
- Penn Computer Assisted Surgery and Outcomes Laboratory, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
6
|
Mahoney LB, Huang JS, Lightdale JR, Walsh CM. Pediatric endoscopy: how can we improve patient outcomes and ensure best practices? Expert Rev Gastroenterol Hepatol 2024; 18:89-102. [PMID: 38465446 DOI: 10.1080/17474124.2024.2328229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Accepted: 03/05/2024] [Indexed: 03/12/2024]
Abstract
INTRODUCTION Strategies to promote high-quality endoscopy in children require consensus around pediatric-specific quality standards and indicators. Using a rigorous guideline development process, the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN) was developed to support continuous quality improvement efforts within and across pediatric endoscopy services. AREAS COVERED This review presents a framework, informed by the PEnQuIN guidelines, for assessing endoscopist competence, granting procedural privileges, audit and feedback, and for skill remediation, when required. As is critical for promoting quality, PEnQuIN indicators can be benchmarked at the individual endoscopist, endoscopy facility, and endoscopy community levels. Furthermore, efforts to incorporate technologies, including electronic medical records and artificial intelligence, into endoscopic quality improvement processes can aid in creation of large-scale networks to facilitate comparison and standardization of quality indicator reporting across sites. EXPERT OPINION PEnQuIN quality standards and indicators provide a framework for continuous quality improvement in pediatric endoscopy, benefiting individual endoscopists, endoscopy facilities, and the broader endoscopy community. Routine and reliable measurement of data, facilitated by technology, is required to identify and drive improvements in care. Engaging all stakeholders in endoscopy quality improvement processes is crucial to enhancing patient outcomes and establishing best practices for safe, efficient, and effective pediatric endoscopic care.
Collapse
Affiliation(s)
- Lisa B Mahoney
- Division of Gastroenterology, Hepatology and Nutrition, Boston Children's Hospital, Boston, MA, USA
| | - Jeannie S Huang
- Rady Children's Hospital, San Diego, CA and University of California San Diego, La Jolla, CA, USA
| | - Jenifer R Lightdale
- Division of Gastroenterology, Hepatology and Nutrition, Boston Children's Hospital, Boston, MA, USA
| | - Catharine M Walsh
- Division of Gastroenterology, Hepatology and Nutrition and the Research and Learning Institutes, The Hospital for Sick Children, Department of Paediatrics and the Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
7
|
Yule S, Dearani JA, Pugh C. Surgical Instant Replay-A National Video-Based Performance Assessment Toolbox. JAMA Surg 2023; 158:1344-1345. [PMID: 37755836 DOI: 10.1001/jamasurg.2023.1803] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/28/2023]
Abstract
This article discusses the widespread implementation of surgical video replay to improve technical and nontechnical performance of surgeons.
Collapse
Affiliation(s)
- Steven Yule
- School of Surgery, University of Edinburgh, Edinburgh, Scotland
| | - Joseph A Dearani
- Department of Cardiovascular Surgery, Mayo Clinic, Rochester, Minnesota
| | - Carla Pugh
- Department of Surgery, Stanford University School of Medicine, Stanford, California
| |
Collapse
|
8
|
Mahoney LB, Walsh CM, Lightdale JR. Promoting Research that Supports High-Quality Gastrointestinal Endoscopy in Children. Curr Gastroenterol Rep 2023; 25:333-343. [PMID: 37782450 DOI: 10.1007/s11894-023-00897-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/17/2023] [Indexed: 10/03/2023]
Abstract
PURPOSE OF REVIEW Defining and measuring the quality of endoscopic care is a key component of performing gastrointestinal endoscopy in children. The purpose of this review is to discuss quality metrics for pediatric gastrointestinal endoscopy and identify where additional research is needed. RECENT FINDINGS Pediatric-specific standards and indicators were recently defined by the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN) working group through a rigorous guideline consensus process. Although the aim of these guidelines is to facilitate best practices for safe and high-quality gastrointestinal endoscopy in children, they highlight the pressing need to expand upon the body of evidence supporting these standards and indicators as predictors of clinically relevant outcomes. In this review, we propose and discuss ideas for several high-yield research topics to engage pediatric endoscopists and promote best practices in pediatric endoscopy.
Collapse
Affiliation(s)
- Lisa B Mahoney
- Division of Gastroenterology, Hepatology and Nutrition, Boston Children's Hospital, 300 Longwood Ave, Boston, MA, 02115, USA
| | - Catharine M Walsh
- Division of Gastroenterology, Hepatology and Nutrition and the Research and Learning Institutes, The Hospital for Sick Children, Department of Paediatrics and the Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Jenifer R Lightdale
- Division of Gastroenterology, Hepatology and Nutrition, Boston Children's Hospital, 300 Longwood Ave, Boston, MA, 02115, USA.
| |
Collapse
|
9
|
Toale C, O'Byrne A, Morris M, Kavanagh DO. Characterizing individual trainee learning curves in surgical training: Challenges and opportunities. Surgeon 2023; 21:285-288. [PMID: 36446700 DOI: 10.1016/j.surge.2022.11.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Accepted: 11/09/2022] [Indexed: 11/27/2022]
Abstract
The surgical learning curve is an observable and measurable phenomenon. In the era of competency-based approaches to surgical training, monitoring the trajectory of individual trainee competence attainment could represent a meaningful method of formative and summative assessment. While technology can assist this approach, a number of significant barriers to the implementation of such assessment methods remain, including: accurate data collection, standard setting, and reliable assessment. Translating individual learning curve data into quantifiable case minimum targets in training poses further difficulties, and may not be possible for all procedures, particularly those that are less frequently performed and assessed. In spite of these challenges, significant benefits could be realized through an individualized approach to competency assessment using trainee learning curve data. Tracking competence acquisition against criterion-referenced standards could allow for targeted training and remediation, conforming with modern theories of adult education and empowering trainees to take control of their own learning. Learning curve data could also be used to assess the effects of educational interventions such as simulation-based training on subsequent competence acquisition rates. Ultimately, the individual learning curves of trainees could be used to inform personalised decisions regarding entrustment, credentialing, and certification, allowing training programmes to move beyond minimum operative experience targets as a crude proxy measure of competence.
Collapse
Affiliation(s)
- C Toale
- Department of Surgical Affairs, Royal College of Surgeons in Ireland, Ireland.
| | - A O'Byrne
- School of Medicine, Trinity College Dublin, The University of Dublin, Ireland; Department of Surgery, Tallaght University Hospital, Dublin, Ireland
| | - M Morris
- Department of Surgical Affairs, Royal College of Surgeons in Ireland, Ireland
| | - D O Kavanagh
- Department of Surgical Affairs, Royal College of Surgeons in Ireland, Ireland; Department of Surgery, Tallaght University Hospital, Dublin, Ireland
| |
Collapse
|
10
|
Gunn EGM, Ambler OC, Nallapati SC, Smink DS, Tambyraja AL, Yule S. Coaching with audiovisual technology in acute-care hospital settings: systematic review. BJS Open 2023; 7:zrad017. [PMID: 37794777 PMCID: PMC10551776 DOI: 10.1093/bjsopen/zrad017] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Accepted: 01/24/2023] [Indexed: 10/06/2023] Open
Abstract
BACKGROUND Surgical coaching programmes are a means of improving surgeon performance. Embedded audiovisual technology has the potential to further enhance participant benefit and scalability of coaching. The objective of this systematic review was to evaluate how audiovisual technology has augmented coaching in the acute-care hospital setting and to characterize its impact on outcomes. METHODS A systematic review was conducted, searching PubMed, Ovid MEDLINE, Embase, PsycInfo, and CINAHL databases using PRISMA. Eligible studies described a coaching programme that utilized audiovisual technology, involved at least one coach-coachee interaction, and included healthcare professionals from the acute-care hospital environment. The risk of bias 2 tool and grading of recommendations, assessment, development, and evaluations (GRADE) framework were used to evaluate studies. Synthesis without meta-analysis was performed, creating harvest plots of three coaching outcomes: technical skills, self-assessment/feedback, and non-technical skills. RESULTS Of 10 458 abstracts screened, 135 full texts were reviewed, and 21 studies identified for inclusion. Seventeen studies were conducted within surgical specialties and six classes of audiovisual technology were utilized. An overall positive direction of effect was demonstrated for studies measuring improvement of either technical skills or non-technical skills. Direction of effect for self-assessment/feedback was weakly positive. CONCLUSION Audiovisual technology has been used successfully in coaching programmes within acute-care hospital settings to facilitate or assess coaching, with a positive impact on outcome measures. Future studies may address the additive benefits of video over in-person observation and enhance the certainty of evidence that coaching impacts on surgeon performance, surgeon well-being, and patient outcomes.
Collapse
Affiliation(s)
- Eilidh G M Gunn
- Department of Vascular Surgery, NHS Lothian, Royal Infirmary of Edinburgh, Edinburgh, UK
- Department of Clinical Surgery, University of Edinburgh, Edinburgh, UK
| | - Olivia C Ambler
- Department of Vascular Surgery, NHS Lothian, Royal Infirmary of Edinburgh, Edinburgh, UK
| | - Siri C Nallapati
- Edinburgh Medical School, College of Medicine and Veterinary Medicine, The University of Edinburgh, Edinburgh, UK
| | - Douglas S Smink
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Andrew L Tambyraja
- Department of Vascular Surgery, NHS Lothian, Royal Infirmary of Edinburgh, Edinburgh, UK
- Department of Clinical Surgery, University of Edinburgh, Edinburgh, UK
| | - Steven Yule
- Department of Clinical Surgery, University of Edinburgh, Edinburgh, UK
| |
Collapse
|
11
|
Montgomery KB, Lindeman B. Using Graduating Surgical Resident Milestone Ratings to Predict Patient Outcomes: A Blunt Instrument for a Complex Problem. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:765-768. [PMID: 36745875 PMCID: PMC10329982 DOI: 10.1097/acm.0000000000005165] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
In 2013, U.S. general surgery residency programs implemented a milestones assessment framework in an effort to incorporate more competency-focused evaluation methods. Developed by a group of surgical education leaders and other stakeholders working with the Accreditation Council for Graduate Medical Education and recently updated in a version 2.0, the surgery milestones framework is centered around 6 "core competencies": patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. While prior work has focused on the validity of milestones as a measure of resident performance, associations between general surgery resident milestone ratings and their post-training patient outcomes have only recently been explored in an analysis in this issue of Academic Medicine by Kendrick et al. Despite their well-designed efforts to tackle this complex problem, no relationships were identified. This accompanying commentary discusses the broader implications for the use of milestone ratings beyond their intended application, alternative assessment methods, and the challenges of developing predictive assessments in the complex setting of surgical care. Although milestone ratings have not been shown to provide the specificity needed to predict clinical outcomes in the complex settings studied by Kendrick et al, hope remains that utilization of other outcomes, assessment frameworks, and data analytic tools could augment these models and further our progress toward a predictive assessment in surgical education. Evaluation of residents in general surgery residency programs has grown both more sophisticated and complicated in the setting of increasing patient and case complexity, constraints on time, and regulation of resident supervision in the operating room. Over the last decade, surgical education research efforts related to resident assessment have focused on measuring performance through accurate and reproducible methods with evidence for their validity, as well as on attempting to refine decision making about resident preparedness for unsupervised practice.
Collapse
Affiliation(s)
- Kelsey B Montgomery
- K.B. Montgomery is a general surgery resident, Department of Surgery, University of Alabama at Birmingham, Birmingham, Alabama; ORCID: https://orcid.org/0000-0002-1284-1830
| | - Brenessa Lindeman
- B. Lindeman is associate professor, Department of Surgery, and assistant dean, Graduate Medical Education, University of Alabama at Birmingham, Birmingham, Alabama
| |
Collapse
|
12
|
Wu S, Chen Z, Liu R, Li A, Cao Y, Wei A, Liu Q, Liu J, Wang Y, Jiang J, Ying Z, An J, Peng B, Wang X. SurgSmart: an artificial intelligent system for quality control in laparoscopic cholecystectomy: an observational study. Int J Surg 2023; 109:1105-1114. [PMID: 37039533 PMCID: PMC10389595 DOI: 10.1097/js9.0000000000000329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Accepted: 02/22/2023] [Indexed: 04/12/2023]
Abstract
BACKGROUND The rate of bile duct injury in laparoscopic cholecystectomy (LC) continues to be high due to low critical view of safety (CVS) achievement and the absence of an effective quality control system. The development of an intelligent system enables the automatic quality control of LC surgery and, eventually, the mitigation of bile duct injury. This study aims to develop an intelligent surgical quality control system for LC and using the system to evaluate LC videos and investigate factors associated with CVS achievement. MATERIALS AND METHODS SurgSmart, an intelligent system capable of recognizing surgical phases, disease severity, critical division action, and CVS automatically, was developed using training datasets. SurgSmart was also applied in another multicenter dataset to validate its application and investigate factors associated with CVS achievement. RESULTS SurgSmart performed well in all models, with the critical division action model achieving the highest overall accuracy (98.49%), followed by the disease severity model (95.45%) and surgical phases model (88.61%). CVSI, CVSII, and CVSIII had an accuracy of 80.64, 97.62, and 78.87%, respectively. CVS was achieved in 4.33% in the system application dataset. In addition, the analysis indicated that surgeons at a higher hospital level had a higher CVS achievement rate. However, there was still considerable variation in CVS achievement among surgeons in the same hospital. CONCLUSIONS SurgSmart, the surgical quality control system, performed admirably in our study. In addition, the system's initial application demonstrated its broad potential for use in surgical quality control.
Collapse
Affiliation(s)
- Shangdi Wu
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Zixin Chen
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Runwen Liu
- ChengDu Withai Innovations Technology Company
| | - Ang Li
- Division of Pancreatic Surgery, Department of General Surgery
- Guang’an People’s Hospital, Guang’an, Sichuan Province, China
| | - Yu Cao
- Operating Room
- West China School of Nursing, Sichuan University
| | - Ailin Wei
- Guang’an People’s Hospital, Guang’an, Sichuan Province, China
| | | | - Jie Liu
- ChengDu Withai Innovations Technology Company
| | - Yuxian Wang
- ChengDu Withai Innovations Technology Company
| | - Jingwen Jiang
- West China Biomedical Big Data Center, West China Hospital of Sichuan University
- Med-X Center for Informatics, Sichuan University, Chengdu
| | - Zhiye Ying
- West China Biomedical Big Data Center, West China Hospital of Sichuan University
- Med-X Center for Informatics, Sichuan University, Chengdu
| | - Jingjing An
- Operating Room
- West China School of Nursing, Sichuan University
| | - Bing Peng
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| | - Xin Wang
- Division of Pancreatic Surgery, Department of General Surgery
- West China School of Medicine
| |
Collapse
|
13
|
Axelrod C, Walker M, Swift B, Farrugia M, Sobel M, Tannenbaum E. What is the role of video-based assessment of surgical skill in residency training? A qualitative study of trainee and faculty perspectives. JOURNAL OF OBSTETRICS AND GYNAECOLOGY CANADA 2023:S1701-2163(23)00312-2. [PMID: 37120146 DOI: 10.1016/j.jogc.2023.04.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 04/13/2023] [Accepted: 04/14/2023] [Indexed: 05/01/2023]
Abstract
PURPOSE Surgical training programs are starting to experiment with video-based assessment (VBA) of residents' technical skills. VBA may limit the effect of interpersonal bias on assessment scores. However, before VBA is implemented widely, stakeholders' perceptions ought to be explored, including potential benefits and challenges. METHODS Using the qualitative methods of hermeneutical phenomenology, the authors explored both trainee and faculty educators' perspectives on VBA using semi-structured interviews. Participants were recruited from the Department of Obstetrics and Gynecology at the XX. Data underwent thematic analysis and was validated by investigator and theoretical triangulation. RESULTS The authors interviewed nine physicians (5 faculty and 4 residents). Four dominant themes were identified, including: advantages compared to traditional methods, the role of feedback and coaching, challenges integrating VBA, and considerations for implementation. CONCLUSIONS Surgical trainees and faculty feel that VBA is a worthy tool to advance equity and fairness in assessment, but felt it was better as a vehicle for feedback and coaching. VBA cannot be used as a standalone assessment metric without additional evidence for its validity. If implemented, residency programs can use VBA as an adjunct to other evaluation measures to facilitate coaching, provide asynchronous feedback, and limit assessment bias.
Collapse
Affiliation(s)
| | - Melissa Walker
- University of Toronto, Temerty Faculty of Medicine, Toronto, ON
| | - Brenna Swift
- University of Toronto, Temerty Faculty of Medicine, Toronto, ON
| | | | - Mara Sobel
- University of Toronto, Temerty Faculty of Medicine, Toronto, ON
| | - Evan Tannenbaum
- University of Toronto, Temerty Faculty of Medicine, Toronto, ON
| |
Collapse
|
14
|
Pryor AD, Lendvay T, Jones A, Ibáñez B, Pugh C. An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery. Ann Surg 2023; 277:591-595. [PMID: 36645875 DOI: 10.1097/sla.0000000000005804] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
OBJECTIVE The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills. BACKGROUND Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery. METHODS Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS. RESULTS To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns. CONCLUSIONS VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.
Collapse
Affiliation(s)
- Aurora D Pryor
- Department of Surgery, Hofstra University/Northwell Health, Long Island, NY
| | - Thomas Lendvay
- Department of Urology, University of Washington, Seattle, WA
| | | | | | - Carla Pugh
- Department of Surgery, Stanford University, Palo Alto, CA
| |
Collapse
|
15
|
Taylor C, Ikiroma A, Crowe A, Felix DH, Grant G, Mitchell L, Ross T, Saunderson M, Young L. Using live stream technology to conduct workplace observation assessment of trainee dental nurses: an evaluation of effectiveness and user experience. BDJ Open 2023; 9:4. [PMID: 36750549 PMCID: PMC9904864 DOI: 10.1038/s41405-023-00132-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Revised: 12/22/2022] [Accepted: 01/13/2023] [Indexed: 02/09/2023] Open
Abstract
AIM/OBJECTIVE This study evaluates the effectiveness and users' experience of using live stream technology to conduct workplace observation assessments of trainee dental nurses. Information on the usability, accessibility, and general satisfaction of this technological technique were collected. MATERIALS AND METHODS This nationwide cross-sectional survey was conducted in Scotland and included one focus group and three online questionnaires with qualitative and quantitative questions. The quantitative responses were described using standard descriptive analysis, while the quantitative data were investigated using thematic analysis. RESULTS Eighty-one trainee dental nurses, 35 clinicians and 19 assessors participated in this study. Live stream observation was generally well received by the trainee dental nurses and clinicians, who thought that it had helped increase their confidence to perform practical skills. The assessors also stated that overall satisfaction was high, and that live stream observation met their expectations for efficacy. However, several technical challenges, such as network issues were brought up by responders. CONCLUSION This study provides evidence that workplace observation assessments can be performed in the future by using live stream technology. However, additional investigation and comparison will aid in determining the most effective way of using this approach and providing feedback to promote learning among dental trainees.
Collapse
Affiliation(s)
- Caroline Taylor
- Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK.
| | - Adalia Ikiroma
- grid.451102.30000 0001 0164 4922Dental Clinical Effectiveness Workstream, NHS Education for Scotland, Edinburgh, UK
| | - Anne Crowe
- grid.451102.30000 0001 0164 4922Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK
| | - David H Felix
- grid.451102.30000 0001 0164 4922Dental Directorate, NHS Education for Scotland, Edinburgh, UK
| | - Gillian Grant
- grid.451102.30000 0001 0164 4922Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK
| | - Lucy Mitchell
- grid.451102.30000 0001 0164 4922Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK
| | - Teresa Ross
- grid.451102.30000 0001 0164 4922Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK
| | - Margaret Saunderson
- grid.451102.30000 0001 0164 4922Dental Care Professionals Workstream, NHS Education for Scotland, Edinburgh, UK
| | - Linda Young
- grid.451102.30000 0001 0164 4922Dental Clinical Effectiveness Workstream, NHS Education for Scotland, Edinburgh, UK
| |
Collapse
|
16
|
Video-based formative and summative assessment of surgical tasks using deep learning. Sci Rep 2023; 13:1038. [PMID: 36658186 PMCID: PMC9852463 DOI: 10.1038/s41598-022-26367-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2022] [Accepted: 12/13/2022] [Indexed: 01/20/2023] Open
Abstract
To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated-none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA is manual, time-intensive, and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds and low-stakes formative assessment to guide surgical skill acquisition. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance. Hence, the DL model paves the way for the quantitative and reproducible evaluation of surgical tasks from videos with the potential for broad dissemination in surgical training, certification, and credentialing.
Collapse
|
17
|
Sankaranarayanan G, Parker LM, Jacinto K, Demirel D, Halic T, De S, Fleshman JW. Development and Validation of Task-Specific Metrics for the Assessment of Linear Stapler-Based Small Bowel Anastomosis. J Am Coll Surg 2022; 235:881-893. [PMID: 36102520 PMCID: PMC9669227 DOI: 10.1097/xcs.0000000000000389] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
INTRODUCTION Task-specific metrics facilitate the assessment of surgeon performance. This 3-phased study was designed to (1) develop task-specific metrics for stapled small bowel anastomosis, (2) obtain expert consensus on the appropriateness of the developed metrics, and (3) establish its discriminant validity. METHODS In Phase I, a hierarchical task analysis was used to develop the metrics. In Phase II, a survey of expert colorectal surgeons established the importance of the developed metrics. In Phase III, to establish discriminant validity, surgical trainees and surgeons, divided into novice and experienced groups, constructed a side-to-side anastomosis on porcine small bowel using a linear cutting stapler. The participants' performances were videotaped and rated by 2 independent observers. Partial least squares regression was used to compute the weights for the task-specific metrics to obtain weighted total score. RESULTS In Phase II, a total of 45 colorectal surgeons were surveyed: 28 with more than 15 years, 13 with 5 to 15 years, and 4 with less than 5 years of experience. The consensus was obtained on all the task-specific metrics in the more experienced groups. In Phase III, 20 subjects participated equally in both groups. The experienced group performed better than the novice group regardless of the rating scale used: global rating scale (p = 0.009) and the task-specific metrics (p = 0.012). After partial least squares regression, the weighted task-specific metric score continued to show that the experienced group performed better (p < 0.001). CONCLUSION Task-specific metric items were developed based on expert consensus and showed good discriminant validity compared with a global rating scale between experienced and novice operators. These items can be used for evaluating technical skills in a stapled small bowel anastomosis model.
Collapse
Affiliation(s)
| | - Lisa M Parker
- Department of Surgery, Baylor University Medical Center, Dallas, TX
| | - Kimberly Jacinto
- Department of Surgery, Baylor University Medical Center, Dallas, TX
| | - Doga Demirel
- Department of Computer Science, Florida Polytechnic University, Lakeland, FL
| | - Tansel Halic
- Department of Computer Science, University of Central Arkansas, Conway, AR
| | - Suvranu De
- Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Troy, NY
| | - James W Fleshman
- Department of Surgery, Baylor University Medical Center, Dallas, TX
| |
Collapse
|
18
|
Assessing VATS competence based on simulated lobectomies of all five lung lobes. Surg Endosc 2022; 36:8067-8075. [PMID: 35467146 DOI: 10.1007/s00464-022-09235-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2021] [Accepted: 04/02/2022] [Indexed: 01/06/2023]
Abstract
OBJECTIVES To determine the number of procedures and expert raters necessary to provide a reliable assessment of competence in Video-Assisted Thoracoscopic Surgery (VATS) lobectomy. METHODS Three randomly selected VATS lobectomies were performed on a virtual reality simulator by participants with varying experience in VATS. Video recordings of the procedures were independently rated by three blinded VATS experts using a modified VATS lobectomy assessment tool (VATSAT). The unitary framework of validity was used to describe validity evidence, and generalizability theory was used to explore the reliability of different assessment options. RESULTS Forty-one participants (22 novices, 10 intermediates, and 9 experienced) performed a total of 123 lobectomies. Internal consistency reliability, inter-rater reliability, and test-retest reliability were 0.94, 0.85, and 0.90, respectively. Generalizability theory found that a minimum of two procedures and four raters or three procedures and three raters were needed to ensure the overall reliability of 0.8. ANOVA showed significant differences in test scores between the three groups (P < 0.001). A pass/fail level of 19 out of 25 points was established using the contrasting groups' standard setting method, leaving one false positive (one novice passed) and zero false negatives (all experienced passed). CONCLUSION We demonstrated validity evidence for a VR simulator test with different lung lobes, and a credible pass/fail level was identified. Our results can be used to implement a standardized mastery learning training program for trainees in VATS lobectomies that ensures that everyone reaches basic competency before performing supervised operations on patients.
Collapse
|
19
|
Mascagni P, Alapatt D, Sestini L, Altieri MS, Madani A, Watanabe Y, Alseidi A, Redan JA, Alfieri S, Costamagna G, Boškoski I, Padoy N, Hashimoto DA. Computer vision in surgery: from potential to clinical value. NPJ Digit Med 2022; 5:163. [PMID: 36307544 PMCID: PMC9616906 DOI: 10.1038/s41746-022-00707-5] [Citation(s) in RCA: 35] [Impact Index Per Article: 17.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Accepted: 10/10/2022] [Indexed: 11/09/2022] Open
Abstract
Hundreds of millions of operations are performed worldwide each year, and the rising uptake in minimally invasive surgery has enabled fiber optic cameras and robots to become both important tools to conduct surgery and sensors from which to capture information about surgery. Computer vision (CV), the application of algorithms to analyze and interpret visual data, has become a critical technology through which to study the intraoperative phase of care with the goals of augmenting surgeons' decision-making processes, supporting safer surgery, and expanding access to surgical care. While much work has been performed on potential use cases, there are currently no CV tools widely used for diagnostic or therapeutic applications in surgery. Using laparoscopic cholecystectomy as an example, we reviewed current CV techniques that have been applied to minimally invasive surgery and their clinical applications. Finally, we discuss the challenges and obstacles that remain to be overcome for broader implementation and adoption of CV in surgery.
Collapse
Affiliation(s)
- Pietro Mascagni
- Gemelli Hospital, Catholic University of the Sacred Heart, Rome, Italy.
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France.
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada.
| | - Deepak Alapatt
- ICube, University of Strasbourg, CNRS, IHU, Strasbourg, France
| | - Luca Sestini
- ICube, University of Strasbourg, CNRS, IHU, Strasbourg, France
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, Italy
| | - Maria S Altieri
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA
| | - Amin Madani
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada
- Department of Surgery, University Health Network, Toronto, ON, Canada
| | - Yusuke Watanabe
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada
- Department of Surgery, University of Hokkaido, Hokkaido, Japan
| | - Adnan Alseidi
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada
- Department of Surgery, University of California San Francisco, San Francisco, CA, USA
| | - Jay A Redan
- Department of Surgery, AdventHealth-Celebration Health, Celebration, FL, USA
| | - Sergio Alfieri
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Guido Costamagna
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Ivo Boškoski
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Nicolas Padoy
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France
- ICube, University of Strasbourg, CNRS, IHU, Strasbourg, France
| | - Daniel A Hashimoto
- Global Surgical Artificial Intelligence Collaborative, Toronto, ON, Canada
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA
| |
Collapse
|
20
|
Vedula SS, Ghazi A, Collins JW, Pugh C, Stefanidis D, Meireles O, Hung AJ, Schwaitzberg S, Levy JS, Sachdeva AK. Artificial Intelligence Methods and Artificial Intelligence-Enabled Metrics for Surgical Education: A Multidisciplinary Consensus. J Am Coll Surg 2022; 234:1181-1192. [PMID: 35703817 PMCID: PMC10634198 DOI: 10.1097/xcs.0000000000000190] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND Artificial intelligence (AI) methods and AI-enabled metrics hold tremendous potential to advance surgical education. Our objective was to generate consensus guidance on specific needs for AI methods and AI-enabled metrics for surgical education. STUDY DESIGN The study included a systematic literature search, a virtual conference, and a 3-round Delphi survey of 40 representative multidisciplinary stakeholders with domain expertise selected through purposeful sampling. The accelerated Delphi process was completed within 10 days. The survey covered overall utility, anticipated future (10-year time horizon), and applications for surgical training, assessment, and feedback. Consensus was agreement among 80% or more respondents. We coded survey questions into 11 themes and descriptively analyzed the responses. RESULTS The respondents included surgeons (40%), engineers (15%), affiliates of industry (27.5%), professional societies (7.5%), regulatory agencies (7.5%), and a lawyer (2.5%). The survey included 155 questions; consensus was achieved on 136 (87.7%). The panel listed 6 deliverables each for AI-enhanced learning curve analytics and surgical skill assessment. For feedback, the panel identified 10 priority deliverables spanning 2-year (n = 2), 5-year (n = 4), and 10-year (n = 4) timeframes. Within 2 years, the panel expects development of methods to recognize anatomy in images of the surgical field and to provide surgeons with performance feedback immediately after an operation. The panel also identified 5 essential that should be included in operative performance reports for surgeons. CONCLUSIONS The Delphi panel consensus provides a specific, bold, and forward-looking roadmap for AI methods and AI-enabled metrics for surgical education.
Collapse
Affiliation(s)
- S Swaroop Vedula
- From the Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD (Vedula)
| | - Ahmed Ghazi
- the Department of Urology, University of Rochester Medical Center, Rochester, NY (Ghazi)
| | - Justin W Collins
- the Division of Surgery and Interventional Science, Research Department of Targeted Intervention and Wellcome/Engineering and Physical Sciences Research Council Center for Interventional and Surgical Sciences, University College London, London, UK (Collins)
| | - Carla Pugh
- the Department of Surgery, Stanford University, Stanford, CA (Pugh)
| | | | - Ozanan Meireles
- the Department of Surgery, Massachusetts General Hospital, Boston, MA (Meireles)
| | - Andrew J Hung
- the Artificial Intelligence Center at University of Southern California Urology, Department of Urology, University of Southern California, Los Angeles, CA (Hung)
| | | | - Jeffrey S Levy
- Institute for Surgical Excellence, Washington, DC (Levy)
| | - Ajit K Sachdeva
- Division of Education, American College of Surgeons, Chicago, IL (Sachdeva)
| |
Collapse
|
21
|
Outcomes of the First Virtual General Surgery Certifying Exam of the American Board of Surgery. Ann Surg 2021; 274:467-472. [PMID: 34183516 DOI: 10.1097/sla.0000000000004988] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022]
Abstract
OBJECTIVE To Study the Outcomes of the First Virtual General Surgery Certifying Exam of the American Board of Surgery. SUMMARY OF BACKGROUND DATA The ABS General Surgery CE is normally an in-person oral examination. Due to the COVID-19 outbreak, the ABS was required to reschedule these. After 2 small pilots, the CE's October administration represented the first large-scale remote virtual exam. The purpose of this report is to compare the outcomes of this virtual and the previous in-person CEs. METHODS CE candidates were asked to provide feedback on their experience via a survey. The passing rate was compared to the 1025 candidates who took the 2019-2020 in-person CEs. RESULTS Of the 308 candidates who registered for the virtual CE, 306 completed the exam (99.4%) and 188 completed the survey (61.4%). The majority had a very positive experience. They rated the virtual CE as very good/excellent in security (90%), ease of exam platform (77%), audio quality (71%), video quality (69%), and overall satisfaction (86%). Notably, when asked their preference, 78% preferred the virtual exam. There were no differences in the passing rates between the virtual or in-person exams. CONCLUSIONS The first virtual CE by the ABS was completed using available internet technology. There was high satisfaction, with the majority preferring the virtual platform. Compared to past in-person CEs, there was no difference in outcomes as measured by passing rates. These data suggest that expansion of the virtual CE may be desirable.
Collapse
|
22
|
Can Deep Learning Algorithms Help Identify Surgical Workflow and Techniques? J Surg Res 2021; 268:318-325. [PMID: 34399354 DOI: 10.1016/j.jss.2021.07.003] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Revised: 07/14/2021] [Accepted: 07/15/2021] [Indexed: 12/21/2022]
Abstract
BACKGROUND Surgical videos are now being used for performance review and educational purposes; however, broad use is still limited due to time constraints. To make video review more efficient, we implemented Artificial Intelligence (AI) algorithms to detect surgical workflow and technical approaches. METHODS Participants (N = 200) performed a simulated open bowel repair. The operation included two major phases: (1) Injury Identification and (2) Suture Repair. Accordingly, a phase detection algorithm (MobileNetV2+GRU) was implemented to automatically detect the two phases using video data. In addition, participants were noted to use three different technical approaches when running the bowel: (1) use of both hands, (2) use of one hand and one tool, or (3) use of two tools. To discern the three technical approaches, an object detection (YOLOv3) algorithm was implemented to recognize objects that were commonly used during the Injury Identification phase (hands versus tools). RESULTS The phase detection algorithm achieved high precision (recall) when segmenting the two phases: Injury Identification (86 ± 9% [81 ± 12%]) and Suture Repair (81 ± 6% [81 ± 16%]). When evaluating three technical approaches in running the bowel, the object detection algorithm achieved high average precisions (Hands [99.32%] and Tools [94.47%]). The three technical approaches showed no difference in execution time (Kruskal-Wallis Test: P= 0.062) or injury identification (not missing an injury) (Chi-squared: P= 0.998). CONCLUSIONS The AI algorithms showed high precision when segmenting surgical workflow and identifying technical approaches. Automation of these techniques for surgical video databases has great potential to facilitate efficient performance review.
Collapse
|
23
|
Mohamadipanah H, Wise B, Witt A, Goll C, Yang S, Perumalla C, Huemer K, Kearse L, Pugh C. Performance assessment using sensor technology. J Surg Oncol 2021; 124:200-215. [PMID: 34245582 PMCID: PMC8855881 DOI: 10.1002/jso.26519] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Revised: 04/13/2021] [Accepted: 04/14/2021] [Indexed: 11/10/2022]
Abstract
Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills.
Collapse
Affiliation(s)
- Hossein Mohamadipanah
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Brett Wise
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Anna Witt
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Cassidi Goll
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Su Yang
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Calvin Perumalla
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Kayla Huemer
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - LaDonna Kearse
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| | - Carla Pugh
- Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
24
|
Ward TM, Mascagni P, Madani A, Padoy N, Perretta S, Hashimoto DA. Surgical data science and artificial intelligence for surgical education. J Surg Oncol 2021; 124:221-230. [PMID: 34245578 DOI: 10.1002/jso.26496] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2021] [Revised: 03/29/2021] [Accepted: 04/02/2021] [Indexed: 11/11/2022]
Abstract
Surgical data science (SDS) aims to improve the quality of interventional healthcare and its value through the capture, organization, analysis, and modeling of procedural data. As data capture has increased and artificial intelligence (AI) has advanced, SDS can help to unlock augmented and automated coaching, feedback, assessment, and decision support in surgery. We review major concepts in SDS and AI as applied to surgical education and surgical oncology.
Collapse
Affiliation(s)
- Thomas M Ward
- Department of Surgery, Surgical AI & Innovation Laboratory, Massachusetts General Hospital, Boston, Massachusetts
| | - Pietro Mascagni
- ICube, University of Strasbourg, CNRS, France.,Fondazione Policlinico A. Gemelli IRCCS, Rome, Italy.,IHU Strasbourg, Strasbourg, France
| | - Amin Madani
- Department of Surgery, University Health Network, Toronto, Canada
| | - Nicolas Padoy
- ICube, University of Strasbourg, CNRS, France.,IHU Strasbourg, Strasbourg, France
| | | | - Daniel A Hashimoto
- Department of Surgery, Surgical AI & Innovation Laboratory, Massachusetts General Hospital, Boston, Massachusetts
| |
Collapse
|
25
|
Cheng K, You J, Wu S, Chen Z, Zhou Z, Guan J, Peng B, Wang X. Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc 2021; 36:3160-3168. [PMID: 34231066 DOI: 10.1007/s00464-021-08619-3] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Accepted: 06/14/2021] [Indexed: 02/08/2023]
Abstract
BACKGROUND Artificial intelligence and computer vision have revolutionized laparoscopic surgical video analysis. However, there is no multi-center study focused on deep learning-based laparoscopic cholecystectomy phases recognizing. This work aims to apply artificial intelligence in recognizing and analyzing phases in laparoscopic cholecystectomy videos from multiple centers. METHODS This observational cohort-study included 163 laparoscopic cholecystectomy videos collected from four medical centers. Videos were labeled by surgeons and a deep-learning model was developed based on 90 videos. Thereafter, the performance of the model was tested in additional ten videos by comparing it with the annotated ground truth of the surgeon. Deep-learning models were trained to identify laparoscopic cholecystectomy phases. The performance of models was measured using precision, recall, F1 score, and overall accuracy. With a high overall accuracy of the model, additional 63 videos as an analysis set were analyzed by the model to identify different phases. RESULTS Mean concordance correlation coefficient for annotations of the surgeons across all operative phases was 92.38%. Also, the overall phase recognition accuracy of laparoscopic cholecystectomy by the model was 91.05%. In the analysis set, there was an average surgery time of 2195 ± 896 s, with a huge individual variance of different surgical phases. Notably, laparoscopic cholecystectomy in acute cholecystitis cases had prolonged overall durations, and the surgeon would spend more time in mobilizing the hepatocystic triangle phase. CONCLUSION A deep-learning model based on multiple centers data can identify phases of laparoscopic cholecystectomy with a high degree of accuracy. With continued refinements, artificial intelligence could be utilized in huge data surgery analysis to achieve clinically relevant future applications.
Collapse
Affiliation(s)
- Ke Cheng
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China.,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China
| | - Jiaying You
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China.,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China
| | - Shangdi Wu
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China.,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China
| | - Zixin Chen
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China.,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China
| | - Zijian Zhou
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China.,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China
| | - Jingye Guan
- ChengDu Withai Innovations Technology Company, Chengdu, China
| | - Bing Peng
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China. .,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China.
| | - Xin Wang
- West China School of Medicine, West China Hospital of Sichuan University, Chengdu, China. .,Department of Pancreatic Surgery, West China Hospital, Sichuan University, No. 37, Guoxue Alley, Chengdu, 610041, Sichuan Province, China.
| |
Collapse
|
26
|
Affiliation(s)
- Daniel A Hashimoto
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114
| |
Collapse
|
27
|
Mascagni P, Alapatt D, Urade T, Vardazaryan A, Mutter D, Marescaux J, Costamagna G, Dallemagne B, Padoy N. A Computer Vision Platform to Automatically Locate Critical Events in Surgical Videos: Documenting Safety in Laparoscopic Cholecystectomy. Ann Surg 2021; 274:e93-e95. [PMID: 33417329 DOI: 10.1097/sla.0000000000004736] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
OBJECTIVE The aim of this study was to develop a computer vision platform to automatically locate critical events in surgical videos and provide short video clips documenting the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). BACKGROUND Intraoperative events are typically documented through operator-dictated reports that do not always translate the operative reality. Surgical videos provide complete information on surgical procedures, but the burden associated with storing and manually analyzing full-length videos has so far limited their effective use. METHODS A computer vision platform named EndoDigest was developed and used to analyze LC videos. The mean absolute error (MAE) of the platform in automatically locating the manually annotated time of the cystic duct division in full-length videos was assessed. The relevance of the automatically extracted short video clips was evaluated by calculating the percentage of video clips in which the CVS was assessable by surgeons. RESULTS A total of 155 LC videos were analyzed: 55 of these videos were used to develop EndoDigest, whereas the remaining 100 were used to test it. The time of the cystic duct division was automatically located with a MAE of 62.8 ± 130.4 seconds (1.95% of full-length video duration). CVS was assessable in 91% of the 2.5 minutes long video clips automatically extracted from the considered test procedures. CONCLUSIONS Deep learning models for workflow analysis can be used to reliably locate critical events in surgical videos and document CVS in LC. Further studies are needed to assess the clinical impact of surgical data science solutions for safer laparoscopic cholecystectomy.
Collapse
Affiliation(s)
- Pietro Mascagni
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, France
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Deepak Alapatt
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, France
| | - Takeshi Urade
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France
| | | | - Didier Mutter
- IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France
- Institute for Research against Digestive Cancer (IRCAD), Strasbourg, France
- Department of Digestive and Endocrine Surgery, University of Strasbourg, Strasbourg, France
| | - Jacques Marescaux
- Institute for Research against Digestive Cancer (IRCAD), Strasbourg, France
| | - Guido Costamagna
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Bernard Dallemagne
- Institute for Research against Digestive Cancer (IRCAD), Strasbourg, France
- Department of Digestive and Endocrine Surgery, University of Strasbourg, Strasbourg, France
| | - Nicolas Padoy
- ICube, University of Strasbourg, CNRS, IHU Strasbourg, France
| |
Collapse
|
28
|
Ward TM, Fer DM, Ban Y, Rosman G, Meireles OR, Hashimoto DA. Challenges in surgical video annotation. Comput Assist Surg (Abingdon) 2021; 26:58-68. [PMID: 34126014 DOI: 10.1080/24699322.2021.1937320] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022] Open
Abstract
Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.
Collapse
Affiliation(s)
- Thomas M Ward
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| | - Danyal M Fer
- Department of Surgery, University of California San Francisco East Bay, Hayward, CA, USA
| | - Yutong Ban
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA.,Distributed Robotics Laboratory, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Guy Rosman
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA.,Distributed Robotics Laboratory, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Ozanan R Meireles
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| | - Daniel A Hashimoto
- Surgical AI & Innovation Laboratory, Department of Surgery, Massachusetts General Hospital, Boston, MA, USA
| |
Collapse
|
29
|
Yule S, Janda A, Likosky DS. Surgical Sabermetrics: Applying Athletics Data Science to Enhance Operative Performance. ANNALS OF SURGERY OPEN 2021; 2:e054. [PMID: 34179890 PMCID: PMC8221711 DOI: 10.1097/as9.0000000000000054] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Accepted: 02/13/2021] [Indexed: 12/03/2022] Open
Abstract
Mini-abstract: Surgical sabermetrics is advanced analytics of digitally recorded surgical training and operative procedures to enhance insight, support professional development, and optimize clinical and safety outcomes. This perspectives article illustrates how surgery can leverage data science approaches in athletics and industry to transform individual and team performance in the operating room.
Collapse
Affiliation(s)
- Steven Yule
- From the Department of Clinical Surgery, University of Edinburgh, Edinburgh, Scotland
- Department of Surgery, Brigham & Women’s Hospital/Harvard Medical School, Boston, MA
| | - Allison Janda
- Department of Anesthesiology, Michigan Medicine, Ann Arbor, MI
| | | |
Collapse
|
30
|
From the Editor-in-Chief: Featured Papers in the January Issue. Am J Surg 2021; 221:1. [PMID: 33303128 DOI: 10.1016/j.amjsurg.2020.11.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
31
|
Welton ML. Invited commentary for "The what? How? And Who? Of video based assessment. Am J Surg 2020; 221:11-12. [PMID: 32778400 DOI: 10.1016/j.amjsurg.2020.07.010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 07/15/2020] [Accepted: 07/15/2020] [Indexed: 02/02/2023]
Affiliation(s)
- Mark Lane Welton
- Fairview Health Services, United States; Section of Colon and Rectal Surgery, University of Minnesota, United States.
| |
Collapse
|