1
|
Boal MWE, Anastasiou D, Tesfai F, Ghamrawi W, Mazomenos E, Curtis N, Collins JW, Sridhar A, Kelly J, Stoyanov D, Francis NK. Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg 2024; 111:znad331. [PMID: 37951600 PMCID: PMC10771126 DOI: 10.1093/bjs/znad331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 09/18/2023] [Accepted: 09/19/2023] [Indexed: 11/14/2023]
Abstract
BACKGROUND There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.
Collapse
Affiliation(s)
- Matthew W E Boal
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
| | - Dimitrios Anastasiou
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Freweini Tesfai
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
| | - Walaa Ghamrawi
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
| | - Evangelos Mazomenos
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Medical Physics and Biomedical Engineering, UCL, London, UK
| | - Nathan Curtis
- Department of General Surgey, Dorset County Hospital NHS Foundation Trust, Dorchester, UK
| | - Justin W Collins
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Ashwin Sridhar
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - John Kelly
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Danail Stoyanov
- Wellcome/ESPRC Centre for Interventional Surgical Sciences (WEISS), University College London (UCL), London, UK
- Computer Science, UCL, London, UK
| | - Nader K Francis
- The Griffin Institute, Northwick Park & St Marks’ Hospital, London, UK
- Division of Surgery and Interventional Science, Research Department of Targeted Intervention, UCL, London, UK
- Yeovil District Hospital, Somerset Foundation NHS Trust, Yeovil, Somerset, UK
| |
Collapse
|
2
|
Outcome prediction in bariatric surgery through video-based assessment. Surg Endosc 2022; 37:3113-3118. [PMID: 35927353 DOI: 10.1007/s00464-022-09480-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2022] [Accepted: 07/13/2022] [Indexed: 10/16/2022]
Abstract
INTRODUCTION The relationship between intraoperative surgical performance scores and patient outcomes has not been demonstrated at a single-case level. The GEARS score is a Likert-based scale that quantifies robotic surgical proficiency in 5 domains. Given that even highly skilled surgeons can have variability in their skill among their cases, we hypothesized that at a patient level, higher surgical skill as determined by the GEARS score will predict individual patient outcomes. METHODS Patients undergoing robotic sleeve gastrectomy between July 2018 and January 2021 at a single-health care system were captured in a prospective database. Bivariate Pearson's correlation was used to compare continuous variables, one-way ANOVA for categorical variables compared with a continuous variable, and chi-square for two categorical variables. Significant variables in the univariable screen were included in a multivariable linear regression model. Two-tailed p-value < 0.05 was considered significant. RESULTS Of 162 patients included, 9 patients (5.5%) experienced a serious morbidity within 30 days. The average excess weight loss (EWL) was 72 ± 12% at 6 months and 74 ± 15% at 12 months. GEARS score was not significantly correlated with EWL at 6 months (p = 0.349), 12 months (p = 0.468), or serious morbidity (p = 0.848) on unadjusted analysis. After adjusting, total GEARS score was not correlated with serious morbidity (p = 0.914); however, GEARS score did predict EWL at 6 (p < 0.001) and 12 months (p < 0.001). All GEARS subcomponent scores, bimanual dexterity, depth perception, efficiency, force sensitivity, and robotic control were predictive of EWL at 6 months (p < 0.001) and 12 months (p < 0.001) on multivariable analysis. CONCLUSION For patients undergoing sleeve gastrectomy, surgical skill as assessed by the GEARS score was correlated with EWL, suggesting that better performance of a sleeve gastrectomy can result in improved postoperative weight loss.
Collapse
|
3
|
Olsen RG, Genét MF, Konge L, Bjerrum F. Crowdsourced assessment of surgical skills: A systematic review. Am J Surg 2022; 224:1229-1237. [DOI: 10.1016/j.amjsurg.2022.07.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Revised: 05/30/2022] [Accepted: 07/14/2022] [Indexed: 11/25/2022]
|
4
|
Azadi S, Green IC, Arnold A, Truong M, Potts J, Martino MA. Robotic Surgery: The Impact of Simulation and Other Innovative Platforms on Performance and Training. J Minim Invasive Gynecol 2020; 28:490-495. [PMID: 33310145 DOI: 10.1016/j.jmig.2020.12.001] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2020] [Revised: 11/25/2020] [Accepted: 12/02/2020] [Indexed: 01/30/2023]
Abstract
OBJECTIVE To review the current status of robotic training and the impact of various training platforms on the performance of robotic surgical trainees. DATA SOURCES Literature review of Google Scholar and PubMed. The search terms included a combination of the following: "robotic training," "simulation," "robotic curriculum," "obgyn residency robotic training," "virtual reality robotic training," "DaVinci training," "surgical simulation," "gyn surgical training." The sources considered for inclusion included peer-reviewed articles, literature reviews, textbook chapters, and statements from various institutions involved in resident training. METHODS OF STUDY SELECTION A literature search of Google Scholar and PubMed using terms related to robotic surgery and robotics training, as mentioned in the "Data Sources" section. RESULTS Multiple novel platforms that use machine learning and real-time video feedback to teach and evaluate robotic surgical skills have been developed over recent years. Various training curricula, virtual reality simulators, and other robotic training tools have been shown to enhance robotic surgical education and improve surgical skills. The integration of didactic learning, simulation, and intraoperative teaching into more comprehensive training curricula shows positive effects on robotic skills proficiency. Few robotic surgery training curricula have been validated through peer-reviewed study, and there is more work to be completed in this area. In addition, there is a lack of information about how the skills obtained through robotics curricula and simulation translate into operating room performance and patient outcomes. CONCLUSION Data collected to date show promising advances in the training of robotic surgeons. A diverse array of curricula for training robotic surgeons continue to emerge, and existing teaching modalities are evolving to keep up with the rapidly growing demand for proficient robotic surgeons. Futures areas of growth include establishing competency benchmarks for existing training tools, validating existing curricula, and determining how to translate the acquired skills in simulation into performance in the operating room and patient outcomes. Many surgical training platforms are beginning to expand beyond discrete robotic skills training to procedure-specific and team training. There is still a wealth of research to be done to understand how to create an effective training experience for gynecologic surgical trainees and robotics teams.
Collapse
Affiliation(s)
- Shirin Azadi
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino)
| | - Isabel C Green
- Department of Gynecology and Obstetric, Mayo Clinic, Rochester, Minnesota (Dr. Green)
| | - Anne Arnold
- American College of Obstetricians and Gynecologists, University of Pennsylvania Graduate School of Education, Philadelphia, PA (Ms. Arnold)
| | - Mireille Truong
- Division of Minimally Invasive Gynecologic Surgery, Department of Obstetrics and Gynecology, Cedars-Sinai Medical Center, Los Angeles, California (Dr. Truong)
| | - Jacqueline Potts
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino)
| | - Martin A Martino
- Department of Obstetrics and Gynecology, Lehigh Valley Health Network, Allentown, Pennsylvania (Drs. Azadi, Potts, and Martino); Department of Obstetrics and Gynecology, University of South Florida, Tampa, Florida (Dr. Martino).
| |
Collapse
|
5
|
Prebay ZJ, Peabody JO, Miller DC, Ghani KR. Video review for measuring and improving skill in urological surgery. Nat Rev Urol 2020; 16:261-267. [PMID: 30622365 DOI: 10.1038/s41585-018-0138-2] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Interest is growing within the urological surgery community for objective assessments of technical skill. Surgical video review relies on the use of objective assessment tools to evaluate both global and procedure-specific skill. These evaluations provide structured feedback to surgeons with the aim of improving technique, which has been associated with patient outcomes. Currently, skill assessments can be performed by using expert peer-review, crowdsourcing or computer-based methods. Given the relationship between skill and patient outcomes, surgeons might be required in the future to provide empirical evidence of their technical skill for certification, employment, credentialing and quality improvement. Interventions such as coaching and skills workshops incorporating video review might help surgeons improve their skill, with the ultimate goal of improving patient outcomes.
Collapse
Affiliation(s)
- Zachary J Prebay
- School of Medicine, Medical College of Wisconsin, Milwaukee, WI, USA
| | - James O Peabody
- Center for Outcomes Research, Analytics and Evaluation, Vattikuti Urology Institute, Henry Ford Hospital, Detroit, MI, USA
| | - David C Miller
- Department of Urology, University of Michigan, Ann Arbor, MI, USA
| | - Khurshid R Ghani
- Department of Urology, University of Michigan, Ann Arbor, MI, USA.
| |
Collapse
|
6
|
Evidence that surgical performance predicts clinical outcomes. World J Urol 2019; 38:1595-1597. [PMID: 31256249 DOI: 10.1007/s00345-019-02857-w] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2019] [Accepted: 06/22/2019] [Indexed: 10/26/2022] Open
Abstract
PURPOSE Assessment of surgeon performance in the operating room has been identified as a direct method of measuring surgical quality. Studies published in urology and other surgical disciplines have investigated this link directly by measuring surgeon and team performance using methodology supported by validity evidence. This article highlights the key findings of these studies and associated underlying concepts. METHODS Seminal literature from urology and related areas of research was used to inform this review of the performance-outcome relationship in surgery. Current efforts to further our understanding of this concept are discussed, including relevant quality improvement and educational interventions that utilize this relationship. RESULTS Evidence from multiple surgical specialties and procedures has established the association between surgeon skill and clinically significant patient outcomes. Novel methods of measuring performance utilize surgeon kinematics and artificial intelligence techniques to more reliably and objectively quantify surgical performance. CONCLUSIONS Future directions include the use of this data to create interventions for quality improvement, as well as innovate the credentialing and recertification process for practicing surgeons.
Collapse
|