1
|
Desir A, Marques C, Farah E, R Hegde S, Holcomb C, Scott DJ, Sankaranarayanan G. Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication. Surg Endosc 2024; 38:2219-2230. [PMID: 38383688 DOI: 10.1007/s00464-024-10675-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Accepted: 12/30/2023] [Indexed: 02/23/2024]
Abstract
BACKGROUND Laparoscopic hiatal hernia repair (LHHR) is a complex operation requiring advanced surgical training. Surgical simulation offers a potential solution for learning complex operations without the need for high surgical volume. Our goal is to develop a virtual reality (VR) simulator for LHHR; however, data supporting task-specific metrics for this procedure are lacking. The purpose of this study was to develop and assess validity and reliability evidence of task-specific metrics for the fundoplication phase of LHHR. METHODS In phase I, structured interviews with expert foregut surgeons were conducted to develop task-specific metrics (TSM). In phase II, participants with varying levels of surgical expertise performed a laparoscopic Nissen fundoplication procedure on a porcine stomach explant. Video recordings were independently assessed by two blinded graders using global and TSM. An intraclass correlation coefficient (ICC) was used to assess interrater reliability (IRR). Performance scores were compared using a Kruskal-Wallis test. Spearman's rank correlation was used to evaluate the association between global and TSM. RESULTS Phase I of the study consisted of 12 interviews with expert foregut surgeons. Phase II engaged 31 surgery residents, a fellow, and 6 attendings in the simulation. Phase II results showed high IRR for both global (ICC = 0.84, p < 0.001) and TSM (ICC = 0.75, p < 0.001). Significant between-group differences were detected for both global (χ2 = 24.01, p < 0.001) and TSM (χ2 = 18.4, p < 0.001). Post hoc analysis showed significant differences in performance between the three groups for both metrics (p < 0.05). There was a strong positive correlation between the global and TSM (rs = 0.86, p < 0.001). CONCLUSION We developed task-specific metrics for LHHR and using a fundoplication model, we documented significant reliability and validity evidence. We anticipate that these LHHR task-specific metrics will be useful in our planned VR simulator.
Collapse
Affiliation(s)
- Alexis Desir
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
- Artificial Intelligence and Medical Simulation Lab, Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Carolina Marques
- Artificial Intelligence and Medical Simulation Lab, Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Emile Farah
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Shruti R Hegde
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
- Artificial Intelligence and Medical Simulation Lab, Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Carla Holcomb
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Daniel J Scott
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA
| | - Ganesh Sankaranarayanan
- Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA.
- Artificial Intelligence and Medical Simulation Lab, Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd., Dallas, TX, 75390, USA.
| |
Collapse
|
2
|
Dwyer T, Chahal J, Murnaghan L, Theodoropoulos J, Cheung J, McParland A, Ogilvie-Harris D. Development of a certification examination for orthopedic sports medicine fellows. Can J Surg 2020; 63:E110-E117. [PMID: 32142243 DOI: 10.1503/cjs.015418] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022] Open
Abstract
Background The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship. Methods Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Results Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair. Conclusion There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.
Collapse
Affiliation(s)
- Tim Dwyer
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - Jaskarndip Chahal
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - Lucas Murnaghan
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - John Theodoropoulos
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - Jeffrey Cheung
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - Aidan McParland
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| | - Darrell Ogilvie-Harris
- From Mount Sinai Hospital, Toronto, Ont. (Dwyer, Theodoropoulos); Toronto Western Hospital, Toronto, Ont. (Chahal, Ogilvie-Harris); and Women’s College Hospital, Toronto, Ont. (Dwyer, Chahal, Murnaghan, Theodoropoulos, Cheung, McParland, Ogilvie-Harris)
| |
Collapse
|
3
|
Bull NB, Silverman CD, Bonrath EM. Targeted surgical coaching can improve operative self-assessment ability: A single-blinded nonrandomized trial. Surgery 2019; 167:308-313. [PMID: 31570149 DOI: 10.1016/j.surg.2019.08.002] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2019] [Revised: 08/04/2019] [Accepted: 08/07/2019] [Indexed: 01/17/2023]
Abstract
BACKGROUND The aim of the study was to investigate the effect of targeted surgical coaching on self-assessment of laparoscopic operative skill. Accurate self-assessment is vital for autonomous professional development. Surgical coaching can be used for performance improvement, but its role in this domain has been insufficiently investigated. METHODS This was a single site, nonrandomized, interrupted time series design trial. Participants were residents, fellows, and attending surgeons regularly performing laparoscopic general surgery operations. Each participant was enrolled in an individualized coaching program using review of personal and peer laparoscopic videos. The program involved 3 to 5 sessions over a period of 6 to 19 weeks. Coaching used case debriefing to target self-assessment proficiency, with a focus on objective interpretation of observations and facilitative capacity building. The primary outcome measure was self-assessment accuracy and correlation to expert ratings. The Objective Structured Assessment of Technical Skill global rating scale was utilized for evaluation. RESULTS Twelve participants were recruited and completed the coaching program. At baseline, there was no correlation between self-assessment and expert ratings. After completion of the coaching program there was correlation between self-assessment and expert ratings (P = .003) and improved self-assessment accuracy compared to baseline (P = .041). CONCLUSION This study has demonstrated that targeted coaching using video review of laparoscopic cases can improve operative self-assessment accuracy using the Objective Structured Assessment of Technical Skill.
Collapse
Affiliation(s)
- Nicholas B Bull
- Department of General Surgery, The Tweed Hospital, Tweed Heads, NSW, Australia.
| | - Candice D Silverman
- Department of General Surgery, The Tweed Hospital, Tweed Heads, NSW, Australia; Practice of Candice D. Silverman, John Flynn Medical Centre, Tugun, QLD, Australia
| | - Esther M Bonrath
- Practice of Candice D. Silverman, John Flynn Medical Centre, Tugun, QLD, Australia; Department of General and Visceral Surgery, Krankenhaus Barmherzige Brueder, Regensburg, Germany
| |
Collapse
|
4
|
Ghaderi I, Farrell TM. Toward effective feedback: From concept to practice. Surgery 2019; 167:685-688. [PMID: 31353077 DOI: 10.1016/j.surg.2019.06.014] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2019] [Revised: 05/27/2019] [Accepted: 06/17/2019] [Indexed: 11/17/2022]
Abstract
Feedback is the building block of assessment and an essential component of effective teaching. In this article, we will provide a comprehensive definition of feedback and a relevant conceptual framework for learning to explain how feedback can be used to improve performance. We will discuss the process of providing feedback, and the role teacher and learner can play to make it more effective. We will also examine the misunderstandings and pitfalls around feedback, as well as generational differences that may influence its impact.
Collapse
Affiliation(s)
- Iman Ghaderi
- Department of Surgery, University of Arizona, Tucson, AZ.
| | - Timothy M Farrell
- Department of Surgery, University of North Carolina at Chapel Hill, NC
| |
Collapse
|
5
|
Bilgic E, Al Mahroos M, Landry T, Fried GM, Vassiliou MC, Feldman LS. Assessment of surgical performance of laparoscopic benign hiatal surgery: a systematic review. Surg Endosc 2019; 33:3798-3805. [PMID: 30671670 DOI: 10.1007/s00464-019-06662-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2018] [Accepted: 01/14/2019] [Indexed: 01/19/2023]
Abstract
BACKGROUND Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument. METHODS A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences). RESULTS There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF. CONCLUSION There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.
Collapse
Affiliation(s)
- Elif Bilgic
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Mohammed Al Mahroos
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Tara Landry
- Montreal General Hospital Medical Library, McGill University Health Centre, 1650, Cedar Avenue, L9. 309, Montréal, QC, H3G 1A4, Canada
| | - Gerald M Fried
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Melina C Vassiliou
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada
| | - Liane S Feldman
- Steinberg-Bernstein Centre for Minimally Invasive Surgery and Innovation, McGill University Health Centre, Montreal, QC, Canada.
| |
Collapse
|
6
|
A scoping review of assessment tools for laparoscopic suturing. Surg Endosc 2018; 32:3009-3023. [PMID: 29721749 DOI: 10.1007/s00464-018-6199-8] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2018] [Accepted: 04/21/2018] [Indexed: 10/17/2022]
Abstract
BACKGROUND A needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions. METHODS Bibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills. RESULTS Forty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences. CONCLUSION Through identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.
Collapse
|
7
|
Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D. The Feedback Tango: An Integrative Review and Analysis of the Content of the Teacher-Learner Feedback Exchange. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:657-663. [PMID: 28991848 DOI: 10.1097/acm.0000000000001927] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
PURPOSE To conduct an integrative review and analysis of the literature on the content of feedback to learners in medical education. METHOD Following completion of a scoping review in 2016, the authors analyzed a subset of articles published through 2015 describing the analysis of feedback exchange content in various contexts: audiotapes, clinical examination, feedback cards, multisource feedback, videotapes, and written feedback. Two reviewers extracted data from these articles and identified common themes. RESULTS Of the 51 included articles, about half (49%) were published since 2011. Most involved medical students (43%) or residents (43%). A leniency bias was noted in many (37%), as there was frequently reluctance to provide constructive feedback. More than one-quarter (29%) indicated the feedback was low in quality (e.g., too general, limited amount, no action plans). Some (16%) indicated faculty dominated conversations, did not use feedback forms appropriately, or provided inadequate feedback, even after training. Multiple feedback tools were used, with some articles (14%) describing varying degrees of use, completion, or legibility. Some articles (14%) noted the impact of the gender of the feedback provider or learner. CONCLUSIONS The findings reveal that the exchange of feedback is troubled by low-quality feedback, leniency bias, faculty deficient in feedback competencies, challenges with multiple feedback tools, and gender impacts. Using the tango dance form as a metaphor for this dynamic partnership, the authors recommend ways to improve feedback for teachers and learners willing to partner with each other and engage in the complexities of the feedback exchange.
Collapse
Affiliation(s)
- Robert Bing-You
- R. Bing-You is professor, Tufts University School of Medicine, Boston, Massachusetts, and vice president for medical education, Maine Medical Center, Portland, Maine. K. Varaklis is clinical associate professor, Tufts University School of Medicine, Boston, Massachusetts, and designated institutional official, Maine Medical Center, Portland, Maine. V. Hayes is clinical assistant professor, Tufts University School of Medicine, Boston, Massachusetts, and faculty member, Department of Family Medicine, Maine Medical Center, Portland, Maine. R. Trowbridge is associate professor, Tufts University School of Medicine, Boston, Massachusetts, and director of undergraduate medical education, Department of Medicine, Maine Medical Center, Portland, Maine. H. Kemp is medical librarian, Maine Medical Center, Portland, Maine. D. McKelvy is manager of library and knowledge services, Maine Medical Center, Portland, Maine
| | | | | | | | | | | |
Collapse
|
8
|
Faculty feedback versus residents' self-assessment of operative performance: Different but complementary. Am J Surg 2018; 215:288-292. [DOI: 10.1016/j.amjsurg.2017.11.016] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2017] [Revised: 11/07/2017] [Accepted: 11/10/2017] [Indexed: 01/07/2023]
|
9
|
|