1
|
Niznik T, Chen S, Gernsback J. Through the Looking Glass: Comparing Resident and Attending Views on Competency and Autonomy. World Neurosurg 2024; 186:e290-e295. [PMID: 38548051 DOI: 10.1016/j.wneu.2024.03.125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Accepted: 03/21/2024] [Indexed: 04/26/2024]
Abstract
BACKGROUND Surgical residencies act as an apprenticeship to train new surgeons, ideally with increasing trainee autonomy and competency. Objective methods of measuring these two qualities are needed. METHODS The objective of this pilot study is to use the Surgical Autonomy Program to compare resident and attending perception of competency, via Zones of Proximal Development (ZPDs), and autonomy, via Teach, Advise, Guide, Solo (TAGS) scale for a common pediatric neurosurgical procedure. The most common pediatric neurosurgical procedure, a ventriculoperitoneal shunt, was broken down into four ZPDs. After each procedure, the resident and the attending rated the competency of the resident by ZPD and their autonomy using TAGS. These ratings were then compared to see if the attending and the resident agreed on their levels of competency and autonomy. RESULTS There were no significant differences in the rating of competency or autonomy between the residents and the attendings. Postgraduate year was significantly associated with differences in TAGS ratings with those having less experience showing a positive perception gap. Other covariates (age, gender) were not associated with differences in rating. The residents found that the ZPDs checklist was a useful tool for communicating with the attending and for getting appropriate autonomy in the operating room. CONCLUSIONS Pediatric neurosurgical trainees and attendings had concordance on their perceptions of resident autonomy and competency, suggesting that this tool could be a useful way of objectively grading the progress of neurosurgical residents in surgery.
Collapse
Affiliation(s)
- Taylor Niznik
- Department of Neurosurgery, University of Oklahoma Health Sciences Center, Oklahoma City, OK, USA; Department of Neurosurgery, Section of Pediatric Neurosurgery, Oklahoma Children's Hospital, University of Oklahoma School of Medicine, Oklahoma City, OK, USA
| | - Sixia Chen
- Department of Biostatistics and Epidemiology, University of Oklahoma Health Sciences Center, Oklahoma City, OK, USA
| | - Joanna Gernsback
- Department of Neurosurgery, University of Texas Health Science Center San Antonio, San Antonio, TX, USA.
| |
Collapse
|
2
|
Guthrie E, Levy D, Del Carmen G. The Operating and Anesthetic Reference Assistant (OARA): A fine-tuned large language model for resident teaching. Am J Surg 2024:S0002-9610(24)00106-5. [PMID: 38365551 DOI: 10.1016/j.amjsurg.2024.02.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2023] [Revised: 02/03/2024] [Accepted: 02/08/2024] [Indexed: 02/18/2024]
Abstract
OBJECTIVE This study aimed to fine-tune a large language model (LLM) for domain-specific text generation in surgical and anesthesia residency education. SUMMARY BACKGROUND DATA With growing interest in artificial intelligence (AI) for medical training, the potential of LLMs to transform residency education is explored. METHODS The 7-billion parameter base model "Vicuna v1.5" was trained on 266,342 lines of text from 821 peer-reviewed documents. We evaluated the model with 150 surgical or anesthesia queries and assessed accuracy, token count, and inference speed across various reasoning tasks. Tests of significance were conducted using ANOVA and chi-square analysis. RESULTS Our model achieved 65.3% accuracy, excelling in surgical case-based tasks. We found no significant difference in accuracy between knowledge domains (P=0.081), though longer response generation demonstrated poorer accuracy, with significant accuracy variation based on output length (P = 0.002). CONCLUSIONS LLMs show potential in enhancing residency education. Our model's efficiency and task-specific accuracy highlights such promise, though limits in parameter count diminishes accuracy of longer response generation. Our findings showcase how AI may be integrated effectively within future residency training.
Collapse
Affiliation(s)
- Estefania Guthrie
- McGovern Medical School at the University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Dominique Levy
- McGovern Medical School at the University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Gabriel Del Carmen
- McGovern Medical School at the University of Texas Health Science Center at Houston, Houston, TX, USA.
| |
Collapse
|
3
|
Sevestre A, Dochez V, Souron R, Deschamps T, Winer N, Thubert T. Evaluation Tools for Assessing Autonomy of Surgical Residents in the Operating Room and Factors Influencing Access to Autonomy: A Systematic Literature Review. JOURNAL OF SURGICAL EDUCATION 2024; 81:182-192. [PMID: 38160113 DOI: 10.1016/j.jsurg.2023.11.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 10/16/2023] [Accepted: 11/01/2023] [Indexed: 01/03/2024]
Abstract
BACKGROUND Surgical residents in France lack a clear pedagogical framework for achieving autonomy in the operating room. The progressive acquisition of surgical autonomy is a determining factor in the confidence of operators for their future independent practice. Currently, there is no autonomy scale commonly used in Europe. The objective of this study is to identify existing tools for quantifying the autonomy of residents and the factors that influence it. MATERIALS AND METHODS We conducted a qualitative systematic review following the recommendations of the Systematic Review Without Meta-Analysis (SWiM) guidelines. Publications were extracted from the MEDLINE (PubMed), EMBASE, and PSYCINFO databases. All publications without date restrictions up to July 2022 were identified. RESULTS Among the 231 identified publications, 21 met the inclusion criteria. Seventeen publications used a graded autonomy assessment tool by the student and/or the teacher, while 4 used evaluations by an observing third party. We found 8 different autonomy scales, with the Zwisch Scale representing 57.1% of the cases. Factors influencing autonomy were diverse, including the work context, experience, and gender of the resident and their teacher. DISCUSSION We found heterogeneity in the tools used to "measure" the autonomy of a resident in the operating room. The SIMPL tool or the Zwisch Scale appear to be the most frequently used tools. The relationship between autonomy, performance, confidence, and knowledge may require multidimensional tools that encompass various areas of competence, but this could make their daily application more challenging. The factors influencing autonomy are numerous; and understanding them would improve teaching in the operating room. There is a significant lack of data on surgical autonomy in France, as well as a lack of evaluation in the field of gynecology-obstetrics worldwide.
Collapse
Affiliation(s)
- Anaïs Sevestre
- Nantes Université, CHU Nantes, Service de Gynécologie-Obstétrique, Nantes, France; Nantes Université, CHU Nantes, Movement - Interactions - Performance, Nantes, France
| | - Vincent Dochez
- Nantes Université, CHU Nantes, Service de Gynécologie-Obstétrique, Nantes, France; Nantes Université, CHU Nantes, Movement - Interactions - Performance, Nantes, France.
| | - Robin Souron
- Nantes Université, CHU Nantes, Movement - Interactions - Performance, Nantes, France
| | - Thibault Deschamps
- Nantes Université, CHU Nantes, Movement - Interactions - Performance, Nantes, France
| | - Norbert Winer
- Nantes Université, CHU Nantes, Service de Gynécologie-Obstétrique, Nantes, France; Nantes Université, CHU Nantes, INRAE, Nantes, France
| | - Thibault Thubert
- Nantes Université, CHU Nantes, Service de Gynécologie-Obstétrique, Nantes, France; Nantes Université, CHU Nantes, Movement - Interactions - Performance, Nantes, France
| |
Collapse
|
4
|
Tatebe LC, Golisch KB, Janczewski LM, Krueger M, Halverson A. Autonomy Maps: Building a Shared Mental Model for Progressive Resident Operative Autonomy on the Road to Entrustable Professional Activities (EPAs). JOURNAL OF SURGICAL EDUCATION 2023; 80:1351-1354. [PMID: 37537103 DOI: 10.1016/j.jsurg.2023.07.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Revised: 05/12/2023] [Accepted: 07/08/2023] [Indexed: 08/05/2023]
Abstract
Our residents expressed dissatisfaction with operative autonomy and faculty feedback regarding technical skills. They reported variability among faculty regarding allowed operative autonomy. Our goals were to establish a shared mental model among residents and faculty regarding intraoperative performance expectations. We asked faculty to assign a level of expected autonomy (Zwisch scale) for various steps of common procedures according to the resident post-graduate year. Through an iterative process, the maps were standardized across service lines. The resulting "Autonomy Maps" were distributed to the faculty and residents. We held educational sessions and set expectations for use. Selected benchmarks were incorporated into resident end-of-rotation assessment forms. Initial operative case mapping identified variability in faculty expectations for a given post-graduate year and procedure. Residents reported improved satisfaction with understanding expectations regarding operative performance. Establishing autonomy benchmarks facilitated more specific feedback regarding residents' technical skills. Faculty expectations for resident operative autonomy are variable. Autonomy Maps provide structure for a shared mental model between faculty and residents for progressive operative autonomy and serve as a framework for expectations that improve resident satisfaction. Case-specific technical benchmarks are useful tools for assessing residents' technical milestones.
Collapse
Affiliation(s)
- Leah C Tatebe
- Feinberg School of Medicine, Northwestern University, Chicago, Illinois.
| | | | | | - Mackenzie Krueger
- Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Amy Halverson
- Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| |
Collapse
|
5
|
Helliwell LA, Hyland CJ, Gonte MR, Malapati SH, Bain PA, Ranganathan K, Pusic AL. Bias in Surgical Residency Evaluations: A Scoping Review. JOURNAL OF SURGICAL EDUCATION 2023; 80:922-947. [PMID: 37142488 DOI: 10.1016/j.jsurg.2023.04.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Revised: 03/21/2023] [Accepted: 04/13/2023] [Indexed: 05/06/2023]
Abstract
OBJECTIVE Given widespread disparities in the surgical workforce and the advent of competency-based training models that rely on objective evaluations of resident performance, this review aims to describe the landscape of bias in the evaluation methods of residents in surgical training programs in the United States. DESIGN A scoping review was conducted within PubMed, Embase, Web of Science, and ERIC in May 2022, without a date restriction. Studies were screened and reviewed in duplicate by 3 reviewers. Data were described descriptively. SETTING/PARTICIPANTS English-language studies conducted in the United States that assessed bias in the evaluation of surgical residents were included. RESULTS The search yielded 1641 studies, of which 53 met inclusion criteria. Of the included studies, 26 (49.1%) were retrospective cohort studies, 25 (47.2%) were cross-sectional studies, and 2 (3.8%) were prospective cohort studies. The majority included general surgery residents (n = 30, 56.6%) and nonstandardized examination modalities (n = 38, 71.7%), such as video-based skills evaluations (n = 5, 13.2%). The most common performance metric evaluated was operative skill (n = 22, 41.5%). Overall, the majority of studies demonstrated bias (n = 38, 73.6%) and most investigated gender bias (n = 46, 86.8%). Most studies reported disadvantages for female trainees regarding standardized examinations (80.0%), self-evaluations (73.7%), and program-level evaluations (71.4%). Four studies (7.6%) assessed racial bias, of which all reported disadvantages for trainees underrepresented in surgery. CONCLUSIONS Evaluation methods for surgery residents may be prone to bias, particularly with regard to female trainees. Research is warranted regarding other implicit and explicit biases, such as racial bias, as well as for nongeneral surgery subspecialties.
Collapse
Affiliation(s)
| | | | - Madeleine R Gonte
- Harvard T.H. Chan School of Public Health, Boston, Massachusetts; Wayne State University School of Medicine, Detroit, Michigan
| | | | - Paul A Bain
- Countway Library, Harvard Medical School, Boston, Massachusetts
| | | | | |
Collapse
|
6
|
Allen K, Najjar M, Ostermeier A, Washington N, Genies MC, Hazle M, Hardy C, Lewis K, McDaniel L, McFarlane DJ, Macias C, Molloy MJ, Perry MF, Piper L, Sevov C, Titus L, Toth H, Unaka NI, Weisgerber MC, Kasick R. The Autonomy Toolbox: A Multicenter Collaborative to Promote Resident Autonomy. Hosp Pediatr 2023; 13:490-503. [PMID: 37153964 DOI: 10.1542/hpeds.2022-006827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
OBJECTIVES Autonomy is necessary for resident professional development and well-being. A recent focus on patient safety has increased supervision and decreased trainee autonomy. Few validated interventions exist to improve resident autonomy. We aimed to use quality improvement methods to increase our autonomy metric, the Resident Autonomy Score (RAS), by 25% within 1 year and sustain for 6 months. METHODS We developed a bundled-intervention approach to improve senior resident (SR) perception of autonomy on Pediatric Hospital Medicine (PHM) services at 5 academic children's hospitals. We surveyed SR and PHM faculty perceptions of autonomy and targeted interventions toward areas with the highest discordance. Interventions included SR and faculty development, expectation-setting huddles, and SR independent rounding. We developed a Resident Autonomy Score (RAS) index to track SR perceptions over time. RESULTS Forty-six percent of SRs and 59% of PHM faculty completed the needs assessment survey querying how often SRs were afforded opportunities to provide autonomous medical care. Faculty and SR ratings were discordant in these domains: SR input in medical decisions, SR autonomous decision-making in straightforward cases, follow-through on SR plans, faculty feedback, SR as team leader, and level of attending oversight. The RAS increased by 19% (3.67 to 4.36) 1 month after SR and faculty professional development and before expectation-setting and independent rounding. This increase was sustained throughout the 18-month study period. CONCLUSIONS SRs and faculty perceive discordant levels of SR autonomy. We created an adaptable autonomy toolbox that led to sustained improvement in perception of SR autonomy.
Collapse
Affiliation(s)
- Karen Allen
- Division of Hospital Medicine, Department of Pediatrics
| | | | - Austen Ostermeier
- Division of Hospital Medicine, Department of Pediatrics, Washington University in St Louis School of Medicine, St Louis, Missouri
| | - Nicole Washington
- Section of Hospital Medicine, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
- Department of Pediatrics, The University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania
| | - Marquita C Genies
- Division of Hospital Medicine, Department of Pediatrics, John Hopkins University School of Medicine, Baltimore, Maryland
| | - Matthew Hazle
- Division of Hospital Medicine, Department of Pediatrics, University of Michigan, Ann Arbor, Michigan
| | | | - Kristen Lewis
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Lauren McDaniel
- Division of Hospital Medicine, Department of Pediatrics, Seattle Children's Hospital, Seattle, Washington
| | - Daniel J McFarlane
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Charlie Macias
- Planning and Business Development, Nationwide Children's Hospital, Columbus, Ohio
| | - Matthew J Molloy
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | | | - Laura Piper
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Claire Sevov
- Division of Hospital Medicine, Department of Pediatrics
- Division of Hospital Medicine, Department of Internal Medicine, Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Lauren Titus
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
| | - Heather Toth
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
- Internal Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin
| | - Ndidi I Unaka
- Division of Hospital Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio
- Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio
| | - Michael C Weisgerber
- Division of Hospital Medicine, Children's Wisconsin, Milwaukee, Wisconsin
- Departments of Pediatrics
| | - Rena Kasick
- Division of Hospital Medicine, Department of Pediatrics
| |
Collapse
|
7
|
Shebrain S, Coster S, Alfred A, De Cecco D, Khalil S, Munene G, Elian A, Timmons J, Sawyer RG. Resident Autonomy and Performance Independence in Surgical Training Are Time- and Skill-Dependent. J Surg Res 2022; 279:285-295. [DOI: 10.1016/j.jss.2022.06.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2022] [Revised: 06/11/2022] [Accepted: 06/14/2022] [Indexed: 11/28/2022]
|
8
|
Eruchalu CN, He K, Etheridge JC, Wu C, Ashley SW, Nitzschke SL, Smink DS, Cho NL. Gender and Racial/Ethnic Disparities in Operative Volumes of Graduating General Surgery Residents. J Surg Res 2022; 279:104-112. [PMID: 35759927 DOI: 10.1016/j.jss.2022.05.020] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Revised: 04/25/2022] [Accepted: 05/22/2022] [Indexed: 10/31/2022]
Abstract
INTRODUCTION Gender disparities in resident operative experience have been described; however, their etiology is poorly understood, and racial/ethnic disparities have not been explored. This study investigated the relationship between gender, race/ethnicity, and surgery resident case volumes. MATERIALS AND METHODS A retrospective analysis of graduating general surgery resident case logs (2010-2020) at an academic medical center was performed. Self-reported gender and race/ethnicity data were collected from program records. Residents were categorized as underrepresented in medicine (URM) (Black, Hispanic, Native American) or non-URM (White, Asian). Associations between gender and URM status and major, chief, and teaching assistant (TA) mean case volumes were analyzed using t-tests. RESULTS The cohort included 80 residents: 39 female (48.8%) and 17 URM (21.3%). Compared to male residents, female residents performed fewer TA cases (33 versus 47, P < 0.001). Compared to non-URM residents, URM residents graduated with fewer major (948 versus 1043, P = 0.008) and TA cases (32 versus 42, P = 0.038). Male URM residents performed fewer TA cases than male non-URM residents (32 versus 50, P = 0.031). Subanalysis stratified by graduation year demonstrated that from 2010 to 2015, female residents performed fewer chief (218 versus 248, P = 0.039) and TA cases (29 versus 50, P = 0.001) than male residents. However, from 2016 to 2020, when gender parity was achieved, no significant associations were observed between gender and case volumes. CONCLUSIONS Female and URM residents perform fewer TA and major cases than male non-URM residents, which may contribute to reduced operative autonomy, confidence, and entrustment. Prioritizing gender and URM parity may help decrease case volume gaps among underrepresented residents.
Collapse
Affiliation(s)
- Chukwuma N Eruchalu
- Harvard Medical School, Boston, Massachusetts; Center for Surgery and Public Health, Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Katherine He
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - James C Etheridge
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts; Ariadne Labs, Harvard T.H. Chan School of Public Health, Brigham and Women's Hospital, Boston, Massachusetts
| | - Christine Wu
- Center for Surgery and Public Health, Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts; Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Stanley W Ashley
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Stephanie L Nitzschke
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Douglas S Smink
- Center for Surgery and Public Health, Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts; Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts
| | - Nancy L Cho
- Department of Surgery, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts.
| |
Collapse
|
9
|
Abbott KL, Krumm AE, Clark MJ, Kendrick DE, Kelley JK, George BC. Representativeness of Workplace-Based Operative Performance Assessments for Resident Operative Experience. JOURNAL OF SURGICAL EDUCATION 2022; 79:769-774. [PMID: 34996745 DOI: 10.1016/j.jsurg.2021.12.010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Revised: 11/04/2021] [Accepted: 12/12/2021] [Indexed: 06/14/2023]
Abstract
OBJECTIVE Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees. Therefore, analyses of these samples may be biased. We aimed to benchmark patterns of procedures logged in the SIMPL operative performance assessment system against records of trainee experience in Accreditation Council for Graduate Medical Education (ACGME) case logs. DESIGN We analyzed SIMPL longitudinal intraoperative performance assessments from categorical trainees in US general surgery residency programs. We compared overall patterns of how procedures are logged in SIMPL and in ACGME case logs using a Pearson correlation, and we examined differences in how individual procedures are logged in each system using Fisher's exact test. RESULTS Total procedure frequency from the SIMPL dataset was strongly correlated with total procedure frequency from ACGME case logs (r = 0.86, 95% CI 0.80-0.90). A subset of these procedures (10 of 116 procedures) was logged more frequently in the SIMPL dataset. These 10 procedures accounted for 56% of SIMPL observations and 30% of ACGME logged cases. Case complexity was comparable for assessments initiated by residents and faculty. CONCLUSIONS Samples of intraoperative performance ratings gathered using the SIMPL application largely resemble ACGME case logs. There is no evidence to indicate that residents preferentially select fewer complex cases for assessment.
Collapse
Affiliation(s)
- Kenneth L Abbott
- Department of Surgery, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan; University of Michigan Medical School, Ann Arbor, Michigan
| | - Andrew E Krumm
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan
| | - Michael J Clark
- Consulting for Statistics, Computing, and Analytics Research, University of Michigan, Ann Arbor, Michigan
| | - Daniel E Kendrick
- Department of Surgery, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan
| | - Jesse K Kelley
- Department of Surgery, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan; University of Michigan Medical School, Ann Arbor, Michigan
| | - Brian C George
- Department of Surgery, Center for Surgical Training and Research, University of Michigan, Ann Arbor, Michigan; Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan.
| |
Collapse
|
10
|
Williams J, Thelen AE, Luckoski J, Chen X, George BC. How Do Resident Surgeons Identify Operative Case Complexity? An Analysis of Resident versus Attending Perceptions. JOURNAL OF SURGICAL EDUCATION 2022; 79:469-474. [PMID: 34602380 DOI: 10.1016/j.jsurg.2021.09.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/25/2021] [Revised: 07/28/2021] [Accepted: 09/16/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE Accurate recognition of patient-related complexity of an operation is critical for appropriate surgical decision making. It is not yet understood whether general surgery residents are able to accurately assess the relative complexity of a given operative case. This study investigates the agreement of case complexity ratings between residents and attending surgeons and explores whether resident-related factors correlate with any discordance in perception of patient-related operative complexity. DESIGN Residents and attending surgeons rated the relative complexity of completed cases on a 3 point scale via the SIMPL (Society for Improving Medical Professional Learning) operative assessment smartphone app. Additional trainee demographic data, autonomy ratings, and performance ratings were also obtained from the SIMPL registry for each rated case. Complexity agreement was defined as an equal rating between the resident and attending and assigned a value of zero. Over-estimate ratings were assigned a positive value and under-estimate ratings were assigned a negative value. Trends in complexity agreement were analyzed using descriptive statistics and mixed-effects models. RESULTS A total of 43,179 general surgery cases were rated by 1946 categorical general surgery residents and 1520 attending surgeons between 2015 and 2020. Residents and attendings agreed on case complexity in 63.23% of cases, while the residents overestimated complexity in 13.37% of cases and underestimated complexity in 23.40% of cases. Every level of resident except post-graduate year 2 had similar rates of agreement about the complexity of a procedure, while residents who received a higher autonomy rating were more likely to be in agreement with the faculty raters (OR 1.12, 95% CI 1.06-1.19). CONCLUSIONS The results of this study suggest that general surgery residents inaccurately perceive the patient-related complexity of a given case approximately one third of the time. Greater experience and operative autonomy appear to be associated with higher complexity agreement. Future research into factors influencing perceived case complexity may provide insight into how to best implement new teaching for surgical residents regarding the concept of case complexity.
Collapse
Affiliation(s)
- Jonathan Williams
- Department of Surgery, Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, Michigan.
| | - Angela E Thelen
- Department of Surgery, Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, Michigan
| | - John Luckoski
- Department of Surgery, Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, Michigan
| | - Xilin Chen
- Department of Surgery, Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, Michigan
| | - Brian C George
- Department of Surgery, Center for Surgical Training and Research, Michigan Medicine, Ann Arbor, Michigan
| |
Collapse
|
11
|
Wojcik BM, McKinley SK, Fong ZV, Mansur A, Bloom JP, Amari N, Hamdi I, Chang DC, Petrusa E, Mullen JT, Phitayakorn R. The Resident-Run Minor Surgery Clinic: A Four-Year Analysis of Patient Outcomes, Satisfaction, and Resident Education. JOURNAL OF SURGICAL EDUCATION 2021; 78:1838-1850. [PMID: 34092535 DOI: 10.1016/j.jsurg.2021.04.003] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Revised: 03/04/2021] [Accepted: 04/01/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVE A resident-run minor surgery clinic was developed to increase resident procedural autonomy. We evaluated whether 1) there was a significant difference between complications and patient satisfaction when procedures were independently performed by surgical residents vs. a surgical attending and 2) if participation was associated with an increase in resident procedural confidence. DESIGN Third year general surgery residents participated in a weekly procedure clinic from 2014-2018. Post-procedure complications and patient satisfaction were compared between patients operated on by residents vs. the staff surgeon. Residents were surveyed regarding their confidence in independently performing a variety of clinic-based patient care tasks. SETTING Massachusetts General Hospital General in Boston, MA; an academic tertiary care general surgery residency program. PARTICIPANTS Post-graduate year three general surgery residents that ran the clinic as part of a general surgery rotation. RESULTS 1230 patients underwent 1592 procedures (612 in resident clinic, 980 in attending clinic). There was no significant difference in the 30-day complication rate between patients operated on by the resident vs. attending (2.5% vs. 1.9%, p = 0.49). 459 patient satisfaction surveys were administered with a 79.1% response rate. There was no significant difference in the overall quality of care rating between residents and the attending surgeon (87.5% top-box rating vs. 93.1%, p = 0.15). Twenty-one residents completed both a pre- and post-rotation survey (77.8% response rate). The proportion of residents indicating that they could independently perform a variety of patient care tasks significantly increased across the rotation (all p < 0.05). CONCLUSION Mid-level general surgery residents can independently perform office-based procedures without detriment to safety or patient satisfaction. The resident-run procedure clinic serves as an environment for residents to grow in confidence in both technical and non-technical skills. Given the high rate at which patients provide resident feedback, future work may investigate how to best incorporate patient derived evaluations into resident assessment.
Collapse
Affiliation(s)
- Brandon M Wojcik
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Sophia K McKinley
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Zhi Ven Fong
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Arian Mansur
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Jordan P Bloom
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Noor Amari
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Isra Hamdi
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - David C Chang
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Emil Petrusa
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - John T Mullen
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts
| | - Roy Phitayakorn
- Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts.
| |
Collapse
|
12
|
Yildirim M, Saglam AI, Ugurlu C, Angin YS, Dasiran F, Okan I. Performance of Residents During Laparoscopic Cholecystectomy: Is Self-assessment Reliable? Surg Laparosc Endosc Percutan Tech 2021; 31:414-420. [PMID: 34156187 DOI: 10.1097/sle.0000000000000959] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 04/02/2021] [Indexed: 11/27/2022]
Abstract
BACKGROUND To compare the self-assessment of surgical residents and observers (faculty members and nurses) during laparoscopic cholecystectomy (LC). MATERIALS AND METHODS A prospective observational study was conducted between February 2020 and July 2020 at a medical school hospital. Seventy-four LC surgeries were performed by surgical residents in the presence of faculty members. A self-assessment of the technical and nontechnical performance of the residents was requested. The self-assessment of residents was compared with observer evaluations using the Kruskal-Wallis test. Gwet AC2 fit coefficient was used to determine the consistency between the observers' and residents' assessments. Bland-Altman plots were generated with 95% limits of agreement to describe the agreement between the total scores of the observers. RESULTS The self-assessment of residents had a statistically significant higher score when compared with observers (faculty and nurses) (P<0.001). However, no significant difference was observed between the total scores given by the observers (faculty members and nurses) (P>0.05). There was a moderate agreement between the resident versus faculty members [0.503; 95% confidence interval (CI), 0.430-0.576] and resident versus nurse (0.518; 95% CI, 0.432-0.605) when evaluating technical skills. However, there was substantial agreement between faculty members and nurses (0.736; 95% CI, 0.684-0.789). Postoperative pain was significantly correlated with resident self-assessment (P=0.022). CONCLUSION The self-assessment scores of surgical residents in LC operations were overestimated compared with observer assessments.
Collapse
Affiliation(s)
- Murat Yildirim
- Department of General Surgery, Faculty of Medicine, Gaziosmanpasa University, Tokat
| | - Ali I Saglam
- Department of General Surgery, Faculty of Medicine, Gaziosmanpasa University, Tokat
| | - Celil Ugurlu
- Department of General Surgery, Faculty of Medicine, Gaziosmanpasa University, Tokat
| | - Yavuz S Angin
- General Surgery Department, Çankiri State Hospital, Çankiri, Turkey
| | - Fatih Dasiran
- Department of General Surgery, Faculty of Medicine, Gaziosmanpasa University, Tokat
| | - Ismail Okan
- Department of General Surgery, Faculty of Medicine, Gaziosmanpasa University, Tokat
| |
Collapse
|
13
|
Listijono DR, Rosen DMB, Choi S, Bukhari M, Cario GM, Chou D. Cultivating the Apprentice-Mentor Model for Minimally Invasive Gynaecology in the Era of Surgically Scarce Training: A Case Report of Laparoscopic Cornuostomy for Interstitial Ectopic Pregnancy by a Trainee. Case Rep Obstet Gynecol 2021; 2021:5560309. [PMID: 33747585 PMCID: PMC7960069 DOI: 10.1155/2021/5560309] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2021] [Revised: 02/13/2021] [Accepted: 02/24/2021] [Indexed: 11/29/2022] Open
Abstract
Over the last few years, there is an apparent growing concern amongst O&G trainees of the inadequacy in exposure to minimally invasive gynaecology surgical training, which has been inadvertently compounded by the more stringent working hour regulations and disproportionately increasing number of trainees relative to surgical volume. Therefore, it is vitally important for trainees to maximise opportunities in the operating theatre and develop autonomy in carrying out more complex surgical procedures. This case report outlines the step-by-step approach of laparoscopic excision of a cornual ectopic pregnancy performed by a trainee under the supervision of a surgical mentor. This manuscript highlights key characteristic traits of a trainee that serve to foster surgical trust and simple but effective steps to foster surgical preparedness.
Collapse
Affiliation(s)
- Dave R. Listijono
- Sydney Women's Endosurgery Centre (SWEC), Sydney, Australia
- IVF Australia, Sydney, Australia
| | | | - Sarah Choi
- Sydney Women's Endosurgery Centre (SWEC), Sydney, Australia
| | | | | | - Danny Chou
- Sydney Women's Endosurgery Centre (SWEC), Sydney, Australia
| |
Collapse
|
14
|
Kendrick DE, Clark MJ, Fischer I, Bohnen JD, Kim GJ, George BC. The reliability of resident self-evaluation of operative performance. Am J Surg 2020; 222:341-346. [PMID: 33309252 DOI: 10.1016/j.amjsurg.2020.11.054] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2020] [Revised: 11/06/2020] [Accepted: 11/29/2020] [Indexed: 11/26/2022]
Abstract
BACKGROUND Self-assessment is critical to professional self-regulation yet many trainees may not reliably self-evaluate. We examine the gap between resident and faculty perceptions of trainee operative performance and contributing factors. METHODS Surgery resident and faculty evaluations of trainee performance were collected from 14 academic institutions using smartphone-based performance assessments. Differences in resident/faculty ratings evaluating the same procedure were analyzed using descriptive statistics and Bayesian mixed models. RESULTS Of 7382 evaluations, 46% trainees and faculty performance ratings were discrepant (r = 0.47), with 80% residents rating themselves lower than faculty in those cases. This gap existed regardless of case complexity and widened as trainees gained experience. Trainees who overrated themselves had the lowest mean performance scores from faculty. CONCLUSION Half of residents perceived their performance differently from faculty, and this difference widened for senior residents. Future focus should be to provide opportunity for trainees to improve skills to reliably assess themselves before graduation.
Collapse
Affiliation(s)
- Daniel E Kendrick
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, 010-A193 North Campus Research Complex, 2800 Plymouth Road Ann Arbor, MI, 48109, United States.
| | - Michael J Clark
- Consulting for Statistics, Computing and Analytics Research, University of Michigan, 3550 Rackham, 915 E. Washington St. Ann Arbor, MI, 48109, United States
| | - Ilana Fischer
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, 010-A193 North Campus Research Complex, 2800 Plymouth Road Ann Arbor, MI, 48109, United States
| | - Jordan D Bohnen
- Department of Surgery, Harvard Medical School, Massachusetts General Hospital, White 506, 55 Fruit Street, Boston, MA, 02114, United States
| | - Grace J Kim
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, 010-A193 North Campus Research Complex, 2800 Plymouth Road Ann Arbor, MI, 48109, United States
| | - Brian C George
- Center for Surgical Training and Research, Department of Surgery, University of Michigan, 010-A193 North Campus Research Complex, 2800 Plymouth Road Ann Arbor, MI, 48109, United States
| |
Collapse
|