1
|
Haischer-Rollo G, Geringer JL, Thomas K, Hale D. Do You See What I See? A Comparison of CCC and Self-Assigned Milestones Across Military Medical Specialties. JOURNAL OF SURGICAL EDUCATION 2024; 81:647-655. [PMID: 38553366 DOI: 10.1016/j.jsurg.2024.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/22/2023] [Revised: 01/08/2024] [Accepted: 02/02/2024] [Indexed: 04/26/2024]
Abstract
BACKGROUND The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to complete competency-based assessments of medical trainees based on nationally established Milestones. Previous research demonstrates a strong correlation between CCC and resident scores on the Milestones in surgery, but little is known if this is true between specialties. In this study, we investigated a variety of specialties and sought to determine what factors affect self-assessment of milestones. In addition, a post-hoc analysis was completed on the COVID-19 pandemic effects on self-evaluation. METHODS This is an IRB approved observational study on prospectively collected self-evaluation milestone data that is used within each ACGME program's Clinical Competency Committees. Medical trainees within the San Antonio Uniformed Services Health Education Consortium were approached for possible participation in this study with permission from program directors. RESULTS There was no significant difference between self-assessments and CCC-assessments based on self-identified gender or residency type (surgical versus nonsurgical) for any milestone domain. Within the postgraduate year (PGY) groups, the PGY5 and PGY6 tended to rate themselves higher than CCC. Chiefs (Internal Medicine PGY2/3, and General Surgery PGY5/6) tended to be more accurate in scoring themselves than the interns (PGY1) within the milestone of Interpersonal Skills and Communication (chiefs 0.5 vs. interns 0.62, p = 0.03). On post hoc analysis of self-rating, during the first wave of the COVID 19 pandemic, Post-Covid residents were more likely to underrate themselves in Systems-Based Practice compared to the Pre-Covid cohort (-0.49 vs 0.10; p = 0.007) and more likely to rate themselves higher in Professionalism (-0.54 vs. -0.10, p = 0.012). CONCLUSION Unique to this study and our institution, there was no gender difference found in self vs CCC evaluations. With the change in learning environment from COVID, there was also a change in ability for some learners to self-assess accurately. As medical educators, we should understand the importance of both encouraging learners to practice self-assessment as well as give feedback to trainees on their progress. We also need to educate our faculty on the use of milestones for assessment to create a true gold standard in the CCC.
Collapse
Affiliation(s)
| | - Jamie Lynn Geringer
- Uniformed Services University, Bethesda, Maryland, USA; Department of Medicine, Brooke Army Medical Center, San Antonio, Texas
| | - Katryna Thomas
- Department of Surgery, Brooke Army Medical Center, San Antonio, Texas, Bethesda, Maryland, USA
| | - Diane Hale
- Department of Surgery, Brooke Army Medical Center, San Antonio, Texas, Bethesda, Maryland, USA.
| |
Collapse
|
2
|
Baynouna AlKetbi L, Nagelkerke N, AlZarouni AA, AlKuwaiti MM, AlDhaheri R, AlNeyadi AM, AlAlawi SS, AlKuwaiti MH. Assessing the impact of adopting a competency-based medical education framework and ACGME-I accreditation on educational outcomes in a family medicine residency program in Abu Dhabi Emirate, United Arab Emirates. Front Med (Lausanne) 2024; 10:1257213. [PMID: 38259827 PMCID: PMC10802161 DOI: 10.3389/fmed.2023.1257213] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Accepted: 11/24/2023] [Indexed: 01/24/2024] Open
Abstract
Background Competency-Based Medical Education (CBME) is now mandated by many graduate and undergraduate accreditation standards. Evaluating CBME is essential for quantifying its impact, finding supporting evidence for the efforts invested in accreditation processes, and determining future steps. The Ambulatory Healthcare Services (AHS) family medicine residency program has been accredited by the Accreditation Council of Graduate Medical Education-International (ACGME-I) since 2013. This study aims to report the Abu Dhabi program's experience in implementing CBME and accreditation. Objectives Compare the two residents' cohorts' performance pre-and post-ACGME-I accreditation.Study the bi-annually reported milestones as a graduating residents' performance prognostic tool. Methods All residents in the program from 2008 to 2019 were included. They are called Cohort one-the intake from 2008 to 2012, before the ACGME accreditation, and Cohort two-the intake from 2013 to 2019, after the ACGME accreditation, with the milestones used. The mandatory annual in-training exam was used as an indication of the change in competency between the two cohorts. Among Cohort two ACGME-I, the biannual milestones data were studied to find the correlation between residents' early and graduating milestones. Results A total of 112 residents were included: 36 in Cohort one and 76 in Cohort two. In Cohort one, before the ACGME accreditation, no significant associations were identified between residents' graduation in-training exam and their early performance indicators, while in Cohort two, there were significant correlations between almost all performance metrics. Early milestones are correlated with the graduation in-training exam score. Linear regression confirmed this relationship after controlling the residents' undergraduate Grade Point Average (GPA). Competency development continues to improve even after residents complete training at Post Graduate Year, PGY4, as residents' achievement in PGY5 continues to improve. Conclusion Improved achievement of residents after the introduction of the ACGME-I accreditation is evident. Additionally, the correlation between the graduation in-training exam and graduation milestones, with earlier milestones, suggests a possible use of early milestones in predicting outcomes.
Collapse
Affiliation(s)
| | - Nico Nagelkerke
- Community Medicine Department, UAEU, Al Ain, United Arab Emirates
| | - Amal A. AlZarouni
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| | - Mariam M. AlKuwaiti
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| | - Ruwaya AlDhaheri
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| | - Amna M. AlNeyadi
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| | - Shamma S. AlAlawi
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| | - Mouza H. AlKuwaiti
- Abu Dhabi Healthcare Services, Ambulatory Healthcare Services, Al Ain, United Arab Emirates
| |
Collapse
|
3
|
Reed S, Mink R, Stanek J, Tyrrell L, Li STT. Are Final Residency Milestones Correlated With Early Fellowship Performance in Pediatrics? ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:1069-1075. [PMID: 36972134 DOI: 10.1097/acm.0000000000005215] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
PURPOSE Milestones have been used to assess trainees across graduate medical education programs and reflect a developmental continuum from novice to expert. This study examined whether residency milestones are correlated with initial fellowship milestone performance in pediatrics. METHOD This retrospective cohort study used descriptive statistics to assess milestone scores from pediatric fellows who began fellowship training between July 2017 and July 2020. Milestone scores were obtained at the end of residency (R), middle of the first fellowship year (F1), and end of the first fellowship year (F2). RESULTS Data represent 3,592 unique trainees. High composite R scores, much lower F1 scores, and slightly higher F2 scores were found over time for all pediatric subspecialities. R scores were positively correlated with F1 scores (Spearman ρ = 0.12, P < .001) and F2 scores (Spearman ρ = 0.15, P < .001). Although scores are negligibly different when trainees graduate from residency, there were differences in F1 and F2 scores among fellows in different specialties. Those who trained at the same institution for residency and fellowship had higher composite milestone F1 and F2 scores compared with those who trained at different institutions ( P < .001). The strongest associations were between R and F2 scores for the professionalism and communication milestones, although associations were still relatively weak overall (r s = 0.13-0.20). CONCLUSIONS This study found high R scores and low F1 and F2 scores across all shared milestones, with weak association of scores within competencies, indicating that milestones are context dependent. Although professionalism and communication milestones had a higher correlation compared with the other competencies, the association was still weak. Residency milestones may be useful for individualized education in early fellowship, but fellowship programs should be cautious about overreliance on R scores due to the weak correlation with F1 and F2 scores.
Collapse
Affiliation(s)
- Suzanne Reed
- S. Reed is associate professor and pediatric residency associate program director, Department of Pediatrics, The Ohio State University College of Medicine/Nationwide Children's Hospital, Columbus, Ohio
| | - Richard Mink
- R. Mink is professor, University of California, Los Angeles, Los Angeles, California, and director, Association of Pediatric Program Directors Subspecialty Pediatrics Investigator Network, McLean, Virginia
| | - Joseph Stanek
- J. Stanek is a biostatistician, Division of Hematology, Oncology, and Bone Marrow Transplantation and the Biostatistics Resource, Nationwide Children's Hospital, Columbus, Ohio
| | - Laura Tyrrell
- L. Tyrrell is a pediatric hematologist and medical education specialist, Indiana Hemophilia & Thrombosis Center, Indianapolis, Indiana
| | - Su-Ting T Li
- S.-T.T. Li is professor, vice chair of education, and residency program director, Department of Pediatrics, University of California, Davis, Sacramento, California
| |
Collapse
|
4
|
Kendrick DE, Thelen AE, Chen X, Gupta T, Yamazaki K, Krumm AE, Bandeh-Ahmadi H, Clark M, Luckoscki J, Fan Z, Wnuk GM, Ryan AM, Mukherjee B, Hamstra SJ, Dimick JB, Holmboe ES, George BC. Association of Surgical Resident Competency Ratings With Patient Outcomes. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:813-820. [PMID: 36724304 DOI: 10.1097/acm.0000000000005157] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
PURPOSE Accurate assessment of clinical performance is essential to ensure graduating residents are competent for unsupervised practice. The Accreditation Council for Graduate Medical Education milestones framework is the most widely used competency-based framework in the United States. However, the relationship between residents' milestones competency ratings and their subsequent early career clinical outcomes has not been established. It is important to examine the association between milestones competency ratings of U.S. general surgical residents and those surgeons' patient outcomes in early career practice. METHOD A retrospective, cross-sectional study was conducted using a sample of national Medicare claims for 23 common, high-risk inpatient general surgical procedures performed between July 1, 2015, and November 30, 2018 (n = 12,400 cases) by nonfellowship-trained U.S. general surgeons. Milestone ratings collected during those surgeons' last year of residency (n = 701 residents) were compared with their risk-adjusted rates of mortality, any complication, or severe complication within 30 days of index operation during their first 2 years of practice. RESULTS There were no associations between mean milestone competency ratings of graduating general surgery residents and their subsequent early career patient outcomes, including any complication (23% proficient vs 22% not yet proficient; relative risk [RR], 0.97, [95% CI, 0.88-1.08]); severe complication (9% vs 9%, respectively; RR, 1.01, [95% CI, 0.86-1.19]); and mortality (5% vs 5%; RR, 1.07, [95% CI, 0.88-1.30]). Secondary analyses yielded no associations between patient outcomes and milestone ratings specific to technical performance, or between patient outcomes and composites of operative performance, professionalism, or leadership milestones ratings ( P ranged .32-.97). CONCLUSIONS Milestone ratings of graduating general surgery residents were not associated with the patient outcomes of those surgeons when they performed common, higher-risk procedures in a Medicare population. Efforts to improve how milestones ratings are generated might strengthen their association with early career outcomes.
Collapse
Affiliation(s)
- Daniel E Kendrick
- D.E. Kendrick is assistant professor, Department of Surgery, University of Minnesota, Minneapolis, Minnesota
| | - Angela E Thelen
- A.E. Thelen is research fellow, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Xilin Chen
- X. Chen is research analyst, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Tanvi Gupta
- T. Gupta is research analyst, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Kenji Yamazaki
- K. Yamazaki is senior data analyst, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | - Andrew E Krumm
- A.E. Krumm is assistant professor, Department of Learning Health Sciences, University of Michigan, Ann Arbor, Michigan
| | - Hoda Bandeh-Ahmadi
- H. Bandeh-Ahmadi is project manager, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Michael Clark
- M. Clark is a biostatistician, Consulting for Statistics, Computing, and Analytics Research, University of Michigan, Ann Arbor, Michigan
| | - John Luckoscki
- J. Luckoscki is research fellow, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Zhaohui Fan
- Z. Fan is research analyst, Center for Healthcare Outcomes and Policy, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Greg M Wnuk
- G.M. Wnuk is program manager, Center for Surgical Training and Research, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Andrew M Ryan
- A.M. Ryan is professor, Department of Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, Michigan
| | - Bhramar Mukherjee
- B. Mukherjee is professor and chair, Division of Biostatistics, School of Public Health, University of Michigan, Ann Arbor, Michigan
| | - Stanley J Hamstra
- S.J. Hamstra is professor, Department of Surgery, University of Toronto, Toronto, Ontario, Canada
| | - Justin B Dimick
- J.B. Dimick is professor and chair, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| | - Eric S Holmboe
- E.S. Holmboe is chief research, Milestone Development, and evaluation officer, Accreditation Council for Graduate Medical Education, Chicago, Illinois
| | - Brian C George
- B.C. George is director, Center for Surgical Training and Research, and assistant professor, Department of Surgery, University of Michigan, Ann Arbor, Michigan
| |
Collapse
|
5
|
Tyrrell LJ, Stanek JR, Stewart C, Reed S. Accreditation Council for Graduate Medical Education Milestone Scores in Pediatrics: Pilot Study Exploring the Relationship Between Residency and Early Fellowship Scores. Acad Pediatr 2023; 23:178-184. [PMID: 35934278 DOI: 10.1016/j.acap.2022.06.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 06/22/2022] [Accepted: 06/28/2022] [Indexed: 01/19/2023]
Abstract
BACKGROUND The Accreditation Council for Graduate Medical Education requires Milestone-based assessments of residents and fellows. In pediatrics, 11 subcompetencies are common to both residency and subspecialty fellowship training. It is unknown whether Milestone scores achieved during residency are related to Milestone scores achieved in early fellowship. OBJECTIVE To assess the relationship between final residency Milestones scores and first-year fellowship Milestones scores in the 11 common subcompetencies (CSCs) across pediatric subspecialties. METHODS This was a retrospective single-institution cohort study of pediatric fellows beginning fellowship training between July 2016 and July 2019. De-identified Milestone score sets for final residency scores (R), mid-year first-year fellowship scores (F1), and final first-year fellowship scores (F2) were collected. Spearman correlation and regression analyses were used to assess score relationships. RESULTS Data for 85 of 98 eligible fellows were collected. Consistently, the F1 scores were lowest, and the R scores were highest. There was a weak positive correlation between the composite R scores and the composite F1 scores. There was a weak positive correlation between residency and fellowship scores for 6 CSCs and no significant correlation for the remaining 5. CONCLUSION For the 11 pediatric CSCs, the final residency Milestone scores are consistently higher than and only weakly associated with early fellowship Milestone scores. There may be limitations to the use of residency scores for fellowship program directors in guiding individualized education for early fellows. This study provides groundwork for additional study of Milestone relationships and may help inform the next iteration of pediatric subspecialty Milestones.
Collapse
Affiliation(s)
- Laura J Tyrrell
- Division of Hematology/Oncology/BMT, Nationwide Children's Hospital (LJ Tyrrell, JR Stanek, and S Reed), Columbus, Ohio.
| | - Joseph R Stanek
- Division of Hematology/Oncology/BMT, Nationwide Children's Hospital (LJ Tyrrell, JR Stanek, and S Reed), Columbus, Ohio; Biostatistics Resource at Nationwide Children's Hospital (JR Stanek), Columbus, Ohio
| | - Claire Stewart
- Division of Pediatric Critical Care Medicine, Nationwide Children's Hospital (C Stewart), Columbus, Ohio
| | - Suzanne Reed
- Division of Hematology/Oncology/BMT, Nationwide Children's Hospital (LJ Tyrrell, JR Stanek, and S Reed), Columbus, Ohio
| |
Collapse
|
6
|
St-Onge C, Boileau E, Langevin S, Nguyen LHP, Drescher O, Bergeron L, Thomas A. Stakeholders' perception on the implementation of Developmental Progress Assessment: using the Theoretical Domains Framework to document behavioral determinants. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022; 27:735-759. [PMID: 35624332 DOI: 10.1007/s10459-022-10119-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Accepted: 04/23/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND The widespread implementation of longitudinal assessment (LA) to document trainees' progression to independent practice rests more on speculative rather than evidence-based benefits. We aimed to document stakeholders' knowledge of- and attitudes towards LA, and identify how the supports and barriers can help or hinder the uptake and sustainable use of LA. METHODS We interviewed representatives from four stakeholder groups involved in LA. The interview protocols were based on the Theoretical Domains Framework (TDF), which contains a total of 14 behaviour change determinants. Two team members coded the interviews deductively to the TDF, with a third resolving differences in coding. The qualitative data analysis was completed with iterative consultations and discussions with team members until consensus was achieved. Saliency analysis was used to identify dominant domains. RESULTS Forty-one individuals participated in the study. Three dominant domains were identified. Participants perceive that LA has more positive than negative consequences and requires substantial ressources. All the elements and characteristics of LA are present in our data, with differences between stakeholders. CONCLUSION Going forward, we could develop and implement tailored and theory driven interventions to promote a shared understanding of LA, and maintain potential positive outcomes while reducing negative ones. Furthermore, ressources to support LA implementation need to be addressed to facilitate its uptake.
Collapse
Affiliation(s)
- Christina St-Onge
- Université de Sherbrooke, Christina St-Onge, 3001 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada.
| | - Elisabeth Boileau
- Université de Sherbrooke, Christina St-Onge, 3001 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada
| | - Serge Langevin
- Université de Sherbrooke, Christina St-Onge, 3001 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada
| | | | | | - Linda Bergeron
- Université de Sherbrooke, Christina St-Onge, 3001 12e Avenue Nord, Sherbrooke, QC, J1H 5N4, Canada
| | | |
Collapse
|
7
|
Feeney C, Hotez E, Wan L, Bishop L, Timmerman J, Haley M, Kuo A, Fernandes P. A Multi-Institutional Collaborative To Assess the Knowledge and Skills of Medicine-Pediatrics Residents in Health Care Transition. Cureus 2022; 13:e20327. [PMID: 35028223 PMCID: PMC8748002 DOI: 10.7759/cureus.20327] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 12/09/2021] [Indexed: 01/23/2023] Open
Abstract
Background Pediatric to adult health care transition (HCT) is an essential process in the care of youth with special health care needs (YSHCN). Many internal medicine-pediatrics (med-peds) residency programs have developed curricula to teach transition knowledge and skills for the care of YSHCN. Objective Using a national med-peds program director quality improvement collaborative to improve transition curriculum, we aim to identify curricular content areas of improvement by describing baseline trainee knowledge and skills taught through existing transition curricula in med-peds programs. Methods We analyzed data collected during the 2018-2019 national med-peds program director quality improvement collaborative to improve transition curriculum. Program directors assessed their programs, and trainees assessed themselves on five transition goals by completing a Likert-scale questionnaire. In addition, trainees received an objective assessment of their knowledge through a multiple-choice questionnaire (MCQ). Results All 19 programs in the collaborative, and 193 of 316 trainees from these programs, completed the questionnaires. Most programs were based at academic centers (68%) and provided transition training via didactics (63%) and/or subspecialty rotations (58%). More programs had high confidence (95%) than trainees (58%) in goal 1 (knowledge and skills of the issues around transition), whereas more trainees had high confidence (60%) than programs (47%) in goal 2 (understanding the developmental and psychosocial aspects of transition). Programs and trainees self-assessed lower in goals related to health insurance, educational and vocational needs, and application of health care system knowledge to the practice environment (goals 3, 4, and 5, respectively). Conclusions Using the assessments of the program directors and resident trainees, we identified subject areas for improvement of transition curricula, including health insurance and the application of health care system knowledge to the practice environment.
Collapse
Affiliation(s)
- Colby Feeney
- Medicine and Pediatrics, Duke University School of Medicine, Durham, USA
| | - Emily Hotez
- Internal Medicine, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA
| | - Lori Wan
- Medicine and Pediatrics, University of California San Diego, San Diego, USA
| | - Laura Bishop
- Medicine and Pediatrics, University of Louisville School of Medicine, Louisville, USA
| | - Jason Timmerman
- Internal Medicine, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA
| | - Madeline Haley
- Internal Medicine and Pediatrics, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA
| | - Alice Kuo
- Internal Medicine and Pediatrics, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA
| | - Priyanka Fernandes
- Internal Medicine and Pediatrics, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA
| |
Collapse
|
8
|
Ottum S, Chao C, Tamakuwala S, Dean J, Shafi A, Kramer KJ, Kaur S, Recanati MA. Can ACGME Milestones predict surgical specialty board passage: an example in Obstetrics and Gynecology. CLIN EXP OBSTET GYN 2021; 48:1048-1055. [PMID: 34720368 PMCID: PMC8555913 DOI: 10.31083/j.ceog4805168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Background: Multiple tools including Accreditation Council for Graduate Medical Education (ACGME) standardized milestones can be utilized to assess trainee and residency program performance. However, little is known regarding the objective validation of these tools in predicting written board passage. Methods: In this retrospective study, data was gathered on n = 45 Wayne State University Obstetrics and Gynecology program graduates over the five-year period ending July 2018. United States Medical Licensing Examination (USMLE) scores, Council on Resident Education in Obstetrics and Gynecology (CREOG) in-training scores and ACGME milestones were used to predict American Board of Obstetrics and Gynecology (ABOG) board passage success on first attempt. Significance was set at p < 0.05. Results: Written board passage was associated with average CREOGs (p = 0.01) and milestones (p = 0.008) while USMLE1 was not significantly associated (p = 0.055). USMLE1 <217 (Positive predictive value (PPV) = 96%). CREOGs <197 (PPV = 100%) and milestones <3.25 (PPV = 100%), particularly practice-based learning and systems-based practice milestones were most strongly correlated with board failure. Using a combination of these two milestones, it is possible to correctly predict board passage using our model (PPV = 86%). Discussion: This study is the first validating the utility of milestones in a surgical specialty by demonstrating their ability to predict board passage. Residents with CREOGs or milestones below thresholds are at risk for board failure and may warrant early intervention.
Collapse
Affiliation(s)
- Sarah Ottum
- Department of Obstetrics and Gynecology, University of Cincinnati, Cincinnati, OH 45267, USA
| | - Conrad Chao
- Department of Maternal-Fetal Medicine, University of New Mexico School of Medicine, Albuquerque, NM 87131, USA
| | - Sejal Tamakuwala
- Department of Obstetrics and Gynecology, Emory University, Atlanta, GA 30322, USA
| | - Joshua Dean
- Department of Obstetrics and Gynecology, Beaumont, Royal Oak, MI 48073, USA
| | - Adib Shafi
- Department of Computer Sciences, Wayne State University, Detroit, MI 48201, USA
| | - Katherine Jennifer Kramer
- Department of Obstetrics and Gynecology, St. Vincent's Catholic Medical Centers, New York, NY 10011, USA
| | - Satinder Kaur
- Department of Obstetrics and Gynecology, Wayne State University, Detroit, MI 48201, USA
| | - Maurice-Andre Recanati
- Department of Obstetrics and Gynecology and NIH-Women's Reproductive Health Research Scholar, Wayne State University, Detroit, MI 48201, USA
| |
Collapse
|
9
|
Renaud JS, Lacasse M, Côté L, Théorêt J, Rheault C, Simard C. Psychometric validation of the Laval developmental benchmarks scale for family medicine. BMC MEDICAL EDUCATION 2021; 21:357. [PMID: 34176475 PMCID: PMC8237442 DOI: 10.1186/s12909-021-02797-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Accepted: 06/17/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND With the implementation of competency-based education in family medicine, there is a need for summative end-of-rotation assessments that are criterion-referenced rather than normative. Laval University's family residency program therefore developed the Laval Developmental Benchmarks Scale for Family Medicine (DBS-FM), based on competency milestones. This psychometric validation study investigates its internal structure and its relation to another variable, two sources of validity evidence. METHODS We used assessment data from a cohort of residents (n = 1432 assessments) and the Rasch Rating Scale Model to investigate its reliability, dimensionality, rating scale functioning, targeting of items to residents' competency levels, biases (differential item functioning), items hierarchy (adequacy of milestones ordering), and score responsiveness. Convergent validity was estimated by its correlation with the clinical rotation decision (pass, in difficulty/fail). RESULTS The DBS-FM can be considered as a unidimensional scale with good reliability for non-extreme scores (.83). The correlation between expected and empirical items hierarchies was of .78, p < .0001.Year 2 residents achieved higher scores than year 1 residents. It was associated with the clinical rotation decision. CONCLUSION Advancing its validation, this study found that the DBS-FM has a sound internal structure and demonstrates convergent validity.
Collapse
Affiliation(s)
- Jean-Sébastien Renaud
- Department of Family and Emergency Medicine, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada.
- Office of Education and Continuing Professional Development, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada.
- Primary Care Research Centre affiliated with Laval University (CERSSPL-U, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada.
| | - Miriam Lacasse
- Department of Family and Emergency Medicine, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
- Educational Leadership Chair in Health Professions Education CMA-MDM, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
| | - Luc Côté
- Department of Family and Emergency Medicine, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
| | - Johanne Théorêt
- Department of Family and Emergency Medicine, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
| | - Christian Rheault
- Department of Family and Emergency Medicine, Laval University, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
| | - Caroline Simard
- Educational Leadership Chair in Health Professions Education CMA-MDM, 1050, avenue de la Médecine, Université Laval, Québec, G1V 0A6, Canada
| |
Collapse
|
10
|
Hanrahan JG, Sideris M, Pasha T, Dedeilia A, Papalois A, Papalois V. Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:285-295. [PMID: 32889945 DOI: 10.1097/acm.0000000000003712] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
PURPOSE Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries. METHOD Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries. RESULTS From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified. CONCLUSIONS This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.
Collapse
Affiliation(s)
- John Gerrard Hanrahan
- J.G. Hanrahan is academic foundation doctor, Department of General Surgery, Lister Hospital, Stevenage, United Kingdom
| | - Michail Sideris
- M. Sideris is a specialty trainee in obstetrics and gynaecology, Women's Health Research Unit, Queen Mary University of London, London, United Kingdom
| | - Terouz Pasha
- T. Pasha is a final-year medical student, Faculty of Life Sciences and Medicine, King's College London, London, United Kingdom
| | - Aikaterini Dedeilia
- A. Dedeilia is a final-year medical student, School of Medicine, National and Kapodistrian University of Athens, Athens, Greece
| | - Apostolos Papalois
- A. Papalois is director, ELPEN Research & Experimental Centre, Athens, Greece
| | - Vassilios Papalois
- V. Papalois is professor of transplant surgery, Department of Surgery and Cancer, Renal Transplant Directorate, Imperial College Healthcare NHS Trust, London, United Kingdom
| |
Collapse
|
11
|
Naples R, French JC, Thomas JD, Khandelwal C, Rosen MJ, Lipman JM. Utilization of a quality reporting system to increase faculty participation in resident operative assessment. Surgery 2020; 169:483-487. [PMID: 33328137 DOI: 10.1016/j.surg.2020.11.014] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2020] [Revised: 10/07/2020] [Accepted: 11/10/2020] [Indexed: 11/29/2022]
Abstract
BACKGROUND A quality collaborative across our hospital system was initiated to track surgical outcomes. We sought to determine whether incorporating a resident operative performance assessment into this quality collaborative would increase the quantity and quality of these assessments and impact relevant milestones. METHODS A resident operative assessment was added to a quality reporting system required to be completed by faculty at the completion of 2 operations. Three milestones directly related to operative performance were analyzed-Patient Care 3, Medical Knowledge 2, and Interpersonal and Communication Skills 3. Residents were divided in 2 groups: quality collaborative (≥10 operative assessments) and no quality collaborative (<10 operative assessments). Milestones from Spring 2019 and Fall 2019 were analyzed. RESULTS Faculty participation was 86% with 407 assessments completed from February to October 2019. A difference in the rate of change in resident performance for Patient Care 3 (+0.95 vs +0.55; P = .04) and Interpersonal and Communication Skills 3 (+1.05 vs +0.52; P = .02) was observed for those residents in the quality collaborative group (n = 20) compared with baseline data. CONCLUSION Addition of an operative assessment to a mandatory quality collaborative increases faculty participation and impacts resident milestone determination. These findings highlight opportunities to find innovative and efficient methods to improve faculty engagement.
Collapse
Affiliation(s)
- Robert Naples
- Department of General Surgery, Cleveland Clinic, Cleveland, OH.
| | - Judith C French
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | - Jonah D Thomas
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | | | - Michael J Rosen
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| | - Jeremy M Lipman
- Department of General Surgery, Cleveland Clinic, Cleveland, OH
| |
Collapse
|
12
|
St-Onge C, Vachon Lachiver É, Langevin S, Boileau E, Bernier F, Thomas A. Lessons from the implementation of developmental progress assessment: A scoping review. MEDICAL EDUCATION 2020; 54:878-887. [PMID: 32083743 DOI: 10.1111/medu.14136] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Revised: 01/21/2020] [Accepted: 02/06/2020] [Indexed: 06/10/2023]
Abstract
OBJECTIVES Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts. METHODS We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results. RESULTS Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences. CONCLUSIONS Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.
Collapse
Affiliation(s)
- Christina St-Onge
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Élise Vachon Lachiver
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Serge Langevin
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Elisabeth Boileau
- Department of Family and Emergency Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Frédéric Bernier
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Research Center - Sherbrooke University Hospital Center (CHUS), Integrated Health and Social Service Centers (CISSS) and Integrated University Health and Social Service Centres (CIUSSS), Sherbrooke, Québec, Canada
| | - Aliki Thomas
- School of Physical and Occupational Therapy, McGill University, Montreal, Québec, Canada
| |
Collapse
|
13
|
Application Factors May Not Be Predictors of Success Among General Surgery Residents as Measured by ACGME Milestones. J Surg Res 2020; 253:34-40. [DOI: 10.1016/j.jss.2020.03.029] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2019] [Revised: 12/16/2019] [Accepted: 03/10/2020] [Indexed: 11/22/2022]
|
14
|
Perkins SQ, Dabaja A, Atiemo H. Best Approaches to Evaluation and Feedback in Post-Graduate Medical Education. Curr Urol Rep 2020; 21:36. [PMID: 32789759 DOI: 10.1007/s11934-020-00991-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
PURPOSE OF REVIEW The objectives of this literature review are to appraise current approaches and assess new technologies that have been utilized for evaluation and feedback of residents, with focus on surgical trainees. RECENT FINDINGS In 1999, the Accreditation Council for Graduate Medical Education introduced the Milestone system as a tool for summative evaluation. The organization allows individual program autonomy on how evaluation and feedback are performed. In the past, questionnaire evaluations and informal verbal feedback were employed. However, with the advent of technology, they have taken a different shape in the form of crowdsourcing, mobile platforms, and simulation. Limited data is available on new methods but studies show promise citing low cost and positive impact on resident education. No one "best approach" exists for evaluation and feedback. However, it is apparent that a multimodal approach that is based on the ACGME Milestones can be effective and aid in guiding programs.
Collapse
Affiliation(s)
- Sara Q Perkins
- Henry Ford Health System, 2799 W Grand Blvd, K9, Detroit, MI, 48202, USA
| | - Ali Dabaja
- Henry Ford Health System, 2799 W Grand Blvd, K9, Detroit, MI, 48202, USA
| | - Humphrey Atiemo
- Henry Ford Health System, 2799 W Grand Blvd, K9, Detroit, MI, 48202, USA.
| |
Collapse
|
15
|
Utility of Residency Milestones Reported to Fellowship Directors: A National Survey of Pediatric Fellowship Program Directors. Acad Pediatr 2020; 20:696-702. [PMID: 31978601 DOI: 10.1016/j.acap.2020.01.004] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/06/2019] [Revised: 01/07/2020] [Accepted: 01/10/2020] [Indexed: 02/02/2023]
Abstract
OBJECTIVE The Accreditation Council for Graduate Medical Education recently made available final residency Milestones for first-year fellows to fellowship program directors (FPDs). Usefulness of residency Milestones for fellows is unknown. Our objective was to determine how many pediatric FPDs downloaded final residency Milestones for their first-year fellows and FPD perspectives about usefulness of residency Milestones. METHODS Mixed methods survey of pediatric FPDs, assessing FPD use of residency Milestones for first-year fellows, and FPD opinions about utility of residency Milestones for fellowship, including during fellow recruitment. Quantitative data were analyzed using descriptive statistics. Qualitative data were analyzed using content analysis. RESULTS The response rate was 67.8% (544 of 802). Only 39.3% (209 of 532) of FPDs downloaded final residency Milestones for their first-year fellows. Twenty-four percent (129 of 532) of all FPDs thought residency Milestones were useful. Forty-one percent (218 of 532) thought residency Milestones would be useful during recruitment; others believed this may harm applicants. Of FPDs that downloaded and reviewed residency Milestones, 27% (50 of 185) used them for individualized education. FPDs felt residency Milestones might allow for identification of trainee needs and baseline assessments, but thought that residency Milestones had limited usefulness during fellowship due to concerns about lack of validity evidence, relevance, and how Milestones are assessed and reported. CONCLUSIONS Most FPDs find residency Milestones to be of limited utility for their fellows and do not use residency Milestones to tailor education for their first-year fellows. Improving relevance of residency Milestones to fellowship training, validity, and how Milestones are assessed and reported may improve their usefulness for fellow training.
Collapse
|
16
|
Abstract
Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.
Collapse
|
17
|
Tekian A, Park YS, Tilton S, Prunty PF, Abasolo E, Zar F, Cook DA. Competencies and Feedback on Internal Medicine Residents' End-of-Rotation Assessments Over Time: Qualitative and Quantitative Analyses. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2019; 94:1961-1969. [PMID: 31169541 PMCID: PMC6882536 DOI: 10.1097/acm.0000000000002821] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
PURPOSE To examine how qualitative narrative comments and quantitative ratings from end-of-rotation assessments change for a cohort of residents from entry to graduation, and explore associations between comments and ratings. METHOD The authors obtained end-of-rotation quantitative ratings and narrative comments for 1 cohort of internal medicine residents at the University of Illinois at Chicago College of Medicine from July 2013-June 2016. They inductively identified themes in comments, coded orientation (praising/critical) and relevance (specificity and actionability) of feedback, examined associations between codes and ratings, and evaluated changes in themes and ratings across years. RESULTS Data comprised 1,869 assessments (828 comments) on 33 residents. Five themes aligned with ACGME competencies (interpersonal and communication skills, professionalism, medical knowledge, patient care, and systems-based practice), and 3 did not (personal attributes, summative judgment, and comparison to training level). Work ethic was the most frequent subtheme. Comments emphasized medical knowledge more in year 1 and focused more on autonomy, leadership, and teaching in later years. Most comments (714/828 [86%]) contained high praise, and 412/828 (50%) were very relevant. Average ratings correlated positively with orientation (β = 0.46, P < .001) and negatively with relevance (β = -0.09, P = .01). Ratings increased significantly with each training year (year 1, mean [standard deviation]: 5.31 [0.59]; year 2: 5.58 [0.47]; year 3: 5.86 [0.43]; P < .001). CONCLUSIONS Narrative comments address resident attributes beyond the ACGME competencies and change as residents progress. Lower quantitative ratings are associated with more specific and actionable feedback.
Collapse
Affiliation(s)
- Ara Tekian
- A. Tekian is professor and associate dean for international affairs, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-9252-1588
| | - Yoon Soo Park
- Y.S. Park is associate professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois; ORCID: http://orcid.org/0000-0001-8583-4335
| | - Sarette Tilton
- S. Tilton is a PharmD candidate, University of Illinois at Chicago College of Pharmacy, Chicago, Illinois
| | - Patrick F. Prunty
- P.F. Prunty is a PharmD candidate, University of Illinois at Chicago College of Pharmacy, Chicago, Illinois
| | - Eric Abasolo
- E. Abasolo is a PharmD candidate, University of Illinois at Chicago College of Pharmacy, Chicago, Illinois
| | - Fred Zar
- F. Zar is professor and program director, Department of Medicine, University of Illinois at Chicago College of Medicine, Chicago, Illinois
| | - David A. Cook
- D.A. Cook is professor of medicine and medical education and associate director, Office of Applied Scholarship and Education Science, and consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota; ORCID: https://orcid.org/0000-0003-2383-4633
| |
Collapse
|
18
|
Cortez R, Lynch K, Charpentier K, Cioffi W, Harrington D, Beard R. An Assessment of Fine Surgical KnotTying. JOURNAL OF SURGICAL EDUCATION 2019; 76:808-813. [PMID: 30824231 DOI: 10.1016/j.jsurg.2018.10.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2018] [Revised: 09/14/2018] [Accepted: 10/30/2018] [Indexed: 06/09/2023]
Abstract
OBJECTIVE Operating room simulation exercises have been well established as an effective means of improving confidence, task engagement, and learning retention among surgical residents. We have established a cost-effective model and scoring system assessing resident skills to tie secure surgical knots with minimal tension. DESIGN A circular grid divided into 18 segments was placed underlying an aluminum can. Trainees tie 20 surgical square knots scored for time and total knot length. Movement of the can outside the grid served as a scoring penalty. Recorded were time, length of the 20 knots, and number of segments exposed at exercise end. A score was developed to identify a progression of skills with PGY level. All outcomes were compared between classes using ANOVA. SETTING Brown University/Rhode Island Hospital Department of Surgery. PARTICIPANTS Surgical residents (PGY1-PGY5) and participating attending surgeons employed by Rhode Island Hospital. RESULTS Knot length and exposed segments showed trends of improved scores with ascending PGY level. Only average time attained statistical significance. Overall scores improved with PGY level: Composite scores significantly improved when comparing PGY1 to PGY3, PGY5, and Attending surgeons (p = 0.016, 0.011, and 0.011, respectively). Time significantly improved when comparing PGY1 to PGY3 and Attending surgeons (77vs. 50 and 47 seconds, p = 0.019 and 0.022 respectively). Composite scores were not significantly different above PGY3. CONCLUSIONS A low fidelity, high impact knot tying model has been developed to assess the ability to securely tie surgical knots while minimizing tension, with linear increases in scores that appear to plateau at the PGY3 level.
Collapse
Affiliation(s)
- Roberto Cortez
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island.
| | - Kenneth Lynch
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - Kevin Charpentier
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - William Cioffi
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - David Harrington
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - Rachel Beard
- Rhode Island Hospital Department of Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island
| |
Collapse
|
19
|
Program Director Perceptions of Usefulness of the Accreditation Council for Graduate Medical Education Milestones System for Urology Resident Evaluation. Urology 2019; 124:28-32. [DOI: 10.1016/j.urology.2018.10.042] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2018] [Revised: 10/05/2018] [Accepted: 10/11/2018] [Indexed: 11/18/2022]
|
20
|
What do quantitative ratings and qualitative comments tell us about general surgery residents' progress toward independent practice? Evidence from a 5-year longitudinal cohort. Am J Surg 2018; 217:288-295. [PMID: 30309619 DOI: 10.1016/j.amjsurg.2018.09.031] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2018] [Revised: 09/12/2018] [Accepted: 09/28/2018] [Indexed: 11/21/2022]
Abstract
BACKGROUND This study examines the alignment of quantitative and qualitative assessment data in end-of-rotation evaluations using longitudinal cohorts of residents progressing throughout the five-year general surgery residency. METHODS Rotation evaluation data were extracted for 171 residents who trained between July 2011 and July 2016. Data included 6069 rotation evaluations forms completed by 38 faculty members and 164 peer-residents. Qualitative comments mapped to general surgery milestones were coded for positive/negative feedback and relevance. RESULTS Quantitative evaluation scores were significantly correlated with positive/negative feedback, r = 0.52 and relevance, r = -0.20, p < .001. Themes included feedback on leadership, teaching contribution, medical knowledge, work ethic, patient-care, and ability to work in a team-based setting. Faculty comments focused on technical and clinical abilities; comments from peers focused on professionalism and interpersonal relationships. CONCLUSIONS We found differences in themes emphasized as residents progressed. These findings underscore improving our understanding of how faculty synthesize assessment data.
Collapse
|
21
|
Affiliation(s)
- Su-Ting T. Li
- Corresponding author: Su-Ting T. Li, MD, MPH, University of California, Davis, Room 220, 2516 Stockton Boulevard, Sacramento, CA 95817, 916.734.2428, fax 916.734.0342,
| |
Collapse
|