Competency champions in the clinical competency committee: a successful strategy to implement milestone evaluations and competency coaching

Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation

PURPOSE

This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME).

METHODS

A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design.

RESULTS

Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions.

CONCLUSIONS

Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Competence committees: The steep climb from concept to implementation

INTRODUCTION

Competence committees (CCs) are groups of educators tasked with reviewing resident progress throughout their training, making decisions regarding the achievement of Entrustable Professional Activities and recommendations regarding promotion and remediation. CCs have been mandated as part of competency-based medical education programmes worldwide; however, there has yet to be a thorough examination of the implementation challenges they face and how this impacts their functioning and decision-making processes. This study examined CC implementation at a Canadian institution, documenting the shared and unique challenges that CCs faced and overcame over a 3-year period.

METHODS

This study consisted of three phases, which were conceptually and analytically linked using Moran-Ellis and colleagues' notion of 'following a thread.' Phase 1 examined the early perceptions and experiences of 30 key informants using a survey and semi-structured interviews. Phase 2 provided insight into CCs' operations through a survey sent to 35 CC chairs 1-year post-implementation. Phase 3 invited 20 CC members to participate in semi-structured interviews to follow up on initial themes 2 years post-implementation. Detailed observation notes from 16 CC meetings across nine disciplines were used to corroborate the findings from each phase.

RESULTS

Response rates in each phase were 83% (n = 25), 43% (n = 15) and 60% (n = 12), respectively. Despite the high degree of support for CCs among faculty and resident members, several ongoing challenges were highlighted: adapting to programme size, optimising membership, engaging residents, maintaining capacity among members, sharing and aggregating data and developing a clear mandate.

DISCUSSION

Findings of this study reinforce the importance of resident engagement and information sharing between disciplines. Challenges faced by CCs are discussed in relation to the existing literature to inform a better understanding of group decision-making processes in medical education. Future research could compare implementation practices across sites and explore which adaptations lead to better or worse decision-making outcomes.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

Use of Resident-Sensitive Quality Measure Data in Entrustment Decision Making: A Qualitative Study of Clinical Competency Committee Members at One Pediatric Residency

PURPOSE

Resident-sensitive quality measures (RSQMs) are quality measures that are likely performed by an individual resident and are important to care quality for a given illness of interest. This study sought to explore how individual clinical competency committee (CCC) members interpret, use, and prioritize RSQMs alongside traditional assessment data when making a summative entrustment decision.

METHOD

In this constructivist grounded theory study, 19 members of the pediatric residency CCC at Cincinnati Children's Hospital Medical Center were purposively and theoretically sampled between February and July 2019. Participants were provided a deidentified resident assessment portfolio with traditional assessment data (milestone and/or entrustable professional activity ratings as well as narrative comments from 5 rotations) and RSQM performance data for 3 acute, common diagnoses in the pediatric emergency department (asthma, bronchiolitis, and closed head injury) from the emergency medicine rotation. Data collection consisted of 2 phases: (1) observation and think out loud while participants reviewed the portfolio and (2) semistructured interviews to probe participants' reviews. Analysis moved from close readings to coding and theme development, followed by the creation of a model illustrating theme interaction. Data collection and analysis were iterative.

RESULTS

Five dimensions for how participants interpret, use, and prioritize RSQMs were identified: (1) ability to orient to RSQMs: confusing to self-explanatory, (2) propensity to use RSQMs: reluctant to enthusiastic, (3) RSQM interpretation: requires contextualization to self-evident, (4) RSQMs for assessment decisions: not sticky to sticky, and (5) expectations for residents: potentially unfair to fair to use RSQMs. The interactions among these dimensions generated 3 RSQM data user profiles: eager incorporation, willing incorporation, and disinclined incorporation.

CONCLUSIONS

Participants used RSQMs to varying extents in their review of resident data and found such data helpful to varying degrees, supporting the inclusion of RSQMs as resident assessment data for CCC review.

Lessons from the implementation of developmental progress assessment: A scoping review

OBJECTIVES

Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts.

METHODS

We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results.

RESULTS

Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences.

CONCLUSIONS

Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.

Amant L, Glover Takahashi S. Managing residents in difficulty within CBME residency educational systems: a scoping review

BACKGROUND

Best practices in managing residents in difficulty (RID) in the era of competency-based medical education (CBME) are not well described. This scoping review aimed to inventory the current literature and identify major themes in the articles that address or employ CBME as part of the identification and remediation of residents in difficulty.

METHODS

Articles published between 2011 to 2017 were included if they were about postgraduate medical education, RID, and offered information to inform the structure and/or processes of CBME. All three reviewers performed a primary screening, followed by a secondary screening of abstracts of the chosen articles, and then a final comprehensive sub-analysis of the 11 articles identified as using a CBME framework.

RESULTS

Of 165 articles initially identified, 92 qualified for secondary screening; the 63 remaining articles underwent full-text abstracting. Ten themes were identified from the content analysis with "identification of RID" (41%) and "defining and classifying deficiencies" (30%) being the most frequent. In the CBME article sub-analysis, the most frequent themes were: need to identify RID (64%), improving assessment tools (45%), and roles and responsibilities of players involved in remediation (27%). Almost half of the CBME articles were published in 2016-2017.

CONCLUSIONS

Although CBME programs have been implemented for many years, articles have only recently begun specifically addressing RID within a competency framework. Much work is needed to describe the sequenced progression, tailored learning experiences, and competency-focused instruction. Finally, future research should focus on the outcomes of remediation in CBME programs.

Developing Academic Advisors and Competence Committees members: A community approach to developing CBME faculty leaders

INTRODUCTION

Implementing competency-based medical education (CBME) at the institutional level poses many challenges including having to rapidly enable faculty to be facilitators and champions of a new curriculum which utilizes feedback, coaching, and models of programmatic assessment. This study presents the necessary competencies required for Academic Advisors (AA) and Competence Committee (CC) members, as identified in the literature and as perceived by faculty members at Queen's University.

METHODS

This study integrated a review of available literature (n=26) yielding competencies that were reviewed by the authors followed by an external review consisting of CBME experts (n=5). These approved competencies were used in a cross-sectional community consultation survey distributed one year before (n=83) and one year after transitioning to CBME (n=144).

FINDINGS

Our newly identified competencies are a useful template for other institutions. Academic Advisor competencies focused on mentoring and coaching, whereas Competence Committee member's competencies focused on integrating assessments and institutional policies. Competency discrepancies between stakeholder groups existing before the transition had disappeared in the post-implementation sample.

CONCLUSIONS

We found value in taking an active community-based approach to developing and validating faculty leader competencies sooner rather than later when transitioning to CBME. The evolution of Competence Committees members and Academic Advisors requires the investment of specialized professional development and the sustained engagement of a collaborative community with shared concerns.

A Developmental Approach to Internal Medicine Residency Education: Lessons Learned from the Design and Implementation of a Novel Longitudinal Coaching Program

BACKGROUND

Resident physicians' achievement of professional competencies requires reflective practice skills and faculty coaching. Graduate medical education programs, however, struggle to operationalize these activities.

OBJECTIVE

To (1) describe the process and strategies for implementing an Internal Medicine (IM) resident coaching program that evolved in response to challenges, (2) characterize residents' professional learning plans (PLPs) and their alignment with EPAs, and, (3) examine key lessons learned.

DESIGN

The program began in 2013 and involved all postgraduate years (PGY) residents (n = 60, 100%), and 20 faculty coaches who were all IM trained and practicing in an IM-related specialty. One coach was linked with 3-4 residents for three years. Through 1:1 meetings, resident-coach pairs identified professional challenges ('disorienting dilemmas' or 'worst days'), reviewed successes ('best days'), and co-created professional learning plans. Typed summaries were requested following meetings. Coaches met monthly for professional development and to discuss program challenges/successes, which informed programmatic improvements; additionally, a survey was distributed after three program years. Data were analyzed using quantitative and qualitative methodologies.

RESULTS

Disorienting dilemmas and professional learning plans mapped to all 16 EPAs and four additional themes: work-life balance, career planning, teaching skills, and research/scholarship. The most-frequently mapped topics included: PGY1 - leading and working within interprofessional care teams (EPA 10), research and scholarship, and work-life balance; PGY2 - improving quality of care (EPA 13), demonstrating personal habits of lifelong learning (EPA15), and research and scholarship; PGY3 - lifelong learning (EPA15); career planning was common across all years.

CONCLUSIONS

Lessons learned included challenges in coordination of observations, identifying disorienting dilemmas, and creating a shared mental model between residents, faculty, and program leadership. The coaching program resulted in professional learning plans aligned with IM EPAs, in addition to other professional development topics. Operationalization of aspects of these results can inform the development of similar programs in residency education.

Group Assessment of Resident Performance: Valuable for Program Director Judgment?

BACKGROUND

Group discussion of resident performance is an emerging assessment approach in postgraduate medical education. However, groups do not necessarily make better decisions than individuals.

OBJECTIVE

This study examined how group meetings concerning the assessment of residents take place, what information is shared during the meetings, and how this influences program directors' judgment of resident performance.

METHODS

In 2017, the researchers observed 10 faculty group meetings where resident performance was discussed and interviewed the program directors within a month after the meetings. We used a thematic framework analysis to identify themes from the transcribed meetings and interviews.

RESULTS

The information shared by group members during the meetings had 2 aims: (1) forming a judgment about the residents, and (2) faculty development. Most group members shared information without written notes, most discussions were not structured by the program director, the major focus of discussions was on residents with performance concerns, and there was a lack of a shared mental model of resident performance. The program directors who benefited most from the meetings were those who thought group members were engaged and summarized the information after every discussion.

CONCLUSIONS

Unstructured discussions and a lack of a shared mental model among group members impede effective information sharing about resident performance with a developmental approach. Structured discussions with an equal amount of discussion time for every resident and creating a shared mental model about the purpose of the discussions and the assessment approach could enhance use of a developmental approach to assessing resident performance.

Key Factors in Clinical Competency Committee Members' Decisions Regarding Residents' Readiness to Serve as Supervisors: A National Study

PURPOSE

Entrustment has become a popular assessment framework in recent years. Most research in this area has focused on how frontline assessors determine when a learner can be entrusted. However, less work has focused on how these entrustment decisions are made. The authors sought to understand the key factors that pediatric residency program clinical competency committee (CCC) members consider when recommending residents to a supervisory role.

METHOD

CCC members at 14 pediatric residency programs recommended residents to one of five progressive supervisory roles (from not serving as a supervisory resident to serving as a supervisory resident in all settings). They then responded to a free-text prompt, describing the key factors that led them to that decision. The authors analyzed these responses, by role recommendation, using a thematic analysis.

RESULTS

Of the 155 CCC members at the participating programs, 84 completed 769 supervisory role recommendations during the 2015-2016 academic year. Four themes emerged from the thematic analysis: (1) Determining supervisory ability follows from demonstrated trustworthiness; (2) demonstrated performance matters, but so does experience; (3) ability to lead a team is considered; and (4) contextual considerations external to the resident are at play.

CONCLUSIONS

CCC members considered resident and environmental factors in their summative entrustment decision making. The interplay between these factors should be considered as CCC processes are optimized and studied further.

Design and evaluation of a clinical competency committee

INTRODUCTION

In postgraduate medical education, group decision-making has emerged as an essential tool to evaluate the clinical progress of residents. Clinical competency committees (CCCs) have been set up to ensure informed decision-making and provide feedback regarding performance of residents. Despite this important task, it remains unclear how CCCs actually function in practice and how their performance should be evaluated.

METHODS

In the prototyping phase of a design-based approach, a CCC meeting was developed, using three theoretical design principles: (1) data from multiple assessment tools and multiple perspectives, (2) a shared mental model and (3) structured discussions. The meetings were held in a university children's hospital and evaluated using observations, interviews with CCC members and an open-ended questionnaire among residents.

RESULTS

The structured discussions during the meetings provided a broad outline of resident performance, including identification of problematic and excellent residents. A shared mental model about the assessment criteria had developed over time. Residents were not always satisfied with the feedback they received after the meeting. Feedback that had been provided to a resident after the first CCC meeting was not addressed in the second meeting.

DISCUSSION

The principles that were used to design the CCC meeting were feasible in practice. Structured discussions, based on data from multiple assessment tools and multiple perspectives, provided a broad outline of resident performance. Residency programs that wish to implement CCCs can build on our design principles and adjust the prototype to their particular context. When running a CCC, it is important to consider feedback that has been provided to a resident after the previous meeting and to evaluate whether it has improved the resident's performance.

Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education

Medical education has shifted to a competency-based paradigm, leading to calls for improved learner assessment methods and validity evidence for how assessment data are interpreted. Clinical competency committees (CCCs) use the collective input of multiple people to improve the validity and reliability of decisions made and actions taken based on assessment data. Significant heterogeneity in CCC structure and function exists across postgraduate medical education programs and specialties, and while there is no "one-size-fits-all" approach, there are ways to maximize value for learners and programs. This paper collates available evidence and the authors' experiences to provide practical tips on CCC purpose, membership, processes, and outputs. These tips can benefit programs looking to start a CCC and those that are improving their current CCC processes.

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions

BACKGROUND

Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.

OBJECTIVE

We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.

METHODS

We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015-2016. Individual CCC members biannually reported characteristics of their review process and Accreditation Council for Graduate Medical Education milestone levels and recommended supervisory role categorizations assigned to residents. Relationships among characteristics of CCC member reviews, mean milestone levels, and supervisory role categorizations were analyzed using mixed-effects linear regression, reported as mean differences with 95% confidence intervals (CIs), and Bayesian mixed-effects ordinal regression, reported as odds ratios (ORs) and 95% credible intervals (CrIs).

RESULTS

A total of 155 CCC members participated. Members who provided milestones or other professional development feedback after CCC meetings assigned significantly lower mean milestone levels (mean 1.4 points; CI -2.2 to -0.6; P < .001) and were significantly less likely to recommend supervisory responsibility in any setting (OR = 0.23, CrI 0.05-0.83) compared with CCC members who did not. Members recommended less supervisory responsibility when they reviewed more residents (OR = 0.96, 95% CrI 0.94-0.99) and participated in more review cycles (OR = 0.22, 95% CrI 0.07-0.63).

CONCLUSIONS

This study explored the association between characteristics of individual CCC member reviews and their summative assessment decisions about residents. Further study is needed to gain deeper understanding of factors influencing CCC members' summative assessment decisions.

Progressive Surgical Autonomy Observed in a Hand Surgery Resident Clinic Model

OBJECTIVE

Resident clinics (RCs) are intended to catalyze the achievement of educational milestones through progressively autonomous patient care. However, few studies quantify their effect on competency-based surgical education, and no previous publications focus on hand surgery RCs (HRCs). We demonstrate the achievement of progressive surgical autonomy in an HRC model.

DESIGN

A retrospective review of all patients seen in a weekly half-day HRC from October 2010 to October 2015 was conducted. Investigators compiled data on patient demographics, provider encounters, operational statistics, operative details, and dictated surgical autonomy on an ascending 5 point scoring system.

SETTING

A tertiary hand surgery referral center.

RESULTS

A total of 2295 HRC patients were evaluated during the study period in 5173 clinic visits. There was an average of 22.6 patients per clinic, including 9.0 new patients with 6.5 emergency room referrals. Totally, 825 operations were performed by 39 residents. Trainee autonomy averaged 2.1/5 (standard deviation [SD] = 1.2), 3.4/5 (SD = 1.3), 2.1/5 (SD = 1.3), 3.4/5 (SD = 1.2), 3.2/5 (SD = 1.5), 3.5/5 (SD = 1.5), 4.0/5 (SD = 1.2), 4.1/5 (SD = 1.2), in postgraduate years 1 to 8, respectively. Linear mixed model analysis demonstrated training level significantly effected operative autonomy (p = 0.0001). Continuity of care was maintained in 79.3% of cases, and patients were followed an average of 3.9 clinic encounters over 12.4 weeks.

CONCLUSIONS

Our HRC appears to enable surgical trainees to practice supervised autonomous surgical care and provide a forum in which to observe progressive operative competency achievement during hand surgery training. Future studies comparing HRC models to non-RC models will be required to further define quality-of-care delivery within RCs.

Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns

BACKGROUND

Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention.

METHODS

Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis.

RESULTS

Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed.

CONCLUSIONS

Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.

How do clinical competency committees use different sources of data to assess residents' performance on the internal medicine milestones?A mixed methods pilot study

PURPOSE

This study examines how Clinical Competency Committees (CCCs) synthesize assessment data to make judgments about residents' clinical performances.

METHODS

Between 2014 and 2015, after four six-month reporting periods to the Accreditation Council for Graduate Medical Education (ACGME), 7 of 16 CCC faculty at Rush University Medical Center completed questionnaires focused on their perspectives about rating residents on their achievement of the milestones and participated in a focus group. Qualitative data were analyzed using grounded theory. Milestones ratings for two six-month ACGME reporting cycles (n = 100 categorical residents) were also analyzed.

RESULTS

CCC members weighted resident rotation ratings highest (weight = 37%), followed by faculty rotation comments (weight = 27%) and personal experience with residents (weight = 14%) for making judgments about learner's milestone levels. Three assessment issues were identified from qualitative analyses: (1) "design issues" (e.g. problems with available data or lack thereof); (2) "synthesis issues" (e.g. factors influencing ratings and decision-making processes) and (3) "impact issues" (e.g. how CCC generated milestones ratings are used).

CONCLUSIONS

Identifying factors that affect assessment at all stages of the CCC process can contribute to improving assessment systems, including support for faculty development for CCCs. Recognizing challenges in synthesizing first and second-hand assessment data is an important step in understanding the CCC decision-making process.

Initial Comparison of Resident and Attending Milestones Evaluations in Plastic Surgery

BACKGROUND

Graduate medical education has recently undergone a major archetypal shift toward competency-based evaluations of residents' performance. The implementation of the Milestones program by the Accreditation Council for Graduate Medical Education (ACGME) is a core component of the shift, designed to ensure uniformity in measuring residency knowledge using a series of specialty-specific achievements. This study evaluates the correlation between residents' self-evaluations and program directors' assessments of their performance.

METHODS

The study population comprised 12 plastic surgery residents, ranging from postgraduate year 1 to postgraduate year 6, enrolled in an integrated residency program at a single institution.

RESULTS

Overall, average attending scores were lower than average resident scores at all levels except postgraduate year 6. Correlation between resident and attending evaluations ranged from 0.417 to 0.957, with the correlation of average scores of Patient Care (0.854) and Medical Knowledge (0.816) Milestones significantly higher than those of professional skillsets (0.581). "Patient care, facial esthetics" was the Milestone with the lowest average scores from both groups. Residents scored themselves notably higher than their attendings' evaluations in Practice-based Learning and Improvement categories (+0.958) and notably lower in Medical Knowledge categories such as "Cosmetic Surgery, Trunk and Lower Extremities" (-0.375) and "Non-trauma hand" (-0.208). The total possible number of participants in this study was 12. The actual number of participants was 12 (100%).

CONCLUSIONS

The remarkable range of correlations suggests that expectations for performance standards may vary widely between residents and program directors. Understanding gaps between expectations and performance is vital to inform current and future residents as the restructuring of the accreditation process continues.

How do small groups make decisions? : A theoretical framework to inform the implementation and study of clinical competency committees

In the competency-based medical education (CBME) approach, clinical competency committees are responsible for making decisions about trainees' competence. However, we currently lack a theoretical model for group decision-making to inform this emerging assessment phenomenon. This paper proposes an organizing framework to study and guide the decision-making processes of clinical competency committees.This is an explanatory, non-exhaustive review, tailored to identify relevant theoretical and evidence-based papers related to small group decision-making. The search was conducted using Google Scholar, Web of Science, MEDLINE, ERIC, and PsycINFO for relevant literature. Using a thematic analysis, two researchers (SC & JP) met four times between April-June 2016 to consolidate the literature included in this review.Three theoretical orientations towards group decision-making emerged from the review: schema, constructivist, and social influence. Schema orientations focus on how groups use algorithms for decision-making. Constructivist orientations focus on how groups construct their shared understanding. Social influence orientations focus on how individual members influence the group's perspective on a decision. Moderators of decision-making relevant to all orientations include: guidelines, stressors, authority, and leadership.Clinical competency committees are the mechanisms by which groups of clinicians will be in charge of interpreting multiple assessment data points and coming to a shared decision about trainee competence. The way in which these committees make decisions can have huge implications for trainee progression and, ultimately, patient care. Therefore, there is a pressing need to build the science of how such group decision-making works in practice. This synthesis suggests a preliminary organizing framework that can be used in the implementation and study of clinical competency committees.

Using the ACGME Milestones for Resident Self-Evaluation and Faculty Engagement

BACKGROUND

Since July 2014 General Surgery residency programs have been required to use the Accreditation Council for Graduate Medical Education milestones twice annually to assess the progress of their trainees. We felt this change was a great opportunity to use this new evaluation tool for resident self-assessment and to furthermore engage the faculty in the educational efforts of the program.

METHODS

We piloted the milestones with postgraduate year (PGY) II and IV residents during the 2013/2014 academic year to get faculty and residents acquainted with the instrument. In July 2014, we implemented the same protocol for all residents. Residents meet with their advisers quarterly. Two of these meetings are used for milestones assessment. The residents perform an independent self-evaluation and the adviser grades them independently. They discuss the evaluations focusing mainly on areas of greatest disagreement. The faculty member then presents the resident to the clinical competency committee (CCC) and the committee decides on the final scores and submits them to the Accreditation Council for Graduate Medical Education website. We stored all records anonymously in a MySQL database. We used Anova with Tukey post hoc analysis to evaluate differences between groups. We used intraclass correlation coefficients and Krippendorff's α to assess interrater reliability.

RESULTS

We analyzed evaluations for 44 residents. We created scale scores across all Likert items for each evaluation. We compared score differences by PGY level and raters (self, adviser, and CCC). We found highly significant increases of scores between most PGY levels (p < 0.05). There were no significant score differences per PGY level between the raters. The interrater reliability for the total score and 6 competency domains was very high (ICC: 0.87-0.98 and α: 0.84-0.97). Even though this milestone evaluation process added additional work for residents and faculty we had very good participation (93.9% by residents and 92.9% by faculty) and feedback was generally positive.

CONCLUSION

Even though implementation of the milestones has added additional work for general surgery residency programs, it has also opened opportunities to furthermore engage the residents in reflection and self-evaluation and to create additional venues for faculty to get involved with the educational process within the residency program. Using the adviser as the initial rater seems to correlate closely with the final CCC assessment. Self-evaluation by the resident is a requirement by the RRC and the milestones seem to be a good instrument to use for this purpose. Our early assessment suggests the milestones provide a useful instrument to track trainee progression through their residency.

Essentials of Pediatric Emergency Medicine Fellowship: Part 6: Program Administration

This article is the sixth in a 7-part series that aims to comprehensively describe the current state and future directions of pediatric emergency medicine (PEM) fellowship training from the essential requirements to considerations for successfully administering and managing a program to the careers that may be anticipated upon program completion. This article provides a broad overview of administering and supervising a PEM fellowship program. It explores 3 topics: the principles of program administration, committee management, and recommendations for minimum time allocated for PEM fellowship program directors to administer their programs.

Effect of Process Changes in Surgical Training on Quantitative Outcomes From Surgery Residency Programs

OBJECTIVES

The purpose of this article is to review the literature on process changes in surgical training programs and to evaluate their effect on the Accreditation Council of Graduate Medical Education (ACGME) Core Competencies, American Board of Surgery In-Training Examination (ABSITE) scores, and American Board of Surgery (ABS) certification.

DESIGN

A literature search was obtained from MEDLINE via PubMed.gov, ScienceDirect.com, Google Scholar on all peer-reviewed studies published since 2003 using the following search queries: surgery residency training, surgical education, competency-based surgical education, ACGME core competencies, ABSITE scores, and ABS pass rate.

RESULTS

Our initial search list included 990 articles on surgery residency training models, 539 on competency-based surgical education, 78 on ABSITE scores, and 33 on ABS pass rate. Overall, 31 articles met inclusion criteria based on their effect on ACGME Core Competencies, ABSITE scores, and ABS certification. Systematic review showed that 5/31, 19/31, and 6/31 articles on process changes in surgical training programs had a positive effect on patient care, medical knowledge, and ABSITE scores, respectively. ABS certification was not analyzed. The other ACGME core competencies were addressed in only 6 studies.

CONCLUSIONS

Several publications on process changes in surgical training programs have shown a positive effect on patient care, medical knowledge, and ABSITE scores. However, the effect on ABS certification, and other quantitative outcomes from residency programs, have not been addressed. Studies on education strategies showing evidence that residency program objectives are being achieved are still needed. This article addresses the 6 ACGME Core Competencies.

How Do Emergency Medicine Residency Programs Structure Their Clinical Competency Committees? A Survey

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) recently has mandated the formation of a clinical competency committee (CCC) to evaluate residents across the newly defined milestone continuum. The ACGME has been nonproscriptive of how these CCCs are to be structured in order to provide flexibility to the programs.

OBJECTIVES

No best practices for the formation of CCCs currently exist. We seek to determine common structures of CCCs recently formed in the Council of Emergency Medicine Residency Directors (CORD) member programs and identify unique structures that have been developed.

METHODS

In this descriptive study, an 18-question survey was distributed via the CORD listserv in the late fall of 2013. Each member program was asked questions about the structure of its CCC. These responses were analyzed with simple descriptive statistics.

RESULTS

A total of 116 of the 160 programs responded, giving a 73% response rate. Of responders, most (71.6%) CCCs are chaired by the associate or assistant program director, while a small number (14.7%) are chaired by a core faculty member. Program directors (PDs) chair 12.1% of CCCs. Most CCCs are attended by the PD (85.3%) and selected core faculty members (78.5%), leaving the remaining committees attended by any core faculty. Voting members of the CCC consist of the residency leadership either with the PD (53.9%) or without the PD (36.5%) as a voting member. CCCs have an average attendance of 7.4 members with a range of three to 15 members. Of respondents, 53.1% of CCCs meet quarterly while 37% meet monthly. The majority of programs (76.4%) report a system to match residents with a faculty mentor or advisor. Of respondents, 36% include the resident's faculty mentor or advisor to discuss a particular resident. Milestone summaries (determination of level for each milestone) are the primary focus of discussion (93.8%), utilizing multiple sources of information.

CONCLUSIONS

The substantial variability and diversity found in our CORD survey of CCC structure and function suggest that there are myriad strategies that residency programs can use to match individual program needs and resources to requirements of the ACGME. Identifying a single protocol for CCC structure and development may prove challenging.

Assessment of resident operative performance using a real-time mobile Web system: preparing for the milestone age

OBJECTIVE

To satisfy trainees' operative competency requirements while improving feedback validity and timeliness using a mobile Web-based platform.

DESIGN

The Southern Illinois University Operative Performance Rating Scale (OPRS) was embedded into a website formatted for mobile devices. From March 2013 to February 2014, faculty members were instructed to complete the OPRS form while providing verbal feedback to the operating resident at the conclusion of each procedure. Submitted data were compiled automatically within a secure Web-based spreadsheet. Conventional end-of-rotation performance (CERP) evaluations filed 2006 to 2013 and OPRS performance scores were compared by year of training using serial and independent-samples t tests. The mean CERP scores and OPRS overall resident operative performance scores were directly compared using a linear regression model. OPRS mobile site analytics were reviewed using a Web-based reporting program.

SETTING

Large university-based general surgery residency program.

PARTICIPANTS

General Surgery faculty used the mobile Web OPRS system to rate resident performance. Residents and the program director reviewed evaluations semiannually.

RESULTS

Over the study period, 18 faculty members and 37 residents logged 176 operations using the mobile OPRS system. There were 334 total OPRS website visits. Median time to complete an evaluation was 45 minutes from the end of the operation, and faculty spent an average of 134 seconds on the site to enter 1 assessment. In the 38,506 CERP evaluations reviewed, mean performance scores showed a positive linear trend of 2% change per year of training (p = 0.001). OPRS overall resident operative performance scores showed a significant linear (p = 0.001), quadratic (p = 0.001), and cubic (p = 0.003) trend of change per year of clinical training, reflecting the resident operative experience in our training program. Differences between postgraduate year-1 and postgraduate year-5 overall performance scores were greater with the OPRS (mean = 0.96, CI: 0.55-1.38) than with CERP measures (mean = 0.37, CI: 0.34-0.41). Additionally, there were consistent increases in each of the OPRS subcategories.

CONCLUSIONS

In contrast to CERPs, the OPRS fully satisfies the Accreditation Council for Graduate Medical Education and American Board of Surgery operative assessment requirements. The mobile Web platform provides a convenient interface, broad accessibility, automatic data compilation, and compatibility with common database and statistical software. Our mobile OPRS system encourages candid feedback dialog and generates a comprehensive review of individual and group-wide operative proficiency in real time.