A multisite, multistakeholder validation of the Accreditation Council for Graduate Medical Education competencies

Multi-stakeholder validation of entrustable professional activities for a family medicine care of the elderly residency program: A focus group study

Entrustable Professional Activities (EPAs) have become widely used within Competency-Based Medical Education (CBME) for the training and evaluation of residents. Little is known about the effectiveness of incorporating multiple stakeholder groups in the validation of EPAs. Here, we seek to validate an EPA framework developed for the University of Manitoba Care of the Elderly Enhanced Skills program using online focus groups consisting of five stakeholder groups. Participants were recruited to take part in one of five online focus groups, one for each stakeholder group (physician faculty, residents, non-physician healthcare professionals, administrators/managers, and patients). Each group met one time for 90 minutes over ZOOM®. The themes arising from stakeholder feedback suggest that successful EPAs must neither be too specific nor too expansive in scope, clearly delineate appropriate means of evaluation, and indicate specific clinical settings in which each EPA should be evaluated. Cross-cutting themes included requiring trainees to collaborate with other professionals when it would optimize patient care, and preparing trainees to advocate for their patients' health (Advocacy). The present study demonstrates that multi-stakeholder analysis yields diverse feedback that can help make EPAs more clear, easier to use in evaluation, and more socially accountable.

Medical education in Brazil

This paper aims to describe and analyze medical education in Brazil, a history of over 200 years. As in most European countries and influenced by the Flexner Report, an undergraduate medical course in Brazil takes 6 years. Recently, medical education research has been advocating a shift from a teacher-centered and hospital-based approach to student-centered and community-based education. Nevertheless, a huge variation exists among Brazilian medical schools. The physicians' supply program known as "More Physicians" has strongly impacted the number of medical schools in Brazil, which is growing rapidly. Professors of medicine from several institutions and other stakeholders have alerted authorities to the risks of operating so many schools without adequate time to prepare teachers, clinician-educators, curricula, and sufficient pedagogical structure to ensure quality medical education. The possibility of an imminent catastrophe in medical education has united stakeholders in pursuit of a guarantee of quality maintenance. This effort has resulted in the creation of an independent accreditation system approved by the World Federation of Medical Education. The study of the unbalanced relationship between stakeholders in medical education in Brazil until now has provided valuable information concerning the importance of having their roles and limits clear. It is possible that these findings might be replicable around the world.

Embracing Multiple Stakeholder Perspectives in Defining Trainee Competence

PURPOSE

To explore how multiple stakeholder groups contribute to the understanding of trainee competence.

METHOD

The authors conducted a constructivist qualitative study in 2015 using focus group discussions to explore the perceptions of different stakeholder groups (patients, nurses/nurse practitioners, supervisors/senior physicians, leaders/administrators, trainees) regarding trainee competence in the emergency department. The authors used a conventional content analysis, a comparative analysis of supervisors'/senior physicians' versus other stakeholders' perspectives, and a directed analysis informed by stakeholder theory to analyze the focus group transcripts.

RESULTS

Forty-six individuals participated in nine focus groups. Four categories of competence were identified: Core Clinical Activities, Patient Centeredness, Aligning Resources, and Code of Conduct. Stakeholders generally agreed in their overall expectations regarding trainee competence. Within individual categories, each stakeholder group identified new considerations, details, and conflicts, which were a replication, elaboration, or complication of a previously identified theme. All stakeholders stressed those aspects of trainee competence that were relevant to their work or values. Trainees were less aware of the patient perspective than that of the other stakeholder groups.

CONCLUSIONS

Considering multiple stakeholder perspectives enriched the description and conceptualization of trainee competence. It also can inform the development of curricula and assessment tools and guide learning about inter- and intradisciplinary conflicts. Further research should explore how trainees' perceptions of value are influenced by their organizational context and, in particular, how trainees adapt their learning goals in response to the divergent demands of key stakeholders.

Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools

PURPOSE

To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs).

METHOD

Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores.

RESULTS

Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlations), and consequences (composite score). Student performance varied by case and task. In the PNs, justification of differential diagnosis was the most discriminating task. G-studies showed that schools accounted for less than 1% of total variance; however, for the PNs, there were differences in scores for varying cases and tasks across schools, indicating a school effect. Composite score reliability was maximized when the PN was weighted between 30% and 40%. Raters preferred using case-specific scoring guidelines with clear point-scoring systems.

CONCLUSIONS

This multisite study presents validity evidence for PN scores based on scoring rubric and case-specific scoring guidelines that offer rigor and feedback for learners. Variability in PN scores across participating sites may signal different approaches to teaching clinical reasoning among medical schools.

[Competency-based Neurosurgery Residency Programme]

A programme proposal for competency-based Neurosurgery training adapted to the specialization project is presented. This proposal has been developed by a group of neurosurgeons commissioned by the SENEC (Spanish Society of Neurosurgery) and could be modified to generate a final version that could come into force coinciding with the implementation of the specialization programme. This document aims to facilitate the test of the new programme included in the online version of our journal.

DURATION OF THE PROGRAMME

Total training period is 6 years; initial 2 years belong to the surgery specialization and remaining 4 years belong to core specialty period.

STRUCTURE OF THE PROGRAMME

It is a competency-based programmed based on the map used by the US Accreditation Council for Graduate Medical Education (ACGME) including the following domains of clinical competency: Medical knowledge, patient care, communication skills, professionalism, practice-based learning and improvement, health systems, interprofessional collaboration and professional and personal development. Subcompetencies map in the domains of Knowledge and Patient care (including surgical competencies) was adapted to the one proposed by AANS and CNS (annex 1 of the programme). A subcompetency map was also used for the specialization rotations.

INSTRUCTION METHODS

Resident's training is based on personal study (self-learning) supported by efficient use of information sources and supervised clinical practice, including bioethical instruction, clinical management, research and learning techniques.

EVALUATION METHODS

Resident evaluation proposal includes, among other instruments, theoretical knowledge tests, objective and structured evaluation of the level of clinical competency with real or standardised patients, global competency scales, 360-degree evaluation, clinical record audits, milestones for residents progress and self-assessment (annex 2). Besides, residents periodically assess the teaching commitment of the department's neurosurgeons and other professors participating in rotations, and annually assess the overall operation of the programme. Results of evaluations are registered, together with other relevant data, in the Resident's Book.

PROGRAMME'S NATIONAL COMMITTEE

The creation of a Programme Committee directly attached to the SENEC (National Commission) that, aside from generating a final version of the programme, monitors its implementation (level of adherence and operation in the different departments), assumes the creation of test banks and the centralized administration of knowledge tests (in the middle of the residency and/or at the end of it) and centralizes information collected by tutors that could be used for re-accreditation of the services, is proposed.

Perceptions of Leaders and Clinician Educators on the Impact of International Accreditation

BACKGROUND

Graduate medical education (GME) is responding to calls for reform by adopting competency-based frameworks and, in some countries, by rapidly implementing external accreditation systems. The Accreditation Council for Graduate Medical Education International (ACGME-I) began accrediting institutions in 2009. This study aimed to describe ACGME-I-accredited institutions and explore perceptions of their leaders and clinician educators (CEs) regarding preparedness, challenges, and initial impact of accreditation.

METHOD

Cross-sectional surveys of all ACGME-I-accredited institutions' leaders and CEs were conducted from June 2013 to June 2014. Eligible participants were identified through institution Web sites and GME offices. Combinations of Web- and paper-based surveys were employed.

RESULTS

Completed surveys were received from 24 (70.6%) of 34 institutional leaders and 274 (76.3%) of 359 CEs, representing 3 countries, 8 academic medical centers, 2 affiliated teaching hospitals, and 47 residency programs. Leaders and CEs felt prepared in the domains of knowledge and implementation of the competencies. Top challenges were excessive "demands on faculty time" and "bureaucratic procedures." The majority of both groups perceived a positive impact of accreditation on all learner, faculty, institution, and patient outcomes; most perceived no impact on patient satisfaction. Overall, 79.2% of leaders and 75.8% of CEs agreed or strongly agreed that seeking ACGME-I accreditation was worthwhile.

CONCLUSIONS

This study indicates that despite the challenges identified, initial perceptions of the impact of ACGME-I accreditation are positive. Findings from this study may be useful to institutions and countries considering similar GME reform, though long-term outcome data are needed.

Implementation of competency-based medical education: are we addressing the concerns and challenges?

CONTEXT

Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation.

OBJECTIVES

Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community.

METHODS

The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised.

CONCLUSIONS

The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.

Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment

PURPOSE

Clinical competency committees (CCCs) are now required in graduate medical education. This study examined how residency programs understand and operationalize this mandate for resident performance review.

METHOD

In 2013, the investigators conducted semistructured interviews with 34 residency program directors at five public institutions in California, asking about each institution's CCCs and resident performance review processes. They used conventional content analysis to identify major themes from the verbatim interview transcripts.

RESULTS

The purpose of resident performance review at all institutions was oriented toward one of two paradigms: a problem identification model, which predominated; or a developmental model. The problem identification model, which focused on identifying and addressing performance concerns, used performance data such as red-flag alerts and informal information shared with program directors to identify struggling residents.In the developmental model, the timely acquisition and synthesis of data to inform each resident's developmental trajectory was challenging. Participants highly valued CCC members' expertise as educators to corroborate the identification of struggling residents and to enhance credibility of the committee's outcomes. Training in applying the milestones to the CCC's work was minimal.Participants were highly committed to performance review and perceived the current process as adequate for struggling residents but potentially not for others.

CONCLUSIONS

Institutions orient resident performance review toward problem identification; a developmental approach is uncommon. Clarifying the purpose of resident performance review and employing efficient information systems that synthesize performance data and engage residents and faculty in purposeful feedback discussions could enable the meaningful implementation of milestones-based assessment.

Medical education practice-based research networks: Facilitating collaborative research

BACKGROUND

Research networks formalize and institutionalize multi-site collaborations by establishing an infrastructure that enables network members to participate in research, propose new studies, and exploit study data to move the field forward. Although practice-based clinical research networks are now widespread, medical education research networks are rapidly emerging.

AIMS

In this article, we offer a definition of the medical education practice-based research network, a brief description of networks in existence in July 2014 and their features, and a more detailed case study of the emergence and early growth of one such network, the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN).

METHODS

We searched for extant networks through peer-reviewed literature and the world-wide web.

RESULTS

We identified 15 research networks in medical education founded since 2002 with membership ranging from 8 to 120 programs. Most focus on graduate medical education in primary care or emergency medicine specialties.

CONCLUSIONS

We offer four recommendations for the further development and spread of medical education research networks: increasing faculty development, obtaining central resources, studying networks themselves, and developing networks of networks.