Competency-Based, Time-Variable Education in the Health Professions: Crossroads

Competency-Based Education: Developing an Advanced Competency Framework for Indonesian Pharmacists

Introduction: Pharmacists need to be adaptable, flexible, and capable of advancing their practice to adapt to rapidly changing population health needs. We describe an educational approach to pharmacy workforce transformation in Indonesia through an advanced practice competency framework development using an "adopt and adapt" methodology. Methods: The competency framework development process comprised a translation phase, an adopt and adapt phase, validation through a nationwide mapping survey, and a completion phase through leadership consensus panels. We conducted a forward-backwards translation of a previously validated Advanced to Consultancy Level Framework (ACLF) to yield the Indonesian Advanced Development Framework (IADF) draft. The subsequent adoption and adaptation process was conducted through a series of consensus panels. We validated the IADF through a nationwide workforce survey. The final phase included leadership consensus panels with the professional leadership body in Indonesia. We analyzed the qualitative data thematically and the quantitative data using a Multiple Correspondence Analysis (MCA) technique. Results: We identified conceptual challenges in adopting and adapting the existing ACLF, which were addressed by providing a national glossary and concrete examples. A total of 6,212 pharmacists participated in the national workforce survey, of which 43% had <2 years of post-license (post-registration) experience. The MCA results showed that practitioner self-assessment to the IADF could discriminate their career development stages. The results also indicated a four-stage career model (including early years career training). Embedding this model in a structured national training program will enhance the professional workforce development through a more structured career journey. Conclusions: We describe the first validation of an advanced competency development framework for the pharmacy workforce in a non-Anglophone country, showing the possibility of transnational applicability of this framework. We argue that this methodology can be used in Low and Middle-income countries (LMICs) for the more rapid advancement of pharmaceutical care practice.

Content and Face Validation of a Novel, Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) to Evaluate Registered Dietitians' Competence: A Delphi Consensus from the United States

The nutrition-focused physical examination (NFPE) is an integral component of nutrition assessment performed by registered dietitian nutritionists (RDNs) to determine signs of malnutrition and other nutrition-related complications. Increased use of this essential skill among RDNs and the transformation of dietetics education to a competency-based model in the near future calls for appropriately validated tools to measure RDNs' NFPE competence. To fill the need for a validated competency tool, this study developed an Interactive Nutrition-Specific Physical Exam Competency Tool (INSPECT) utilizing the initial 70 items identified in the first phase of the study. The second phase of this study aimed to test the preliminary version of the INSPECT for content and face validity. An expert panel of 17 members provided consensus recommendations through the Delphi process. Internal consistency of the consensus was measured with Cronbach's alpha (α) and α of ≥0.70 was defined as acceptable a priori. Inter-rater agreement among the expert panel was determined using the intraclass correlation coefficient (ICC) and an a priori ICC of 0.75 to 0.9 was established as good and >0.9 as excellent agreement. The results showed acceptable face validity (α = 0.71) and excellent content validity for the INSPECT, with an internal consistency of α = 0.97 in the first round and α = 0.96 in the second round. The inter-rater agreement was also excellent with ICC = 0.95 for each of the Delphi rounds. A total of 52 items were retained from the preliminary version of the INSPECT. Open feedback from the experts allowed for the consolidation of 11 similar items for better scoring and evaluation and thus, a total of 41 items were included in the final version of the INSPECT. The final version of the INSPECT is currently being studied in real-life, multi-site clinical settings among practicing RDNs to examine construct validity, reliability, and item-level psychometric properties. Ultimately, the validated INSPECT will be available for the competency evaluation of RDNs practicing in clinical settings.

A Dynamic Risk Management Approach for Reducing Harm From Invasive Bedside Procedures Performed During Residency

Internal medicine (IM) residents frequently perform invasive bedside procedures during residency training. Bedside procedure training in IM programs may compromise patient safety. Current evidence suggests that IM training programs rely heavily on the number of procedures completed during training as a proxy for resident competence instead of using objective postprocedure patient outcomes. The authors posit that the results of procedural training effectiveness should be reframed with outcome metrics rather than process measures alone. This article introduces the as low as reasonably achievable (ALARA) approach, which originated in the nuclear industry to increase safety margins, to help assess and reduce bedside procedural risks. Training program directors are encouraged to use ALARA calculations to define the risk trade-offs inherent in current procedural training and assess how best to reliably improve patient outcomes. The authors describe 5 options to consider: training all residents in bedside procedures, training only select residents in bedside procedures, training no residents in bedside procedures, deploying 24-hour procedure teams supervised by IM faculty, and deploying 24-hour procedure teams supervised by non-IM faculty. The authors explore how quality improvement approaches using process maps, fishbone diagrams, failure mode effects and analyses, and risk matrices can be effectively implemented to assess training resources, choices, and aims. Future research should address the drivers behind developing optimal training programs that support independent practice, correlations with patient outcomes, and methods that enable faculty to justify their supervisory decisions while adhering to ALARA risk management standards.

OpTrust: An Effective Educational Bundle for Enhancing Faculty-resident Intraoperative Entrustment

OBJECTIVE

The purpose of this study was to measure the efficacy of a novel faculty and resident educational bundle focused on development of faculty-resident behaviors and entrustment in the operating room.

SUMMARY BACKGROUND DATA

As surgical training environments are orienting to entrustable professional activities (EPAs), successful transitions to this model will require significant faculty and resident development. Identifying an effective educational initiative which prepares faculty and residents for optimizing assessment, teaching, learning, and interacting in this model is critical.

METHODS

From September 2015 to June 2017, an experimental study was conducted in the Department of Surgery at the University of Michigan Health System (UMHS). Case observations took place across general, plastic, thoracic, and vascular surgical specialties. A total of 117 operating room observations were conducted during Phase I of the study and 108 operating room observations were conducted during Phase II following the educational intervention. Entrustment behaviors were rated for 56 faculty and 73 resident participants using OpTrust, a validated intraoperative entrustment instrument.

RESULTS

Multiple regression analysis showed a significant increase in faculty entrustment (Phase I = 2.32 vs Phase II = 2.56, P < 0.027) and resident entrustability (Phase I = 2.16 vs Phase II = 2.40, P < 0.029) scores following exposure to the educational intervention.

CONCLUSIONS

Our study shows improved intraoperative entrustment following implementation of faculty and resident development, indicating the efficacy of this innovative educational bundle. This represents a crucial component in the implementation of a competency-based assessment framework like EPAs.

Usability and reproducibility of three tools to assess medical students and residents in emergency medicine

OBJECTIVES

It is critical to assess competency of medical students and residents in emergency medicine (EM) during undergraduate and graduate medical education. However, very few valid tools exist to assess both technical and nontechnical skills in the specific context of EM. Three Acute Care Assessment Tools (ACAT 1, 2, and 3) have been previously developed for three acute care conditions: cardiac arrest (1), coma (2), and acute respiratory failure (3). This study aimed to evaluate the reproducibility of the tools.

METHODS

The tool was tested using recorded videos of simulation sessions of fourth year medical students and first year residents in EM. Raters independently reviewed the videos two times in a 3-month interval, and interrater and intrarater reliability using intraclass correlation (ICC) was calculated. Secondary endpoints included the completeness rate and relevance of the ACAT.

RESULTS

Sixty-two sessions were recorded and 48 videos analyzed (18 for CA and 15 for both respiratory failure and coma. The learners were residents in 32 (66%) of videos. Interrater reliability was excellent (ICC >0.9 for all three contexts) and so was the intrarater reliability (>0.88), both upon first review (month 0, M0) and at 3 months (M3). The usability of the ACAT was good, with a completeness of the items that ranged from 96% to 100%. Only one item of the ACAT 1 had a relevance of 27%, as it could not be completed in 13 scenarios out of 18.

CONCLUSIONS

The results demonstrate educators can evaluate students similarly utilizing video recordings of simulated medical scenario. The excellent completeness of the rated items advocated for good usability. The three ACATs can be utilized to assess for completeness of predefined tasks in three acute care broad scenario in a competency-based medical education framework.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Validity Evidence for Assessing Entrustable Professional Activities During Undergraduate Medical Education

PURPOSE

To explore validity evidence for the use of entrustable professional activities (EPAs) as an assessment framework in medical education.

METHOD

Formative assessments on the 13 Core EPAs for entering residency were collected for 4 cohorts of students over a 9- to 12-month longitudinal integrated clerkship as part of the Education in Pediatrics Across the Continuum pilot at the University of Minnesota Medical School. The students requested assessments from clinical supervisors based on direct observation while engaging in patient care together. Based on each observation, the faculty member rated the student on a 9-point scale corresponding to levels of supervision required. Six EPAs were included in the present analyses. Student ratings were depicted as curves describing their performance over time; regression models were employed to fit the curves. The unit of analyses for the learning curves was observations rather than individual students.

RESULTS

(1) Frequent assessments on EPAs provided a developmental picture of competence consistent with the negative exponential learning curve theory; (2) This finding was true across a variety of EPAs and across students; and (3) The time to attain the threshold level of performance on the EPA for entrustment varied by student and EPA.

CONCLUSIONS

The results provide validity evidence for an EPA-based program of assessment. Students assessed using multiple observations performing the Core EPAs for entering residency demonstrate classic developmental progression toward the desired level of competence resulting in entrustment decisions. Future work with larger data samples will allow further psychometric analyses of assessment of EPAs.

Outcomes of competency-based medical education: A taxonomy for shared language

As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.

Growth mindset in competency-based medical education

The ongoing adoption of competency-based medical education (CBME) across health professions training draws focus to learner-centred educational design and the importance of fostering a growth mindset in learners, teachers, and educational programs. An emerging body of literature addresses the instructional practices and features of learning environments that foster the skills and strategies necessary for trainees to be partners in their own learning and progression to competence and to develop skills for lifelong learning. Aligned with this emerging area is an interest in Dweck's self theory and the concept of the growth mindset. The growth mindset is an implicit belief held by an individual that intelligence and abilities are changeable, rather than fixed and immutable. In this paper, we present an overview of the growth mindset and how it aligns with the goals of CBME. We describe the challenges associated with shifting away from the fixed mindset of most traditional medical education assumptions and practices and discuss potential solutions and strategies at the individual, relational, and systems levels. Finally, we present future directions for research to better understand the growth mindset in the context of CBME.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

Essential competencies for physical therapist managing individuals with spinal muscular atrophy: A delphi study

BACKGROUND AND PURPOSE

With the availability and development of disease-modifying therapies for individuals with spinal muscular atrophy (SMA), new emerging phenotypes must be characterized, and potential new treatment paradigms tested. There is an urgent demand to develop an educational program that provides physical therapists (PTs) worldwide the necessary knowledge and training to contribute to best-practice care and clinical research. A competency based education framework is one that would focus on outcomes not process and where progression of learners would occur only after competencies are demonstrated. The first step toward such a framework is defining outcomes. The purpose of this Delphi study was to develop consensus on those competencies deemed essential within the SMA PT community.

METHODS

Purposive selection and snowball sampling techniques were used to recruit expert SMA PTs. Three web-based survey rounds were used to achieve consensus, defined as agreement among >80% of respondents. The first round gathered demographic information on participants as well as information on clarity and redundancy on a list of competencies; the second round, collected the same information on the revised list and whether or not participants agreed if the identified domains captured the essence of a SMA PT as well as the definitions for each; and the third asked participants to rank their agreement with each competency.

RESULTS

Consensus revealed 35 competencies, organized under 6 domains, which were deemed essential for a PT working with persons with SMA.

DISCUSSION

In order to develop a curriculum to meet the physical therapy needs of persons with SMA, it is imperative to establish defined outcomes and to achieve consensus on those outcomes within the SMA community.

CONCLUSIONS

This study identified essential competencies that will help to provide guidance in development of a formal education program to meet these defined outcomes. This can foster best-practice care and clinical decision-making for all PTs involved in the care of persons with SMA in a clinical and research setting.

Clinical Competence Assessed Using Simulation: Development of a Standardized Tool to Assess Nutrition-Focused Physical Exam Skill Competence for Registered Dietitian Nutritionists

OBJECTIVE

This study details development and testing of a tool designed to evaluate clinical competence by registered dietitian nutritionists (RDNs) on the nutrition-focused physical exam (NFPE). The process demonstrated in this study can be adapted for assessment of other clinical skills in competency-based education programs, such as Accreditation Council on Education in Nutrition and Dietetics' Future Model of Dietetics Education.

DESIGN AND SETTING

The tool was tested for content validity, then used to assess the videotaped performance of the NFPE with standardized patients in a simulated clinical environment.

PARTICIPANTS

Seven expert raters assessed the relevance of the tool's items for content validity. For other tests, the tool was used by 3 dietetics faculty to score a group of 18 RDNs.

VARIABLES AND ANALYSIS

Content validity was tested using the item-level Content Validity Index. Interrater reliability and internal consistency of the tool were calculated and assessed.

RESULTS

The interrater reliability results indicated excellent agreement between raters, both overall (kappa statistic = 0.86; 95% confidence interval, 0.86-0.87; P < 0.001) and on individual items. Cronbach α statistic was calculated at α = .91, indicating excellent internal consistency.

CONCLUSIONS

The tool represents an instrument that can assess competence in the NFPE among RDNs and may be used as a template for development of other competency-based education assessment tools.

Medical Education's Wicked Problem: Achieving Equity in Assessment for Medical Learners

Despite a lack of intent to discriminate, physicians educated in U.S. medical schools and residency programs often take actions that systematically disadvantage minority patients. The approach to assessment of learner performance in medical education can similarly disadvantage minority learners. The adoption of holistic admissions strategies to increase the diversity of medical training programs has not been accompanied by increases in diversity in honor societies, selective residency programs, medical specialties, and medical school faculty. These observations prompt justified concerns about structural and interpersonal bias in assessment. This manuscript characterizes equity in assessment as a "wicked problem" with inherent conflicts, uncertainty, dynamic tensions, and susceptibility to contextual influences. The authors review the underlying individual and structural causes of inequity in assessment. Using an organizational model, they propose strategies to achieve equity in assessment and drive institutional and systemic improvement based on clearly articulated principles. This model addresses the culture, systems, and assessment tools necessary to achieve equitable results that reflect stated principles. Three components of equity in assessment that can be measured and evaluated to confirm success include intrinsic equity (selection and design of assessment tools), contextual equity (the learning environment in which assessment occurs), and instrumental equity (uses of assessment data for learner advancement and selection and program evaluation). A research agenda to address these challenges and controversies and demonstrate reduction in bias and discrimination in medical education is presented.

Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based Advancement to Residency

PURPOSE

To evaluate response process validity evidence for clinical competency committee (CCC) assessments of first-year residents on a subset of General Pediatrics Entrustable Professional Activities (EPAs) and milestones in the context of a national pilot of competency-based, time-variable (CBTV) advancement from undergraduate to graduate medical education.

METHOD

Assessments of 2 EPAs and 8 milestones made by the trainees' actual CCCs and 2 different blinded "virtual" CCCs for 48 first-year pediatrics residents at 4 residency programs between 2016 and 2018 were compared. Residents had 3 different training paths from medical school to residency: time-variable graduation at the same institution as their residency, time-fixed graduation at the same institution, or time-fixed graduation from a different institution. Assessments were compared using ordinal mixed-effects models.

RESULTS

Actual CCCs assigned residents higher scores than virtual CCCs on milestones and one EPA's supervision levels. Residents who graduated from a different institution than their residency received lower milestone ratings than either group from the same institution; CBTV residents received higher ratings on one milestone (ICS4) and similar ratings on all others compared with non-CBTV residents who completed medical school at the same institution.

CONCLUSIONS

First-year residents who graduated from CBTV medical school programs were assessed as having the same level of competence as residents who graduated from traditional medical school programs, but response process evidence suggests that members of CCCs may also draw on undocumented personal knowledge of the learner to draw conclusions about resident competence.

Experiential Learning in Project-Based Quality Improvement Education: Questioning Assumptions and Identifying Future Directions

PURPOSE

Project-based experiential learning is a defining element of quality improvement (QI) education despite ongoing challenges and uncertainties. The authors examined stakeholders' perceptions and experiences of QI project-based learning to increase understanding of factors that influence learning and project experiences.

METHOD

The authors used a case study approach to examine QI project-based learning in 3 advanced longitudinal QI programs, 2 at the University of Toronto and 1 at an academic tertiary-care hospital. From March 2016 to June 2017, they undertook 135 hours of education program observation and 58 interviews with learners, program directors, project coaches, and institutional leaders and reviewed relevant documents. They analyzed data using a conventional and directed data analysis approach.

RESULTS

The findings provide insight into 5 key factors that influenced participants' project-based learning experiences and outcomes: (1) variable emphasis on learning versus project objectives and resulting benefits, tensions, and consequences; (2) challenges integrating the QI project into the curriculum timeline; (3) project coaching factors (e.g., ability, capacity, role clarity); (4) participants' differing access to resources and ability to direct a QI project given their professional roles; and (5) workplace environment influence on project success.

CONCLUSIONS

The findings contribute to an empirical basis toward more effective experiential learning in QI by identifying factors to target and optimize. Expanding conceptualizations of project-based learning for QI education beyond learner-initiated, time-bound projects, which are at the core of many QI educational initiatives, may be necessary to improve learning and project outcomes.

Effective accreditation in postgraduate medical education: from process to outcomes and back

BACKGROUND

The accreditation of medical educational programs is thought to be important in supporting program improvement, ensuring the quality of the education, and promoting diversity, equity, and population health. It has long been recognized that accreditation systems will need to shift their focus from processes to outcomes, particularly those related to the end goals of medical education: the creation of broadly competent, confident professionals and the improvement of health for individuals and populations. An international group of experts in accreditation convened in 2013 to discuss this shift.

MAIN TEXT

Participants unequivocally supported the inclusion of more outcomes-based criteria in medical education accreditation, specifically those related to the societal accountability of the institutions in which the education occurs. Meaningful and feasible outcome metrics, however, are hard to identify. They are regionally variable, often temporally remote from the educational program, difficult to measure, and susceptible to confounding factors. The group identified the importance of health outcomes of the clinical milieu in which education takes place in influencing outcomes of its graduates. The ability to link clinical data with individual practice over time is becoming feasible with large repositories of assessment data linked to patient outcomes. This was seen as a key opportunity to provide more continuous oversight and monitoring of program impact. The discussants identified several risks that might arise should outcomes measures completely replace process issues. Some outcomes can be measured only by proxy process elements, and some learner experience issues may best be measured by such process elements: in brief, the "how" still matters.

CONCLUSIONS

Accrediting bodies are beginning to view the use of practice outcome measures as an important step toward better continuous educational quality improvement. The use of outcomes will present challenges in data collection, aggregation, and interpretation. Large datasets that capture clinical outcomes, experience of care, and health system performance may enable the assessment of multiple dimensions of program quality, assure the public that the social contract is being upheld, and allow identification of exemplary programs such that all may improve. There remains a need to retain some focus on process, particularly those related to the learner experience.

Academic coaching for learners in medical education: Twelve tips for the learning specialist

As medical education moves toward a competency-based model of teaching and learning, the need for individual academic coaching has increased. However, coaching for improving academic performance is underexplored relative to coaching for the improvement of technical skills. This manuscript introduces 12 tips for developing academic coaching skills (in the context of one-on-one coaching encounters as well as that of the broader learning and teaching environment) that promote lifelong learning habits. These tips were developed by drawing upon relevant literature from coaching (across disciplines), educational linguistics, the learning sciences, and my experiences as a medical education learning specialist. Key considerations for impactful academic coaching include critical self-reflection (for both the learner and learning specialist), dialogic interpersonal communication, collaborating with student leaders and faculty, and facilitating the development of self-regulated, lifelong learning skills.

Reconceptualizing Ethics Through Morbidity and Mortality Rounds

BACKGROUND

Surgeons face ethical tensions daily, yet ethics education continues to prove challenging. Two possible reasons for these challenges may be the different conceptions of knowledge between technical training vs those that underpin ethical practice, and the potential devaluing of ethics as a focus for education given false assumptions about its inherent nature. This study implemented and evaluated an innovation meant to prioritize and contextualize ethics in surgical learning and practice.

STUDY DESIGN

After implementation of Ethics Morbidity and Mortality (M&M) rounds as an educational intervention, a qualitative evaluation consisted of interviews with 12 residents and 9 faculty. Analysis was informed by principles of constructivist grounded theory and the theoretical framework of Habermas' 3 types of knowledge: technical, practical, and emancipatory. For comparative purposes, analysis was conducted of how participants described ethics and ethics education and learning in relation to the traditional ethics teaching model vs the M&Ms.

RESULTS

In the traditional model, ethics teaching was seen as disconnected from real life, and not valuable. Within M&Ms, ethics was viewed as integral to practice, engaging, valuable, and relevant. In the traditional model, ethics principles were seen as acquired through role modeling and as a fixed part of character. Within M&Ms, ethics principles were seen as learnable and transformable parts of identity.

CONCLUSIONS

Traditional teaching of surgical ethics may result in physicians armed with knowledge, but unable to apply it. Our findings suggest that incorporating ethics into M&Ms allows not only learning the tools of ethics, but the knowledge that ethical principles were becoming integrated into professional identity.

A New Paradigm for Testing Clinical Competence in Chaplaincy Certification

This article describes the rationale, evolution, implementation, and evaluation of a process for testing clinical competence in health care chaplaincy certification. The process developed by the Spiritual Care Association uses Zoom technology, simulated patient actors, and evidence-based behaviors. Evaluation of the process by users has been very positive. The process seems to have acceptable validity, reliability, and usability and should be considered as an alternative to self-report clinical encounters as a test of clinical competence.

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities

IMPORTANCE

Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence.

OBJECTIVE

To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data.

DESIGN, SETTING, AND PARTICIPANTS

Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018.

INTERVENTIONS

Longitudinal, prospective assessment using EPAs.

MAIN OUTCOMES AND MEASURES

Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA.

RESULTS

Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician).

CONCLUSIONS AND RELEVANCE

This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Guidelines: The dos, don'ts and don't knows of remediation in medical education

INTRODUCTION

Two developing forces have achieved prominence in medical education: the advent of competency-based assessments and a growing commitment to expand access to medicine for a broader range of learners with a wider array of preparation. Remediation is intended to support all learners to achieve sufficient competence. Therefore, it is timely to provide practical guidelines for remediation in medical education that clarify best practices, practices to avoid, and areas requiring further research, in order to guide work with both individual struggling learners and development of training program policies.

METHODS

Collectively, we generated an initial list of Do's, Don'ts, and Don't Knows for remediation in medical education, which was then iteratively refined through discussions and additional evidence-gathering. The final guidelines were then graded for the strength of the evidence by consensus.

RESULTS

We present 26 guidelines: two groupings of Do's (systems-level interventions and recommendations for individual learners), along with short lists of Don'ts and Don't Knows, and our interpretation of the strength of current evidence for each guideline.

CONCLUSIONS

Remediation is a high-stakes, highly complex process involving learners, faculty, systems, and societal factors. Our synthesis resulted in a list of guidelines that summarize the current state of educational theory and empirical evidence that can improve remediation processes at individual and institutional levels. Important unanswered questions remain; ongoing research can further improve remediation practices to ensure the appropriate support for learners, institutions, and society.

Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase

In the move toward competency-based medical education, leaders have called for standardization of learning outcomes and individualization of the learning process. Significant progress has been made in establishing defined expectations for the knowledge, skills, attitudes, and behaviors required for successful transition to residency training, but individualization of educational processes to assist learners in reaching these competencies has been predominantly conceptual to date. The traditional time-based structure of medical education has posed a challenge to individualization within the curriculum and has led to more attention on innovations that facilitate transition from medical school to residency. However, a shift of focus to the clerkship-to-postclerkship transition point in the undergraduate curriculum provides an opportunity to determine how longitudinal competency-based assessments can be used to facilitate intentional and individualized structuring of the long-debated fourth year.This Perspective demonstrates how 2 institutions-the University of Virginia School of Medicine and the University of Michigan Medical School-are using competency assessments and applying standardized outcomes in decisions about individualization of the postclerkship learning process. One institution assesses Core Entrustable Professional Activities for Entering Residency, whereas the other has incorporated Accreditation Council for Graduate Medical Education core competencies and student career interests to determine degrees of flexibility in the postclerkship phase. Individualization in addition to continued assessment of performance presents an opportunity for intentional use of curriculum time to develop each student to be competently prepared for the transition to residency.

What Can the Giant Do? Defining the Path to Unsupervised Primary Care Practice by Competence, Not Time

In this issue of Academic Medicine, Dewan and Norcini examine the significant variability of time-in-training between patient care "giants"-the physicians, nurse practitioners, and physician assistants who practice primary care-and they call for further studies to determine optimal training duration and eventual scope of practice. They ask, what is the minimum education and training required to practice primary care, or "how tall is the shortest giant?" In this Invited Commentary, the authors reframe the question from identifying the minimum length of training required, to identifying desired patient care outcomes. Primary care is not a uniform entity. It ranges from complex elderly chronically ill patients, to twentysomething millennials with acute problems, to pregnant women, to families, and everything in between. The authors argue that training should be fit for purpose and produce high-quality outcomes for patients. Competence should be defined by these outcomes. Drawing parallels with Major League Baseball, the authors note that time to competence development will be variable for different training programs depending on purpose, and also variable for people within those programs, even with shared purpose. While time is a tool for competence attainment, it should not be the metric by which readiness for practice is measured.

Redefining the Acting Internship in the Era of Entrustment: One Institution's Approach to Reforming the Acting Internship

Creating a Core Entrustable Professional Activities (Core EPA) curriculum requires a longitudinal approach. Current curricular efforts have focused primarily on the pre-clerkship and clerkship phases of training; however, the role of the Acting Internship (AI) has not been explored. The AI experience offers opportunities for students to have enhanced clinical responsibility, demonstrate proficiency, and allows for assessment of Core EPAs that are beyond the focus of clerkships. We share our experience developing an interdepartmental AI experience designed to assess designated Core EPAs and highlight tensions that should be considered when incorporating an AI experience into a longitudinal Core EPA-oriented curriculum.

Competency and Its Many Meanings

Competency and competency assessment are central to much of professional education, regulation, and practice. In the name of safe and effective professional practice, elaborate competency education and competency assessment systems have evolved, and consume significant time, energy, and financial resources. This paper will review the evolution of competing competency discourses in pharmacy and discuss implications of these approaches on professional practice, with particular emphasis on understanding the consequences of outsized focus on competency at the expense of other potential lenses for understanding professional practice.

Medical education today: all that glitters is not gold

BACKGROUND

The medical education system based on principles advocated by Flexner and Osler has produced generations of scientifically grounded and clinically skilled physicians whose collective experiences and contributions have served medicine and patients well. Yet sweeping changes launched around the turn of the millennium have constituted a revolution in medical education. In this article, a critique is presented of the new undergraduate medical education (UME) curricula in relationship to graduate medical education (GME) and clinical practice.

DISCUSSION

Medical education has changed and will continue to change in response to scientific advances and societal needs. However, enthusiasm for reform needs to be tempered by a more measured approach to avoid unintended consequences. Movement from novice to master in medicine cannot be rushed. An argument is made for a shoring up of biomedical science in revised curricula with the beneficiaries being nascent practitioners, developing physician-scientists --and the public.

CONCLUSION

Unless there is further modification, the new integrated curricula are at risk of produce graduates deficient in the characteristics that have set physicians apart from other healthcare professionals, namely high-level clinical expertise based on a deep grounding in biomedical science and understanding of the pathologic basis of disease. The challenges for education of the best possible physicians are great but the benefits to medicine and society are enormous.

A Mastery Learning Capstone Course to Teach and Assess Components of Three Entrustable Professional Activities to Graduating Medical Students

PROBLEM

Thirteen measurable Entrustable Professional Activities (EPAs) have been proposed by the Association of American Medical Colleges as a means to operationalize medical school graduates' patient care qualifications. Mastery learning is an effective method for boosting clinical skills, but its applicability to the EPAs remains to be studied. The authors designed this study to evaluate a mastery learning intervention to teach and assess components of 3 of the 13 EPAs in a 4th-year capstone course.

INTERVENTION

The course featured mastery learning principles and addressed three EPA-based skills: (a) obtain informed consent, (b) develop a differential diagnosis and write admission orders, and (c) write discharge prescriptions. All students underwent a baseline skills assessment, received feedback, engaged in deliberate practice with actionable feedback, and completed a similar skills-based posttest assessment. Students continued with practice and testing until the minimum passing standards (MPSs) were reached for each posttest.

CONTEXT

All medical students at a single medical school (Northwestern University, Feinberg School of Medicine) who matriculated in 2012 and graduated with the class of 2016 participated in a required transition to residency course immediately prior to graduation.

OUTCOME

There were 134 students eligible to participate, and 130 (97.0%) completed all curricular requirements and assessments. All 130 medical students who completed the course met or exceeded the MPS for each of the three EPA-based clinical skills. Reliability coefficients for outcome data were uniformly high. Measures for each of the three clinical skills showed statistically significant improvement.

LESSONS LEARNED

The capstone course was an effective approach to teach and assess components of three EPA-based clinical skills to mastery learning standards in a 4th-year capstone course. We learned that this approach for implementation is feasible and results in significant improvement in components of EPA skill performance. Next steps will include developing assessments incorporating the mastery model into components of additional EPAs, identifying the best location within the curriculum to insert this content, and expanding the number of assessments as part of a larger assessment system.

Cognitive load theory for training health professionals in the workplace: A BEME review of studies among diverse professions: BEME Guide No. 53

AIM

Cognitive load theory (CLT) is of increasing interest to health professions education researchers. CLT has intuitive applicability to workplace settings, yet how CLT should inform teaching, learning, and research in health professions workplaces is unclear.

METHOD

To map the existing literature, we performed a scoping review of studies involving cognitive load, mental effort and/or mental workload in professional workplace settings within and outside of the health professions. We included actual and simulated workplaces and workplace tasks.

RESULT

Searching eight databases, we identified 4571 citations, of which 116 met inclusion criteria. Studies were most often quantitative. Methods to measure cognitive load included psychometric, physiologic, and secondary task approaches. Few covariates of cognitive load or performance were studied. Overall cognitive load and intrinsic load were consistently negatively associated with the level of experience and performance. Studies consistently found distractions and other aspects of workplace environments as contributing to extraneous load. Studies outside the health professions documented similar findings to those within the health professions, supporting relevance of CLT to workplace learning.

CONCLUSION

The authors discuss implications for workplace teaching, curricular design, learning environment, and metacognition. To advance workplace learning, the authors suggest future CLT research should address higher-level questions and integrate other learning frameworks.

Integrated Pediatric Health Care in Pennsylvania: A Survey of Primary Care and Behavioral Health Providers

This study reports on a statewide survey of medical and behavioral health professionals to advance the knowledge base on the benefits and obstacles to delivering integrated pediatric health care. Surveys distributed in 3 statewide provider networks were completed by 110 behavioral health specialists (BHSs) and 111 primary care physicians (PCPs). Survey content documented their perceptions about key services, benefits, barriers, and needed opportunities related to integrated care. Factor analyses identified 8 factors, and other items were examined individually. We compared responses by specialty group (BHS vs PCP) and integrated care experience (no vs yes). The findings revealed differences across domains by specialty subgroup. In several cases, BHS (vs PCP) respondents, especially those with integrated care experience, reported lower benefits, higher barriers, and fewer resource requests. The implications of these results for enhancing care integration development, delivery, training, and research are discussed along with the study's limitations and empirical literature.

Monitoring communication skills progress of medical students: Establishing a baseline has value, predicting the future is difficult

OBJECTIVE

To provide evidence for the validity of an Introductory Clinical Experience (ICE) that was implemented as a baseline assessment of medical students' clinical communication skills to support progression of skills over time.

METHODS

In this longitudinal study of communication skills, medical students completed the ICE, then a Practice of Medicine (POM) Objective Structured Clinical Exam 8 months later, and the Comprehensive Clinical Skills Exam (CCSE) 25 months later. At each experience, trained Standardized Patients assessed students, using the same behaviorally anchored checklist in 3 domains: Information Gathering, Relationship Development, and Patient Education and Counseling (PEC) with good internal reliability (.70-.87). Skills development patterns were described. ICE as a predictor of later performance was explored. Students' perspectives were elicited.

RESULTS

140 (80%) medical students consented to include their data in this study. Overall communication scores increased over time (eta² = .17, medium effect) mostly attributable to increase in PEC skills (eta² = .48, large effect), in 4 patterns. ICE and POM scores predicted future communication skills. Most students recognized the educational value of ICE.

CONCLUSION

Entering medical students' clinical communication skills increase over time on average and may predict future performance.

PRACTICE IMPLICATIONS

Implementing an ICE is likely a valid strategy for monitoring progress and facilitating communication skills development.

Bridging the gap: The intersection of entrustability and perceived autonomy for surgical residents in the OR

BACKGROUND

Faculty entrustment decisions affect resident entrustability behaviors and surgical autonomy. The relationship between entrustability and autonomy is not well understood. This pilot study explores that relationship.

METHODS

108 case observations were completed. Entrustment behaviors were rated using OpTrust. Residents completed a Zwisch self-assessment to measure surgical autonomy. Resident perceived autonomy was collected for 67 cases used for this pilot study.

RESULTS

Full entrustability was observed in 5 of the 108 observed cases. Residents in our study did not report full autonomy. Spearman's rank correlation coefficient identified that resident entrustability was positively correlated with perceived resident autonomy (ρ = 0.66, p < 0.05). Ordinal logistic regression assessed the relationship between resident entrustability and autonomy. The relationship persisted while controlling for PGY level, gender, and case complexity (OR = 8.42, SEM = 4.54, p < 0.000).

CONCLUSIONS

Resident entrustability is positively associated with perceived autonomy, yet full entrustability is not translating to the perception of full autonomy for residents.

Preparing educators for adaptive education (AE) programs

CONTEXT

There has long been awareness that educational experiences should be individualized. In the health professions, this tenet remains inconsistently implemented. Adapting to the widely diverse characteristics of different learners requires educators who are prepared for planning and implementing adaptive education (AE).

RATIONALE

Learning experiences, for both educators and health professions students, can be substantially enhanced by mirroring clinical care, where we respect the uniqueness of each person and increasingly approach care as a collaboration. We are continuously "diagnostic," striving to understand our patients' clinical and life circumstances, adjusting to new findings. Learners are also unique in multiple, relevant ways. They deserve educators who work with them collaboratively and "diagnostically," adapting to changing information.

IMPLEMENTATION

Until recently, having educational programs that adapt to learner uniqueness was logistically and economically unrealistic. Now, thanks to deeper understandings of the learning process and new technologies, individualization is feasible. Here, we focus on the foundation step of preparing educators. Educator development: Suitably prepared educators are indispensable to success in becoming appropriately adaptive to learners' needs. For some educators, becoming more adaptive can be contrary to long-held assumptions and habits. We offer recommendations for effective educator development, without which authentic AE is unlikely.

The EPA-based Utrecht undergraduate clinical curriculum: Development and implementation

AIM

As reports of the application of entrustable professional activities (EPAs) increase, not only for postgraduate but also for undergraduate medical education, there is a need for descriptions of what a UME curriculum with EPAs could look like. We provide such a description based on the experiences at University Medical Center Utrecht, the Netherlands, which can be used as an example by other curriculum developers.

METHODS

In a three-year process, the UMC Utrecht Curriculum Committee developed a clinical workplace curriculum with an EPA structure, taking into account examples, such as the US Core EPAs for Entering Residency, and recommendations to integrate and increase the length of clerkships.

RESULTS

In the resulting curriculum, operational from 2016, students train to be trusted with indirect supervision before graduation in five broad EPAs: the clinical consultation; general medical procedures; informing, advising and guiding patients and families; communicating and collaborating with colleagues; and extraordinary patient care. Each of these integrates smaller (nested) EPAs that receive focused training attention in integrated clerkships at various moments and must be signed off for entrustment with indirect supervision to complete the clerkship.

DISCUSSION

The framework of EPAs went through many iterations before it was consolidated. Among the issues that required special attention was the application of a supervision levels scale for sign-off, the necessity to cover all relevant clinical content while not labeling too many small tasks each as a separate EPA, methods of EPA-focused assessment in the workplace and the creation of an e-portfolio model to serve assessment and entrustment.