Competency milestones for medical students: Design, implementation, and analysis at one medical school

Current context and evaluation of medical residency programs: experience of six Brazilian Medical Societies

This article shows the most recent data on the evolution of the number of resident physicians and vacancies from R1 to R5, as well as the number of programs and institutions accredited by the National Commission for Medical Residency (CNRM). It also discusses the types and modalities of evaluation of medical residency, with a focus on the assessment of competencies throughout in-service training, which were incorporated into the most recent CNRM resolution, at the end of 2023, which amended and updated the 2006 directives. Finally, it shows the experience of six Medical Societies that conduct periodic evaluation of resident physicians, presented at a Brazilian Medical Association event.

Relevance of Competency Based Education for Architectural Education in India

Background

A holistic architectural education is the culmination of learning knowledge, skills, attitudes, and values, which eventually reflects in the quality of graduates. Even though different schools of thought have made various kinds of qualitative contributions towards the evolution of architectural education in India, it has largely been dominated by the quantitative and technical aspects of its regulating framework. Architects engage with the demanding contradictions between responsibilities of an ethical nature, the dynamic challenges of practice, and the intricacies of architectural imagination. The aesthetical and imaginative foundations of the field make it incumbent upon the architects to possess a balance of ecumenical proficiencies for accountability and personalization. The purpose of the study is to identify relevant attributes of Competency Based Education (CBE) that can be adopted for architectural education in India.

Methods

This research follows a narrative review approach and a descriptive-analytic method to broadly understand the attributes of CBE and its potential relevance to architectural education in India. 323 articles were searched on various search strings based on their relevance to the inquiry. 76 documents written in English language were included and appraised through the Scale of the Assessment for Narrative Review Articles (SANRA) tool to avoid any risk of bias. The PRISMA 2020 checklist and flow diagram has been used to report the findings of this narrative review.

Result

The study identifies eight critical parameters of CBE with respect to its definitions, origins, transitions, regulatory environment, characteristics, approaches and implications on teaching-learning, frameworks and models of assessment; and challenges, which makes a case for the relevance of CBE for architectural education in India, which hasn't been explored yet.

Conclusion

The broader expectations of 'being competent' can be addressed through a conscious adoption of strategies of relevant attributes of CBE which can encourage building attitudes and temperament for life-long learning.

Advancing anti-oppression and social justice in healthcare through competency-based medical education (CBME)

Competency-based medical education (CBME) focuses on preparing physicians to improve the health of patients and populations. In the context of ongoing health disparities worldwide, medical educators must implement CBME in ways that advance social justice and anti-oppression. In this article, authors describe how CBME can be implemented to promote equity pedagogy, an approach to education in which curricular design, teaching, assessment strategies, and learning environments support learners from diverse groups to be successful. The five core components of CBME programs - outcomes competency framework, progressive sequencing of competencies, learning experiences tailored to learners' needs, teaching focused on competencies, and programmatic assessment - enable individualization of learning experiences and teaching and encourage learners to partner with their teachers in driving their learning. These educational approaches appreciate each learner's background, experiences, and strengths. Using an exemplar case study, the authors illustrate how CBME can afford opportunities to enhance anti-oppression and social justice in medical education and promote each learner's success in meeting the expected outcomes of training. The authors provide recommendations for individuals and institutions implementing CBME to enact equity pedagogy.

Evaluating the Role of Competency-Based Behavioral Interviewing in Holistic Medical School Admissions

PROBLEM

Holistic review is a multifaceted concept that aims to increase diversity and applicant fit with program needs by complementing traditional academic requirements with appraisal of a wider range of personal characteristics and experiences. Behavioral interviewing has been practiced and studied in human resources, business, and organizational psychology for over 50 years. Its premise is that future performance can be anticipated from past actions. However, many of the interview approaches within the holistic framework are resource intensive and logistically challenging.

APPROACH

The Vanderbilt University School of Medicine instituted a competency-based behavioral interview (CBBI) to augment the selection process in 2012. Behavioral interviews are based on key competencies needed for entering students and require applicants to reflect on their actual experiences and what they learned from them. The authors reviewed 5 years of experience (2015-2019) to evaluate how CBBI scores contributed to the overall assessment of applicants for admission.

OUTCOMES

The final admission committee decision for each applicant was determined by reviewing multiple factors, with no single assessment determining the final score. The CBBI and summary interview scores showed a strong association ( P < .005), suggesting that the summary interviewer, who had access to the full applicant file, and the CBBI interviewer, who did not, assessed similar strengths despite the 2 different approaches, or that the strengths assessed tracked in the same direction. Students whose 2 interview scores were not aligned were less likely to be accepted to the school.

NEXT STEPS

The review raised awareness about the cultural aspects of interpreting the competencies and the need to expand our cultural framework throughout interviewer training. Findings indicate that CBBIs have the potential to reduce bias related to overreliance on standardized metrics; however, additional innovation and research are needed.

Using ACGME milestones as a formative assessment for the internal medicine clerkship: a consecutive two-year outcome and follow-up after graduation

BACKGROUND

This study evaluated the utility of using Accreditation Council for Graduate Medical Education (ACGME) Milestones as a formative assessment tool for the fifth- and sixth-grade medical students' performance in their internal medicine (IM) clerkship and the same students' performance in their post-graduate year (PGY) IM training.

METHODS

Retrospective data were collected from 65 medical students completing the two-year IM clerkship in the academic years 2019 and 2020 and 26 of the above students completing their PGY-1 training at the same university hospital in the academic year 2021. Data included the assessment results of 7 of the ACGME IM Milestones, information on admitted patients assigned to the students, and surveys of the students' satisfaction.

RESULTS

The analysis included 390 assessment results during the IM clerkship and 78 assessment results during the PGY-1 training. Clinical teachers commonly rated level 3 to medical students in the IM clerkship, with PC-2 subcompetency receiving the lowest rating among seven subcompetencies. The levels of most subcompetencies showed stationary in the two-year IM clerkship. Significant improvement was observed in all subcompetencies during the PGY-1 training. The medical students in the second-year IM clerkship expressed higher satisfaction with implementing Milestones than in their first-year IM clerkship and perceived Milestones assessments' usefulness as learning feedback.

CONCLUSIONS

Using ACGME Milestones as a formative assessment tool in the IM clerkship yielded promising outcomes. Longitudinal follow-up of subcompetencies facilitated tracking students' development and providing constructive feedback.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Competency-based pharmacy education definition: What components need to be defined to implement it?

OBJECTIVES

This study aimed to define the essential elements in the proposed competency-based pharmacy education (CBPE) definition, provide the key defining components of each essential element on the basis of educational theory and evidence, and define how the essential elements meet the identified needs for CBPE.

METHODS

best-practice integrative review was conducted as part of the work of the American Association of Colleges of Pharmacy CBPE Task Force to define the essential elements in the CBPE definition and how these elements fit with the need for CBPE. The definition was compared with other published competency-based education definitions across K-12, higher education, medical education, and veterinary education. Task Force members then met to develop a consensus on the core components of the 5 essential elements in the definition. Next, the Task Force evaluated the fit of CBPE by matching the identified needs, discussed in detail elsewhere, across each of the stakeholder perspectives with the core components of the 5 essential elements in the derived definition of CBPE.

FINDINGS

Upon review of the proposed CBPE definition, the Task Force identified 5 essential elements. These elements include the following: meeting health care and societal needs, outcomes-based curricular model, de-emphasized time, learner-centered culture, and authentic teaching and learning strategies aligned to assessments.

SUMMARY

This article helps to establish a common language for CBPE by defining the essential elements of the core components of the definition, and provides a starting point for further exploration of CBPE.

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

OBJECTIVES

Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations.

METHODS

Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment.

RESULTS

Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants.

CONCLUSION

We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

Appropriate antibiotic use and antimicrobial resistance: knowledge, attitudes and behaviour of medical students and their needs and preferences for learning

BACKGROUND

The impact of an appropriate use of antibiotics on the prevention of antimicrobial resistance (AMR) has been demonstrated. Surveys have shown, however, that medical students do not feel sufficiently trained to use antibiotics wisely. The aims of our study were (1) to describe what medical students currently know about appropriate antibiotic use, and (2) to identify students' learning preferences as a basis for developing student-centred teaching modules to convey the basics of AMR prevention.

METHODS

We performed an online survey at Charité Universitätsmedizin Berlin and the Julius-Maximilians-University Würzburg on the knowledge, attitudes, and behaviour (KAB) of medical students concerning AMR, antibiotic treatment options, and their perceptions of AMR topics addressed in the medical curriculum. Participants were able to fill out an online questionnaire between December 2019 and February 2020. In addition, we conducted focus group discussions with lecturers and medical students in winter 2019/2020 to identify AMR-related learning needs and preferences. Data were analysed descriptively.

RESULTS

Overall, 356 students (response rate 5.1%) participated in the KAB survey. Of these, 192 (54%) strongly agreed that the topic of AMR is relevant to students' clinical practice and 48% (171/355) stated that their future antibiotic prescription behaviour will have an influence on AMR development in their region. Participating students seemed to be interested in the topic of AMR and antibiotic therapy. But even of them, only 46% answered the question about the length of antibiotic use for community-acquired pneumonia correctly and 57% the question about the appropriate use of antibiotics in Staphylococcus aureus infections. Focus group discussions with students (n = 7) and lecturers (n = 9) identified a lack of competence in the responsible use of antibiotics and the prevention of AMR. Respondents stated that the teaching formats and AMR-related content should emphasize clinical applications, interaction with peers/clinicians, and repeated formative feedback from instructors.

CONCLUSIONS

Our results show that even medical students who were interested in the AMR problem were not able to use antibiotics appropriately due to gaps in knowledge and a lack of clinical skills. Based on the insights gained in the learning preferences of students and their content priorities, improved student-centred teaching materials should be developed.

A Competency-Guided Approach to Optimizing a Physician-Scientist Curriculum

Physician-scientists are uniquely positioned to achieve significant biomedical advances to improve the human condition. Their clinical and scientific training allows them to bridge fields and contribute to cutting-edge, clinically relevant research. The need for a highly skilled physician-scientist workforce has never been more acute. We propose a competency-guided program design (CGPD) framework that focuses on core skills to enhance the physician-scientist training curriculum. In partnership with clinical and graduate curricula, the CGPD framework can be employed as a tool to meaningfully integrate physician-scientist training, address barriers to attract and sustain the physician-scientist workforce, and avoid overprogramming that detracts from a solid foundation of clinical and graduate research training.

Development of an electronic learning progression dashboard to monitor student clinical experiences

INTRODUCTION

Clinical experience tracking mechanisms for students at dental schools provide patient assignment, student experience, and learning progression feedback. The purpose of this study was to evaluate dental students' clinical experiences following the implementation of a learning progression dashboard (LPD).

METHODS

After developing and deploying an electronic LPD using PHP, secondary data analysis on dental students' clinical experiences from 2017-2019 was conducted. Student experience differences were compared between the year before continuous use of the LPD and the first year using it. LPD data contained the required clinical procedures dentistry students must perform across all disciplines and the number of planned, in progress, and completed tasks each student has accomplished. Using two time points, the students' experiences were compared. Univariate statistics and independent t-tests were conducted in R for detecting the differences in the number and categories of codes.

RESULTS

The number and category of codes showed significant differences between the academic year 2017-2018 and 2018-2019 for both third- and fourth-year dental students after one and two terms. Overall, students recorded a 26% greater number of treatment codes and experienced a 26% greater number of code categories compared to the previous year.

CONCLUSION

Applying information management methods such as dashboards can better inform educators on student clinical experiences and improve clinical learning outcomes for students.

Competences in the training of nurses to assist the airway of adult patients in urgency and emergency situations

Objective:

construction and validation in appearance and content of the competence frameworks and of the Entrustable Professional Activities to develop skills in the training of nurses to assist the airway of adult patients in urgency and emergency situations.

Method:

a descriptive and methodological study developed in four phases: in the first, a workshop was held, composed of experts, for the construction of the competence frameworks; in the second, the material was validated using the Snowball Technique and the Delphi Technique, in the third, content analysis and calculation of the Content Validation Index were conducted; and in the fourth phase, the Entrustable Professional Activities were built, validated in simulated workshops.

Results:

the competence frameworks were built and validated, with a resulting CVI≥0.85 in all the items. The Entrustable Professional Activities were validated by experts regarding their applicability; of these, 44% stated they were applicable in simulated environments, 100% that they were useful content and with appropriate language, 22% suggested the insertion of new items to assess competence, 11% reported the difficulty of assessing competence individually in the clinical settings, and 11% of the experts referred to the need for prior training of the teacher/facilitator to use it.

Conclusion:

the study resulted in the construction of competence frameworks and six Entrustable Professional Activities relating them to the domains of essential competences in the training of nurses to assist the airway of adult patients in urgency and emergency situations. The participation of experts in the construction and validation of this material was essential to guarantee the theoretical and practical relevance of the result.

Realizing the vision of the Lancet Commission on Education of Health Professionals for the 21st Century: Transforming medical education through the Accelerating Change in Medical Education Consortium

In the last two decades, prompted by the anticipated arrival of the 21st Century and on the centenary of the publication of the Flexner Report, many in medical education called for change to address the expanding chasm between the requirements of the health care system and the educational systems producing the health care workforce. Calls were uniform. Curricular content was missing. There was a mismatch in where people trained and where they were needed to practice, legacy approaches to pedagogical methods that needed to be challenged, an imbalance in diversity of trainees, and a lack of research on educational outcomes, resulting in a workforce that was described as ill-equipped to provide health care in the current and future environment. The Lancet Commission on Education of Health Professionals for the 21st Century published a widely acclaimed report in 2010 that called for a complete and authoritative re-examination of health professional education. This paper describes the innovations of the American Medical Association Accelerating Change in Medical Education Consortium schools as they relate to the recommendations of the Lancet Commission. We outline the successes, challenges, and lessons learned in working to deeply reform medical education.

The Accelerating Change in Medical Education Consortium: Key Drivers of Transformative Change

The American Medical Association's (AMA's) Accelerating Change in Medical Education (ACE) initiative, launched in 2013 to foster advancements in undergraduate medical education, has led to the development and scaling of innovations influencing the full continuum of medical training. Initial grants of $1 million were awarded to 11 U.S. medical schools, with 21 schools joining the consortium in 2016 at a lower funding level. Almost one-fifth of all U.S. MD- and DO-granting medical schools are represented in the 32-member consortium. In the first 5 years, the consortium medical schools have delivered innovative educational experiences to approximately 19,000 medical students, who will provide a potential 33 million patient care visits annually. The core initiative objectives focus on competency-based approaches to medical education and individualized pathways for students, training in health systems science, and enhancing the learning environment. At the close of the initial 5-year grant period, AMA leadership sought to catalogue outputs and understand how the structure of the consortium may have influenced its outcomes. Themes from qualitative analysis of stakeholder interviews as well as other sources of evidence aligned with the 4 elements of the transformational leadership model (inspirational motivation, intellectual stimulation, individualized consideration, and idealized influence) and can be used to inform future innovation interventions. For example, the ACE initiative has been successful in stimulating change at the consortium schools and propagating those innovations broadly, with outputs involving medical students, faculty, medical schools, affiliated health systems, and the broader educational landscape. In summary, the ACE initiative has fostered a far-reaching community of innovation that will continue to drive change across the continuum of medical education.

Exploiting the power of information in medical education

The explosion of medical information demands a thorough reconsideration of medical education, including what we teach and assess, how we educate, and whom we educate. Physicians of the future will need to be self-aware, self-directed, resource-effective team players who can synthesize and apply summarized information and communicate clearly. Training in metacognition, data science, informatics, and artificial intelligence is needed. Education programs must shift focus from content delivery to providing students explicit scaffolding for future learning, such as the Master Adaptive Learner model. Additionally, educators should leverage informatics to improve the process of education and foster individualized, precision education. Finally, attributes of the successful physician of the future should inform adjustments in recruitment and admissions processes. This paper explores how member schools of the American Medical Association Accelerating Change in Medical Education Consortium adjusted all aspects of educational programming in acknowledgment of the rapid expansion of information.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

Psychometric validation of the Laval developmental benchmarks scale for family medicine

BACKGROUND

With the implementation of competency-based education in family medicine, there is a need for summative end-of-rotation assessments that are criterion-referenced rather than normative. Laval University's family residency program therefore developed the Laval Developmental Benchmarks Scale for Family Medicine (DBS-FM), based on competency milestones. This psychometric validation study investigates its internal structure and its relation to another variable, two sources of validity evidence.

METHODS

We used assessment data from a cohort of residents (n = 1432 assessments) and the Rasch Rating Scale Model to investigate its reliability, dimensionality, rating scale functioning, targeting of items to residents' competency levels, biases (differential item functioning), items hierarchy (adequacy of milestones ordering), and score responsiveness. Convergent validity was estimated by its correlation with the clinical rotation decision (pass, in difficulty/fail).

RESULTS

The DBS-FM can be considered as a unidimensional scale with good reliability for non-extreme scores (.83). The correlation between expected and empirical items hierarchies was of .78, p < .0001.Year 2 residents achieved higher scores than year 1 residents. It was associated with the clinical rotation decision.

CONCLUSION

Advancing its validation, this study found that the DBS-FM has a sound internal structure and demonstrates convergent validity.

Precision Cancer Medicine: Dynamic Learning of Cancer Biology in a Clinically Meaningful Context

PURPOSE

Precision medicine is revolutionizing healthcare practices, most notably in oncology. With cancer being the second leading cause of death in the USA, it is important to integrate precision oncology content in undergraduate medical education.

METHODS

In 2015, we launched a Clinical Cancer Medicine Integrated Science Course (ISC) for post-clerkship medical students at Vanderbilt University School of Medicine (VUSM). In this ISC, students learned cancer biology and clinical oncology concepts through a combination of classroom and patient care activities. Student feedback from mid- and end-of-course surveys and student match data were analyzed and used to develop ongoing course improvements.

RESULTS

To date, 72 medical students have taken the Clinical Cancer Medicine ISC. Over 90% of students who completed end-of-course surveys agreed or strongly agreed that this course advanced their foundational science knowledge in clinical cancer medicine, that clinical relevance was provided during non-clinical foundational science learning activities, and that foundational science learning was embedded in course clinical experiences. Students who took this course most commonly matched in Internal Medicine, Pathology, Pediatrics, and Radiation Oncology. VUSM students who matched into Pathology and Radiation Oncology were more likely to take this ISC than students who matched in other specialties.

CONCLUSION

The Clinical Cancer Medicine ISC serves as a model for incorporating precision oncology, cancer biology foundational science, and oncology patient care activities in undergraduate medical education. The course prepares students to care for oncology patients in their fields of interests during their future career in medicine.

Designing health professional education curricula using systems thinking perspectives

BACKGROUND

Medical students navigate complex personal learning pathways from entry into medical school, through an educational program, and into life-long practice. However, many stakeholders have called for substantive reforms in contemporary curricula, citing concerns about the lack of key abilities amongst newly graduated doctors to work in complex healthcare environments. Despite the need for educators to focus on curricula design, there is a paucity of overarching perspectives that allow synthesis of the various curricular elements in a way that lends meaningfulness and appreciation to the students in terms of navigating the immediate program requirements and beyond. Without such guidance, educators risk creating fragmented program designs that can lead to both unintended and unactionable outcomes for students as well as curriculum designers. Using systems thinking, we set out to address this gap by providing an overarching perspective for curriculum designers to appreciate the relationships and the interactions of the various curricular elements that inform and impact student's preparedness for practice.

METHODS

By framing a curriculum as a complex adaptive system, we used soft systems thinking to develop an initial prototype of a conceptual curricular toolkit, underpinned by an appraisal of relevant literature within health professional education and the broader educational context. The prototype was further refined iteratively after critical reflection by the authors with a diverse range of national and international colleagues via posters, short communications, and workshops at several conferences, and through social media.

RESULTS

We describe how the 3P-6Cs toolkit captures a learner's personal journey through an educational program into a field of practice by logically linking the three key elements: the personal, the program, and the practice. We demonstrate its application in three examples related to contemporary health profession education curricula. These are: creating integrated educational designs to capture students' developmental continua, conceptualising immersive clinical placements in non-traditional settings, and complexity-consistent evaluation of curricular interventions.

CONCLUSION

Applying the 3P-6Cs curricular toolkit to problems of curricula (re)design can provide overarching perspectives that enable educators to have a better understanding of how integration of elements within education programs can inform and impact student's preparation for lifelong practice.

Embedding Ethics Education in Clinical Clerkships by Identifying Clinical Ethics Competencies: The Vanderbilt Experience

The clinical clerkships in medical school are the first formal opportunity for trainees to apply bioethics concepts to clinical encounters. These clerkships are also typically trainees' first sustained exposure to the "reality" of working in clinical teams and the full force of the challenges and ethical tensions of clinical care. We have developed a specialized, embedded ethics curriculum for Vanderbilt University medical students during their second (clerkship) year to address the unique experience of trainees' first exposure to clinical care. Our embedded curriculum is centered around core "ethics competencies" specific to the clerkship: for Medicine, advanced planning and end-of-life discussions; for Surgery, informed consent; for Pediatrics, the patient-family-provider triad; for Obstetrics and Gynecology, women's autonomy, unborn child's interests, and partner's rights; and for Neurology/Psychiatry, decision-making capacity. In this paper, we present the rationale for these competencies, how we integrated them into the clerkships, and how we assessed these competencies. We also review the additional ethical issues that have been identified by rotating students in each clerkship and discuss our strategies for continued evolution of our ethics curriculum.

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.

Integrating Specialty-Specific Clinical Anatomy Education into the Post-Clerkship Curriculum

Anatomy is a foundational science mainstay of undergraduate medical school education, particularly in the pre-clerkship curriculum. During the post-clerkship curriculum, students closer to graduate medical education may benefit from a focused concentration on human anatomy related to their specific clinical interests. Here, we describe a course for post-clerkship students that uniquely incorporates a multimodal approach of dissection, didactics, and clinical correlation to radiologic imaging, with the opportunity to personalize student learning on a specialty-specific anatomic region. The course increased students' confidence of anatomical knowledge and its clinical relevance. Other institutions may benefit from establishing a similar multimodal integrated post-clerkship anatomy curriculum.

A Responsible Educational Handover: Improving Communication to Improve Learning

An important tenet of competency-based medical education is that the educational continuum should be seamless. The transition from undergraduate medical education (UME) to graduate medical education (GME) is far from seamless, however. Current practices around this transition drive students to focus on appearing to be competitively prepared for residency. A communication at the completion of UME-an educational handover-would encourage students to focus on actually preparing for the care of patients. In April 2018, the American Medical Association's Accelerating Change in Medical Education consortium meeting included a debate and discussion on providing learner performance measures as part of a responsible educational handover from UME to GME. In this Perspective, the authors describe the resulting 5 recommendations for developing such a handover: (1) The purpose of the educational handover should be to provide medical school performance data to guide continued improvement in learner ability and performance, (2) the process used to create an educational handover should be philosophically and practically aligned with the learner's continuous quality improvement, (3) the educational handover should be learner driven with a focus on individualized learning plans that are coproduced by the learner and a coach or advisor, (4) the transfer of information within an educational handover should be done in a standardized format, and (5) together, medical schools and residency programs must invest in adequate infrastructure to support learner improvement. These recommendations are shared to encourage implementation of the educational handover and to generate a potential research agenda that can inform policy and best practices.

Fundamental Curriculum Change with 1-Year Pre-Clerkship Phase and Effect on Stress Associated with Residency Specialty Selection

Vanderbilt University School of Medicine recently changed from 2 pre-clerkship years (Traditional curriculum) to a 1.5-year pre-clerkship phase for one class (Hybrid curriculum) to a 1-year pre-clerkship phase (Curriculum 2.0). This study investigated the relationship between shortened pre-clerkship training and stress associated with selecting a residency field. The surveyed graduating medical student population included one cohort from the Traditional and Hybrid curricula, and the first two cohorts from Curriculum 2.0. The authors modeled recollected stress levels using a Zero-Inflated Linear Mixed Model with additional covariate and random effects adjustments. Specialty decision-related stress levels increased in the Hybrid curriculum by 10.208 points [p = 0.0115, 95% CI 2.293, 18.122] on a 0-100 point scale. Curriculum 2.0 students had an insignificant increase in stress of 4.062 points [p = 0.304, 95% CI - 3.690, 11.814] relative to the Traditional curriculum. Time since starting medical school and time when a specialty was chosen were the largest factors associated with stress. While this study only evaluated a single facet of the potential downstream effects of curricular change, these data should inspire confidence for reform efforts as a significant increase in specialty decision-related stress present in Hybrid curriculum resolved in both cohorts of Curriculum 2.0.

Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase

In the move toward competency-based medical education, leaders have called for standardization of learning outcomes and individualization of the learning process. Significant progress has been made in establishing defined expectations for the knowledge, skills, attitudes, and behaviors required for successful transition to residency training, but individualization of educational processes to assist learners in reaching these competencies has been predominantly conceptual to date. The traditional time-based structure of medical education has posed a challenge to individualization within the curriculum and has led to more attention on innovations that facilitate transition from medical school to residency. However, a shift of focus to the clerkship-to-postclerkship transition point in the undergraduate curriculum provides an opportunity to determine how longitudinal competency-based assessments can be used to facilitate intentional and individualized structuring of the long-debated fourth year.This Perspective demonstrates how 2 institutions-the University of Virginia School of Medicine and the University of Michigan Medical School-are using competency assessments and applying standardized outcomes in decisions about individualization of the postclerkship learning process. One institution assesses Core Entrustable Professional Activities for Entering Residency, whereas the other has incorporated Accreditation Council for Graduate Medical Education core competencies and student career interests to determine degrees of flexibility in the postclerkship phase. Individualization in addition to continued assessment of performance presents an opportunity for intentional use of curriculum time to develop each student to be competently prepared for the transition to residency.

Fill the gap between traditional and new era: The medical educational reform in Taiwan

The 7-year medical education program in Taiwan has been established since 1949. More than 60 years later, many medical professionals have observed and voiced its deficiencies following the outbreak of severe acute respiratory syndrome. The deficiencies are three-fold: (1) specialties are excessively institutionalized, (2) students engage in passive learning and memorization, and (3) passing one written national examination serves as the means of granting permanent physician qualification. The situation has aroused concerns and discussions among medical professionals and educators for a new medical education program. Authorized by the Conference of Deans of Medical Schools in Taiwan, Prof. Chyi-Her Lin assembled a team for planning medical curricular reform. Subsequently, Prof. Shan-Chwen Chang organized a task force team which has been monitoring the new 6-year program since 2013. The aims of medical reform by Prof. Lin are (1) to eliminate the specialty training part, (2) to use innovative teaching methods to motivate students to learn proactively, and (3) to implement competency-based medical education. Now, the first class of physicians will enter the workplace in 2019, subject to various clinical challenges.

Perceptions of competency-based medical education from medical student discussion forums

BACKGROUND

Competency-based medical education (CBME) is becoming widely implemented in medical education. Trainees' perceptions of CBME are important factors in the implementation and acceptance of CBME. Online discussion groups allow unique insight into trainees' perceptions of CBME during residency training.

METHODS

We analysed 867 posts from 20 discussion threads in Premed 101 (Canadian) and 2756 posts from 50 threads in Student Doctor Network (SDN) (American) using NVivo 11. Inductive content analysis was used to develop a data-driven coding scheme that evolved throughout the analysis. Measures were taken to ensure the trustworthiness of findings, including co-coding of a subsample of 600 posts, peer debriefing, consensus-based analytical decision making and the maintenance of an audit trial.

RESULTS

Medical residents and students participating in the discussion forums emphasised select themes regarding the implementation of CBME in residency training. Concerns about CBME in Canada primarily involved its implications for the length of residency and post-residency opportunities. Posts on the American forum had a prominent focus on differing areas, such as the subjectivity in the assessment of core competencies and the role of CBME in termination of a resident's position.

CONCLUSIONS

Online discussion groups have the potential to provide unique insight into perceptions of CBME. The presented concerns may have implications for refining the model of CBME and illustrate the importance of providing clarification for trainees regarding length of training and evaluation structures from those involved in designing of CBME programmes.

Student perspective on outcomes and process - Recommendations for implementing competency-based medical education

PURPOSE

Competency-based medical education (CBME) seeks to prepare undergraduate and postgraduate trainees for clinical practice. Its major emphasis is on outcomes, but questions about how best to reach these remain. One key issue is the need to integrate what matters most to students when setting educational goals: this is crucial if we are to design curricula that trainees understand and engage with, and that promote successful achievement of competencies.

METHOD

We interviewed medical students in years 4 and 6 of a 6-year medical degree and used thematic analysis to understand their main educational priorities and how these fit with the aims of CBME.

RESULTS

Two major themes emerged: features of content and process. For content, students wanted clear guidance on what constitutes competence, finding broad outcome statements abstract and difficult to understand as novices. They also attach critical importance to features of process such as being welcomed, included in clinical teams and being known personally - these promote motivation, understanding, and professional development.

CONCLUSIONS

We present recommendations for those designing CBME curricula to emphasize the student perspective: what kind of guidance on outcomes is required, and features of process that must not be neglected if competence is to be achieved.

The challenges of detecting progress in generic competencies in the clinical setting

CONTEXT

Competency-based medical education has spurred the implementation of longitudinal workplace-based assessment (WBA) programmes to track learners' development of competencies. These hinge on the appropriate use of assessment instruments by assessors. This study aimed to validate our assessment programme and specifically to explore whether assessors' beliefs and behaviours rendered the detection of progress possible.

METHODS

We implemented a longitudinal WBA programme in the third year of a primarily rotation-based clerkship. The programme used the professionalism mini-evaluation exercise (P-MEX) to detect progress in generic competencies. We used mixed methods: a retrospective psychometric examination of student assessment data in one academic year, and a prospective focus group and interview study of assessors' beliefs and reported behaviours related to the assessment.

RESULTS

We analysed 1662 assessment forms for 186 students. We conducted interviews and focus groups with 21 assessors from different professions and disciplines. Scores were excellent from the outset (3.5-3.7/4), with no meaningful increase across blocks (average overall scores: 3.6 in block 1 versus 3.7 in blocks 2 and 3; F = 8.310, d.f. 2, p < 0.001). The main source of variance was the forms (47%) and only 1% of variance was attributable to students, which led to low generalisability across forms (Eρ²  = 0.18). Assessors reported using multiple observations to produce their assessments and were reluctant to harm students by consigning anything negative to writing. They justified the use of a consistent benchmark across time by citing the basic nature of the form or a belief that the 'competencies' assessed were in fact fixed attributes that were unlikely to change.

CONCLUSIONS

Assessors may purposefully deviate from instructions in order to meet their ethical standards of good assessment. Furthermore, generic competencies may be viewed as intrinsic and fixed rather than as learnable. Implementing a longitudinal WBA programme is complex and requires careful consideration of assessors' beliefs and values.

Construction and validation of competency frameworks for the training of nurses in emergencies

Objective:

to build and validate competency frameworks to be developed in the training of nurses for the care of adult patients in situations of emergency with a focus on airway, breathing and circulation approach.

Method:

this is a descriptive and methodological study that took place in three phases: the first phase consisted in a literature review and a workshop involving seven experts for the creation of the competency frameworks; in the second phase, 15 experts selected through the Snowball Technique and Delphi Technique participated in the face and content validation, with analysis of the content of the suggestions and calculation of the Content Validation Index to assess the agreement on the representativeness of each item; in the third phase, 13 experts participated in the final agreement of the presented material.

Results:

the majority of the experts were nurses, with graduation and professional experience in the theme of the study. Competency frameworks were developed and validated for the training of nurses in the airway, breathing and circulation approach.

Conclusion:

the study made it possible to build and validate competency frameworks. We highlight its originality and potentialities to guide teachers and researchers in an efficient and objective way in the practical development of skills involved in the subject approached.

Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter

PURPOSE

Many factors influence the reliable assessment of medical students' competencies in the clerkships. The purpose of this study was to determine how many clerkship competency assessment scores were necessary to achieve an acceptable threshold of reliability.

METHOD

Clerkship student assessment data were collected during the 2015-2016 academic year as part of the medical school assessment program at the University of Michigan Medical School. Faculty and residents assigned competency assessment scores for third-year core clerkship students. Generalizability (G) and decision (D) studies were conducted using balanced, stratified, and random samples to examine the extent to which overall assessment scores could reliably differentiate between students' competency levels both within and across clerkships.

RESULTS

In the across-clerkship model, the residual error accounted for the largest proportion of variance (75%), whereas the variance attributed to the student and student-clerkship effects was much smaller (7% and 10.1%, respectively). D studies indicated that generalizability estimates for eight assessors within a clerkship varied across clerkships (G coefficients range = 0.000-0.795). Within clerkships, the number of assessors needed for optimal reliability varied from 4 to 17.

CONCLUSIONS

Minimal reliability was found in competency assessment scores for half of clerkships. The variability in reliability estimates across clerkships may be attributable to differences in scoring processes and assessor training. Other medical schools face similar variation in assessments of clerkship students; therefore, the authors hope this study will serve as a model for other institutions that wish to examine the reliability of their clerkship assessment scores.

Integrating Foundational Sciences in a Clinical Context in the Post-Clerkship Curriculum

PURPOSE

To design, implement, and launch courses that integrate foundational science learning and clinical application in a post-clerkship undergraduate medical school curriculum.

METHOD

In Academic Year (AY) 15-16, as part of a comprehensive curricular revision, Vanderbilt University School of Medicine (VUSM) formally implemented "Integrated Science Courses" (ISCs) that combined rigorous training in the foundational sciences with meaningful clinical experiences. These courses integrated foundational sciences that could be leveraged in the clinical environment, utilized a variety of instructional modalities, and included quantitative and qualitative (competency-based milestones) student assessments. Each ISC underwent a rigorous quality improvement process that required input on foundational science content, student experience, and student performance assessment.

RESULTS

Eleven ISCs were delivered to 173 students in AY15-16, with some students taking more than one ISC. Immediately after completing each course, 93% (n=222) of ISC enrollees completed a course evaluation. Students (91%; n=201) 'agreed' or 'strongly agreed' that foundational science learning informed and enriched the clinical experiences. Furthermore, 94% (n=209) of students thought that the clinical experiences informed and enriched the foundational science learning. Ninety-four percent of the students anticipated using the foundational science knowledge acquired in future clinical training and practice.

CONCLUSION

The teaching of foundational sciences in the clinical workplace in the post-clerkship medical curriculum is challenging and resource-intensive, yet feasible. Additional experience with the model will inform the mix of courses as well as the breadth and depth of foundational science instruction that is necessary to foster scientifically-based clinical reasoning skills in each student.

Competency-Based, Time-Variable Education in the Health Professions: Crossroads

Health care systems around the world are transforming to align with the needs of 21st-century patients and populations. Transformation must also occur in the educational systems that prepare the health professionals who deliver care, advance discovery, and educate the next generation of physicians in these evolving systems. Competency-based, time-variable education, a comprehensive educational strategy guided by the roles and responsibilities that health professionals must assume to meet the needs of contemporary patients and communities, has the potential to catalyze optimization of educational and health care delivery systems. By designing educational and assessment programs that require learners to meet specific competencies before transitioning between the stages of formal education and into practice, this framework assures the public that every physician is capable of providing high-quality care. By engaging learners as partners in assessment, competency-based, time-variable education prepares graduates for careers as lifelong learners. While the medical education community has embraced the notion of competencies as a guiding framework for educational institutions, the structure and conduct of formal educational programs remain more aligned with a time-based, competency-variable paradigm.The authors outline the rationale behind this recommended shift to a competency-based, time-variable education system. They then introduce the other articles included in this supplement to Academic Medicine, which summarize the history of, theories behind, examples demonstrating, and challenges associated with competency-based, time-variable education in the health professions.

Chiropractic student choices in relation to indications, non-indications and contra-indications of continued care

Background

The quality of health care provider clinical decisions has long been recognized as variable. Research has focused on clinical decision making with the aim of improving patient outcomes. No studies have looked at chiropractic students´ abilities in this regard.

Method

In 2016, advanced students from two Australian chiropractic programs (N = 444) answered a questionnaire on patient case scenarios for neck and low back pain (LBP). We selected 7 scenarios representing the three categories; continuing care, non-indicated care, and contraindicated care. This represented a total of 21 tested scores. Comparisons of correct answers were made a) for program years 3, 4 and 5, and b) between the three categories of care.

Results

In almost 1/3 of scenarios, correct scores were 70% or greater. Best results were for two neck pain cases (simple and with spinal cord involvement). Continued care showed most improvements with study year. However, the scenarios that reflected non-indication for continued care had much worse results and did not improve in higher years. For the obvious contraindicated neck scenario, the results were good from the beginning and progressively improved and for a contraindicated LBP scenario the results started poorly in year 3 but improved over the program years.

Conclusions

Although student responses were generally good, there is still room for improvement, especially for non-indicated care. The quality of students’ clinical decisions can be measured and thus has the potential to be used by chiropractic educators and regulatory bodies to identify student’s in need of assistance as well as to monitor chiropractic programs in relation to student competence.

Trial registration

Not applicable.

Electronic supplementary material

The online version of this article (10.1186/s12998-017-0170-y) contains supplementary material, which is available to authorized users.