A systems approach for institutional CBME adoption at Queen's University

Exploring stakeholder perspectives regarding the implementation of competency-based medical education: a qualitative descriptive study

Introduction

Competency-based medical education (CBME) offers perceived advantages and benefits for postgraduate medical education (PGME) and the training of competent physicians. The purpose of our study was to gain insights from those involved in implementing CBME in two residency programs to inform ongoing implementation practices.

Methods

We conducted a qualitative descriptive study to explore the perspectives of multiple stakeholders involved in the implementation of CBME in two residency programs (the first cohort) to launch the Royal College's Competence by Design model at one Canadian university. Semi-structured interviews were conducted with 17 participants across six stakeholder groups including residents, department chairs, program directors, faculty, medical educators, and program administrators. Data collection and analysis were iterative and reflexive to enhance the authenticity of the results.

Results

The participants' perspectives organized around three key themes including: a) contextualizing curriculum and assessment practices with educational goals of CBME, b) coordinating new administrative requirements to support implementation, and c) adaptability toward a competency-based program structure, each with sub-themes.

Conclusion

By eliciting the perspectives of different stakeholder groups who experienced the implementation processes, we developed a common understanding regarding facilitators and challenges for program directors, program administrators and educational leaders across PGME. Results from our study contribute to the scholarly conversation regarding the key aspects related to CBME implementation and serve to inform its ongoing development and application in various educational contexts.

The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting

Residents and faculty have described a burden of assessment related to the implementation of competency-based medical education (CBME), which may undermine its benefits. Although this concerning signal has been identified, little has been done to identify adaptations to address this problem. Grounded in an analysis of an early Canadian pan-institutional CBME adopter's experience, this article describes postgraduate programs' adaptations related to the challenges of assessment in CBME. From June 2019-September 2022, 8 residency programs underwent a standardized Rapid Evaluation guided by the Core Components Framework (CCF). Sixty interviews and 18 focus groups were held with invested partners. Transcripts were analyzed abductively using CCF, and ideal implementation was compared with enacted implementation. These findings were then shared back with program leaders, adaptations were subsequently developed, and technical reports were generated for each program. Researchers reviewed the technical reports to identify themes related to the burden of assessment with a subsequent focus on identifying adaptations across programs. Three themes were identified: (1) disparate mental models of assessment processes in CBME, (2) challenges in workplace-based assessment processes, and (3) challenges in performance review and decision making. Theme 1 included entrustment interpretation and lack of shared mindset for performance standards. Adaptations included revising entrustment scales, faculty development, and formalizing resident membership. Theme 2 involved direct observation, timeliness of assessment completion, and feedback quality. Adaptations included alternative assessment strategies beyond entrustable professional activity forms and proactive assessment planning. Theme 3 related to resident data monitoring and competence committee decision making. Adaptations included adding resident representatives to the competence committee and assessment platform enhancements. These adaptations represent responses to the concerning signal of significant burden of assessment within CBME being experienced broadly. The authors hope other programs may learn from their institution's experience and navigate the CBME-related assessment burden their invested partners may be facing.

Knowledge and Perceptions of Competency-Based Medical Education in Diagnostic Radiology Post-Graduate Medical Education: Identifying Priorities and Developing a Framework for Professional Development Activities

Introduction: We evaluated knowledge and perceptions of an established Competency-Based Medical Education (CBME) model developed by the Royal College of Physicians and Surgeons of Canada, Competence by Design (CBD), and identified evidence-informed priorities for professional development activities (PDAs). Materials and Methods: Teaching faculty and residents at a single, large diagnostic radiology post-graduate medical education (PGME) program were eligible to participate in this cross-sectional, survey-based study. Knowledge of CBD was evaluated through multiple choice questions (MCQs), which assessed participants' understanding of major principles and terms associated with CBD. Participants' perceptions of the anticipated impact of CBD on resident education and patient care were evaluated and priorities for PDAs were identified, which informed a framework for CBD PDAs. Results: Fifty faculty and residents participated. The faculty and resident response rates were 11.6% (n = 29/249) and 55.3% (n = 21/38), respectively. The mean ± standard deviation overall score on MCQs was 39.0% ± 20.4%. The majority of participants perceived the impact of CBD on resident education to be equivocal and to not impact patient care. Knowledge of CBD was not statistically significantly associated with participants' perceptions of the impact of CBD on either resident education or patient care (P > .05). Delivery of high-quality feedback was the greatest priority identified for PDAs. Discussion: Our results and proposed CBD PDAs framework may help to guide diagnostic radiology PGME programs in designing evidence-informed PDAs, which may meaningfully contribute to the successful implementation of CBD in diagnostic radiology PGME. As diagnostic radiology PGME programs throughout the world increasingly implement CBME models, evidence-informed PDAs will become of increasing importance.

Competency-Based Medical Education in Canadian Radiation Oncology Residency Training: an Institutional Implementation Pilot Study

Canadian radiation oncology (RO) residency programs transitioned to a competency-based medical education (CBME) training model named Competence by Design (CBD) in July 2019. Prior to this, CBD was piloted in a single RO training program to characterize assessment completion and challenges of implementation. Six residents and seven staff participated in a mixed-methods study and were oriented to CBD. Four Entrustable Professional Activities were assessed over a 4-week-long block and documented using online assessment forms. Anonymized assessments were analyzed to characterize completion. Post-pilot surveys were completed by 4/6 residents and 5/7 staff. Semi-structured post-pilot focus groups were conducted with all residents. Assessments were requested and documented on a weekly basis. Narrative comments were found in 68.1% of assessments, of which 26.7% described specific examples of observed competence or recommendations for improvement. Three of five staff believed that assessments have a negative impact on clinical workflow. Three themes were identified: (1) direct observation is the most challenging aspect of CBD to implement; (2) feedback content can be improved; and (3) staff attitude, clinical workflow, and inaccessibility of assessment forms are the primary barriers to completing assessments. This study demonstrates that CBD assessments can be completed regularly in an outpatient radiation oncology setting and that implementation challenges include improving feedback quality, promoting direct observation, and continuing faculty development to improve perceptions of this assessment model. Further study is required to identify best practices and expectations for the discipline in the era of CBME.

Review of Challenges to the Implementation of Competence by Design in Post-Graduate Medical Education: What Can Diagnostic Radiology Learn from the Experience of Other Specialty Disciplines?

Competence by Design (CBD) is a medical education initiative instituted by the Royal College of Physicians and Surgeons of Canada to improve the training of resident physicians in specialty disciplines. CBD integrates Competency Based Medical Education with traditional specialty discipline post-graduate medical education (PGME) training through the application of an organizational framework of competencies. Various specialty disciplines in Canada have transitioned to CBD since 2017 in a staggered approach. Diagnostic radiology PGME programs in Canada are expected to transition to CBD in 2022 for the incoming resident physician cohort. This article reviews potential challenges to the implementation of CBD in diagnostic radiology PGME programs and proposes evidence-informed targeted strategies and solutions to address these challenges. It is important for diagnostic radiology PGME programs to understand the challenges pertaining to the implementation of CBD so that they may be able to successfully implement this or similar medical education initiatives in their programs. Moreover, as radiology subspecialty PGME programs, such as nuclear medicine, interventional radiology, neuroradiology, and pediatric radiology, likewise transition to CBD and diagnostic radiology PGME programs internationally increasingly implement other Competency Based Medical Education models, the implications of the challenges pertaining to the implementation of CBD will further become of increasing importance.

Preparedness of Residents and Medical Students for the Transition to Competence by Design in Diagnostic Radiology Post-Graduate Medical Education

Introduction: This needs assessment evaluated residents' and medical students' knowledge of Competence by Design (CBD), perceived benefits of and challenges or barriers to the transition to CBD for residents, and perceived overall preparedness for the transition to CBD in diagnostic radiology. Materials and Methods: All diagnostic radiology residents and medical students in Canada were eligible to participate in this national cross-sectional, questionnaire-based needs assessment. Knowledge of CBD was evaluated through participants' self-reported rating of their knowledge of CBD on a 5-point Likert scale. Perceived benefits of and challenges or barriers to the transition to CBD for residents were rank ordered. Participants' overall self-reported preparedness for the transition to CBD was assessed on a 5-point Likert scale. Data were summarized by descriptive statistics and bivariate analyses were conducted as appropriate. Results: Ninety-four residents (n = 77) and medical students (n = 17) participated in this needs assessment. Participants' mean ± standard deviation self-reported rating of their overall knowledge of CBD was 2.86 ± .94. Provision of meaningful feedback to learners and learners' ability to identify their own educational needs were among the highest ranked perceived benefits of the transition to CBD, while demands on time and increased frequency of evaluation were among the highest ranked perceived challenges or barriers to the transition to CBD. Few participants reported being either "prepared" (4.7%) or "somewhat prepared" (14.0%) for the transition to CBD. Conclusion: Preparedness for the transition to CBD in diagnostic radiology may be improved. Targeted interventions to augment the preparedness of residents and medical students should be considered.

What is Transferred and How Much is Retained? A Simulation Study of Complex Surgical Skills

INTRODUCTION

Studies indicate that learning surgical skills on low-fidelity models is equally beneficial to learning on high-fidelity models in terms of skills retention and transfer. However, it is unclear how low-fidelity simulation training impacts retention and transfer in novice learners, particularly on complex surgical tasks that incorporate multiple challenging skills. This study explores the capacity of complete novices to learn and transfer complex surgical skills from a low-fidelity model to a high-fidelity simulation after a delay.

METHODS

Task-naïve medical and nonmedical undergraduate students (n = 62) participated in a three-phase prospective double-arm randomized (2:1) experimental study. Participants completed two skills training sessions (end-to-side anastomosis) on a low-fidelity bench model. After a 4-week delay, participants completed the task again either using the low-fidelity model or a high-fidelity model (cadaver) and were assessed using a validated checklist.

RESULTS

There was a significant time × fidelity group interaction (P = 0.004). Simple effects analysis indicated the high-fidelity group (M_diff = 4.18, P < 0.001) performed significantly worse (P = 0.003) in phase 3 relative to phase 2 compared to the low-fidelity group (M_diff = 0.75, P = 0.39). Post hoc logistic regression analysis indicated that radial suturing technique and economy of motion skills were less likely to be completed correctly for those in the high-fidelity group.

CONCLUSIONS

These findings suggest that for novice populations, relying on low-fidelity simulation training as a source of teaching complex skills may not provide a reliable transfer to high-fidelity models and in turn clinical settings.

Rivalries for attention: insights from a realist evaluation of a postgraduate competency-based medical education implementation in Canada

BACKGROUND

Implementing competency-based medical education (CBME) in post-graduate medical education (PGME) is a complex process that requires multiple systemic changes in a complex system that is simultaneously engaged in multiple initiatives. These initiatives often compete for attention during the implementation of CBME and produce unintended and unanticipated consequences. Understanding the impact of this context is necessary for evaluating the effectiveness of CBME. The purpose of the study was to identify factors, such as contexts and processes, that contribute to the implementation of CBME.

METHODS

We conducted a realist evaluation using data collected from 15 programs through focus groups with residents (2 groups, n = 16) and faculty (one group, n = 8), and semi-structured interviews with program directors (n = 18), and program administrators (n = 12) from 2018 to 2021. Data were analyzed using a template analysis based on a coding framework that was developed from a sample of transcripts, the context-mechanism-outcomes framework for realist evaluations, and the core components of CBME.

RESULTS

The findings demonstrate that simultaneous initiatives in the academic health sciences system creates a key context for CBME implementation - rivalries for attention - and specifically, the introduction of curricular management systems (CMS) concurrent to, but separate from, the implementation of CBME. This context influenced participants' participation, communication, and adaptation during CBME implementation, which led to change fatigue and unmet expectations for the collection and use of assessment data.

CONCLUSIONS

Rival initiatives, such as the concurrent implementation of a new CMS, can have an impact on how programs implement CBME and greatly affect the outcomes of CBME. Mitigating the effects of rivals for attention with flexibility, clear communication, and training can facilitate effective implementation of CBME.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Transitioning to virtual ambulatory care during the COVID-19 pandemic: a qualitative study of faculty and resident physician perspectives

BACKGROUND

The COVID-19 pandemic resulted in a rapid shift from in-person to virtual care delivery for many medical specialties across Canada. The purpose of this study was to explore the lived experiences of resident physicians and faculty related to teaching, learning and assessment during ambulatory virtual care encounters within the competency-based medical education model.

METHODS

In this qualitative phenomenological study, we recruited resident physicians (postgraduate year [PGY] 1-5 trainees) and faculty from the Departments of Surgery and Medicine at Queen's University, Ontario, via purposive sampling. Participants were not required to have exposure to virtual care. Interviews were conducted from September 2020 to March 2021 by 1 researcher, and 2 researchers conducted focus groups via Zoom to explore participants' experiences with the transition to virtual care. These were audio-recorded and transcribed verbatim; qualitative data were analyzed thematically.

RESULTS

There were 18 male and 19 female participants; 20 were resident physicians and 17 were faculty; 19 were from the Department of Surgery and 18 from the Department of Medicine. All faculty participants had participated in virtual care during ambulatory care; 2 PGY-1 residents in surgery had not actively participated in virtual care, although they had participated in clinics where faculty were using virtual care. The mean age of faculty participants was 38 (standard deviation [SD] 8.6) years, and the mean age of resident physicians was 29 (SD 5.4) years. Overall, 28 interviews and 4 focus groups (range 2-3 participants per group) were conducted, and 4 themes emerged: teaching and learning, assessment, logistical considerations, and suggestions. Barriers to teaching included the lack of direct observations and teaching time, and barriers to assessment included an absence of specific Entrustable Professional Activities (EPAs) and feedback focused on virtual care-related competencies. Logistical challenges included lack of technological infrastructure, insufficient private office space and administrative burdens. Both resident physicians and faculty did not foresee virtual care limiting resident physicians' ability to progress within competency-based medical education. Benefits of virtual care included increased accessibility to patients for follow-up visits, for disclosing patients' results and for out-of-town visits. Suggestions included faculty development, improved access to technology and space, educational guidelines for conducting virtual care encounters, and development of virtual care-specific competencies and EPAs.

INTERPRETATION

In the postgraduate program we studied, virtual care imposed substantial barriers on teaching, learning and assessment during the first year of the COVID-19 pandemic. Adapting to new circumstances such as virtual care with suggestions from resident physicians and faculty may help to ensure the continuity of postgraduate medical education throughout the COVID-19 pandemic.

An adaptation-focused evaluation of Canada's first competency-based medical education implementation in radiology

OBJECTIVES

Systematic program evaluation of the Queen's University diagnostic radiology residency program following transition to a competency-based medical education (CBME) curriculum.

METHODS

Rapid Evaluation methodology and the Core Components Framework were utilized to measure CBME implementation. A combination of interviews and focus groups were held with program leaders (n = 6), faculty (n = 10), both CBME stream and traditional stream residents (n = 6), and program staff (n = 2). Interviews and focus groups were transcribed and analyzed abductively. Study team met with program leaders to review common themes and plan potential adaptations.

RESULTS

Strengths of CBME implementation included more frequent and timely feedback as well as the role of the Academic Advisor. However, frontline faculty felt insufficiently supported with regards to the theory and practical implementation of the new curriculum and found assessment tools unintuitive. The circumstances surrounding the curricular implementation also resulted in some negative sentiment. Additional faculty and resident education workshops were identified as areas for improvement as well as changes to assessment tools for increased clarity. Residents overall viewed the changes favorably, with traditional stream residents indicating that they also had a desire for increased feedback.

CONCLUSIONS

Rapid Evaluation is an effective method for program assessment following curricular change in diagnostic radiology. A departmental champion driving enthusiasm for change from within may be valuable. Adequate resident and faculty education is key to maximize change and smooth the transition. Advances in knowledge: This study provides insights for other radiology training programs transitioning to a CBME framework and provides a structure for programmatic assessment.

Exploring the use of rating scales with entrustment anchors in workplace-based assessment

PURPOSE

Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale.

METHOD

A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes.

RESULTS

Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not.

CONCLUSIONS

By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

The Implementation of Entrustable Professional Activities in Postgraduate Medical Education in the Netherlands: Rationale, Process, and Current Status

Postgraduate medical education in the Netherlands has adopted competency-based education since the turn of the century. In 2006, the CanMEDS competency framework was introduced. A 2013 government plan to reduce the length and budgets of training programs led the Dutch Association of Medical Specialists (DAMS) to respond with a proposal to create more flexibility and individualization rather than a blunt cut in the length across all training programs. DAMS launched a government-funded, nation-wide, 4-year project (2014-2018) to blueprint the reform of postgraduate medical education in this direction. To achieve competency-based individualization, the fixed duration of postgraduate programs was abandoned, and entrustable professional activities (EPAs) were introduced in all specialty programs. Implementation of this new generation of programs took place in 2017-2019 in all disciplines. The project focused on EPA-based individualization of all programs, while addressing issues of the continuity of patient care in time-variable programs and the legal and regulatory consequences of individualization. About 30 specialty programs were revised at national, regional, local, and individual levels to incorporate EPAs; portfolio systems were adapted, clinical competency committees were installed for all programs, and procedures for summative entrustment decision making were elaborated. This paper reports on the rationale and the process that led to a more time-variable postgraduate education landscape, and, on average, a shortening of training length by 3 months.