Developing a dashboard for faculty development in competency-based training programs: a design-based research project

PharmaCORE: Optimizing medical pharmacology education with an innovative instructional dashboard

WHAT WAS THE EDUCATIONAL CHALLENGE?

Diminishing emphasis on pharmacology education in medical schools has resulted in a concerning lack of prescribing knowledge among physician graduates. These concerns mirror our graduates' expressed dissatisfaction with the structure and quality of pharmacology educational experiences over the past 5 years.

WHAT WAS THE SOLUTION?

PharmaCORE, a web-based instructional dashboard, was developed as an interactive faculty development tool to enhance integration and instruction of pharmacology content in pre-clinical curriculum at a US medical school.

HOW WAS THE SOLUTION IMPLEMENTED?

PharmaCORE was introduced in Spring 2022 for instructors teaching pharmacology in the pre-clinical curriculum. Instructors used the dashboard to assess coverage of specific drug topics throughout the curriculum and to apply tailored, learner-centered teaching strategies to optimize learner engagement and comprehension.

WHAT LESSONS WERE LEARNED THAT ARE RELEVANT TO A WIDER GLOBAL AUDIENCE?

The initial assessment indicated that the dashboard was user-friendly and positively influenced instructor awareness of pharmacology content and learner-centered teaching. This faculty development approach underscores the importance of skill-based mapping and maintaining learner-centered teaching standards to address other integrated subjects and broader curricular challenges.

WHAT ARE THE NEXT STEPS?

This study lays the foundation for the broader applicability of instructional dashboards in tracking and addressing curricular challenges across pharmacology and other science subjects. Future steps include more personalized feedback for instructors, creating a student-accessible version, and ongoing monitoring of maintenance measures like milestone exams.

Enabling Implementation of Competency Based Medical Education through an Outcomes-Focused Accreditation System

Competency based medical education is being adopted around the world. Accreditation plays a vital role as an enabler in the adoption and implementation of competency based medical education, but little has been published about how the design of an accreditation system facilitates this transformation. The Canadian postgraduate medical education environment has recently transitioned to an outcomes-based accreditation system in parallel with the adoption of competency based medical education. Using the Canadian example, we characterize four features of an accreditation system that can facilitate the implementation of competency based medical education: theoretical underpinning, quality focus, accreditation standards, and accreditation processes. Alignment of the underlying educational theories within the accreditation system and educational paradigm drives change in a consistent and desired direction. An accreditation system that prioritizes quality improvement over quality assurance promotes educational system development and progressive change. Accreditation standards that achieve the difficult balance of being sufficiently detailed yet flexible foster a high fidelity of implementation without stifling innovation. Finally, accreditation processes that recognize the change process, encourage program development, and are not overly punitive all enable the implementation of competency based medical education. We also discuss the ways in which accreditation can simultaneously hinder the implementation of this approach. As education bodies adopt competency based medical education, particular attention should be paid to the role that accreditation plays in successful implementation.

The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study

Background

Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs.

Methods

Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall's tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients.

Results

Both the faculty's utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees' utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88).

Conclusions

The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.

Competency-based faculty development: applying transformations from lessons learned in competency-based medical education

Faculty development in medical education is often delivered in an ad hoc manner instead of being a deliberately sequenced program matched to data-informed individual needs. In this article, the authors, all with extensive experience in Faculty Development (FD), present a competency-based faculty development (CBFD) framework envisioned to enhance the impact of FD. Steps and principles in the CBFD framework reflect the lessons learned from competency-based medical education (CBME) with its foundational goal to better train physicians to meet societal needs. The authors see CBFD as a similar framework, this one to better train faculty to meet educational needs. CBFD core elements include articulated competencies for the varied educational roles faculty fulfill, deliberately designed curricula structured to build those competencies, and an assessment program and process to support individualized faculty learning and professional growth. The framework incorporates ideas about where and how CBFD should be delivered, the use of coaching to promote reflection and identity formation and the creation of communities of learning. As with CBME, the CBFD framework has included the important considerations of change management, including broad stakeholder engagement, continuous quality improvement and scholarship. The authors have provided examples from the literature as well as challenges and considerations for each step.

Developing and authenticating an electronic health record-based report card for assessing residents' clinical performance

Purpose

The electronic health record (EHR) is frequently identified as a source of assessment data regarding residents' clinical performance. To better understand how to harness EHR data for education purposes, the authors developed and authenticated a prototype resident report card. This report card used EHR data exclusively and was authenticated with various stakeholders to understand individuals' reactions to and interpretations of EHR data when presented in this way.

Methods

Using principles derived from participatory action research and participatory evaluation, this study brought together residents, faculty, a program director, and medical education researchers (n = 19) to develop and authenticate a prototype report card for residents. From February to September 2019, participants were invited to take part in a semistructured interview that explored their reactions to the prototype and provided insights about how they interpreted the EHR data.

Results

Our results highlighted three themes: data representation, data value, and data literacy. Participants varied in terms of the best way to present the various EHR metrics and felt pertinent contextual information should be included. All participants agreed that the EHR data presented were valuable, but most had concerns about using it for assessment. Finally, participants had difficulties interpreting the data, suggesting that these data could be presented more intuitively and that residents and faculty may require additional training to fully appreciate these EHR data.

Conclusions

This work demonstrated how EHR data could be used to assess residents' clinical performance, but it also identified areas that warrant further consideration, especially pertaining to data representation and subsequent interpretation. Providing residents and faculty with EHR data in a resident report card was viewed as most valuable when used to guide feedback and coaching conversations.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project

BACKGROUND

Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation.

METHODS

Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations.

RESULTS

Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard.

CONCLUSIONS

We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

Integrating training, practice, and reflection within a new model for Canadian medical licensure: a concept paper prepared for the Medical Council of Canada

In 2020 the Medical Council of Canada created a task force to make recommendations on the modernization of its practices for granting licensure to medical trainees. This task force solicited papers on this topic from subject matter experts. As outlined within this Concept Paper, our proposal would shift licensure away from the traditional focus on high-stakes summative exams in a way that integrates training, clinical practice, and reflection. Specifically, we propose a model of graduated licensure that would have three stages including: a trainee license for trainees that have demonstrated adequate medical knowledge to begin training as a closely supervised resident, a transition to practice license for trainees that have compiled a reflective educational portfolio demonstrating the clinical competence required to begin independent practice with limitations and support, and a fully independent license for unsupervised practice for attendings that have demonstrated competence through a reflective portfolio of clinical analytics. This proposal was reviewed by a diverse group of 30 trainees, practitioners, and administrators in medical education. Their feedback was analyzed and summarized to provide an overview of the likely reception that this proposal would receive from the medical education community.

The Impact of Electronic Data to Capture Qualitative Comments in a Competency-Based Assessment System

Introduction

Digitalizing workplace-based assessments (WBA) holds the potential for facilitating feedback and performance review, wherein we can easily record, store, and analyze data in real time. When digitizing assessment systems, however, it is unclear what is gained and lost in the message as a result of the change in medium. This study evaluates the quality of comments generated in paper vs. electronic media and the influence of an assessor’s seniority.

Methods

Using a realist evaluation framework, a retrospective database review was conducted with paper-based and electronic medium comments. A sample of assessments was examined to determine any influence of the medium on the word count and the Quality of Assessment for Learning (QuAL) score. A correlation analysis evaluated the relationship between word count and QuAL score. Separate univariate analyses of variance (ANOVAs) were used to examine the influence of the assessor's seniority and medium on word count, QuAL score, and WBA scores.

Results

The analysis included a total of 1,825 records. The average word count for the electronic comments (M=16) was significantly higher than the paper version (M=12; p=0.01). Longer comments positively correlated with QuAL score (r=0.2). Paper-based comments received lower QuAL scores (0.41) compared to electronic (0.51; p<0.01). Years in practice was negatively correlated with QuAL score (r=-0.08; p<0.001) as was word count (r=-0.2; p<0.001).

Conclusion

Digitization of WBAs increased the length of comments and did not appear to jeopardize the quality of WBAs; these results indicate higher-quality assessment data. True digital transformation may be possible by harnessing trainee data repositories and repurposing them to analyze for faculty-relevant metrics.

The Learning Loop: Conceptualizing Just-in-Time Faculty Development

BACKGROUND

As technology advances, the gap between learning and doing continues to close-especially for frontline academic faculty and clinician educators. For busy clinician faculty members, it can be difficult to find time to engage in skills and professional development. Competing interests between clinical care and various forms of academic work (e.g., research, administration, education) all create challenges for traditional group-based and/or didactic faculty development.

METHODS

The authors engaged in a synthetic narrative review of literature from several unrelated fields: learning technologies, medical education/health professions education, general/higher education. The aim for this review was to synthesize this pre-existing literature to propose a new conceptual model.

RESULTS

The authors propose a new conceptual model, the Just-In-Time Learning Loop, to guide the development of online faculty development for just-in-time delivery.

CONCLUSIONS

The Just-In-Time Learning Loop is a new conceptual framework that may be of use to those engaging in online, digital learning design. Faculty developers, especially in emergency medicine, can integrate leading concepts from the technology-enhanced learning field (e.g., microlearning, micro-credentialing, badging) to create new types of learning experiences for their end-users.