1
|
Ellaway RH, O'Brien BC, Sherbino J, Maggio LA, Artino AR, Nimmon L, Park YS, Young M, Thomas A. Is There a Problem With Evidence in Health Professions Education? ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:841-848. [PMID: 38574241 DOI: 10.1097/acm.0000000000005730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/06/2024]
Abstract
ABSTRACT What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume this knowledge might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more system-wide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations and influencing decisions.
Collapse
|
2
|
Johnson D, Chopra S, Bilgic E. Exploring the Use of Natural Language Processing to Understand Emotions of Trainees and Faculty Regarding Entrustable Professional Activity Assessments. J Grad Med Educ 2024; 16:323-327. [PMID: 38882409 PMCID: PMC11173031 DOI: 10.4300/jgme-d-23-00526.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 12/06/2023] [Accepted: 03/25/2024] [Indexed: 06/18/2024] Open
Abstract
Background In medical education, artificial intelligence techniques such as natural language processing (NLP) are starting to be used to capture and analyze emotions through written text. Objective To explore the application of NLP techniques to understand resident and faculty emotions related to entrustable professional activity (EPA) assessments. Methods Open-ended text data from a survey on emotions toward EPA assessments were analyzed. Respondents were residents and faculty from pediatrics (Peds), general surgery (GS), and emergency medicine (EM), recruited for a larger emotions study in 2023. Participants wrote about their emotions related to receiving/completing EPA assessments. We analyzed the frequency of words rated as positive via a validated sentiment lexicon used in NLP studies. Specifically, we were interested if the count of positive words varied as a function of group membership (faculty, resident), specialty (Peds, GS, EM), gender (man, woman, nonbinary), or visible minority status (yes, no, omit). Results A total of 66 text responses (30 faculty, 36 residents) contained text data useful for sentiment analysis. We analyzed the difference in the count of words categorized as positive across group, specialty, gender, and being a visible minority. Specialty was the only category revealing significant differences via a bootstrapped Poisson regression model with GS responses containing fewer positive words than EM responses. Conclusions By analyzing text data to understand emotions of residents and faculty through an NLP approach, we identified differences in EPA assessment-related emotions of residents versus faculty, and differences across specialties.
Collapse
Affiliation(s)
- Devin Johnson
- is a recent graduate of the PhD program, Department of Psychology, Neuroscience & Behaviour, McMaster University, Hamilton, Ontario, Canada
| | - Sonaina Chopra
- is a Research Assistant, McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, Ontario, Canada; and
| | - Elif Bilgic
- is an Assistant Professor and Education Scientist, Department of Pediatrics and McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
3
|
Chopra S, Harley JM, Keuhl A, Bassilious E, Sherbino J, Bilgic E. Understanding Emotions Impacted by New Assessment Mandates Implemented in Medical Education: A Survey of Residents and Faculty Across Multiple Specialties. Cureus 2024; 16:e62013. [PMID: 38983997 PMCID: PMC11233152 DOI: 10.7759/cureus.62013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/31/2024] [Indexed: 07/11/2024] Open
Abstract
Background Previous research findings show that the overall perception of residents regarding the new entrustable professional activity (EPA) assessment mandates is primarily negative. Hence, this study aims to explore the link between EPA assessment experiences and resident and faculty emotions and expectancy of successfully completing residency training. Methods A standardized questionnaire (Medical Emotions Scale (MES)), which measures 20 unique emotions on a 5-point Likert scale, was used to explore the emotions of residents and faculty members regarding EPA assessments and residents' expectancy of success. Data analysis included descriptive statistics and analysis of variance (ANOVA). Results Ninety-one (N=91) participants (46 faculty members and 45 residents) completed the survey. The results revealed that residents have more negative emotions toward EPA assessments compared to faculty. Additionally, resident and faculty emotions regarding EPA assessments vary across specialty and gender. Conclusions These findings will be crucial in providing the Royal College of Physicians and Surgeons of Canada and medical education programs with concrete evidence and guidance in understanding the perspectives and emotions of residents and faculty towards EPA assessments and residents' beliefs about successfully completing their medical training.
Collapse
Affiliation(s)
- Sonaina Chopra
- McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, CAN
| | | | - Amy Keuhl
- McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, CAN
| | | | - Jonathan Sherbino
- McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, CAN
- Emergency Medicine, McMaster University, Hamilton, CAN
| | - Elif Bilgic
- McMaster Education Research, Innovation and Theory (MERIT) Program, McMaster University, Hamilton, CAN
- Pediatrics, McMaster University, Hamilton, CAN
| |
Collapse
|
4
|
Brown A, La J, Keri MI, Hillis C, Razack S, Korah N, Karpinski J, Frank JR, Wong B, Goldman J. In EPAs we trust, is quality and safety a must? A cross-specialty analysis of entrustable professional activity guides. MEDICAL TEACHER 2024:1-9. [PMID: 38527417 DOI: 10.1080/0142159x.2024.2323177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 02/21/2024] [Indexed: 03/27/2024]
Abstract
PURPOSE The inclusion of quality improvement (QI) and patient safety (PS) into CanMEDS reflects an expectation that graduating physicians are competent in these areas upon training completion. To ensure that Canadian postgraduate specialty training achieves this, the translation of QI/PS competencies into training standards as part of the implementation of competency-based medical education requires special attention. METHODS We conducted a cross-specialty, multi-method analysis to examine how QI/PS was incorporated into the EPA Guides across 11 postgraduate specialties in Canada. RESULTS We identify cross-specialty variability in how QI/PS is incorporated, positioned, and emphasized in EPAs and milestones. QI/PS was primarily referenced alongside clinical activities rather than as a sole competency or discrete activity. Patterns were characterized in how QI/PS became incorporated into milestones through repetition and customization. QI/PS was also decoupled, conceptualized, and emphasized differently across specialties. CONCLUSIONS Variability in the inclusion of QI/PS in EPAs and milestones has important implications considering the visibility and influence of EPA Guides in practice. As specialties revisit and revise EPA Guides, there is a need to balance the standardization of foundational QI/PS concepts to foster shared understanding while simultaneously ensuring context-sensitive applications across specialties. Beyond QI/PS, this study illuminates the challenges and opportunities that lie in bridging theoretical frameworks with practical implementation in medical education, prompting broader consideration of how intrinsic roles and emergent areas are effectively incorporated into competency-based medical education.
Collapse
Affiliation(s)
- Allison Brown
- Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Julie La
- Graduate Program in Health Quality, Queen's University, Kingston, Canada
- Department of Surgery, Queen's University, Kingston, Canada
| | | | - Chris Hillis
- Faculty of Health Sciences, McMaster University, Hamilton, Canada
| | - Saleem Razack
- Faculty of Medicine, University of British Columbia, Kelowna, Canada
| | - Nadine Korah
- Faculty of Medicine, McGill University, Montreal, Canada
| | | | - Jason R Frank
- Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Brian Wong
- Centre for Quality Improvement and Patient Safety, Toronto, Canada
- Faculty of Medicine, University of Toronto Temerty, Toronto, Canada
| | - Joanne Goldman
- Centre for Quality Improvement and Patient Safety, Toronto, Canada
- Faculty of Medicine, University of Toronto Temerty, Toronto, Canada
| |
Collapse
|
5
|
Sternszus R, Slattery NK, Cruess RL, Cate OT, Hamstra SJ, Steinert Y. Contradictions and Opportunities: Reconciling Professional Identity Formation and Competency-Based Medical Education. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:507-516. [PMID: 37954041 PMCID: PMC10637293 DOI: 10.5334/pme.1027] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/22/2023] [Accepted: 10/24/2023] [Indexed: 11/14/2023]
Abstract
The widespread adoption of Competency-Based Medical Education (CBME) has resulted in a more explicit focus on learners' abilities to effectively demonstrate achievement of the competencies required for safe and unsupervised practice. While CBME implementation has yielded many benefits, by focusing explicitly on what learners are doing, curricula may be unintentionally overlooking who learners are becoming (i.e., the formation of their professional identities). Integrating professional identity formation (PIF) into curricula has the potential to positively influence professionalism, well-being, and inclusivity; however, issues related to the definition, assessment, and operationalization of PIF have made it difficult to embed this curricular imperative into CBME. This paper aims to outline a path towards the reconciliation of PIF and CBME to better support the development of physicians that are best suited to meet the needs of society. To begin to reconcile CBME and PIF, this paper defines three contradictions that must and can be resolved, namely: (1) CBME attends to behavioral outcomes whereas PIF attends to developmental processes; (2) CBME emphasizes standardization whereas PIF emphasizes individualization; (3) CBME organizes assessment around observed competence whereas the assessment of PIF is inherently more holistic. Subsequently, the authors identify curricular opportunities to address these contradictions, such as incorporating process-based outcomes into curricula, recognizing the individualized and contextualized nature of competence, and incorporating guided self-assessment into coaching and mentorship programs. In addition, the authors highlight future research directions related to each contradiction with the goal of reconciling 'doing' and 'being' in medical education.
Collapse
Affiliation(s)
- Robert Sternszus
- Department of Pediatrics & Institute of Health Sciences Education, McGill University, Montreal, Quebec, CA
| | | | - Richard L. Cruess
- Department of Orthopedic Surgery & Institute of Health Sciences Education, McGill University, Montreal, Quebec, CA
| | - Olle ten Cate
- Utrecht Center for Research and Development of Health Professions Education, University Medical Center Utrecht and Utrecht University, NL
| | - Stanley J. Hamstra
- Department of Surgery, University of Toronto, Toronto, Canada
- Sunnybrook Research Institute, Holland Bone and Joint Program, Toronto, Canada
- ACGME, Chicago, IL, US
- Department of Medical Education, Northwestern University Feinberg School of Medicine, Chicago, IL, US
| | - Yvonne Steinert
- Department of Family Medicine & Institute of Health Sciences Education, McGill University, Montreal, Quebec, CA
| |
Collapse
|
6
|
Gingerich A, Sebok-Syer SS, Lingard L, Watling CJ. The shift from disbelieving underperformance to recognising failure: A tipping point model. MEDICAL EDUCATION 2022; 56:395-406. [PMID: 34668213 DOI: 10.1111/medu.14681] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Revised: 10/05/2021] [Accepted: 10/15/2021] [Indexed: 06/13/2023]
Abstract
CONTEXT Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed. METHODS Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process. RESULTS The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm. CONCLUSION Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.
Collapse
Affiliation(s)
- Andrea Gingerich
- Division of Medical Sciences, University of Northern British Columbia, Prince George, British Columbia, Canada
| | | | - Lorelei Lingard
- Schulich School of Medicine & Dentistry, Centre for Education Research & Innovation, Western University, London, Ontario, Canada
| | - Christopher J Watling
- Schulich School of Medicine & Dentistry, Centre for Education Research & Innovation, Western University, London, Ontario, Canada
| |
Collapse
|
7
|
Canales MK, Drevdahl DJ. A Sisyphean task: Developing and revising public health nursing competencies. Public Health Nurs 2022; 39:1078-1088. [PMID: 35395106 PMCID: PMC9543881 DOI: 10.1111/phn.13077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2021] [Revised: 03/04/2022] [Accepted: 03/10/2022] [Indexed: 11/26/2022]
Abstract
BACKGROUND Competencies are intended to enhance the public health workforce's skills. Competencies used to evaluate public health nursing (PHN) practice and education have been promoted by several nursing organizations. Having multiple sets of competencies raises questions about redundancies and their usefulness in evaluating PHN, as well as the central question about the value of the competencies themselves. METHODS A literature review of psychometric evaluation research of the competencies was performed. Qualitative content analyses were conducted of seven documents: Association of Community Health Nursing Educators', 2000 and 2010 essentials; Quad Council Coalition's 2004, 2011, and 2018 competencies; and the American Nurses Association's, 2013 and the 2021 draft of PHN scope and standards of practice with respect to competency definition, conceptual basis, and use of an established taxonomy. RESULTS No psychometric evaluations of the competency sets were found. Textual content analysis revealed inconsistent and or missing competency definitions and theoretical frameworks with competencies proliferating over time. Taxonomy analysis identified minimal competencies at higher complexity levels according to Bloom's revised taxonomy. CONCLUSIONS Analyzed competencies lack reliability and validity testing, making assessment difficult for PHN educators and practitioners. Multiple and competing competencies further erode PHN's visibility, even among public health nurses. With unending revisions of PHN competencies and lack of supporting evidence regarding their effect and their integration into education or practice, recommendations for future efforts are offered.
Collapse
|
8
|
Nomura O, Itoh T, Mori T, Ihara T, Tsuji S, Inoue N, Carrière B. Creating Clinical Reasoning Assessment Tools in Different Languages: Adaptation of the Pediatric Emergency Medicine Script Concordance Test to Japanese. Front Med (Lausanne) 2021; 8:765489. [PMID: 34950681 PMCID: PMC8688734 DOI: 10.3389/fmed.2021.765489] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2021] [Accepted: 11/12/2021] [Indexed: 11/17/2022] Open
Abstract
Introduction: Clinical reasoning is a crucial skill in the practice of pediatric emergency medicine and a vital element of the various competencies achieved during the clinical training of resident doctors. Pediatric emergency physicians are often required to stabilize patients and make correct diagnoses with limited clinical information, time and resources. The Pediatric Emergency Medicine Script Concordance Test (PEM-SCT) has been developed specifically for assessing physician's reasoning skills in the context of the uncertainties in pediatric emergency practice. In this study, we developed the Japanese version of the PEM-SCT (Jpem-SCT) and confirmed its validity by collecting relevant evidence. Methods: The Jpem-SCT was developed by translating the PEM-SCT into Japanese using the Translation, Review, Adjudication, Pretest, Documentation team translation model, which follows cross-cultural survey guidelines for proper translation and cross-cultural and linguistic equivalences between the English and Japanese version of the survey. First, 15 experienced pediatricians participated in the pre-test session, serving as a reference panel for modifying the test descriptions, incorporating Japanese context, and establishing the basis for the scoring process. Then, a 1-h test containing 60 questions was administered to 75 trainees from three academic institutions. Following data collection, we calculated the item-total correlations of the scores to optimize selection of the best items in the final version of the Jpem-SCT. The reliability of the finalized Jpem-SCT was calculated using Cronbach's α coefficient for ensuring generalizability of the evidence. We also conducted multiple regression analysis of the test score to collect evidence on validity of the extrapolation. Results: The final version of the test, based on item-total correlation data analysis, contained 45 questions. The participant's specialties were as follows: Transitional interns 12.0%, pediatric residents 56.0%, emergency medicine residents 25.3%, and PEM fellows 6.7%. The mean score of the final version of the Jpem-SCT was 68.6 (SD 9.8). The reliability of the optimized test (Cronbach's α) was 0.70. Multiple regression analysis showed that being a transitional intern was a negative predictor of test scores, indicating that clinical experience relates to performance on the Jpem-SCT. Conclusion: This pediatric emergency medicine Script Concordance Test was reliable and valid for assessing the development of clinical reasoning by trainee doctors during residency training.
Collapse
Affiliation(s)
- Osamu Nomura
- Department of Emergency and Disaster Medicine, Hirosaki University, Hirosaki, Aomori, Japan
| | - Taichi Itoh
- Department of Emergency Medicine, University of Michigan, Ann Arbor, MI, United States.,Department of Medical Education, University of Illinois at Chicago, Chicago, IL, United States
| | - Takaaki Mori
- Department of Emergency Medicine, KK Women's and Children's Hospital, Singapore, Singapore.,Division of Pediatric Emergency Medicine, Tokyo Children's Medical Center, Fuchu, Tokyo, Japan
| | - Takateru Ihara
- Division of Pediatric Emergency Medicine, Tokyo Children's Medical Center, Fuchu, Tokyo, Japan
| | - Satoshi Tsuji
- National Center for Child Health and Development, Setagaya, Tokyo, Japan
| | - Nobuaki Inoue
- Department of Human Resources Development, Bureau of International Health Cooperation, National Center for Global Health and Medicine, Shinjuku, Tokyo, Japan
| | - Benoit Carrière
- Department of Pediatrics, Section of Pediatric Emergency Medicine, Centre Hospitalier Universitaire Sainte-Justine, Montréal, QC, Canada
| |
Collapse
|
9
|
|