1
|
Ginsburg S, Stroud L, Brydges R, Melvin L, Hatala R. Dual purposes by design: exploring alignment between residents' and academic advisors' documents in a longitudinal program. Adv Health Sci Educ Theory Pract 2024:10.1007/s10459-024-10318-2. [PMID: 38438699 DOI: 10.1007/s10459-024-10318-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 02/04/2024] [Indexed: 03/06/2024]
Abstract
Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose-to help competence committees make decisions-which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2 year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents' writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.
Collapse
Affiliation(s)
- Shiphra Ginsburg
- Department of Medicine, Mount Sinai Hospital, Toronto, ON, Canada.
- Wilson Centre for Research in Education, University Health Network, Toronto, ON, Canada.
| | - Lynfa Stroud
- Department of Medicine, Sunnybrook Health Sciences Centre, Toronto, ON, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Ryan Brydges
- Wilson Centre for Research in Education, University Health Network, Toronto, ON, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada
| | - Lindsay Melvin
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada
- Department of Medicine, University Health Network, Toronto, ON, Canada
| | - Rose Hatala
- Department of Medicine, University of British Columbia, Vancouver, BC, Canada
- Centre for Health Education Scholarship, University of British Columbia, Vancouver, Canada
| |
Collapse
|
2
|
Cheng A, Molinaro M, Ott M, Cristancho S, LaDonna KA. Set Up to Fail? Barriers Impeding Resident Communication Training in Neonatal Intensive Care Units. Acad Med 2023; 98:S65-S71. [PMID: 37983398 DOI: 10.1097/acm.0000000000005355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/02/2023]
Abstract
PURPOSE Learning to navigate difficult clinical conversations is an essential feature of residency training, yet much of this learning occurs "on the job," often without the formative, multisource feedback trainees need. To generate insight into how on-the-job training influences trainee performance, the perspectives of parents and health care providers (HCPs) who engaged in or observed difficult conversations with Neonatal Intensive Care Unit (NICU) trainees were explored. METHOD The iterative data generation and analysis process was informed by constructivist grounded theory. Parents (n = 14) and HCPs (n = 10) from 2 Canadian NICUs were invited to participate in semistructured interviews informed by rich pictures-a visual elicitation technique useful for exploring complex phenomena like difficult conversations. Themes were identified using the constant comparative approach. The study was conducted between 2018 and 2021. RESULTS According to participants, misalignment between parents' and trainees' communication styles, HCPs intervening to protect parents when trainee-led communication went awry, the absence of feedback, and a culture of sole physician responsibility for communication all conspired against trainees trying to develop communication competence in the NICU. Given beliefs that trainees' experiential learning should not trump parents' well-being, some physicians perceived the art of communication was best learned by observing experts. Sometimes, already limited opportunities for trainees to lead conversations were further constricted by perceptions that trainees lacked the interest and motivation to focus on so-called "soft" skills like communication during their training. CONCLUSIONS Parents and NICU staff described that trainees face multiple barriers against learning to navigate difficult conversations that may set them up to fail. A deeper understanding of the layered challenges trainees face, and the hierarchies and sociocultural norms that interfere with teaching, may be the start of breaking down multiple barriers trainees and their clinician supervisors need to overcome to succeed.
Collapse
Affiliation(s)
- Anita Cheng
- A. Cheng is a neonatologist and assistant professor, Department of Pediatrics, Western University, London, Ontario, Canada; ORCID: https://orcid.org/0000-0002-6787-7275
| | - Monica Molinaro
- M. Molinaro is a banting postdoctoral fellow, Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada; ORCID: https://orcid.org/0000-0002-5629-5974
| | - Mary Ott
- M. Ott is a researcher, Centre for Education Research & Innovation, Western University, London, Ontario, Canada; ORCID: https://orcid.org/0000-0003-4010-6558
| | - Sayra Cristancho
- S. Cristancho is associate professor and scientist, Centre for Education Research & Innovation, Department of Surgery and Faculty of Education, Western University, London, Ontario, Canada; ORCID: https://orcid.org/0000-0002-8738-2130
| | - Kori A LaDonna
- K.A. LaDonna is associate professor, Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada; ORCID: https://orcid.org/0000-0003-4738-0146
| |
Collapse
|
3
|
Sheikh N, Mehta J, Shah R, Brydges R. Feedback that Lands: Exploring How Residents Receive and Judge Feedback During Entrustable Professional Activities. Perspect Med Educ 2023; 12:427-437. [PMID: 37868076 PMCID: PMC10588547 DOI: 10.5334/pme.1020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Accepted: 10/03/2023] [Indexed: 10/24/2023]
Abstract
Introduction Receiving feedback from different types of assessors (e.g., senior residents, staff supervisors) may impact trainees' perceptions of the quantity and quality of data during entrustable professional activity (EPA) assessments. We evaluated the quality of EPA feedback provided by different assessors (senior residents, chief medical residents/subspecialty residents, and staff) and explored residents' judgements of the value of this feedback. Methods From a database of 2228 EPAs, we calculated the frequency of contribution from three assessor groups. We appraised the quality of 60 procedure-related EPAs completed between July 2019 and March 2020 using a modified Completed Clinical Evaluation Report Rating (CCERR) tool. Next, we asked 15 internal medicine residents to sort randomly selected EPAs according to their judgements of value, as an elicitation exercise before a semi-structured interview. Interviews explored participants' perceptions of quality of written feedback and helpful assessors. Results Residents completed over 60% of EPA assessments. We found no difference in modified-CCERR scores between the three groups. When judging EPA feedback value, residents described a process of weighted deliberation, considering perceived assessor characteristics (e.g., credibility, experience with EPA system), actionable written comments, and their own self-assessment. Discussion Like other recent studies, we found that residents contributed most to procedure-related EPA assessments. To the established list of factors influencing residents' judgements of feedback value, we add assessors' adherence to, and their shared experiences of being assessed within, EPA assessment systems. We focus on the implications for how assessors and leaders can build credibility in themselves and in the practices of EPA assessments.
Collapse
Affiliation(s)
- Natasha Sheikh
- Oakville Trafalgar Memorial Hospital, Oakville, Ontario, Canada
| | - Joshua Mehta
- Critical Care Medicine Residency Program, University of Toronto, Toronto, Ontario, Canada
| | - Rupal Shah
- Division of General Internal Medicine, Toronto Western Hospital, University Health Network, Toronto, Ontario, Canada
- Faculty Member of the HoPingKong Centre for Excellence in Education and Practice, Toronto Western Hospital, University Health Network, Toronto, Ontario, Canada
| | - Ryan Brydges
- Technology-Enabled Education at St. Michael’s Hospital, Unity Health Toronto, CA
- Department of Medicine, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
4
|
Margalit N, Pincavage AT. Feedback as dialogue for faculty: From peer review to peer-to-peer coaching. Med Educ 2023; 57:118-120. [PMID: 36419318 PMCID: PMC10099507 DOI: 10.1111/medu.14980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 11/07/2022] [Accepted: 11/15/2022] [Indexed: 06/16/2023]
Abstract
Drawing on the themes identified by Watling et al. regarding feedback in peer review, the authors consider how their findings could be applied more broadly to faculty feedback.
Collapse
Affiliation(s)
- Noam Margalit
- Department of MedicineUniversity of California San FranciscoSan FranciscoCaliforniaUSA
| | | |
Collapse
|
5
|
Ott MC, Pack R, Cristancho S, Chin M, Van Koughnett JA, Ott M. "The Most Crushing Thing": Understanding Resident Assessment Burden in a Competency-Based Curriculum. J Grad Med Educ 2022; 14:583-592. [PMID: 36274774 PMCID: PMC9580312 DOI: 10.4300/jgme-d-22-00050.1] [Citation(s) in RCA: 20] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 04/21/2022] [Accepted: 08/15/2022] [Indexed: 01/28/2023] Open
Abstract
BACKGROUND Competency-based medical education (CBME) was expected to increase the workload of assessment for graduate training programs to support the development of competence. Learning conditions were anticipated to improve through the provision of tailored learning experiences and more frequent, low-stakes assessments. Canada has adopted an approach to CBME called Competence by Design (CBD). However, in the process of implementation, learner anxiety and assessment burden have increased unexpectedly. To mitigate this unintended consequence, we need a stronger understanding of how resident assessment burdens emerge and function. OBJECTIVE This study investigates contextual factors leading to assessment burden on residents within the framework of CBD. METHODS Residents were interviewed about their experiences of assessment using constructivist grounded theory. Participants (n=21) were a purposive sample from operative and perioperative training programs, recruited from 6 Canadian medical schools between 2019 and 2020. Self-determination theory was used as a sensitizing concept to categorize findings on types of assessment burden. RESULTS Nine assessment burdens were identified and organized by threats to psychological needs for autonomy, relatedness, and competence. Burdens included: missed opportunities for self-regulated learning, lack of situational control, comparative assessment, lack of trust, constraints on time and resources, disconnects between teachers and learners, lack of clarity, unrealistic expectations, and limitations of assessment forms for providing meaningful feedback. CONCLUSIONS This study contributes a contextual understanding of how assessment burdens emerged as unmet psychological needs for autonomy, relatedness, and competence, with unintended consequences for learner well-being and intrinsic motivation.
Collapse
Affiliation(s)
- Mary C. Ott
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Mary C. Ott, PhD, is Research Associate, Centre for Education Research and Innovation
| | - Rachael Pack
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Rachael Pack, PhD, is Research Associate, Centre for Education Research and Innovation
| | - Sayra Cristancho
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Sayra Cristancho, PhD, is Scientist, Centre for Education Research and Innovation
| | - Melissa Chin
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Melissa Chin, MD, MHS, FRCPC, is CBME Lead, Department of Anesthesia and Perioperative Medicine
| | - Julie Ann Van Koughnett
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Julie Ann Van Koughnett, MD, MEd, FRCSC, is Program Director, General Surgery, Department of Surgery
| | - Michael Ott
- All authors are with Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
- Mary C. Ott, PhD, is Research Associate, Centre for Education Research and Innovation
| |
Collapse
|
6
|
Matthiesen M, Kelly MS, Dzara K, Begin AS. Medical residents and attending physicians’ perceptions of feedback and teaching in the United States: a qualitative study. J Educ Eval Health Prof 2022; 19:9. [PMID: 35468668 PMCID: PMC9247714 DOI: 10.3352/jeehp.2022.19.9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 04/19/2022] [Indexed: 06/14/2023]
Abstract
PURPOSE Residents and attendings agree on the importance of feedback to resident education. However, while faculty report providing frequent feedback, residents often do not perceive receiving it, particularly in the context of teaching. Given the nuanced differences between feedback and teaching, we aimed to explore resident and attending perceptions of feedback and teaching in the clinical setting. METHODS We conducted a qualitative study of internal medicine residents and attendings from December 2018 through March 2019 at the Massachusetts General Hospital to investigate perceptions of feedback in the inpatient clinical setting. Residents and faculty were recruited to participate in focus groups. Data were analyzed using thematic analysis to explore perspectives and barriers to feedback provision and identification. RESULTS Five focus groups included 33 total participants in 3 attending (n=20) and 2 resident (n=13) groups. Thematic analysis of focus group transcripts identified 7 themes which organized into 3 thematic categories: (1) disentangling feedback and teaching, (2) delivering high-quality feedback, and (3) experiencing feedback in the group setting. Residents and attendings highlighted important themes in discriminating feedback from teaching. They indicated that while feedback is reactive in response to an action or behavior, teaching is proactive and oriented toward future endeavors. CONCLUSION Confusion between the critical concepts of teaching and feedback may be minimized by allowing them to each have their intended impact, either in response to prior events or aimed toward those yet to take place.
Collapse
Affiliation(s)
- Madeleine Matthiesen
- Department of Medicine, Massachusetts General Hospital, Boston, MA, USA
- Department of Pediatrics, Massachusetts General Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Michael S. Kelly
- Department of Medicine, Massachusetts General Hospital, Boston, MA, USA
- Department of Pediatrics, Massachusetts General Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Kristina Dzara
- Department of Biomedical Informatics and Medical Education and Center for Leadership and Innovation in Medical Education, University of Washington School of Medicine, Seattle, WA, USA
- Department of Medical Education, University of Washington School of Medicine, Seattle, WA, USA
- Center for Leadership and Innovation in Medical Education, University of Washington School of Medicine, Seattle, WA, USA
| | - Arabella Simpkin Begin
- Department of Medicine, Massachusetts General Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| |
Collapse
|
7
|
Long S, Rodriguez C, St-Onge C, Tellier PP, Torabi N, Young M. Factors affecting perceived credibility of assessment in medical education: A scoping review. Adv Health Sci Educ Theory Pract 2022; 27:229-262. [PMID: 34570298 DOI: 10.1007/s10459-021-10071-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/29/2020] [Revised: 08/31/2021] [Accepted: 09/11/2021] [Indexed: 06/13/2023]
Abstract
Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner's perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners' perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners' level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.
Collapse
Affiliation(s)
- Stephanie Long
- Department of Family Medicine, McGill University, Montreal, QC, Canada
| | - Charo Rodriguez
- Department of Family Medicine, McGill University, Montreal, QC, Canada
| | - Christina St-Onge
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, QC, Canada
| | | | - Nazi Torabi
- Science Collections, University of Toronto Libraries, Toronto, ON, Canada
| | - Meredith Young
- Institute of Health Sciences Education, McGill University, 1110 Pine Ave West, Montreal, QC, H3A 1A3, Canada.
- Department of Medicine, McGill University, Montreal, QC, Canada.
| |
Collapse
|
8
|
Burm S, Sebok-Syer SS, Van Koughnett JA, Watling CJ. Are we generating more assessments without added value? Surgical trainees' perceptions of and receptiveness to cross-specialty assessment. Perspect Med Educ 2020; 9:201-209. [PMID: 32504448 PMCID: PMC7459015 DOI: 10.1007/s40037-020-00594-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Competency-based medical education (CBME) hinges on robust assessment. However, integrating regular workplace-based assessment within demanding and sometimes chaotic clinical environments remains challenging. Many faculty lack assessment expertise, and some programs lack the infrastructure and faculty numbers to fulfill CBME's mandate. Recognizing this, we designed and implemented an assessment innovation that trains and deploys a cadre of faculty to assess in specialties outside their own. Specifically, we explored trainees' perceptions of and receptiveness to this novel assessment approach. METHODS Within Western University's Surgical Foundations program, 27 PGY‑1 trainees were formatively assessed by trained non-surgeons on a basic laparoscopic surgical skill. These assessments did not impact trainees' progression. Four focus groups were conducted to gauge residents' sentiments about the experience of cross-specialty assessment. Data were then analyzed using a thematic analysis approach. RESULTS While a few trainees found the experience motivating, more often trainees questioned the feedback they received and the practicality of this assessment approach to advance their procedural skill acquisition. What trainees wanted were strategies for improvement, not merely an assessment of performance. DISCUSSION Trainees' trepidation at the idea of using outside assessors to meet increased assessment demands appeared grounded in their expectations for assessment. What trainees appeared to desire was a coach-someone who could break their performance into its critical individual components-as opposed to an assessor whose role was limited to scoring their performance. Understanding trainees' receptivity to new assessment approaches is crucial; otherwise training programs run the risk of generating more assessments without added value.
Collapse
Affiliation(s)
- Sarah Burm
- Division of Medical Education, Dalhousie University, Halifax, Nova Scotia, Canada.
| | | | - Julie Ann Van Koughnett
- Divisions of General Surgery and Surgical Oncology, Western University, London, Ontario, Canada
| | - Christopher J Watling
- Departments of Oncology and Clinical Neurological Sciences, Centre for Education Research and Innovation, Western University, London, Ontario, Canada
| |
Collapse
|