Evaluating the Reliability and Validity Evidence of the RIME (Reporter-Interpreter-Manager-Educator) Framework for Summative Assessments Across Clerkships

Application of the RIME framework in genetic counseling fieldwork training to assess practice-based competencies

Using educational frameworks for learner assessment in genetic counseling (GC) training may help students and supervisors articulate developmentally appropriate clinical skills-based objectives and tasks that align with various stages of training as students work toward achieving entry-level competency. This professional issues case study describes how two GC programs adapted and implemented the RIME (Reporter-Interpreter-Manager-Educator) learner assessment framework, originally designed for medical education, to support and assess students' acquisition of practice-based competencies (PBCs) during clinical fieldwork placements. Each RIME level describes a different set of expectations regarding the skills students should be able to demonstrate based on the level of training they have achieved up to that point in time. In early training, students work mainly on gathering and reporting clinical information (Reporter level). In early to mid-training, students have learned what information to collect from clients and begin to apply the information to generate differential diagnoses (Interpreter level). When students reach the Manager level (typically by mid- to late-training), they can independently develop and implement case management plans tailored to individual cases. The Educator level, which may not be fully attained until after graduation, involves critically evaluating evidence and educating others about new evidence. The following paper describes our experiences incorporating the RIME framework into two GC graduate programs and explains the development of corresponding RIME-based assessment forms that align with the Accreditation Council of Genetic Counseling's 2023 PBCs. Overall, we find that using the RIME framework fosters a growth mindset by enabling students and supervisors to create developmentally appropriate goals and expectations, thereby facilitating assessment and guidance of trainee progress. Despite these perceived benefits, we acknowledge the need for research to evaluate the efficacy of the RIME framework or other learner assessment models in supporting student progression in achieving the GC PBCs.

Clerkship Students' Use of Clinical Reasoning Concepts After a Pre-clinical Reasoning Course

BACKGROUND

Many medical schools have incorporated clinical reasoning (CR) courses into their pre-clinical curricula to address the quality and safety issue of diagnostic error. It is unknown how students use concepts and practices from pre-clinical CR courses once in clerkships.

OBJECTIVE

We sought to understand how students utilize CR concepts from a pre-clinical course during clerkships and to identify facilitators and barriers to the use of reasoning concepts.

DESIGN

We used structured interviews to gain insight into medical students' experiences with CR concepts in clerkships.

PARTICIPANTS

We interviewed 16 students who had completed a pre-clinical CR course and subsequently completed a neurology, internal medicine, or pediatrics clerkship.

APPROACH

We used constructivist grounded theory to perform a qualitative analysis and to develop a theoretical model to describe findings.

KEY RESULTS

Insights fell into three main areas: (1) CR concept carryover, representing concepts taught in the CR course, such as problem representation, illness scripts, schema, and prioritized differential diagnosis, which were utilized in clerkships; (2) CR concept reinforcers, which included the clerkship setting and supervising physicians who emphasized and provided feedback on CR; and (3) CR concept diminishers, which included time constraints and supervisors who were unfamiliar with or did not reinforce CR concepts.

CONCLUSIONS

Concepts taught in a pre-clinical CR course influenced how students prepared for and navigated clinical encounters. Contextual factors both enhanced and inhibited the utilization of CR concepts. Our findings align with social learning theories including social cognitive theory and ecological psychology. This contextual view-taking into account interactions between personal, social, and environmental factors-can help educators integrate CR education from the classroom to the clinical setting.

R.I.M.E. and reason: multi-station OSCE enhancement to neutralize grade inflation

To offset grade inflation, many clerkships combine faculty evaluations with objective assessments including the Medical Examiners Subject Examination (NBME-SE) or Objective Structured Clinical Examination (OSCE), however, standardized methods are not established. Following a curriculum transition removing faculty clinical evaluations from summative grading, final clerkship designations of fail (F), pass (P), and pass-with-distinction (PD) were determined by combined NBME-SE and OSCE performance, with overall PD for the clerkship requiring meeting this threshold in both. At the time, 90% of students achieved PD on the Internal Medicine (IM) OSCE resulting in overall clerkship grades primarily determined by the NBME-SE. The clerkship sought to enhance the OSCE to provide a more thorough objective clinical skills assessment, offset grade inflation, and reduce the NBME-SE primary determination of the final clerkship grade. The single-station 43-point OSCE was enhanced to a three-station 75-point OSCE using the Reporter-Interpreter-Manager-Educator (RIME) framework to align patient encounters with targeted assessments of progressive skills and competencies related to the clerkship rotation. Student performances were evaluated pre- and post-OSCE enhancement. Student surveys provided feedback about the clinical realism of the OSCE and the difficulty. Pre-intervention OSCE scores were more tightly clustered (SD = 5.65%) around a high average performance with scores being highly negatively skewed. Post-intervention OSCE scores were more dispersed (SD = 6.88%) around a lower average with scores being far less skewed resulting in an approximately normal distribution. This lowered the total number of students achieving PD on the OSCE and PD in the clerkship, thus reducing the relative weight of the NMBE-SE in the overall clerkship grade. Student response was positive, indicating the examination was fair and reflective of their clinical experiences. Through structured development, OSCE assessment can provide a realistic and objective measurement of clinical performance as part of the summative evaluation of students.

The 1-minute preceptor to improve diagnostic reasoning in a primary care nurse practitioner residency program

BACKGROUND

The One-Minute Preceptor (OMP) model to teach diagnostic reasoning and Reporter, Interpreter, Manager, and Educator (RIME) framework to measure progress are used in physician training. Little is known about the use of these tools in nurse practitioner (NP) training.

LOCAL PROBLEM

Precepting NP trainees at the Veterans Affairs (VA) is not standardized. A standardized approach to precepting NP residency trainees using the OMP model and RIME scoring was evaluated for improvement and helpfulness.

METHODS

A quality-improvement project with two Plan-Do-Study-Act (PDSA) cycles were conducted over a 12-week period. Mean RIME scores, preceptor self-efficacy, and use of teaching skills were measured preintervention and postintervention. Data were analyzed using a paired sample t -test and descriptive statistics.

INTERVENTIONS

A convenience sample of preceptors and trainees was recruited from a large VA medical center. A 1-hour workshop educated preceptors with role playing and return demonstrations on OMP techniques and RIME scoring. The teachings were applied to standardize precepting and assess diagnostic reasoning. Trainee self-scoring and results triggered conversations to fulfil the identified gaps.

RESULTS

Mean RIME scores improved (1.62 [0.17] vs. 2.23 [0.38], p < .001) post 12-week intervention. Mean RIME scores improved between PDSA cycle 1 and cycle 2 (2.07 [0.25] vs. 2.48 [0.39], p < .001). Preceptors (91%) and trainees (100%) found the OMP model and RIME framework helpful.

CONCLUSION

Use of the OMP improved diagnostic reasoning in NP trainees. The OMP and RIME framework provided standardization of precepting and trainee discussions on improvements.

Large Language Model-Based Neurosurgical Evaluation Matrix: A Novel Scoring Criteria to Assess the Efficacy of ChatGPT as an Educational Tool for Neurosurgery Board Preparation

INTRODUCTION

Technological advancements are reshaping medical education, with digital tools becoming essential in all levels of training. Amidst this transformation, the study explores the potential of ChatGPT, an artificial intelligence model by OpenAI, in enhancing neurosurgical board education. The focus extends beyond technology adoption to its effective utilization, with ChatGPT's proficiency evaluated against practice questions from the Primary Neurosurgery Written Board Exam.

METHODS

Using the Congress of Neurologic Surgeons (CNS) Self-Assessment Neurosurgery (SANS) Exam Board Review Prep questions, we conducted 3 rounds of analysis with ChatGPT. We developed a novel ChatGPT Neurosurgical Evaluation Matrix (CNEM) to assess the output quality, accuracy, concordance, and clarity of ChatGPT's answers.

RESULTS

ChatGPT achieved spot-on accuracy for 66.7% of prompted questions, 59.4% of unprompted questions, and 63.9% of unprompted questions with a leading phrase. Stratified by topic, accuracy ranged from 50.0% (Vascular) to 78.8% (Neuropathology). In comparison to SANS explanations, ChatGPT output was considered better in 19.1% of questions, equal in 51.6%, and worse in 29.3%. Concordance analysis showed that 95.5% of unprompted ChatGPT outputs and 97.4% of unprompted outputs with a leading phrase were aligned.

CONCLUSIONS

Our study evaluated the performance of ChatGPT in neurosurgical board education by assessing its accuracy, clarity, and concordance. The findings highlight the potential and challenges of integrating AI technologies like ChatGPT into medical and neurosurgical board education. Further research is needed to refine these tools and optimize their performance for enhanced medical education and patient care.

Evaluating Family Medicine Resident Narrative Comments Using the RIME Scheme

BACKGROUND

In 2013, the Accreditation Council on Graduate Medical Education (ACGME) launched the Next Accreditation System, which required explicit documentation of trainee competence in six domains. To document narrative comments, the University of North Carolina Family Medicine Residency Program developed a mobile application to document real time observations.

OBJECTIVE

The objective of this work was to assess if the Reporter, Interpreter, Manager, Expert (RIME) framework could be applied to the narrative comments in order to convey a degree of competency.

METHODS

From August to December 2020, 7 individuals analyzed narrative comments of four family medicine residents. The narrative comments were collected from July to December 2019. Each individual applied the RIME framework to the comments and the team met to discuss. Comments where 5/7 individuals agreed were not further discussed. All other comments were discussed until consensus was achieved.

RESULTS

102 unique comments were assessed. Of those comments, 25 (25.5%) met threshold for assessor agreement after independent review. Group discussion about discrepancies led to consensus about the appropriate classification for 92 (90.2%). General comments on performance were difficult to fit into the RIME framework.

CONCLUSIONS

Application of the RIME framework to narrative comments may add insight into trainee progress. Further faculty development is needed to ensure comments have discrete elements needed to apply the RIME framework and contribute to overall evaluation of competence.