The effect of simulation-based mastery learning on thoracentesis referral patterns

Using Simulation-Based Mastery Learning to Achieve Excellent Learning Outcomes in Physical Therapist Education

INTRODUCTION

The 2 aims of this observational study are (a) to describe the implementation and feasibility of a bed mobility skills simulation-based mastery learning (SBML) curricular module for physical therapist students and (b) to measure learning outcomes and student perceptions of this module.

REVIEW OF LITERATURE

Simulation-based mastery learning is an outcome-based educational approach that has been successful in other health professions but has not been explored in physical therapy education.

SUBJECTS

Eighty-seven students in a single cohort of a Doctor of Physical Therapy program.

METHODS

The SBML module in this pretest-posttest study included a pretest, instruction, initial posttest, and additional rounds of instruction and assessment as needed for all learners to achieve the minimum passing standard (MPS) set using the Mastery Angoff and Patient Safety methods. Outcome measures were bed mobility assessment pass rates and scores, additional student and faculty time compared with a traditional approach, and student perceptions of their self-confidence and the module.

RESULTS

All students achieved the MPS after 3 rounds of training and assessment beyond the initial posttest. Mean Total Scores improved from 67.6% (12.9%) at pretest to 91.4% (4.8%) at mastery posttest (P < .001, Cohen's d = 1.8, 95% CI [1.4-2.1]); mean Safety Scores improved from 75.2% (16.0%) at pretest to 100.0% (0.0%) at mastery posttest (P < .001, Cohen's d = 1.5, 95% CI [1.2-1.9]). Students who did not achieve the MPS at the initial posttest (n = 30) required a mean of 1.2 hours for additional instruction and assessment. Survey results revealed an increase in student confidence (P < .001) and positive student perceptions of the module.

DISCUSSION AND CONCLUSION

Implementation of this SBML module was feasible and resulted in uniformly high levels of bed mobility skill acquisition. Based on rigorous learning outcomes, feasible requirements for implementation, and increased student confidence, SBML offers a promising approach for wider implementation in physical therapy education.

Beyond the Procedure Log: Using Individualized Learning Plans to Set Learner-Specific Milestones for Procedural Skills Acquisition

ABSTRACT

Procedural training for nonsurgical fields, such as internal medicine, is an important component of medical education. However, recent changes to accreditation guidelines have resulted in less formal guidance on procedural competency, not only leading to opportunities for individualizing training but also creating potential problems for trainees and training programs. In this article, the authors use internal medicine as an exemplar to review current strategies for procedural education in nonsurgical fields, including procedural simulation, dedicated procedural rotations, and advanced subspecialty training, and highlight an emerging need for learner-specific terminal milestones in procedural training. Individualized learning plans (ILPs), collections of trainee-specific objectives for learning, are arguably a useful strategy for organizing procedural training. The role of ILPs as a framework to support setting learner-specific terminal milestones, guide skill acquisition, and allocate procedural learning opportunities based on trainees' anticipated career plans is subsequently explored, and how an ILP-based approach might be implemented within the complex educational milieu of a clinical training program is examined. The limitations and pitfalls of an ILP-based approach, including the need for development of coaching programs, are considered. The authors conclude that, despite the limitations of ILPs, when combined with other current strategies for building trainees' procedural competence, these plans may help trainees maximize the educational benefits of their training period and can encourage effective, safer, and equitable allocation of procedural practice opportunities.

Education in Focused Assessment With Sonography for Trauma Using Immersive Virtual Reality: A Prospective, Interventional Cohort Study and Non-inferiority Analysis With a Historical Control

OBJECTIVE

Focused assessment with sonography for trauma (FAST) is a valuable ultrasound procedure in emergency settings, and there is a need for evidence-based education in FAST to ensure competencies. Immersive virtual reality (IVR) is a progressive training modality gaining traction in the field of ultrasound training. IVR holds several economic and practical advantages to the common instructor-led FAST courses using screen-based simulation (SBS).

METHODS

This prospective, interventional cohort study investigated whether training FAST using IVR unsupervised and out-of-hospital was non-inferior to a historical control group training at a 90 min SBS course in terms of developing FAST competencies in novices. Competencies were assessed in both groups using the same post-training simulation-based FAST test with validity evidence, and a non-inferiority margin of 2 points was chosen.

RESULTS

A total of 27 medical students attended the IVR course, and 27 junior doctors attended the SBS course. The IVR group trained for a median time of 117 min and scored a mean 14.2 ± 2.0 points, compared with a mean 13.7 ± 2.5 points in the SBS group. As the lower bound of the 95% confidence interval at 13.6 was within the range of the non-inferiority margin (11.7-13.7 points), training FAST in IVR for a median of 117 min was found non-inferior to training at a 90 min SBS course. No significant correlation was found between time spent in IVR and test scores.

CONCLUSION

Within the limitations of the use of a historical control group, the results suggest that IVR could be an alternative to SBS FAST training and suitable for unsupervised, out-of-hospital courses in basic FAST competencies.

Anticipation or avoidance: internal medicine resident experiences performing invasive bedside procedures

Background

Internal Medicine (IM) residents are required to perform bedside procedures for diagnostic and therapeutic purposes. Residents' experiences with procedures vary widely, for unclear reasons.

Objective

To explore IM residents' experiences with performing bedside procedures and to identify barriers and facilitators to obtaining sufficient experience.

Methods

Using an inductive, thematic approach, we conducted five individual semi-structured interviews and one focus group with seven IM residents (12 residents in total) during the 2017-2018 academic year at a Canadian tertiary care centre. We used iterative, open-ended questions to elicit residents' experiences, and barriers and facilitators, to performing bedside procedures. Transcripts were analyzed for themes using Braun and Clarke's method.

Results

We identified four themes 1) Patient-specific factors such as body habitus and procedure urgency; 2) Systems factors such as time constraints and accessibility of materials; 3) Faculty factors including availability to supervise, comfort level, and referral preferences, and 4) Resident-specific factors including preparation, prior experiences, and confidence. Some residents expressed procedure-related anxiety and avoidance.

Conclusion

Educational interventions aimed to improve procedural efficiency and ensure availability of supervisors may help facilitate residents to perform procedures, yet may not address procedure-related anxiety. Further study is required to understand better how procedure-averse residents can gain confidence to seek out procedures.

Simulation-based Mastery Learning Improves Critical Care Skills of Advanced Practice Providers

Background

Advanced practice providers (APPs) are essential members of intensive care unit (ICU) interprofessional teams and are expected to be competent in performing procedures. There are no published criteria for establishing when APPs can independently perform procedures. Simulation-based mastery learning (SBML) is an effective strategy for improving critical care skills but has not been applied to practicing ICU APPs.

Objective

The purpose of this study was to evaluate if an SBML curriculum could improve the critical care skills and procedural self-confidence of ICU APPs.

Methods

We performed a pretest-posttest study of central venous catheter (CVC) insertion, thoracentesis, and mechanical ventilation (MV) management skills among ICU APPs who participated in an SBML course at an academic hospital. For each skill, APPs underwent baseline skills assessments (pretests) on a simulator using previously published checklists, followed by didactic sessions and deliberate practice with individualized feedback. Within 2 weeks, participants were required to meet or exceed previously established minimum passing standards (MPS) on simulated skills assessments (posttests) using the same checklists. Further deliberate practice was provided for those unable to meet the MPS until they retested and met this standard. We compared pretest to posttest skills checklist scores and procedural confidence.

Results

All 12 eligible ICU APPs participated in internal jugular CVC, subclavian CVC, and MV training. Five APPs participated in thoracentesis training. At baseline, no APPs met the MPS on all skills. At training completion, all APPs achieved the mastery standard. Internal jugular CVC pretest performance improved from a mean of 67.2% (standard deviation [SD], 28.8%) items correct to 97.1% (SD, 3.8%) at posttest (P = 0.005). Subclavian CVC pretest performance improved from 29.2% (SD, 32.7%) items correct to 93.1% (SD 3.9%) at posttest (P < 0.001). Thoracentesis pretest skill improved from 63.9% (SD, 30.6%) items correct to 99.2% (SD, 1.7%) at posttest (P = 0.054). Pretest MV skills improved from 54.8% (SD, 19.7%) items correct to 92.3% (SD, 5.0%) at posttest (P < 0.001). APP procedural confidence improved for each skill from pre to posttest.

Conclusion

SBML is effective for training APPs to perform ICU skills. Relying on traditional educational methods does not reliably ensure that APPs are adequately prepared to perform skills such as CVC insertion, thoracentesis, and MV management.

Simulation-based Education Improves Military Trainees' Skill Performance and Self- Confidence in Tourniquet Placement: A Randomized Controlled Trial

BACKGROUND

Tactical Combat Casualty Care (TCCC) is the standard of care for stabilization and treatment of military trauma patients. The DoD has mandated that all Service members receive role-based TCCC training and certification. Simulation education can increase procedural skills by providing opportunities for deliberate practice in safe, controlled environments. We developed and evaluated the effectiveness of a simulation-based TCCC training intervention to improve participants' skill performance and self-confidence in tourniquet placement.

METHODS

This study was a single-blinded, randomized trial with waitlist controls. Army ROTC cadets from a single training battalion comprised the study population. After randomization and baseline assessment of all participants, Group A alone received focused, simulation-based TCCC tourniquet application training. Three months later, all participants underwent repeat testing, and after crossover, the waitlist Group B received the same intervention. Two months later, all cadets underwent a third/final assessment. The primary outcome was tourniquet placement proficiency assessed by total score achieved on a standardized 8-item skill checklist. A secondary outcome was self-confidence in tourniquet application skill as judged by participants' Likert scale ratings.

RESULTS

Forty-three Army ROTC cadets completed the study protocol. Participants in both Group A (n = 25) and Group B (n = 18) demonstrated significantly higher performance from baseline to final assessment at five months and two months, respectively, following the intervention. Mean total checklist score of the entire study cohort increased significantly from 5.53 (SD = 2.00) at baseline to 7.56 (SD = 1.08) at Time 3, a gain of 36.7% (p < 0.001). Both groups rated their self-confidence in tourniquet placement significantly higher following the training.

CONCLUSIONS

A simulation-based TCCC curriculum resulted in significant, consistent, and sustained improvement in participants' skill proficiency and self-confidence in tourniquet placement. Participants maintained these gains two to five months after initial training.

LEVEL OF EVIDENCE

Level II - RCT with significant difference and only one negative criterion (<80% follow-up).

Impact of Simulation-based Mastery Learning on Resident Skill Managing Mechanical Ventilators

Background: Caring for patients requiring mechanical ventilation is complex, and residents may lack adequate skill for managing these patients. Simulation-based mastery learning (SBML) is an educational model that trains clinicians to a high standard and can reduce complications. The mastery learning model has not been applied to ventilator management. Objective: The purpose of this study was to determine whether SBML, as compared with traditional training, is an effective strategy for teaching residents the skills necessary to manage patients requiring mechanical ventilation. Methods: We developed an SBML curriculum and a 47-item skills checklist to test ventilator management for patients with normal, restricted, and obstructed lung physiology. A minimum passing standard (MPS) on the checklist was set using the Mastery Angoff method. Residents rotating through the medical intensive care unit in Academic Year 2017-2018 were assigned to SBML or traditional training based on their medical intensive care unit team. The SBML group was pretested on a ventilator simulator using the skills checklist. They then received a 1.5-hour session (45 min didactic and 45 min deliberate practice on the simulator with feedback). At rotation completion, they were posttested on the simulator using the checklist until the MPS was met. Both SBML-trained and traditionally trained groups received teaching during daily bedside rounds and twice weekly didactic lectures. At rotation completion, traditionally trained residents were tested using the same skills checklist on the simulator. We compared pretest and posttest performance among SBML-trained residents and end of the rotation test performances between the SBML-trained and traditionally trained residents. Results: The MPS was set at 87% on the checklist. Fifty-seven residents were assigned to the SBML-trained group and 49 were assigned to the traditionally trained group. Mean checklist scores for SBML-trained residents improved from 51.4% (standard deviation [SD] = 17.5%) at pretest to 86.1% (SD = 7.6%) at initial posttest and 92.5% (SD = 3.7%) at final (mastery) posttest (both P < 0.001). Forty-two percent of residents required more than one attempt at the posttest to meet or exceed the MPS. At rotation completion, the traditionally trained residents had a mean test score of 60.9% (SD = 13.3%). Conclusion: SBML is an effective strategy to train residents on mechanical ventilator management. An SBML curriculum may augment traditional training methods to further equip residents to safely manage ventilated patients.

Use of a simulation-based mastery learning curriculum for neurology residents to improve the identification and management of status epilepticus

BACKGROUND

Appropriate and timely treatment of status epilepticus (SE) decreases morbidity and mortality. Therefore, skill-based training in the identification and management of SE is crucial for clinicians.

OBJECTIVE

The objective of the study was to develop and evaluate the impact of a simulation-based mastery learning (SBML) curriculum to train neurology residents on the identification and management of SE.

METHODS

We used pretest-posttest design with a retention test on SE skills for this study. Neurology residents in the second postgraduate year (PGY-2) were eligible to participate in the SE SBML curriculum. Learners completed a baseline-simulated SE skills assessment (pretest) using a 26-item dichotomous skills checklist. Next, they participated in a didactic session about the identification and management of SE, followed by deliberate skills practice. Subsequently, participants completed another skills assessment (posttest) using the same 26-item checklist. All participants were required to meet or exceed a minimum passing standard (MPS) determined by a panel of 14 SE experts using the Mastery Angoff standard setting method. After meeting the MPS at posttest, participants were reassessed during an unannounced in situ simulation session on the medical wards. We compared pretest with posttest simulated SE skills performance and posttest with reassessment in situ performance.

RESULTS

The MPS was set at 88% (23/26) checklist items correct. Sixteen neurology residents participated in the intervention. Participant performance improved from a median of 44.23% (Interquartile range (IQR): 34.62-55.77) at pretest to 94.23% (IQR: 92.13-100) at the posttest after SBML (p < .001). There was no significant difference in scores between the posttest and in situ test up to 8 months later (94.23%; IQR: 92.31-100 vs. 92.31%; IQR: 88.46-96.15; p = .13).

CONCLUSIONS

Our SBML curriculum significantly improved residents' SE identification and management skills that were largely retained during an unannounced simulated encounter in the hospital setting.

Development of the Uncertainty Communication Checklist: A Patient-Centered Approach to Patient Discharge From the Emergency Department

Clear communication with patients upon emergency department (ED) discharge is important for patient safety during the transition to outpatient care. Over one-third of patients are discharged from the ED with diagnostic uncertainty, yet there is no established approach for effective discharge communication in this scenario. From 2017 to 2019, the authors developed the Uncertainty Communication Checklist for use in simulation-based training and assessment of emergency physician communication skills when discharging patients with diagnostic uncertainty. This development process followed the established 12-step Checklist Development Checklist framework and integrated patient feedback into 6 of the 12 steps. Patient input was included as it has potential to improve patient-centeredness of checklists related to assessment of clinical performance. Focus group patient participants from 2 clinical sites were included: Thomas Jefferson University Hospital, Philadelphia, PA, and Northwestern University Hospital, Chicago, Illinois.The authors developed a preliminary instrument based on existing checklists, clinical experience, literature review, and input from an expert panel comprising health care professionals and patient advocates. They then refined the instrument based on feedback from 2 waves of patient focus groups, resulting in a final 21-item checklist. The checklist items assess if uncertainty was addressed in each step of the discharge communication, including the following major categories: introduction, test results/ED summary, no/uncertain diagnosis, next steps/follow-up, home care, reasons to return, and general communication skills. Patient input influenced both what items were included and the wording of items in the final checklist. This patient-centered, systematic approach to checklist development is built upon the rigor of the Checklist Development Checklist and provides an illustration of how to integrate patient feedback into the design of assessment tools when appropriate.

Disparities over Time in Volume, Day of the Week, and Patient Complexity between Paracentesis and Thoracentesis Procedures Performed by Radiologists versus Those Performed by Nonradiologists

PURPOSE

To compare the disparities between the paracenteses and thoracenteses performed by radiologists with those performed by nonradiologists over time. Variables included the volume of procedures, the days of the week, and the complexity of the patient's condition.

MATERIALS AND METHODS

Using carrier claims files for a 5% national sample of Medicare beneficiaries from 2004 to 2016, paracentesis and thoracentesis examinations were retrospectively classified by physician specialty (radiologist vs nonradiologist), day of the week (weekday vs weekend), and the complexity of the patient's condition (using Charlson comorbidity index scores). The Pearson chi-square and independent samples t-test were used for statistical analysis.

RESULTS

Between 2004 and 2016, the proportion of all paracentesis and thoracentesis procedures performed by radiologists increased from 70% to 80% and from 47% to 66%, respectively. Although radiologists increasingly performed more of both services on both weekends and weekdays, the share performed by radiologists was lower on weekends. For most of the first 9 years across the study period, radiologists performed paracentesis in patients with more complex conditions than those treated by nonradiologists, but the complexity of patients' conditions was similar during recent years. For thoracentesis, the complexity of patients' conditions was similar for both specialty groups across the study period.

CONCLUSIONS

The proportion of paracentesis and thoracentesis procedures performed in Medicare beneficiaries by radiologists continues to increase, with radiologists increasingly performing most of both services on weekends. Nonetheless, radiologists perform disproportionately more on weekdays than on weekends. Presently, radiologists and nonradiologists perform paracentesis and thoracentesis procedures in patients with similarly complex conditions. These interspecialty differences in timing and complexity of the patient's condition differ from those recently described for several diagnostic imaging services, reflecting the unique clinical and referral patterns for invasive versus diagnostic imaging services.

Simulated training program in abdominal paracentesis for undergraduate medical students

BACKGROUND AND AIMS

Abdominal paracentesis is an area that every general physician should know about, and the current learning model is unsafe for patients. Simulation allows students to develop their skills prior to clinical confrontation with minimal risks. The aims of this study were to design and evaluate a paracentesis simulation workshop for undergraduate students.

METHODS

A workshop was implemented using a specially designed and validated simulation model for abdominal paracentesis. The simulated technique considered the recognition of materials, operator equipment, asepsis, anesthesia, puncture and obtaining liquid, collecting samples for analysis, withdrawal of the material and occlusion. A 24-point direct observation checklist was administered to assess the student. We assessed two students at the beginning of the workshop and all the students at the end. A perception survey was applied to attendees at the end of the workshop.

RESULTS

247 students were included and a workshop that involved 8 students per session was held. Students significantly improved their skills comparing pre- and post-evaluation results [13.36±4.46 (55.7%) vs. 22.3±1.83 (92.9%) respectively (n=69) p<0.001]. The students' perception questionnaire (n=38) showed that the training sessions were highly valued, averaging 4.8±0.38 on a Likert scale of 1-5.

CONCLUSIONS

Simulated training in abdominal paracentesis is a very good teaching method. This teaching methodology should be highly recommended as an educational strategy in medicine because it could accelerate the acquisition of clinical skills in a safe learning environment.

Development and Utilization of 3D Printed Material for Thoracotomy Simulation

Medical simulation is a widely used training modality that is particularly useful for procedures that are technically difficult or rare. The use of simulations for educational purposes has increased dramatically over the years, with most emergency medicine (EM) programs primarily using mannequin-based simulations to teach medical students and residents. As an alternative to using mannequin, we built a 3D printed models for practicing invasive procedures. Repeated simulations may help further increase comfort levels in performing an emergency department (ED) thoracotomy in particular, and perhaps this can be extrapolated to all invasive procedures. Using this model, a simulation training conducted with EM residents at an inner city teaching hospital showed improved confidence. A total of 21 residents participated in each of the three surveys [(1) initially, (2) after watching the educational video, and (3) after participating in the simulation]. Their comfort levels increased from baseline after watching the educational video (9.5%). The comfort level further improved from baseline after performing the hands on simulation (71.4%).

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Simulation-Based Mastery Learning for Thoracentesis Skills Improves Patient Outcomes: A Randomized Trial

PURPOSE

Physicians-in-training often perform bedside thoracenteses in academic medical centers, and complications are more common among less experienced clinicians. Simulation-based mastery learning (SBML) is one potential solution to this problem. This study evaluated the effects of a randomized trial of thoracentesis SBML on patient complications: iatrogenic pneumothorax (IP), hemothorax, and reexpansion pulmonary edema (REPE).

METHOD

The authors randomized internal medicine residents to undergo thoracentesis SBML at a tertiary care academic center from December 2012 to May 2016. They subsequently compared thoracentesis complications from procedures performed by SBML-trained residents, traditionally trained residents (no simulation training), and those referred to pulmonary medicine or interventional radiology (IR).

RESULTS

During the study period, 917 thoracenteses were performed on 709 patients. IP occurred in 60 (6.5%) procedures, of which 7 (11.6%) were clinically meaningful. SBML-trained residents performed procedures with a trend toward lower combined clinically meaningful complications (IP, hemothorax, REPE) compared with traditionally trained residents (7.9% vs. 0%; P = .06). SBML-trained residents caused fewer clinically meaningful IPs compared with traditionally trained residents, pulmonary, and IR referrals (P = .02). Hemothorax occurred after 4 (0.4%) thoracenteses, and SBML-trained residents had a trend toward lower hemothorax (0) compared with other groups (P = .07). REPE occurred after 3 (0.3%) procedures, with no differences between groups. SBML-trained residents performed procedures with lower combined clinically meaningful complications compared with other groups (P = .008).

CONCLUSIONS

Residents randomized to an SBML intervention performed thoracenteses with low rates of clinically meaningful complications. Rigorous education represents a successful quality improvement strategy.

Residents' Procedural Experience Does Not Ensure Competence: A Research Synthesis

BACKGROUND

Many medical certifying bodies require that a minimum number of clinical procedures be completed during residency training to obtain board eligibility. However, little is known about the relationship between the number of procedures residents perform and their clinical competence.

OBJECTIVE

This study evaluated associations between residents' medical procedure skills measured in a simulation laboratory and self-reported procedure experience and year of training.

METHODS

This research synthesis extracted and summarized data from multiple cohorts of internal medicine, emergency medicine, anesthesiology, and neurology resident physicians who performed simulated clinical procedures. The procedures were central venous catheter insertion, lumbar puncture, paracentesis, and thoracentesis. We compared residents' baseline simulated performance to their self-reported procedure experience using data from 7 research reports written by Northwestern University investigators between 2006 and 2016. We also evaluated how performance differed by postgraduate year (PGY).

RESULTS

A total of 588 simulated procedures were performed during the study period. We found significant associations between passing the skills examinations and higher number of self-reported procedures performed (P = .011) and higher PGY (P < .001). However, performance for all procedures was poor, as only 10% of residents passed the assessments with a mean of 48% of checklist items correct (SD = 24.2). The association between passing the skills examination and year of training was mostly due to differences between PGY-1 and subsequent years of training.

CONCLUSIONS

Despite positive associations between self-reported experience and simulated procedure performance, overall performance was poor. Residents' clinical experience is not a proxy for skill.