Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

OBJECTIVE

To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.

METHODS

We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.

RESULTS

Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.

CONCLUSION

Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments.

LEVEL OF EVIDENCE

N/A Laryngoscope, 2024.

Using the Psychiatry Resident-In-Training Examination (PRITE) to Assess the Psychiatry Medical Knowledge Milestones in Psychiatry

OBJECTIVE

The introduction of the Milestone Project underscored the need for objective assessments of resident progress across the competencies. Therefore, the authors examined the Psychiatry Resident-In-Training Examination (PRITE) utility for measuring improvements in medical knowledge (MK).

METHODS

The authors compared the mean performance for each MK subcompetency by resident year for all residents taking the PRITE from 2015 to 2017 (18,175 examination administrations). In addition, they surveyed psychiatry residency program directors regarding how well they thought they teach these subcompetencies.

RESULTS

Increases in MK subcompetencies by resident year were significant for Psychopathology (p < 0.003), Psychotherapy (p < 0.002), and Somatic Therapies (p < 0.000). Development, Clinical Neuroscience, and Practice of Psychiatry did not show statistically significant differences between postgraduate years. Eighty psychiatry program directors responded to the survey and felt optimistic about their ability to teach the Psychopathology, Psychotherapy, Somatic Therapies, and Practice of Psychiatry subcompetencies.

CONCLUSIONS

The PRITE measured significant improvements in medical knowledge for several of the core subcompetencies. The program director's responses would suggest that the lack of statistically significant differences found for Development and Clinical Neuroscience reflects areas in need of curricular development. The disparity between PRITE performance and program director perception of the Practice of Psychiatry subcompetency may reflect difficulties in defining the scope of this subcompetency. Overall, this suggests that structured examinations help measure improvements in certain subcompetencies and may also help identify curricular needs. However, there may be potential problems with the definition of some subcompetencies.

The feasibility of simulation-based high-stakes assessment in emergency medicine settings: A scoping review

Using simulation in high-stakes assessments has been evolving as a method to improve the assessment process. There is a concurrent need to address challenges and establish best practices to ensure the best quality when implementing high-stakes evaluations. The aim of this study is to provide an insight for stakeholders about using multiple modalities of simulation in high-stakes evaluations by presenting challenges, best practices, and future directions described in the relevant literature. A scoping review of original studies (from the year 1994-2021) including the use of common modalities (standardized patients, high-fidelity mannequins, part-task trainers, virtual simulation, and hybrid simulation) was conducted. The search covered the common databases: PubMed, Education Resource Information Center, Cumulative Index to Nursing and Allied Health Literature, and the Cochrane library. Initial screening in the databases resulted in 111,253 articles. After the application of a refining search strategy, 47 articles were included for comprehensive evaluation. Most articles were about credentialing/licensure purposes of assessment. Many articles were specialty-specific, especially focusing on anesthesia. Most challenges described were related to the validity of the assessment that should be considered in the planning phase. Best practices were mostly related to planning for measures to assure the validity of the assessment tools and process. The use of multimodality simulation for high-stakes assessment is growing despite challenges; this growth is associated with the accumulation of experience shared in literature. This growth will help to improve planning, practices, and goals achievement from such an application.

Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review

The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.

Experience Is the Teacher of All Things: Prior Participation in Anesthesiology OSCEs Enhances Communication of Treatment Options With Simulated High-Risk Patients

BACKGROUND

The objective of this study was to investigate whether previous experiences within an anesthesiology-based Objective Structured Clinical Examination (OSCE) assessing communication and professionalism skills was associated with improved performance in a subsequent anesthesiology-based OSCE scenario.

METHODS

This retrospective multi-center study used the performance data of 44 Post Graduate Year 4 clinical anesthesia residents from 3 US anesthesiology residency programs on an OSCE scenario that assessed the residents' effectiveness of discussing anesthesiology-specific treatment options with a high-risk patient. Residents from 2 of the programs had no prior anesthesiology-based OSCE experience. Residents from the third program had previously participated in 4 separate multi-scenario anesthesiology-based OSCE sessions in the 2 years prior to this study. Participating residents completed the same scenario at their respective institutions' simulation center. Ten performances were randomly selected for double rating to assess the interrater reliability of the assessments. Interrater reliability was good for the scenario (intraclass correlation coefficient = 0.66, 95% confidence interval = 0.12-0.90). Performance difference between groups with different OSCE experience status were examined using an independent sample t test, with a Wilcoxon-Mann-Whitney test as a sensitivity analysis.

RESULTS

Independent sample t test found prior OSCE experience was significantly associated with higher performance scores (t = 2.53, P = .02). The Wilcoxon-Mann-Whitney test result confirmed this finding (z = 3.28, P = .001).

CONCLUSIONS

Findings from this study provide preliminary evidence that anesthesiology-based OSCE experience is associated with improved performance in an OSCE scenario, specifically regarding discussions of treatment options with high-risk patients.