Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

Reliability and validity of simulation-based Electrocardiogram assessment rubrics for cardiac life support skills among medical students using generalizability theory

Simulation-based learning (SBL) is effective for EKG interpretation training in the advanced cardiac life support (ACLS) context, enhancing motivation, confidence, and learning outcomes. However, research on the psychometrics of assessment rubrics for ACLS skills among pre-clinical students is limited. This study investigates the validity and reliability of assessment rubrics for ACLS skills, including EKG interpretation, scenario and pharmacological management, and teamwork. An SBL course that integrates basic EKG interpretation into ACLS Stations was conducted at Phramongkutklao College of Medicine, utilizing high-fidelity mannequins to simulate realistic scenarios, enrolling 96 medical students. The course consisted of five independent stations, and each student was assessed once by two raters using ten-item assessment rubrics. The rubrics included three domains: (1) EKG and ACLS algorithm skills, (2) management and mechanisms of action, and (3) affective domains. Validity evidence on the content was gathered, and construct validity was confirmed with confirmatory factor analysis (CFA). Inter-rater and internal consistency reliability were calculated. Generalizability theory was utilized to analyse the data. Three expert reviews yielded an item-objective congruence index of 0.67-1.00, with iterative validation through alpha and beta tests. The CFA demonstrated a good fit, but two questions with loading factors below 0.30 were removed, resulting in an eight-item assessment form. An inter-rater correlation of 0.70 (p < 0.001) and a Cronbach's alpha of 0.76 was demonstrated. To achieve a Phi-coefficient ≥0.80, three raters and at least 10 items are required in a p×i×r crossed design. With eight items, r:(p×i) nested design reliability was 0.69, 0.79, and 0.83 for one, two, and three raters, respectively. While a single rater with 10 items achieved a Phi-coefficient of 0.74. The rubrics for assessing ACLS skills among pre-clinical students demonstrated acceptable validity and reliability. A condensed eight-item rubric with acceptable reliability is proposed as a practical tool for optimizing assessment in future evaluations relevant to the pre-clinical context.

Automatic Final-Product Assessment of Virtual Reality Mastoidectomy Performance: A Validity and Reliability Study

OBJECTIVE

Assessment is key in modern surgical education to monitor progress and document sufficient skills. Virtual reality (VR) temporal bone simulators allow automated tracking of basic metrics such as time, volume removed, and collisions. However, adequate performance assessment further includes compound rating of the stepwise bony excavation, and exposure and preservation of soft tissue structures. Such complex assessment requires further development of automated assessment routines in the VR simulation environment. In this study, we present the integration of automated mastoidectomy final-product assessment with validation against manual rating.

METHODS

At two international temporal bone courses, 33 ORL trainees performed anatomical mastoidectomies in the Visible Ear (VR) Simulator with automatic performance assessment using a newly implemented rating routine based on the modified Welling Scale. Automated assessment was compared with manual ratings by experts using absolute agreement, intraclass correlation, and generalizability analysis to establish validity and reliability.

RESULTS

The overall average agreement between manual and automatic assessment was 83.9% compared with the inter-rater agreement of 88.9%. A majority of items (15 out of 26) showed high agreement between automated and manual rating (>85%). Intraclass correlation coefficients were found to be high. Generalizability analysis with D-studies found that five repetitions per participant are needed for a G coefficient >0.8, which is considered necessary for high-stakes assessments.

CONCLUSION

We have demonstrated the feasibility, validity, and reliability of an automatic assessment system integrated into a VR temporal bone simulator. This can prove to be an important tool for future self-directed training with skills certification.

3D-printed temporal bone models for training: Does material transparency matter?

PURPOSE

To investigate the impact of 3D-printed temporal bone models with two different material transparencies on trainees' mastoidectomy performance.

METHODS

Eleven ORL residents performed two anatomical mastoidectomies with posterior tympanotomy on two 3D-printed models with different transparency and VR simulation training. Participants where divided into two groups based on their experience. Within each group participants were randomized to start with the model printed in a completely opaque material or in a material featuring some degree of transparency. After drilling on 3D-printed models, the participants performed two similar mastoidectomies on human cadavers: one on the left side of one cadaver and one on the right side of another cadaver. After drilling 3D-printed models and cadavers, the final-product performances were evaluated by two experienced raters using the 26-item modified Welling Scale. Participants also evaluated the models using a questionnaire.

RESULTS

Overall, the participants performed 25 % better on the 3D-printed models featuring transparency compared to the opaque models (18.6 points vs 14.9 points, mean difference = 3.7, 95 % CI 2.0-5.3, P < 0.001)). This difference in performance was independent of which material the participants had drilled first. In addition, the residents also subjectively rated the transparent model to be closer to cadaver dissection. The experienced group starting with the 3D-printed models scored 21.5 points (95 % CI 20.0-23.1), while the group starting with VR simulation training score 18.4 points (95 % CI 16.6-20.3).

CONCLUSION

We propose that material used for 3D-printing temporal bone models should feature some degree of transparency, like natural bone, for trainees to learn and exploit key visual cues during drilling.

Exploring the measurement of psychological resilience in Chinese civil aviation pilots based on generalizability theory and item response theory

Understanding and accurately measuring resilience among Chinese civil aviation pilots is imperative, especially concerning the psychological impact of distressing events on their well-being and aviation safety. Despite the necessity, a validated and tailored measurement tool specific to this demographic is absent. Addressing this gap, this study built on the widely used CD-RISC-25 to analyze and modify its applicability to Chinese civil aviation pilots. Utilizing CD-RISC-25 survey data from 231 Chinese pilots, correlational and differential analyses identified items 3 and 20 as incongruent with this population's resilience profile. Subsequently, factor analysis derived a distinct two-factor resilience psychological framework labeled "Decisiveness" and "Adaptability", which diverged from the structure found in American female pilots and the broader Chinese populace. Additionally, to further accurately identify the measurement characteristics of this 2-factor measurement model, this study introduced Generalized Theory and Item Response Theory, two modern measurement analysis theories, to comprehensively analyze the overall reliability of the measurement and issues with individual items. Results showed that the 2-factor model exhibited high reliability, with generalizability coefficient reaching 0.89503 and dependability coefficient reaching 0.88496, indicating the 2-factor measurement questionnaire can be effectively utilized for relative and absolute comparison of Chinese civil aviation pilot resilience. However, items in Factor 2 provided less information and have larger room for optimization than those in Factor 1, implying item option redesign may be beneficial. Consequently, this study culminates in the creation of a more accurate and reliable two-factor psychological resilience measurement tool tailored for Chinese civil aviation pilots, while exploring directions for optimization. By facilitating early identification of individuals with lower resilience and enabling the evaluation of intervention efficacy, this tool aims to positively impact pilot psychological health and aviation safety in the context of grief and trauma following distressing events.

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Haptic simulators accelerate laparoscopic simulator training, but skills are not transferable to a non-haptic simulator: a randomized trial

BACKGROUND

Laparoscopy requires specific psychomotor skills and can be challenging to learn. Most proficiency-based laparoscopic training programs have used non-haptic virtual reality simulators; however, haptic simulators can provide the tactile sensations that the surgeon would experience in the operating room. The objective was to investigate the effect of adding haptic simulators to a proficiency-based laparoscopy training program.

METHODS

A randomized controlled trial was designed where residents (n = 36) were randomized to proficiency-based laparoscopic simulator training using haptic or non-haptic simulators. Subsequently, participants from the haptic group completed a follow-up test, where they had to reach proficiency again using the non-haptic simulator. Participants from the non-haptic group returned to train until reaching proficiency again using the non-haptic simulator.

RESULTS

Mean completion times during the intervention were 120 min (SD 38.7 min) and 183 min (SD 66.3 min) for the haptic group and the non-haptic group, respectively (p = 0.001). The mean times to proficiency during the follow-up test were 107 min (SD 41.0 min) and 58 min (SD 23.7 min) for the haptic and the non-haptic group, respectively (p < 0.001). The haptic group was not faster to reach proficiency in the follow-up test than during the intervention (p = 0.22). In contrast, the non-haptic group reached the required proficiency level significantly faster in the follow-up test (p < 0.001).

CONCLUSION

Haptic virtual reality simulators reduce the time to reach proficiency compared to non-haptic simulators. However, the acquired skills are not transferable to the conventional non-haptic setting.

Am I doing this right? Structured self-assessment during simulation training of mastoidectomy improves cadaver dissection performance: a prospective educational study

PURPOSE

Temporal bone surgery requires excellent surgical skills and simulation-based training can aid novices' skills acquisition. However, simulation-based training is challenged by early stagnation of performance after few performances. Structured self-assessment during practice might enhance learning by inducing reflection and engagement in the learning task. In this study, structured self-assessment was introduced during virtual reality (VR) simulation of mastoidectomy to investigate the effects on subsequent performance during cadaveric dissection.

METHODS

A prospective educational study with comparison with historical controls (reference cohort). At a temporal bone dissection course, eighteen participants performed structured self-assessment during 3 h of VR simulation mastoidectomy training before proceeding to cadaver dissection (intervention cohort). At a previous course, eighteen participants received identical VR simulation training but without the structured self-assessment (reference cohort). Final products from VR simulation and cadaveric dissection were recorded and assessed by two blinded raters using a 19-point modified Welling Scale.

RESULTS

The intervention cohort completed fewer procedures (average 4.2) during VR simulation training than the reference cohort (average 5.7). Nevertheless, the intervention cohort achieved a significantly higher average performance score both in VR simulation (11.1 points, 95% CI [10.6-11.5]) and subsequent cadaveric dissection (11.8 points, 95% CI [10.7-12.8]) compared with the reference cohort, who scored 9.1 points (95% CI [8.7-9.5]) during VR simulation and 5.8 points (95% CI [4.8-6.8]) during cadaveric dissection.

CONCLUSIONS

Structured self-assessment is a valuable learning support during self-directed VR simulation training of mastoidectomy and the positive effect on performance transfers to subsequent cadaveric dissection performance.

Using Self-Rated Examinations to Ensure Competence in Laparoscopy

INTRODUCTION

Asking participants to rate their own performance during unsupervised training in laparoscopy is reliable and may be cost-effective. The objective of the study was to explore the reliability of self-rated examinations where participants rate their own performance and decide themselves when they have passed tasks in basic laparoscopic skills.

METHODS

This prospective observational study was conducted at the Copenhagen Academy for Medical Education and Simulation where simulation-based laparoscopic skill training is offered. Here, participants taking part in a basic laparoscopic skills course were asked to rate their own performance and decide when they had passed the Training and Assessment of Basic Laparoscopic Techniques test. To explore reliability, all examinations were video recorded and rated by a blinded rater after the end of the course.

RESULTS

Thirty-two surgical trainees participated in the course, and 28 completed the study. We found a high reliability when comparing self-rated scores and blinded ratings with an intraclass correlation coefficient of 0.89 ( P < 0.001); self-rated scores compared with blinded ratings were not significantly different (mean = 451 vs. 455, P = 0.28), and the participants did not underestimate nor overestimate their performance.

CONCLUSIONS

Ratings from self-rated examinations in a basic laparoscopic skills course are reliable, and participants neither underestimate nor overestimate their performance. Self-rated examinations may also be beneficial because they also can offer a cost-effective approach to assessment of surgical trainees.

Validation of a novel assessment tool identifying proficiency in Transurethral Bladder Tumour Resection: The OSATURBS assessment tool

BACKGROUND

Competence in transurethral bladder tumour resection (TURB) is critical in bladder cancer management and should be ensured before independent practice.

OBJECTIVE

Develop an assessment tool for TURB and explore validity evidence in a clinical context.

DESIGN, SETTING, AND PARTICIPANTS

July 2019-March 2021, a total of 33 volunteer doctors from three hospitals were included. Participants performed two TURB procedures on patients with bladder tumours. A newly developed assessment tool (OSATURBS) was used for direct observation assessment, self-assessment, and blinded video-assessment. Outcome measurements and statistical analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t-test and independent samples t-test.

RESULTS AND LIMITATIONS

The internal consistency reliability across items Cronbach's alpha was 0.94 (n = 260, p < 0.001). Inter-rater reliability = 0.80 (n = 64, p < 0.001). Test-retest correlation was high, r = 0.71 (n = 32, p < 0.001). Relation to TURB experience was high, r = 0.71 (n = 32, p < 0.001). Pass/fail score = 19 points. Direct observation assessments were strongly correlated with video ratings (r = 0.85, p < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances.

CONCLUSIONS

OSATURBS assessment tool for TURB can be used for assessment of surgical proficiency in the clinical setting. Direct observation assessment and self-assessment are biased, and blinded video-assessment of TURB performances is advised.

Current Evidence for Simulation-Based Training and Assessment of Myringotomy and Ventilation Tube Insertion: A Systematic Review

OBJECTIVE

Myringotomy and ventilation tube insertion (MT) is a key procedure in otorhinolaryngology and can be trained using simulation models. We aimed to systematically review the literature on models for simulation-based training and assessment of MT and supporting educational evidence.

DATABASES REVIEWED

PubMed, Embase, Cochrane Library, Web of Science, Directory of Open Access Journals.

METHODS

Inclusion criteria were MT training and/or skills assessment using all types of training modalities and learners. Studies were divided into 1) descriptive and 2) educational interventional/observational in the analysis. For descriptive studies, we provide an overview of available models including materials and cost. Educational studies were appraised using Kirkpatrick's level of educational outcomes, Messick's framework of validity, and a structured quality assessment tool.

RESULTS

Forty-six studies were included consisting of 21 descriptive studies and 25 educational studies. Thirty-one unique physical and three virtual reality simulation models were identified. The studies report moderate to high realism of the different simulators and trainees and educators perceive them beneficial in training MT skills. Overall, simulation-based training is found to reduce procedure time and errors, and increase performance as measured using different assessment tools. None of the studies used a contemporary validity framework and the current educational evidence is limited.

CONCLUSION

Numerous simulation models and assessment tools have been described in the literature but educational evidence and systematic implementation into training curricula is scarce. There is especially a need to establish the effect of simulation-based training of MT in transfer to the operating room and on patient outcomes.

Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta

BACKGROUND

Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE.

METHODS

This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard.

RESULTS

Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed.

CONCLUSION

Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.

LEVEL OF EVIDENCE

Diagnostic test, no or poor gold standard, level V.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

OpenEar Image Data Enables Case Variation in High Fidelity Virtual Reality Ear Surgery

BACKGROUND

Virtual reality (VR) simulation is an established option for temporal bone surgical training. Most VR simulators are based on computed tomography imaging, whereas the Visible Ear Simulator (VES) is based on high-fidelity cryosections of a single temporal bone specimen. Recently published OpenEar datasets combine cone-beam computed tomography (CBCT) and micro-slicing to achieve similar model quality. This study explores integration of OpenEar datasets into VES to enable case variation in simulation with implications for patient-specific modeling based on CBCT.

METHODS

The OpenEar dataset consists of segmented, coregistered, multimodal imaging sets of human temporal bones. We derived drillable bone segments from the dataset as well as triangulated surface models of critical structures such as facial nerve or dura. Realistic visualization was achieved using coloring from micro-slicing, custom tinting, and texture maps. Resulting models were validated by clinical experts.

RESULTS

Six of the eight OpenEar datasets could be integrated in VES complete with instructional guides for various temporal bone surgical procedures. Resulting models were of high quality because of postprocessing steps taken to increase realism including colorization and imaging artifact removal. Bone artifacts were common in CBCT, resulting in dehiscences that most often could not be found in the ground truth micro-slicing data.

CONCLUSION

New anatomy models are included in VES version 3.5 freeware and provide case variation for training which could help trainees to learn more quickly and transferably under variable practice conditions. The use of CBCT for VR simulation models without postprocessing results in bone artifacts, which should be considered when using clinical imaging for patient-specific simulation, surgical rehearsal, and planning.

Defining the need for faculty development in assessment

BACKGROUND

High-quality workplace-based assessments are essential for competency-based surgical education. We explored education leaders' perceptions regarding faculty competence in assessment.

METHODS

Surgical education leaders were surveyed regarding which areas faculty needed improvement, and knowledge of assessment tools. Respondents were queried on specific skills regarding (a)importance in resident/medical student education (b)competence of faculty in assessment and feedback.

RESULTS

Surveys (n = 636) were emailed, 103 responded most faculty needed improvement in: verbal (86%) and written (83%) feedback, assessing operative skill (49%) and preparation for procedures (50%). Cholecystectomy, trauma laparotomy, inguinal herniorrhaphy were "very-extremely important" in resident education (99%), but 21-24% thought faculty "moderately to not-at-all" competent in assessment. This gap was larger for non-technical skills. Regarding assessment tools, 56% used OSATS, 49% Zwisch; most were unfamiliar with all non-technical tools.

SUMMARY

These data demonstrate a significant perceived gap in competence of faculty in assessment and feedback, and unfamiliarity with assessment tools. This can inform faculty development to support competency-based surgical education.