Reliable Assessment of Performance in Surgery: A Practical Approach to Generalizability Theory

Development and Validity of a Simulation Program for Assessment of Clinical Teaching Skills

Background: Teaching competence is expected of all intensivists, yet experts rarely supervise or assess trainees' teaching skills. Simulation offers an attractive solution. Objective: Develop and validate a simulation-based assessment of clinical teaching skills in pediatric critical care medicine (CCM). Methods: Participants were 128 pediatric CCM trainees, registered nurses, and respiratory therapists. Medical education experts used literature review and consensus to design three scenarios to assess teaching skills. Scenarios were piloted before use, and raters were trained. Teams completed one of three teaching scenarios, followed by a communication scenario. Raters were faculty members and trainees. Evidence for validity was collected and analyzed using Messick's unifying framework under the following domains: content, response processes, internal structure, relationship to other variables, and consequences of the assessment. Results: The scenarios and assessment tools were designed to capture the characteristics of a good teacher as described in the literature. Raters provided feedback that the tools were easy to use. Internal consistency of the scores measured by Cronbach's α was high. Rater agreement measured by interclass correlation was moderate for one of three scenarios. The relationship to other variables was investigated by correlating teaching scores with communication. Pearson's correlation was moderate for two of three scenarios. Consequences evidence was gathered using a retrospective self-assessed learning gain before versus after the training, which was significant for all scenarios. Conclusion: We developed a three-station simulation program for the assessment of teaching skills in pediatric CCM. The validity evidence collected is moderate, which indicates that it is effective for training and feedback on teaching skills.

Gathering Validity Evidence for a Simulation-Based Test of Otoscopy Skills

OBJECTIVE

Otoscopy is a key clinical examination used by multiple healthcare providers but training and testing of otoscopy skills remain largely uninvestigated. Simulator-based assessment of otoscopy skills exists, but evidence on its validity is scarce. In this study, we explored automated assessment and performance metrics of an otoscopy simulator through collection of validity evidence according to Messick's framework.

METHODS

Novices and experienced otoscopists completed a test program on the Earsi otoscopy simulator. Automated assessment of diagnostic ability and performance were compared with manual ratings of technical skills. Reliability of assessment was evaluated using Generalizability theory. Linear mixed models and correlation analysis were used to compare automated and manual assessments. Finally, we used the contrasting groups method to define a pass/fail level for the automated score.

RESULTS

A total of 12 novices and 12 experienced otoscopists completed the study. We found an overall G-coefficient of .69 for automated assessment. The experienced otoscopists achieved a significantly higher mean automated score than the novices (59.9% (95% CI [57.3%-62.6%]) vs. 44.6% (95% CI [41.9%-47.2%]), P < .001). For the manual assessment of technical skills, there was no significant difference, nor did the automated score correlate with the manually rated score (Pearson's r = .20, P = .601). We established a pass/fail standard for the simulator's automated score of 49.3%.

CONCLUSION

We explored validity evidence supporting an otoscopy simulator's automated score, demonstrating that this score mainly reflects cognitive skills. Manual assessment therefore still seems necessary at this point and external video-recording is necessary for valid assessment. To improve the reliability, the test course should include more cases to achieve a higher G-coefficient and a higher pass/fail standard should be used.

Development and Validation of the Breast Cancer Scale QLICP-BR V2.0 Based on Classical Test Theory and Generalizability Theory

Objective

The aim of this study was to develop and validate the breast cancer scale among the system of quality-of-life instruments for cancer patients (QLICP-BR V2.0).

Methods

Programmed decision procedures and theories on instrument development were applied to develop QLICP-BR V2.0. A total of 246 breast cancer inpatients were investigated using QLICP-BR V2.0 from hospital admission until discharge. The reliability, validity, and responsiveness of the QLICP-BR V2.0 scale were evaluated by using the classical test theory combined with the generalizability theory (GT), including correlation analysis, multi-trait scaling analysis, factor analyses, t-tests, and also multivariate generalizability theory analysis.

Results

The test–retest reliability of the total scale is 0.79, the Cronbach coefficient is 0.85, and the intra-class correlations coefficient is 0.88. The item–domain correlation analysis showed that the correlation coefficient between items and their own domain is greater than that with other domains except of item GSO4. The exploratory factor analysis showed that three principal components are obtained in the specific module. The outcome of the factor analysis coincides substantially with our theoretical conception. The score difference of each domain of the scale and the total scale before and after treatment is statistically significant (P &lt; 0.05), with the standardized response mean of the total scale being 0.61. According to GT, the generalization coefficient of the scores in the 5 domains is between 0.626 and 0.768, and the reliability index is between 0.557 and 0.695.

Conclusion

QLICP-BR V2.0 exhibited reasonable degrees of validity, reliability, and responsiveness according to classical test and the generalizability theory. The number of items in the scale is appropriate.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

PURPOSE

Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory.

METHOD

Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability.

RESULTS

Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations).

CONCLUSIONS

Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Using generalizability analysis to estimate parameters for anatomy assessments: A multi-institutional study

With integrated curricula and multidisciplinary assessments becoming more prevalent in medical education, there is a continued need for educational research to explore the advantages, consequences, and challenges of integration practices. This retrospective analysis investigated the number of items needed to reliably assess anatomical knowledge in the context of gross anatomy and histology. A generalizability analysis was conducted on gross anatomy and histology written and practical examination items that were administered in a discipline-based format at Indiana University School of Medicine and in an integrated fashion at the University of Alabama School of Medicine and Rush University Medical College. Examination items were analyzed using a partially nested design s×(i:o) in which items were nested within occasions (i:o) and crossed with students (s). A reliability standard of 0.80 was used to determine the minimum number of items needed across examinations (occasions) to make reliable and informed decisions about students' competence in anatomical knowledge. Decision study plots are presented to demonstrate how the number of items per examination influences the reliability of each administered assessment. Using the example of a curriculum that assesses gross anatomy knowledge over five summative written and practical examinations, the results of the decision study estimated that 30 and 25 items would be needed on each written and practical examination to reach a reliability of 0.80, respectively. This study is particularly relevant to educators who may question whether the amount of anatomy content assessed in multidisciplinary evaluations is sufficient for making judgments about the anatomical aptitude of students. Anat Sci Educ 10: 109-119. © 2016 American Association of Anatomists.

Perioperative feedback in surgical training: A systematic review

BACKGROUND

Changes in surgical training have raised concerns about residents' operative exposure and preparedness for independent practice. One way of addressing this concern is by optimizing teaching and feedback in the operating room (OR). The objective of this study was to perform a systematic review on perioperative teaching and feedback.

METHODS

A systematic literature search identified articles from 1994 to 2014 that addressed teaching, feedback, guidance, or debriefing in the perioperative period. Data was extracted according to ENTREQ guidelines, and a qualitative analysis was performed.

RESULTS

Thematic analysis of the 26 included studies identified four major topics. Observation of teaching behaviors in the OR described current teaching practices. Identification of effective teaching strategies analyzed teaching behaviors, differentiating positive and negative teaching strategies. Perceptions of teaching behaviors described resident and attending satisfaction with teaching in the OR. Finally models for delivering structured feedback cited examples of feedback strategies and measured their effectiveness.

CONCLUSIONS

This study provides an overview of perioperative teaching and feedback for surgical trainees and identifies a need for improved quality and quantity of structured feedback.

Operating Room Performance Improves after Proficiency-Based Virtual Reality Cataract Surgery Training

PURPOSE

To investigate the effect of virtual reality proficiency-based training on actual cataract surgery performance. The secondary purpose of the study was to define which surgeons benefit from virtual reality training.

DESIGN

Multicenter masked clinical trial.

PARTICIPANTS

Eighteen cataract surgeons with different levels of experience.

METHODS

Cataract surgical training on a virtual reality simulator (EyeSi) until a proficiency-based test was passed.

MAIN OUTCOME MEASURES

Technical performance in the operating room (OR) assessed by 3 independent, masked raters using a previously validated task-specific assessment tool for cataract surgery (Objective Structured Assessment of Cataract Surgical Skill). Three surgeries before and 3 surgeries after the virtual reality training were video-recorded, anonymized, and presented to the raters in random order.

RESULTS

Novices (non-independently operating surgeons) and surgeons having performed fewer than 75 independent cataract surgeries showed significant improvements in the OR-32% and 38%, respectively-after virtual reality training (P = 0.008 and P = 0.018). More experienced cataract surgeons did not benefit from simulator training. The reliability of the assessments was high with a generalizability coefficient of 0.92 and 0.86 before and after the virtual reality training, respectively.

CONCLUSIONS

Clinically relevant cataract surgical skills can be improved by proficiency-based training on a virtual reality simulator. Novices as well as surgeons with an intermediate level of experience showed improvement in OR performance score.