"Reflection-Before-Practice" Improves Self-Assessment and End-Performance in Laparoscopic Surgical Skills Training

Study of the performance of resident surgeon-in-training during distal fibula lateral plate placement according to 2 learning methods: naive practice versus deliberate practice

INTRODUCTION

Historically, surgical training has been primarily carried out in the operating theatre, using mentoring for the surgical resident to reach the appropriate skill level (3/5). Other surgical training methods also improve performance, but do not always lead to the highest level of expertise (5/5). Another training method, sometimes termed- deliberate practice, by setting objectives based on feedback, may be more effective. In this study we compared resident learning of osteosynthesis for a fibular fracture model between deliberate practice and traditional teaching or naive practice. The main hypothesis was that deliberate practice would result in better objective performance, better subjective performance and reduced stress levels.

MATERIALS AND METHODS

The study involved a level 3 expert and ten level 1 surgeon-in-training subjects divided into two groups naive practice and deliberate practice. Each subject placed 5 plates on a synthetic fibula model. The deliberate practice group received feedback from the expert after each trial. Stress level was measured using the Analgesia Nociception Index (ANI). Objective performance was assessed by OSATS and subjective performance by self-assessment.

RESULTS

Based on initial performance measurements, the two groups were comparable. The mean progression of objective performance over the five osteosynthesis was 10.3 in the naive practice group and 17.1 in the deliberate practice group, with a strong difference in favor of the deliberate practice group. Subjective performance and ANI improved in both groups, with no significant difference.

DISCUSSION

The main hypothesis was disproven: the improvement in objective performance was not significantly greater (< 97.5%) with deliberate practice. However, there was a substantial difference in favor of the deliberate practice group (93%). Secondary hypotheses were not proven too, as neither PS nor ANI were affected by deliberate practice. In conclusion, deliberate practice complements mentoring but must adhere to strict guidelines to be effective: level 5 expertise, precise criteria for defining OSATS, and the use of high- profile simulators.

Self-assessment, and not continuous training, improves basic open suturing skills

BACKGROUND

To develop and maintain suturing skills, clinical exposure is important. When clinical exposure cannot be guaranteed, an adequate training schedule for suturing skills is required. This study evaluates the effect of continuous training, 'reflection before practice' and self-assessment on basic open suturing skills.

METHODS

Medical students performed four basic suturing tasks on a simulation set up before ('pre-test') and after their surgical rotation ('after-test'). Participants were divided in three groups; the 'clinical exposure group' (n = 44) had clinical exposure during their rotation only, the 'continuous training group' (n = 16) completed a suturing interval training during their rotation and the 'self-assessment group' (n = 16) also completed a suturing interval training, but with the use of reflection before practice and self-assessment. Parameters measured by a tracking system during the suturing tasks and a calculated 'composite score' were compared between groups and test-moments.

RESULTS

A significantly better composite score was found at the after-test compared to the pre-test for all groups for all basic suturing tasks (0.001 ≤ p ≤ 0.049). The self-assessment group scored better at the pre-test than the other two groups for all tasks, except for 'knot tying by hand' (0.004 ≤ p ≤ 0.063). However, this group did not score better at the after-test for all tasks, compared to the other two groups. This resulted in a smaller delta of time ('transcutaneous suture', p = 0.013), distance ('Donati suture' and 'intracutaneous suture', 0.005 ≤ p ≤ 0.009) or composite score (all tasks, except for knot tying by hand, 0.007 ≤ p ≤ 0.061) in the self-assessment group.

CONCLUSION

Reflection before practice and self-assessment during continuous training of basic open suturing tasks, may improve surgical skills at the start of the learning curve.

Informing factors and outcomes of self-assessment practices in medical education: a systematic review

BACKGROUND

Self-assessment (SA) is increasingly recognized not only as an assessment method but also as a vital learning activity that enables learners to identify their strengths and limitations, fostering a crucial skill for lifelong learning. Despite its acknowledged significance, there remains a gap in understanding SA for and as learning, rather than merely of learning. This review, therefore, explores two primary questions: (1) What factors inform the implementation of medical students' SA practices? (2) How does SA practices contribute to various educational outcomes?

METHODS

A systematic review was conducted across five databases, employing a combination of keywords pertinent to SA and medical education. Our selection criteria deliberately excluded articles that solely focused on the accuracy of SA or used SA exclusively as a measure for other outcome variables. As a result, 39 studies met our inclusion criteria and were analyzed for this review. Qualitative narrative synthesis was used to summarize the informing factors. Furthermore, Kirkpatrick's model was employed to categorize and summarize the effects of SA activities across various educational outcome levels, including reactions, learning, and behavioral changes.

RESULTS

Our review reveals that key factors influencing students' SA includes feedback from various stakeholders, peer assessment, the format of SA, and both learner and teacher training. Among the 39 studies examined, the majority demonstrated positive effects of SA on aspects such as attitudes and skills. However, a minority found no significant associations, with these outcomes frequently linked to contextual variables and how SA was implemented.

CONCLUSIONS

Our review does not focus on SA accuracy or its role as an assessment method; instead it delves into the role of SA as an integral learning practice. We explored its influencing factors and its impact across various outcomes. The findings indicate that for SA to effectively enhance student learning outcomes, it must be implemented with ample support, clear guidelines, and within contexts that encourage feedback and reflection.

Study of surgical performance during clavicle plate placements using 2 learning methods: naive practice versus deliberate practice

BACKGROUND

Companionship and simple experience or naive practice (NP) rarely lead to expert level surgery, in contrast to deliberate practice (DP) where an expert analyzes the learner's errors and sets goals to improve performance. The main hypothesis was that using DP for learning would result in faster and/or greater progress than using NP.

OBJECTIVES

The objective of this work was to compare the evolution of the learning curve for clavicle locking plate placement on a sawbone model of a clavicle fracture, by surgical trainees learning via two different methods; NP and DP.

PATIENTS AND METHODS

Ten surgical residents, divided into 2 groups of 5, each placed 6 plates. The 6 trials were filmed. The NP group saw an expert video before each placement. The DP group saw this video once and then received personalized advice from the expert for improvement, by analyzing their own video after each subsequent trial. Objective performance (OP) was measured by a standardized evaluation grid (OSATS, with a score ranging from 10 to 50 points per trial), self-evaluation of performance by a numerical scale (from 0 to 10) and stress by an analgesia-nociception index (ANI, calculated by heart rate recording, from 0 to 100).

RESULTS

The mean OP at the last trial of clavicle plate placement was 41.8 (NP group) and 48.2 (DP group), with a mean progression from the first to last trials of 0.8 in the NP group, and 5.1 in the DP group. The mean progression in self-evaluation between the first and last trials was 3.4 (NP group) and 4.6 (DP group). The mean progression of the ANI between the first and last trials was -4.5 (NP group) and +5 (DP group).

DISCUSSION

The results of learning a clavicle plate osteosynthesis technique measured by OSATS were better with deliberate practice than with naive practice. The progression in self-evaluated performance was better with deliberate practice, but with a higher stress level.

CONCLUSION

Deliberate practice is a technique for learning the surgical procedure which complements companionship and experience. It shortens the learning curve and improves the level of performance of surgical trainees.

LEVEL OF EVIDENCE

IV; non-interventional research.

Differences between resident self-assessments and faculty- assessments on Anesthesiology Milestones and associated factors: a cross-sectional study

BACKGROUND

Accurate self-assessment is crucial for the professional development of physicians. There has been sparse data on the accuracy of self-assessments on Anesthesiology Milestones. The aim of this study was to investigate the differences between resident self-assessments and faculty-assessments on Anesthesiology Milestones and the associated factors.

METHODS

This was a cross-sectional study conducted in a general tertiary university-affiliated hospital. We included anesthesia residents who were enrolled in the standardized residency training program in postgraduate year two and three at the time of the Milestone evaluation. We requested evaluations of competencies from both the Clinical Competency Committee faculty and the anesthesia residents themselves, utilizing the Chinese version of Anesthesiology Milestones in January 2023 and January 2024. The primary outcome was the differences between self- and faculty-assessments, calculated by subtracting the faculty-rated score from the self-rated score on each subcompetency.

RESULTS

A total of 46 and 42 residents were evaluated in year 2023 and 2024, respectively. The self-rated sum score was significantly higher than that rated by faculty [mean (standardized deviation): 120.39 (32.41) vs. 114.44 (23.71), P = 0.008 in paired t test] with an intraclass correlation coefficient of 0.55 [95% confidence interval (CI): 0.31 to 0.70]. The Bland-Altman plots revealed significant overestimation in patient care (bias 0.32, 95% CI: 0.05 to 0.60), practice-based learning and improvement (bias 0.45, 95% CI: 0.07 to 0.84), and professionalism (bias 0.37, 95% CI: 0.02 to 0.72). Ratings from residents with master's degrees (mean difference: -1.06, 95% CI: -1.80 to -0.32, P = 0.005) and doctorate degrees (mean difference: -1.14, 95% CI: -1.91 to -0.38, P = 0.003) were closer to the faculty-assessments than residents with bachelor's degrees. Compared with patient care, the differences between self- and faculty- rated scores were smaller in medical knowledge (mean difference: -0.18, 95% CI: -0.35 to -0.02, P = 0.031) and interpersonal and communication skills (mean difference: -0.41, 95% CI: -0.64 to -0.19, P < 0.001) in the generalized estimating equation logistic regression model.

CONCLUSIONS

This study revealed that residents tended to overestimate themselves, emphasizing the need to improve the accuracy of Milestones self-assessment. The differences between self- and faculty-assessments were associated with residents' degrees and domains of competency.

Reflecting on insight and insights into reflection: a systematic review of insight and reflection in post graduate medical education

BACKGROUND

Clinicians recognize insight as important for safe independent medical practice. Clinical education literature focuses on self-reflection. The aim of this review is to describe how clinical educators conceptualize reflection and ask is it analogous to how clinicians conceptualize insight?

METHODS

Using PRISMA guidelines, a systematic review of the literature around insight and reflection in postgraduate medical education was undertaken. A thematic analysis of the concepts of insight and reflection was performed.

RESULTS

A total of 75 reports were included in the analysis. The literature focussed predominantly on reflection with little discussion of insight. Three main themes were generated: episodic reflection; cyclic reflection; reflection as a state. Reflection as a state seemed to be the professional quality most often aspired to but was less well defined in terms of educational interventions. When more than one model was described, it was often with a reflective state being the ideal that episodic or cyclic reflection may approximate. It is not clear that it is possible to progress up the hierarchy.

CONCLUSION

We present a novel description of a hierarchy from discrete episodes of reflection, to cyclic processes that involve reflection, through to a state in which the practitioner is reflective. There is no unified understanding of how an individual ascends this hierarchy, or a cohesive description of what insight is for an independent medical practitioner. This review highlights the need for research into how practicing clinicians conceptualize and characterize insight in their training and practice.

A scoping review of self-monitoring in graduate medical education

BACKGROUND

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education.

METHODS

We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022.

RESULTS

The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n = 31, 42%). Regardless of task, studies focused on self-judgement (n = 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n = 41, 71%). Most self-monitoring was conducted post-task (n = 65, 84%).

CONCLUSION

Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy

INTRODUCTION

Self-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.

METHODS AND PROCEDURES

Using a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1-3] and senior trainees (PGY 4-5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.

RESULTS

A total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.

CONCLUSIONS

OPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.

The feasibility and benefit of unsupervised at-home training of minimally invasive surgical skills

BACKGROUND

Simulation-based training may be used to acquire MIS skills. While mostly done in a simulation center, it is proposed that this training can be undertaken at-home as well. The aim of this study is to evaluate whether unsupervised at-home training and assessment of MIS skills is feasible and results in increased MIS skills.

METHODS

Medical doctors and senior medical students were tested on their innate abilities by performing a pre-test on a take-home simulator. Henceforth, they followed a two-week interval training practicing two advanced MIS skills (an interrupted suture with knot tying task and a precise peg transfer task) and subsequently performed a post-test. Both tests and all training moments were performed at home. Performance was measured using motion analysis software (SurgTrac) and by expert-assessment and self-assessment using a competency assessment tool for MIS suturing (LS-CAT).

RESULTS

A total of 38 participants enrolled in the study. Participants improved significantly between the pre-test and the post-test for both tasks. They were faster (632 s vs. 213 s, p < 0.001) and more efficient (distance of instrument tips: 9.8 m vs. 3.4 m, p = 0.001) in the suturing task. Total LS-CAT scores, rated by an expert, improved significantly with a decrease from 36 at the pre-test to 20 at the post-test (p < 0.001) and showed a strong correlation with self-assessment scores (R 0.771, p < 0.001). The precise peg transfer task was completed faster (300 s vs. 163 s, p < 0.001) and more efficient as well (14.8 m vs. 5.7 m, p = 0.005). Additionally, they placed more rings correctly (7 vs. 12, p = 0.010).

CONCLUSION

Unsupervised at-home training and assessment of MIS skills is feasible and resulted in an evident increase in skills. Especially in times of less exposure in the clinical setting and less education on training locations this can aid in improving MIS skills.

Validation of a Newly Developed Competency Assessment Tool for the Posterior Sagittal Anorectoplasty

INTRODUCTION

 The correction of an anorectal malformation (ARM) is complex and relatively infrequent. Simulation training and subsequent assessment may result in better clinical outcomes. Assessment can be done using a competency assessment tool (CAT). This study aims to develop and validate a CAT for the posterior sagittal anorectoplasty (PSARP) on a simulation model.

MATERIALS AND METHODS

 The CAT-PSARP was developed after consultation with experts in the field. The PSARP was divided into five steps, while tissue and instrument handling were scored separately. Participants of pediatric colorectal hands-on courses in 2019 and 2020 were asked to participate. They performed one PSARP procedure on an ARM simulation model, while being assessed by two objective observers using the CAT-PSARP.

RESULTS

 A total of 82 participants were enrolled. A fair interobserver agreement was found for general skills (intraclass correlation coefficient [ICC] = 0.524, p < 0.001), a good agreement for specific skills (ICC = 0.646, p < 0.001), and overall performance (ICC = 0.669, p < 0.001). The experienced group scored higher on all steps (p < 0.001), except for "anoplasty (p = 0.540)," compared with an inexperienced group.

CONCLUSION

 The CAT-PSARP is a suitable objective assessment tool for the overall performance of the included steps of the PSARP for repair of an ARM on a simulation model.

The big five: Studying the surgical personality

BACKGROUND

The challenging nature of performing surgery on a personal and professional level demands specific characteristics. Personality traits play an important role in the nature and behavior of humans, which are studied using the five-factor model. Therefore, we investigated the personality of 3 surgical generations.

METHODS

Three distinct surgical populations were approached. The Dutch Big Five Inventory-2 was sent out online to 126 surgical residents (response: n = 69) and 104 surgeons (response: n = 60) in a teaching region in the Netherlands. Moreover, medical students interested in surgery were approached via the students' surgical society (response: n = 54). To obtain a normative Dutch population sample, the Longitudinal Internet studies for the Social Sciences panel was used, creating groups of the following age categories: 18 to 25 (n = 84), 26 to 35 (n = 101), 36 to 67 (n = 432). One-way analysis of variance with Bonferroni correction was used to assess differences in personality scores.

RESULTS

Individuals interested in surgery (ie, surgically-oriented medical students, surgical residents, and surgeons) generally scored significantly higher on extraversion, conscientiousness, agreeableness, open-mindedness, and lower on negative emotionality compared with the normative population sample. Across the surgical generations, surgical residents scored significantly lower on open-mindedness (3.60) compared with surgeons (3.92) and surgically-oriented medical students (3.82). Surgically-oriented medical students scored significantly higher in negative emotionality (2.44) compared with surgical residents (2.12) and surgeons (2.07).

CONCLUSION

Being a surgeon demands particular levels of determination and emotional stability. The surgical population shows a distinct personality pattern compared with the normative population, and more modest differences exist between persons in different stages of their surgical career.

Saudi nursing student satisfaction and evaluation of reflective practice: A cross-sectional study

BACKGROUND

Regardless of the fact that reflection has been documented in nursing literature for several years and is recognized in the nursing profession to improve learners' knowledge and mindfulness, this topic requires investigation in Gulf countries from a student perspective, as no studies have been published in the gulf regarding nursing students' perspectives on reflective practice.

METHODS

A total of 261 nursing students from the third and fourth academic years at one major educational institution were recruited using a quantitative cross-sectional study design.

RESULTS

The participating nursing students reported high scores on the level of all domains of reflective practice as well as on the level of student satisfaction, where 3rd year students scored higher on the level of reflection with others (p = .03), as well as the reflection on action domains (p < .01). A significant correlation was found between reflection and satisfaction (p < .01), where reflection explained 6.87% of the changes in variance of satisfaction.

CONCLUSION

Nursing students who took part in this study has demonstrated a positive regard towards reflective practice and indicated high correlated satisfaction with learning.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice

BACKGROUND

Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods.

METHODS

The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300).

RESULTS

The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6).

CONCLUSION

Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

Examining the accuracy of residents' self-assessments and faculty assessment behaviours in anesthesiology

BACKGROUND

Residents' accurate self-assessment and clinical judgment are essential for optimizing their clinical skills development. Evidence from the medical literature suggests that residents generally do poorly at self-assessing their performance, often due to factors relating to learners' personal backgrounds, cultures, the specific contexts of the learning environment and rater bias or inaccuracies. We evaluated the accuracy of anesthesiology residents' self-assessed Global Entrustment scores and determined whether differences between faculty and resident scores varied by resident seniority, faculty leniency, and/or year of assessment.

METHODS

We employed variance components modeling techniques and analyzed 329 pairs of faculty and self-assessed entrustment scores among 43 faculty assessors and 15 residents. Using faculty scores as the gold standard, we compared faculty scores with residents' scores (x_i(faculty)-x_i(resident)), and determined residents' accuracy, including over- and under-confidence.

RESULTS

The results indicate that residents were respectively over- and under-confident in 10.9% and 54.4% of the assessments but more consistent in their individual self-assessments (rho = 0.70) than faculty assessors. Faculty scores were significantly higher (α = 0.396; z = 4.39; p < 0.001) than residents' self-assessed scores. Being a lenient/dovish (β = 0.121, z = 3.16, p < 0.01) and a neutral (β = 0.137, z = 3.57, p < 0.001) faculty assessor predicted a higher likelihood of resident under-confidence. Senior residents were significantly less likely to be under-confident compared to junior residents (β = -0.182, z =-2.45, p < 0.05). The accuracy of self-assessments did not significantly vary during the two years of the study period.

CONCLUSIONS

The majority of residents' self-assessments were inaccurate. Our findings may help identify the sources of such inaccuracies.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles

INTRODUCTION

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.

METHODS

We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.

RESULTS

Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.

DISCUSSION

Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Development and clinical implementation of a structured, simulation-based training programme in laparoscopic appendectomy: description, validation and evaluation

Background

Laparoscopic appendectomy is a common procedure in general surgery but is likely underused in structured and real-life teaching. This study describes the development, validation and evaluation of implementing a structured training programme for laparoscopic appendectomy.

Study design

A structured curriculum and simulation-based programme for trainees and trainers was developed. All general surgery trainees and trainers were involved in laparoscopic appendectomies. All trainees and trainers underwent the structured preprocedure training programme before real-life surgery evaluation. A standardised form evaluated eight technical steps (skills) of the procedure as well as an overall assessment, and nine elements of communication (feedback), and was used for bilateral evaluation by each trainee and trainer. A consecutive, observational cohort over a 12-month period was used to gauge real-life implementation.

Results

During 277 eligible real-life appendectomies, structured evaluation was performed in 173 (62%) laparoscopic appendectomies, for which 165 forms were completed by 19 trainees. Construct validity was found satisfactory. Inter-rater reliability demonstrated good correlation between trainee and trainer. The trainees’ and trainers’ stepwise and overall assessments of technical skills had an overall good reliability (intraclass correlation coefficient of 0.88). The vast majority (92.2%) of the trainees either agreed or strongly agreed that the training met their expectations.

Conclusion

Structured training for general surgery residents can be implemented for laparoscopic appendectomy. Skills assessment by trainees and trainers indicated reliable self-assessment. Overall, the trainees were satisfied with the training, including the feedback from the trainers.

Are Self-Assessment and Peer Assessment of Added Value in Training Complex Pediatric Surgical Skills?

INTRODUCTION

 Self-assessment aids "reflection-before-practice," which is expected to result in a better understanding of one's strengths and weaknesses and consequently a better overall performance. This is, however, rarely used in surgical training. This study aims to evaluate the correlation between self-, peer-, and expert assessment on surgical skills of pediatric surgical trainees.

MATERIALS AND METHODS

 A competency assessment tool for the posterior sagittal anorectoplasty (CAT-PSARP) was previously developed and validated. During international hands-on pediatric colorectal workshops in 2019 and 2020, participants practiced the PSARP on an inanimate anorectal malformation model. They were assisted by a peer and assessed by two independent expert observers, using the CAT-PSARP. After the training session, both self- and peer assessment were completed, using the same CAT-PSARP.

RESULTS

 A total of 79 participants were included. No correlation was found between the overall CAT-PSARP scores of the expert observers and the self-assessment (r = 0.179, p = 0.116), while a weak correlation was found between experts and peer assessment (r = 0.317, p = 0.006). When comparing the self-assessment scores with peer assessment, a moderate correlation was found for the overall performance score (r = 0.495, p < 0.001). Additionally, the participants who were first to perform the procedure scored significantly better than those who trained second on the overall performance (mean 27.2 vs. 24.4, p < 0.001).

CONCLUSION

 Participants, peers, and experts seemed to have a unique view on the performance during training because there was little correlation between outcomes of the trainees and the experts. Self-assessment may be useful for reflection during training; however, expert assessment seems to be essential for assessment of surgical skills.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Validation of Motion Tracking Software for Evaluation of Surgical Performance in Laparoscopic Cholecystectomy

Motion tracking software for assessing laparoscopic surgical proficiency has been proven to be effective in differentiating between expert and novice performances. However, with several indices that can be generated from the software, there is no set threshold that can be used to benchmark performances. The aim of this study was to identify the best possible algorithm that can be used to benchmark expert, intermediate and novice performances for objective evaluation of psychomotor skills. 12 video recordings of various surgeons were collected in a blinded fashion. Data from our previous study of 6 experts and 23 novices was also included in the analysis to determine thresholds for performance. Video recording were analyzed both by the Kinovea 0.8.15 software and a blinded expert observer using the CAT form. Multiple algorithms were tested to accurately identify expert and novice performances. ½ L + [Formula: see text] A + [Formula: see text] J scoring of path length, average movement and jerk index respectively resulted in identifying 23/24 performances. Comparing the algorithm to CAT assessment yielded in a linear regression coefficient R2 of 0.844. The value of motion tracking software in providing objective clinical evaluation and retrospective analysis is evident. Given the prospective use of this tool the algorithm developed in this study proves to be effective in benchmarking performances for psychomotor skills evaluation.

Understanding the effects of structured self-assessment in directed, self-regulated simulation-based training of mastoidectomy: A mixed methods study

Objective

Self-directed training represents a challenge in simulation-based training as low cognitive effort can occur when learners overrate their own level of performance. This study aims to explore the mechanisms underlying the positive effects of a structured self-assessment intervention during simulation-based training of mastoidectomy.

Methods

A prospective, educational cohort study of a novice training program consisting of directed, self-regulated learning with distributed practice (5x3 procedures) in a virtual reality temporal bone simulator. The intervention consisted of structured self-assessment after each procedure using a rating form supported by small videos. Semi-structured telephone interviews upon completion of training were conducted with 13 out of 15 participants. Interviews were analysed using directed content analysis and triangulated with quantitative data on secondary task reaction time for cognitive load estimation and participants’ self-assessment scores.

Results

Six major themes were identified in the interviews: goal-directed behaviour, use of learning supports for scaffolding of the training, cognitive engagement, motivation from self-assessment, self-assessment bias, and feedback on self-assessment (validation). Participants seemed to self-regulate their learning by forming individual sub-goals and strategies within the overall goal of the procedure. They scaffolded their learning through the available learning supports. Finally, structured self-assessment was reported to increase the participants’ cognitive engagement, which was further supported by a quantitative increase in cognitive load.

Conclusions

Structured self-assessment in simulation-based surgical training of mastoidectomy seems to promote cognitive engagement and motivation in the learning task and to facilitate self-regulated learning.

Applications of the reflective practice questionnaire in medical education

BACKGROUND

We sought to determine whether the Reflective Practice Questionnaire (RPQ) is a reliable measure of reflective capacity and related characteristics in medical students. We also planned to learn how the RPQ could be used in medical education.

METHODS

The RPQ is a 40 item self-report questionnaire that includes a multi-faceted approach to measuring reflective capacity. It also includes sub-scales on several other theoretically relevant constructs such as desire for improvement, confidence, stress, and job satisfaction. The reliabilities of reflective capacity and other sub-scales were determined by calculating their Cronbach alpha reliability values. In the present study, the RPQ was answered by 98 graduating fourth-year medical students from an American University, and these RPQ scores were compared with general public and mental health practitioner samples from a prior study using ANOVA and Bonferroni adjusted comparisons.

RESULTS

Medical students reported a higher reflective capacity than the general public sample, but students were statistically indistinguishable from the mental health practitioner sample. For medical students, reflective capacity was associated with features of confidence, stress, and desire for improvement. Job satisfaction was positively associated with confidence in communication with patients, and negatively associated with stress when interacting with patients. A cluster analysis revealed that around 19% of the medical students exhibited a relatively high level of anxiety interacting with patients, 23% were less engaged, 5% were dissatisfied, and 7% expressed a level of over-confidence in their knowledge and skills that was concerning.

CONCLUSIONS

The RPQ is a reliable measure of reflective capacity (Chronbach's alpha value = 0.84) and related characteristics (Cronbach's alpha values from 0.75 to 0.83) in medical students. The RPQ can be used as part of pre-post evaluations of medical education initiatives, to complement student self-reflection activities in the curriculum, and to identify students who might benefit from targeted intervention.