Impact of a structured skills laboratory curriculum on surgery residents' intraoperative decision-making and technical skills

Non-technical skill assessments across levels of US surgical training

BACKGROUND

To ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool.

METHODS

We undertook an exploratory, prospective cohort study of 32 residents-interns (postgraduate year 1), junior residents (postgraduate years 2-3), and senior residents (postgraduate years 4-5)-across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident's intraoperative performance. Mean NOTSS-US ratings (total range 4-20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1-5, with 1=poor, 3=average, 5=excellent) were stratified by residents' training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression.

RESULTS

For 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3-8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5-6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3-3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents' training levels persisted for individual NOTSS-US behavior categories.

CONCLUSION

These data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees' non-technical skills.

Towards Development of an Open Surgery Competency Assessment for Residents (OSCAR) Tool - A Systematic Review of the Literature and Delphi Consensus

OBJECTIVE

Competency-based education has mandated accurate intra-operative assessment tools. We aimed to define consensus-based open surgical skills perceived by experts as critical for assessment.

DESIGN

A mixed-method design was employed: systematic review and e-Delphi methodology.

SETTING

The study was performed at McGill University-affiliated large tertiary academic centers in Montreal, Quebec, Canada.

PARTICIPANTS

Per PRISMA guidelines, a peer-reviewed search strategy was employed. Studies published in English and those describing technical skill assessment of open abdominal surgery were included; subspecialty-specific skills, conference abstracts, academic memoirs were excluded. Most-cited skills were subjected to e-Delphi methodology to identify those deemed essential by experts, based a 3-point Likert scale. Eighteen McGill University-affiliated general surgeons, representing a variety of subspecialties of General Surgery, were invited to answer the questionnaire.

RESULTS

Around 120 of 4285 references were retained for analysis. The 12 most cited skills included suturing, tissue and instrument handling, movement economy, instrument knowledge, knot tying, flow, knowledge of procedure, completion time, dissection technique, knowledge of anatomy and sterile technique; 6 of these achieved high or perfect scores and agreement after 2 rounds of survey: suturing, sterile technique, knot tying, knowledge of anatomy, knowledge of procedure, and tissue handling. Median standard deviation decreased (0.495 to 0.450) from first to second round, indicating improvement in consensus.

CONCLUSION

These results will help develop and validate the OSCAR (objective structured clinical assessment rubric) assessment tool for immediate intra-operative feedback of open technical skills for surgical trainees.

Does scripting operative plans in advance lead to better preparedness of trainees? A pilot study

BACKGROUND

We pondered if preoperative scripting might better prepare residents for the operating room (OR).

METHODS

Interns rotating on a general surgeon's service were instructed to script randomized cases prior to entering the OR. Scripts contained up to 20 points highlighting patient information perceived important for surgical management. The attending was blinded to the scripting process and completed a feedback sheet (Likert scale) following each procedure. Feedback questions were categorized into "preparedness" (aware of patient specific details, etc.) and "performance" (provided better assistance, etc.).

RESULTS

Eight surgical interns completed 55 scripted and 61 non-scripted cases. Total scores were higher in scripted cases (p = 0.02). Performance scores were higher for scripted cases (3.31 versus 3.13, p = 0.007), while preparedness did not differ (3.65 and 3.62, p = 0.51).

CONCLUSIONS

This pilot study suggests scripting cases may be a useful preoperative planning tool to increase interns' operative and patient care performance but may not affect perceived preparedness.

Understanding Cognitive Performance During Robot-Assisted Surgery

OBJECTIVE

To understand cognitive function of an expert surgeon in various surgical scenarios while performing robot-assisted surgery.

MATERIALS AND METHODS

In an Internal Review Board approved study, National Aeronautics and Space Administration-Task Load Index (NASA-TLX) questionnaire with surgical field notes were simultaneously completed. A wireless electroencephalography (EEG) headset was used to monitor brain activity during all procedures. Three key portions were evaluated: lysis of adhesions, extended lymph node dissection, and urethro-vesical anastomosis (UVA). Cognitive metrics extracted were distraction, mental workload, and mental state.

RESULTS

In evaluating lysis of adhesions, mental state (EEG) was associated with better performance (NASA-TLX). Utilizing more mental resources resulted in better performance as self-reported. Outcomes of lysis were highly dependent on cognitive function and decision-making skills. In evaluating extended lymph node dissection, there was a negative correlation between distraction level (EEG) and mental demand, physical demand and effort (NASA-TLX). Similar to lysis of adhesion, utilizing more mental resources resulted in better performance (NASA-TLX). Lastly, with UVA, workload (EEG) negatively correlated with mental and temporal demand and was associated with better performance (NASA-TLX). The EEG recorded workload as seen here was a combination of both cognitive performance (finding solution) and motor workload (execution). Majority of workload was contributed by motor workload of an expert surgeon. During UVA, muscle memory and motor skills of expert are keys to completing the UVA.

CONCLUSION

Cognitive analysis shows that expert surgeons utilized different mental resources based on their need.

Cognitive Task Analysis

Background. Surgical training is changing and evolving as time, pressure, and legislative demands continue to mount on trainee surgeons. A paradigm change in the focus of training has resulted in experts examining the cognitive steps needed to perform complex and often highly pressurized surgical procedures. Objective. To provide an overview of the collective evidence on cognitive task analysis (CTA) as a surgical training method, and determine if CTA improves a surgeon’s performance as measured by technical and nontechnical skills assessment, including precision, accuracy, and operative errors. Methods. A systematic literature review was performed. PubMed, Cochrane, and reference lists were analyzed for appropriate inclusion. Results. A total of 595 surgical participants were identified through the literature review and a total of 13 articles were included. Of these articles, 6 studies focused on general surgery, 2 focused on practical procedures relevant to surgery (central venous catheterization placement), 2 studies focused on head and neck surgical procedures (cricothyroidotomy and percutaneous tracheostomy placement), 2 studies highlighted vascular procedures (endovascular aortic aneurysm repair and carotid artery stenting), and 1 detailed endovascular repair (abdominal aorta and thoracic aorta). Overall, 92.3% of studies showed that CTA improves surgical outcome parameters, including time, precision, accuracy, and error reduction in both simulated and real-world environments. Conclusion. CTA has been shown to be a more effective training tool when compared with traditional methods of surgical training. There is a need for the introduction of CTA into surgical curriculums as this can improve surgical skill and ultimately create better patient outcomes.

Development of a standardized training course for laparoscopic procedures using Delphi methodology

BACKGROUND

Content, evaluation, and certification of laparoscopic skills and procedure training lack uniformity among different hospitals in The Netherlands. Within the process of developing a new regional laparoscopic training curriculum, a uniform and transferrable curriculum was constructed for a series of laparoscopic procedures. The aim of this study was to determine regional expert consensus regarding the key steps for laparoscopic appendectomy and cholecystectomy using Delphi methodology.

METHODS

Lists of suggested key steps for laparoscopic appendectomy and cholecystectomy were created using surgical textbooks, available guidelines, and local practice. A total of 22 experts, working for teaching hospitals throughout the region, were asked to rate the suggested key steps for both procedures on a Likert scale from 1-5. Consensus was reached with Crohnbach's α ≥ 0.90.

RESULTS

Of the 22 experts, 21 completed and returned the survey (95%). Data analysis already showed consensus after the first round of Delphi on the key steps for laparoscopic appendectomy (Crohnbach's α = 0.92) and laparoscopic cholecystectomy (Crohnbach's α = 0.90). After the second round, 15 proposed key steps for laparoscopic appendectomy and 30 proposed key steps for laparoscopic cholecystectomy were rated as important (≥4 by at least 80% of the expert panel). These key steps were used for the further development of the training curriculum.

CONCLUSION

By using the Delphi methodology, regional consensus was reached on the key steps for laparoscopic appendectomy and cholecystectomy. These key steps are going to be used for standardized training and evaluation purposes in a new regional laparoscopic curriculum.

Task analysis method for procedural training curriculum development

A central venous catheter (CVC) is an important medical tool used in critical care and emergent situations. Integral to proper care in many circumstances, insertion of a CVC introduces the risk of central line-associated blood stream infections and mechanical adverse events; proper training is important for safe CVC insertion. Cognitive task analysis (CTA) methods have been successfully implemented in the medical field to improve the training of postgraduate medical trainees, but can be very time-consuming to complete and require a significant time commitment from many subject matter experts (SMEs). Many medical procedures such as CVC insertion are linear processes with well-documented procedural steps. These linear procedures may not require a traditional CTA to gather the information necessary to create a training curriculum. Accordingly, a novel, streamlined CTA method designed primarily to collect cognitive cues for linear procedures was developed to be used by medical professionals with minimal CTA training. This new CTA methodology required fewer trained personnel, fewer interview sessions, and less time commitment from SMEs than a traditional CTA. Based on this study, a streamlined CTA methodology can be used to efficiently gather cognitive information on linear medical procedures for the creation of resident training curricula and procedural skills assessments.

Coordinating progressive levels of simulation fidelity to maximize educational benefit

PURPOSE

To evaluate the effectiveness of a novel, simulation-based educational model rooted in scaffolding theory that capitalizes on a systematic progressive sequence of simulators that increase in realism (i.e., fidelity) and information content.

METHOD

Forty-five medical students were randomly assigned to practice intravenous catheterization using high-fidelity training, low-fidelity training, or progressive training from low to mid to high fidelity. One week later, participants completed a transfer test on a standardized patient simulation. Blinded expert raters assessed participants' global clinical performance, communication, procedure documentation, and technical skills on the transfer test. Participants' management of the resources available during practice was also recorded. Data were analyzed using multivariate analysis of variance. The study was conducted in fall 2008 at the University of Toronto.

RESULTS

The high-fidelity group scored higher (P < .05) than the low-fidelity group on all measures except procedure documentation. The progressive group scored higher (P < .05) than other groups for documentation and global clinical performance and was equivalent to the high-fidelity group for communication and technical skills. Total practice time was greatest for the progressive group; however, this group required little practice time on the resource-intensive high-fidelity simulator.

CONCLUSIONS

Allowing students to progress in their practice on simulators of increasing fidelity led to superior transfer of a broad range of clinical skills. Further, this progressive group was resource-efficient, as participants concentrated on lower fidelity and lower resource-intensive simulators. It is suggested that clinical training curricula incorporate exposure to multiple simulators to maximize educational benefit and potentially save educator time.

Intra-operative decision making: more than meets the eye

Operating room teams consist of team members with diverse training backgrounds. In addition to differences in training, each team member has unique and complex decision making paths. As such, team members may function in the same environment largely unaware of their team members' perspectives. The goal of our work was to use a theory-based approach to better understand the complexity of knowledge-based intra-operative decision making. Cognitive task analysis methods were used to extract the knowledge, thought processes, goal structures and critical decisions that provide the foundation for surgical task performance. A triangulated and iterative approach is presented.

Use of the Surgical Council on Resident Education (SCORE) curriculum as a template for evaluating and planning a program's clinical curriculum

BACKGROUND

The SCORE curriculum defines surgical operations/procedures that residents are expected to be competent with by the end of the residency.

OBJECTIVE

The purpose of this study was to conduct a gap analysis to determine how well the operative experience in a general surgery residency program approximates the expectations of the SCORE curriculum, especially regarding those procedures considered essential to general surgical training.

SETTING/PARTICIPANTS

Final ACGME resident operative experience reports of recent Northwestern University general surgery program graduates (n = 15) were compared with the specific procedures and procedure levels (ie, Essential-Common, Essential-Uncommon, Complex) defined in the SCORE curriculum. The average numbers of individual SCORE procedures and procedures per SCORE procedure level performed per resident were summarized using descriptive statistics.

RESULTS

During their 5 years of training general surgery residents logged a mean of 1025.7 (SD 152.9) primary procedures per resident. We were able to match 87.1% of these ACGME logged procedures with specific procedures identified in the SCORE curriculum. On average, of the Essential-common procedures, 23 (35%) were performed >10 times and 35 (53%) were performed >five times. Conversely, the number of Essential-uncommon and Complex procedures performed >five times were 3 (5%) and 10 (7%), respectively. Several procedures identified in the SCORE curriculum were performed at very low frequency during residency training.

CONCLUSIONS

This experience suggests that leadership at SCORE and the ACGME need to make the curriculum and logging system compatible and that surgical residents need to be better educated with regards to case logging. Despite these issues, important differences appeared to exist between actual resident operative experiences and expectations set by the SCORE curriculum. Based on these finding we advocate that similar gap analyses be performed at other surgical residency training programs to identify discrepancies between program experience and SCORE curriculum expectations.