Practice does not always make perfect: need for selection curricula in modern surgical training

Innate Versus Acquired: A Review of Predictive Technical Aptitude Assessments in Surgical Trainee Selection

PURPOSE OF REVIEW

The evaluation and selection process of similarly qualified applicants for surgical residency positions in the United States (US) is challenging. Technical aptitude assessments may provide an opportunity to improve the selection process by offering insight into a candidate's technical skills. The use of these assessments prompts consideration of the degree to which technical aptitude in surgery is innate versus acquired. In this narrative review, we review the state of these assessments and the limitations of developing and validating these instruments.

RECENT FINDINGS

Recent evidence suggests that technical aptitude can be quantified in medical students prior to selection for surgical training; however, both technical aptitude and dedicated practice of technical skills influence operative performance. In the US, technical aptitude assessments are inconsistently used in selecting candidates for surgical residency. Internationally, few countries have implemented standardized technical aptitude assessments in their surgical trainee selection processes. The development of technical aptitude assessments is ongoing worldwide. Technical aptitude assessments may help identify individuals particularly well-suited for a surgical career or those who might benefit from additional practice or remediation. The potential role of technical aptitude assessments in the selection process for surgical trainees remains uncertain. Consideration of these instruments requires an understanding of meaningful outcomes associated with particular technical assessments as well as the assessments' limitations.

Factors Associated with the Innate Orthopedic Ability of Veterinary Students

Relatively little is known about the innate surgical ability of veterinary undergraduates. The objective of this study was to investigate if there were differences in the innate surgical ability of a cohort of 142 third-year veterinary undergraduate students to perform a series of simulated orthopedic surgical tasks, and whether specific factors influenced their innate ability. Participants performed four simulated surgical tasks; "depth of plunge"-an assessment of the "plunge" depth through foam when drilling through the trans cortex of a PVC pipe; "3-dimensional drilling"-an assessment of accuracy when drilling through a block of wood; "depth measurement"-an assessment of the ability to correctly measure the depth of holes in PVC pipe; and "fracture reduction"-where the speed and systematic reduction of a simulated fracture was assessed using a rubric score. Performance for each task was compared based on the responses to a survey. Results showed considerable variation in innate ability. Previous experience performing manual tasks and using a drill was associated with an improvement in students' ability to perform one of the four tasks (fracture reduction). Age, gender, handedness, videogame experience, building game experience, exposure to orthopedic surgery, or desire to pursue surgery as a career were not associated with student performance in any task. A learning curve was observed for the depth of plunge task. An increased target angle led to decreased drilling accuracy for the 3D drilling task. The innate ability of veterinary students to undertake simulated surgical tasks was largely unaffected by the previous experiences evaluated.

Current status, challenges, and future directions of laparoscopic training in young surgeons: a nationwide survey in China

BACKGROUND

With the rapid advancement of technology, minimally invasive surgery, particularly laparoscopic surgery, has made significant progress in the field of surgery. Despite the advantages of laparoscopic surgery, a systematic training system for laparoscopic procedures is lacking in Chinese postgraduate medical education. Our study aims to explore the prevalence of laparoscopic training among resident and attending physicians in China and to assess the current state of training programs.

METHODS

A 10-item questionnaire was distributed to 1,750 resident and attending physicians specializing in surgery across China, with 1,324 valid responses (75.7% response rate). The survey focused on demographics, training curriculum content, and feedback on training effectiveness. Data analysis was conducted using Microsoft Excel and IBM SPSS.

RESULTS

Among the 1,324 respondents, 30.7% reported receiving laparoscopic training, primarily at the attending physician stage. Only 4% of resident physicians and 14% of attending physicians could independently perform complex laparoscopic surgeries. Most respondents (76.6%) could only assist in surgeries. The majority expressed a desire for more frequent and longer training sessions, with suture training being identified as the most beneficial.

CONCLUSIONS

This study underscores the critical need for comprehensive laparoscopic training in China. Early, frequent, and structured training programs are essential for developing proficient laparoscopic surgeons. Future initiatives should focus on expanding access to training at all levels of medical education, ensuring continuous skill development and improved surgical care quality.

Development and validation of an objective virtual reality tool for assessing technical aptitude among potential candidates for surgical training

BACKGROUND

Good technical skills are crucial for surgeons. Yet although surgical training programs strive to assess technical aptitude when selecting surgical residents, valid assessments of such aptitude are still lacking. Surgical simulators have been proposed as a potentially effective tool for this purpose. The current study aims to develop a technical aptitude test using a virtual reality surgical simulator, and to validate its use for the selection of surgical residents.

METHODS

The study had three phases. In Phase 1, we developed an initial version of the technical aptitude test using the Lap-X-VR laparoscopic simulator. In Phases 2 and 3 we refined the test and collected empirical data to evaluate four main sources of validity evidence (content, response process, internal structure, and relationships with other variables), and to evaluate the feasibility and acceptability of the test. Specifically, Phase 2 comprised a review of the test by 30 senior surgeons, and in Phase 3 a revised version of the test was administered to 152 interns to determine its psychometric properties.

RESULTS

Both the surgeons and interns rated the test as highly relevant for selecting surgical residents. Analyses of the data obtained from the trial administration of the test supported the appropriateness of the score calculation process and showed good psychometric properties, including reliability (α = 0.83) and task discrimination (mean discrimination = 0.5, SD = 0.1). The correlations between test scores and background variables revealed significant correlations with gender, surgical simulator experience, and video game experience (ps < 0.001). These variables, however, explained together only 10% of the variance in test scores.

CONCLUSIONS

We describe the systematic development of an innovative virtual reality test for assessing technical aptitude in candidates for surgical training, and present evidence for its validity, feasibility and acceptability. Further validation is required to support the application of the test for selection, as well as to discern the impact of gender, surgical simulator experience, and video game experience on the fairness of test results. However, the test appears to be a promising tool that may help training programs assess the suitability of candidates for surgical training.

Prediction of laparoscopic skills: objective learning curve analysis

INTRODUCTION

Prediction of proficiency of laparoscopic skills is essential to establish personalized training programs. Objective assessment of laparoscopic skills has been validated in a laparoscopic box trainer with force, motion and time recognition. The aim of this study is to investigate whether acquiring proficiency of laparoscopic skills can be predicted based on performance in such a training box.

METHODS

Surgical residents in their first year of training performed six different tasks in the Lapron box trainer. Force, motion and time data, three objective measures of tissue manipulation and instrument handling, were collected and analyzed for the six different tasks. Linear regression tests were used to predict the learning curve and the number of repetitions required to reach proficiency.

RESULTS

A total of 6010 practice sessions performed by 42 trainees from 13 Dutch hospitals were assessed and included for analysis. Proficiency level was determined as a mean result of seven experts performing 42 trials. Learning curve graphs and prediction models for each task were calculated. A significant relationship between force, motion and time during six different tasks and prediction of proficiency was present in 17 out of 18 analyses.

CONCLUSION

The learning curve of proficiency of laparoscopic skills can accurately be predicted after three repetitions of six tasks in a training box with force, path length and time recognition. This will facilitate personalized training programs in laparoscopic surgery.

Modeling Technical Skills Learning Curves of Incoming Surgical Trainees

OBJECTIVE

Not all trainees reach technical competency even after completing surgical training. While assessment of technical skill is not part of the residency interview process, identifying under-performers early on may help identify opportunities for individualized, targeted training. The objectives of this study were to (1) create predictive learning curve (LC) models for each of 3 basic laparoscopic tasks to identify performers versus underperformers and (2) evaluate the use of LCs to identify underperformers during selection into surgical training.

DESIGN

Predictive LC models were created for laparoscopic pattern cutting (PC), peg transfer (PT) and intra-corporeal knots (IC) over 40 repetitions by 65 novice trainees in 2014. Trainees were categorized as performers and underperformers. Receiver operator characteristic analysis determined the minimum number of repetitions required to predict individual LCs, which were then used to determine the proportion of underperformers.

SETTING

Technical performance was assessed onsite at the Canadian Residence Matching Service (CaRMS) interviews, after interview completion (January 2015).

PARTICIPANTS

Applicants to general surgery (GS) and gynecology (OBGYN) participated in a skills assessment during.

RESULTS

The PC, PT and IC tasks required a minimum of 8, 10, and 5 repetitions respectively, to predict overall performance. Predictive values for each task had excellent sensitivity and specificity: 1.00, 1.00 (PC); 1.00, 1.00 (PT); and 0.94, 1.00 (IC). Eighty applicants completed 8 PC repetitions; 16% were identified as underperformers.

CONCLUSIONS

Individual LCs for three different laparoscopic tasks can be predicted with excellent sensitivity and specificity based on 10 repetitions or less. This information can be used to identify trainees who may have difficulty with laparoscopic technical skills early on.

Using Job Analysis for Identifying the Desired Competencies of 21st-Century Surgeons for Improving Trainees Selection

OBJECTIVE

The current selection for surgical training is based on ineffective methods. In order to identify or to develop more valid selection tools to improve the selection, it is necessary to first define what are the competencies that are most important for success in contemporary surgery. Therefore, the current study aims to identify what competencies are required for success as a surgeon in the 21st-century and to evaluate their relative importance for selection for surgical training.

METHODS

Job analysis was conducted using a mixed-methods design. First, 104 senior surgeons from all surgical fields from various hospitals in Israel were interviewed in order to query their perceptions of competencies associated with success as a surgeon. Their answers were coded and analyzed to create a list of important competencies. Next, a larger sample of 1,102 surgeons and residents from all surgical fields completed a questionnaire in which they rated the importance of each competency in the list for success as a surgeon and for selection for surgical training in the 21st-century.

RESULTS

Twenty-four competencies (five technical skills, six cognitive abilities, 13 personality characteristics) were identified in the interview analysis. Analysis of the questionnaire's data revealed that all 24 competencies were perceived as important for success as a surgeon in the 21st-century as well as for selection for surgical training. The perceived importance of personality characteristics was higher than both cognitive abilities (p < 0.001) and technical skills (p < 0.001). The results did not differ between different surgical fields.

CONCLUSIONS

Twenty-four competencies were identified as important for 21st-century surgeons and for selection for surgical training. Although all competencies were perceived as important, personality characteristics were perceived as more important than technical skills and cognitive abilities. This updated definition of required competencies may aid in developing more valid selection methods of candidates for surgical training.

Visuospatial ability is associated to 2D laparoscopic simulator performance amongst surgical residents

Background

The technical skills of a surgeon influence surgical outcome. Testing technical aptitude at point of recruitment of surgical residents is only conducted in a few countries. This study investigated the impact of visuospatial ability (VSA), background factors, and manual dexterity on performance in two different laparoscopic surgical simulators amongst applicants and 1st year surgical residents.

Method

Applicants from general surgery, pediatric surgery, and urology were included from seven hospitals in Sweden between 2017 and 2021. Some 73 applicants were invited and 50 completed. Participants filled out a background form, and were tested for manual dexterity, and visuospatial ability. Two laparoscopic simulators were used, one 2D video box trainer and one 3D Virtual Reality Simulator.

Results

A significant association was found between the visuospatial ability test and 2D video box laparoscopic performance (95 % CI: 1.028-1.2, p-value <0.01). For every point on the visuospatial test the odds of accomplishing the task increased by 11 %. No association was found between VSA and performance in a laparoscopic VR simulator using 3D vision. No other background factors were associated with performance in the two laparoscopic simulators.

Conclusion

Visuospatial ability in applicants to surgical residency positions is associated to performance in a 2D video box trainer. Knowledge of a resident's visuospatial ability can be used to tailor individualized laparoscopic training programs, and in the future might function as a selection tool concerning laparoscopic ability.

Key message

Visuospatial ability differs greatly amongst applicants for surgical residency and is associated to laparoscopic simulator performance. Testing applicants' visuospatial ability could possibly be used to tailor individualized laparoscopic training programs or in the future as a selection tool concerning laparoscopic ability.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Aptitude and attitude: predictors of performance during and after basic laparoscopic skills training

Background

Manual dexterity and visual-spatial ability are considered key to the development of superior laparoscopic skills. Nevertheless, these abilities do not reliably explain all the variance found in the technical performance of surgical trainees. Consequently, we must look beyond these abilities to improve our understanding of laparoscopic skills and to better identify/develop surgical potential earlier on.

Purpose

To assess the individual and collective impact of physical, cognitive, visual, and psychological variables on performance during and after basic simulation-based laparoscopic skills training.

Method

Thirty-four medical students (laparoscopic novices) completed a proficiency-based laparoscopic skills training program (using either a 2D or 3D viewing mode). This was followed by one testing session, a follow-up testing session with new (yet similar) tasks, and a series of physical, cognitive, visual, and psychological measures.

Results

The statistical models that best predicted variance in training performance metrics included four variables: viewingmode (2D vs 3D), psychologicalflexibility, perceivedtaskdemands, and manualdexterity (bimanual). In subsequent testing, a model that included viewingmode and manualdexterity (assembly) best predicted performance on the pre-practiced tasks. However, for a highly novel, spatially complex laparoscopic task, performance was best predicted by a model that comprised viewingmode, visual-spatialability, and perceivedtaskdemands. At follow-up, manualdexterity (assembly) alone was the best predictor of performance on new (yet similar) tasks.

Conclusion

By focussing exclusively on physical/cognitive abilities, we may overlook other important predictors of surgical performance (e.g. psychological variables). The present findings suggest that laparoscopic performance may be more accurately explained through the combined effects of physical, cognitive, visual, and psychological variables. Further, the results suggest that the predictors may change with both task demands and the development of the trainee. This study highlights the key role of psychological skills in overcoming initial training challenges, with far-reaching implications for practice.

Assessment of technical skills based on learning curve analyses in laparoscopic surgery training

BACKGROUND

Objective force- and motion-based assessment is currently lacking in laparoscopic skills curricula. This study aimed to evaluate the added value of parameter-based assessment and feedback during training.

METHODS

Laparoscopy-naïve surgical residents that took part in a 3-week skills training curriculum were included. A box trainer equipped with the ForceSense system was used for assessment of tissue manipulation- (MaxForce) and instrument-handling skills (Path length and Time). Learning curves were established using linear regression tests. Pre- and post-course comparisons indicated the overall progression and were compared to predefined proficiency levels. A post-course survey was carried out to assess face validity.

RESULTS

In total, 4,268 trials, executed by 24 residents, were successfully assessed. Median (interquartile range) MaxForce outcomes improved from 2.7 Newton (interquartile range 1.9-3.8) to 1.8 Newton (interquartile range 1.2-2.4) between pre- and post-course assessment (P ≤ .009). Instrument Path length improved from 7,102.2 mm (interquartile range 5,255.2-9,025.9) to 3,545.3 mm (interquartile range 2,842.9-4,563.2) (P ≤.001). Time to execute the task improved from 159.8 seconds (interquartile range 119.8-219.0) to 60.7 seconds (interquartile range 46.0-79.5) (P ≤ .001). The learning curves revealed during what training phase the proficiency benchmarks were reached for each trainee. In the survey outcomes, trainees indicated that this curriculum should be part of a surgical residency program (mean visual analog scale score of 9.2 ± 0.9 standard deviation).

CONCLUSION

Force-, motion-, and time-parameters can be objectively measured during basic laparoscopic skills curricula and do indicate progression of skills over time. The ForceSense parameters enable curricula to be designed for specific proficiency-based training goals and offer the possibility for objective classification of the levels of expertise.

Competency-Based Education in Minimally Invasive and Robotic Colorectal Surgery

Minimally invasive and robotic techniques have become increasingly implemented into surgical practice and are now an essential part of the foundational skills of training colorectal surgeons. Over the past 5 years there has been a shift in the surgical educational paradigm toward competency-based education (CBE). CBE recognizes that trainees learn at different rates but regardless, are required to meet a competent threshold of performance prior to independent practice. Thus, CBE attempts to replace the traditional "time" endpoint of training with "performance." Although conceptually sensible, implementing CBE has proven challenging. This article will define competence, outline appropriate assessment tools to assess technical skill, and review the literature on the number of cases required to achieve competence in colorectal procedures while outlining the barriers to implementing CBE.

Innate Arthroscopic & Laparoscopic Surgical Skills: A Systematic Review of Predictive Performance Indicators Within Novice Surgical Trainees

OBJECTIVE

To query the literature for predictive factors for performance on arthroscopic and laparoscopic surgical simulators in novice surgical trainees. These predictive factors may prove a valuable tool in identifying certain innate potential to becoming a future skilled surgeon that could benefit both surgical residency candidates and program directors alike, while also improving patient care.

DESIGN

Systematic Review.

RESULTS

The initial searches yielded 249 articles. After screening titles/abstracts and implementing inclusion and exclusion criteria, 36 studies were included in the final systematic review.

CONCLUSIONS

Current literature suggests that video game experience/frequency, psychomotor and visuospatial aptitude, and perceptual ability are among the most promising predictive indicators of baseline simulator performance. Study limitations include utilization of different standards for characterizing predictive factors. Future studies should aim to utilize standard guidelines for accurate quantification of innate predictive factors. Future research should also focus on utilizing standardized simulator platforms and aptitude tests to allow for more accurate cross-study comparisons and meta-analyses with larger sample sizes.

Brain activation during laparoscopic tasks in high- and low-performing medical students: a pilot fMRI study

BACKGROUND

Up to 20% of medical students are unable to reach competency in laparoscopic surgery. It is unknown whether these difficulties arise from heterogeneity in neurological functioning across individuals. We sought to examine the differences in neurological functioning during laparoscopic tasks between high- and low-performing medical students using functional magnetic resonance imaging (fMRI).

METHODS

This prospective cohort study enrolled North American medical students who were within the top 20% and bottom 20% of laparoscopic performers from a previous study. Brain activation was recorded using fMRI while participants performed peg-pointing, intracorporeal knot tying (IKT), and the Pictorial Surface Orientation (PicSOr) test. Brain activation maps were created and areas of activation were compared between groups.

RESULTS

In total, 9/12 high and 9/13 low performers completed the study. High performers completed IKT faster and made more successful knot ties than low performers [standing: 23.5 (5.0) sec vs. 37.6 (18.4) sec, p = 0.03; supine: 23.2 (2.5) sec vs. 72.7 (62.8) sec, p = 0.02; number of successful ties supine, 3 ties vs. 1 tie, p = 0.01]. Low performers showed more brain activation than high performers in the peg-pointing task (q < 0.01), with no activation differences in the IKT task. There were no behavioral differences in the PiCSOr task.

CONCLUSIONS

This study is the first to show differences between low and high performers of laparoscopic tasks at the brain level. This pilot study has shown the feasibility of using fMRI to examine laparoscopic surgical skills. Future studies are needed for further exploration of our initial findings.

A machine learning approach to predict surgical learning curves

BACKGROUND

Contemporary surgical training programs rely on the repetition of selected surgical motor tasks. Such methodology is inherently open ended with no control on the time taken to attain a set level of proficiency, given the trainees' intrinsic differences in initial skill levels and learning abilities. Hence, an efficient training program should aim at tailoring the surgical training protocols to each trainee. In this regard, a predictive model using information from the initial learning stage to predict learning curve characteristics should facilitate the whole surgical training process.

METHODS

This paper analyzes learning curve data to train a multivariate supervised machine learning model. One factor is extracted to define the trainees' learning ability. An unsupervised machine learning model is also utilized for trainee classification. When established, the model can predict robustly the learning curve characteristics based on the first few trials.

RESULTS

We show that the information present in the first 10 trials of surgical tasks can be utilized to predict the number of trials required to achieve proficiency (R²=0.72) and the final performance level (R²=0.89). Furthermore, only a single factor, learning index, is required to describe the learning process and to classify learners with unique learning characteristics.

CONCLUSION

Using machine learning models, we show, for the first time, that the first few trials contain sufficient information to predict learning curve characteristics and that a single factor can capture the complex learning behavior. Using such models holds the potential for personalization of training regimens, leading to greater efficiency and lower costs.

The "Twitching Technique": A New Space-Irrespective Laparoscopic Ligation Technique Using a JAiMY Needle Holder

Background: Laparoscopic intracorporeal sutures and knots require advanced techniques in children. The JAiMY^® needle holder (Endocontrol Company, Grenoble, France), a flexible jaw with both clockwise and anticlockwise revolving functions, enables the placement of sutures in very small working spaces. Using this tool, a novel new ligation method, which we named "twitching technique," can be performed. Methods: To perform the "twitching technique," first grasp the long tail of the thread with the JAiMY and the forceps on the needle side. Then, rotate the tip of the needle holder halfway in either direction and bring the forceps closer to the needle holder to make a ring using the thread. Grasp the thread at the crossing point with the forceps, and then catch the short tail through the ring with the needle holder. Results (Case Presentation): A 1-year-old boy presented with vomiting because of esophageal hiatal hernia. He was being treated for single ventricle and asplenia at our hospital. Laparoscopic esophageal hiatus plication was performed. Although the working space was extremely limited, the JAiMY and the "twitching technique" enabled steady suture placement and ligation. Conclusions: The "twitching technique" is an easy, steady, and safe method for performing ligation in difficult situations and may facilitate the performance of many laparoscopic surgeries in pediatric fields.

Striving for Better Medical Education: the Simulation Approach

Medical simulation is a rapidly expanding area within medical education due to advances in technology, significant reduction in training hours and increased procedural complexity. Simulation training aims to enhance patient safety through improved technical competency and eliminating human factors in a risk free environment. It is particularly applicable to a practical, procedure-orientated specialties.

Simulation can be useful for novice trainees, experienced clinicians (e.g. for revalidation) and team building. It has become a cornerstone in the delivery of medical education, being a paradigm shift in how doctors are educated and trained. Simulation must take a proactive position in the development of metric-based simulation curriculum, adoption of proficiency benchmarking definitions, and should not depend on the simulation platforms used.

Conversely, ingraining of poor practice may occur in the absence of adequate supervision, and equipment malfunction during the simulation can break the immersion and disrupt any learning that has occurred. Despite the presence of high technology, there is a substantial learning curve for both learners and facilitators.

The technology of simulation continues to advance, offering devices capable of improved fidelity in virtual reality simulation, more sophisticated procedural practice and advanced patient simulators. Simulation-based training has also brought about paradigm shifts in the medical and surgical education arenas and ensured that the scope and impact of simulation will continue to broaden.