Supervised classification of psychomotor competence in minimally invasive surgery based on instruments motion analysis

Modern artificial intelligence and large language models in graduate medical education: a scoping review of attitudes, applications & practice

BACKGROUND

Artificial intelligence (AI) holds transformative potential for graduate medical education (GME), yet, a comprehensive exploration of AI's applications, perceptions, and limitations in GME is lacking.

OBJECTIVE

To map the current literature on AI in GME, identifying prevailing perceptions, applications, and research gaps to inform future research, policy discussions, and educational practices through a scoping review.

METHODS

Following the Joanna Briggs Institute guidelines and the PRISMA-ScR checklist a comprehensive search of multiple databases up to February 2024 was performed to include studies addressing AI interventions in GME.

RESULTS

Out of 1734 citations, 102 studies met the inclusion criteria, conducted across 16 countries, predominantly from North America (72), Asia (14), and Europe (6). Radiology had the highest number of publications (21), followed by general surgery (11) and emergency medicine (8). The majority of studies were published in 2023. Several key thematic areas emerged from the literature. Initially, perceptions of AI in graduate medical education (GME) were mixed, but have increasingly shifted toward a more favorable outlook, particularly as the benefits of AI integration in education become more apparent. In assessments, AI demonstrated the ability to differentiate between skill levels and offer meaningful feedback. It has also been effective in evaluating narrative comments to assess resident performance. In the domain of recruitment, AI tools have been applied to analyze letters of recommendation, applications, and personal statements, helping identify potential biases and improve equity in candidate selection. Furthermore, large language models consistently outperformed average candidates on board certification and in-training examinations, indicating their potential utility in standardized assessments. Finally, AI tools showed promise in enhancing clinical decision-making by supporting trainees with improved diagnostic accuracy and efficiency.

CONCLUSIONS

This scoping review provides a comprehensive overview of applications and limitations of AI in GME but is limited with potential biases, study heterogeneity, and evolving nature of AI.

Estimating Cognitive Workload Using Task-Related Pupillary Responses in Simulated Drilling in Cochlear Implantation

OBJECTIVES

Training of temporal bone drilling requires more than mastering technical skills with the drill. Skills such as visual imagery, bimanual dexterity, and stress management need to be mastered along with precise knowledge of anatomy. In otorhinolaryngology, these psychomotor skills underlie performance in the drilling of the temporal bone for access to the inner ear in cochlear implant surgery. However, little is known about how psychomotor skills and workload management impact the practitioners' continuous and overall performance.

METHODS

To understand how the practitioner's workload and performance unfolds over time, we examine task-evoked pupillary responses (TEPR) of 22 medical students who performed transmastoid-posterior tympanotomy (TMPT) and removal of the bony overhang of the round window niche in a 3D-printed model of the temporal bone. We investigate how students' TEPR metrics (Average Pupil Size [APS], Index of Pupil Activity [IPA], and Low/High Index of Pupillary Activity [LHIPA]) and time spent in drilling phases correspond to the performance in key drilling phases.

RESULTS

All TEPR measures revealed significant differences between key drilling phases that corresponded to the anticipated workload. Enlarging the facial recess lasted significantly longer than other phases. IPA captured significant increase of workload in thinning of the posterior canal wall, while APS revealed increased workload during the drilling of the bony overhang.

CONCLUSION

Our findings contribute to the contemporary competency-based medical residency programs where objective and continuous monitoring of participants' progress allows to track progress in expertise acquisition. Laryngoscope, 134:5087-5095, 2024.

Craniotomy Simulator with Force Myography and Machine Learning-Based Skills Assessment

Craniotomy is a fundamental component of neurosurgery that involves the removal of the skull bone flap. Simulation-based training of craniotomy is an efficient method to develop competent skills outside the operating room. Traditionally, an expert surgeon evaluates the surgical skills using rating scales, but this method is subjective, time-consuming, and tedious. Accordingly, the objective of the present study was to develop an anatomically accurate craniotomy simulator with realistic haptic feedback and objective evaluation of surgical skills. A CT scan segmentation-based craniotomy simulator with two bone flaps for drilling task was developed using 3D printed bone matrix material. Force myography (FMG) and machine learning were used to automatically evaluate the surgical skills. Twenty-two neurosurgeons participated in this study, including novices (n = 8), intermediates (n = 8), and experts (n = 6), and they performed the defined drilling experiments. They provided feedback on the effectiveness of the simulator using a Likert scale questionnaire on a scale ranging from 1 to 10. The data acquired from the FMG band was used to classify the surgical expertise into novice, intermediate and expert categories. The study employed naïve Bayes, linear discriminant (LDA), support vector machine (SVM), and decision tree (DT) classifiers with leave one out cross-validation. The neurosurgeons' feedback indicates that the developed simulator was found to be an effective tool to hone drilling skills. In addition, the bone matrix material provided good value in terms of haptic feedback (average score 7.1). For FMG-data-based skills evaluation, we achieved maximum accuracy using the naïve Bayes classifier (90.0 ± 14.8%). DT had a classification accuracy of 86.22 ± 20.8%, LDA had an accuracy of 81.9 ± 23.6%, and SVM had an accuracy of 76.7 ± 32.9%. The findings of this study indicate that materials with comparable biomechanical properties to those of real tissues are more effective for surgical simulation. In addition, force myography and machine learning provide objective and automated assessment of surgical drilling skills.

Objective psychomotor laparoscopic skills evaluation using a low-cost wearable device based on accelerometry: construct and concurrent validity study

BACKGROUND

Motion analysis of surgical maneuvers provides useful quantitative information for the objective evaluation of the surgeons. However, surgical simulation laboratories for laparoscopic training do not usually integrate devices that help quantify the level of skills of the surgeons due to their limited resources and the high costs of new technologies. The purpose of this study is to present the construct and concurrent validity of a low-cost motion tracking system, based on a wireless triaxial accelerometer, employed to objectively evaluate psychomotor skills of surgeons during laparoscopic training.

METHODS

An accelerometry system, a wireless three-axis accelerometer with appearance of wristwatch, was placed on the dominant hand of the surgeons to register the motion during the laparoscopy practice with the EndoViS simulator, which simultaneously recorded the motion of the laparoscopic needle driver. This study included the participation of 30 surgeons (6 experts, 14 intermediates and 10 novices) who performed the task of intracorporeal knot-tying suture. Using 11 motion analysis parameters (MAPs), the performance of each participant was assessed. Subsequently, the scores of the three groups of surgeons were statistically analyzed. In addition, a validity study was conducted comparing the metrics between the accelerometry-tracking system and the EndoViS hybrid simulator.

RESULTS

Construct validity was achieved for 8 of the 11 metrics examined with the accelerometry system. Concurrent validity demonstrated that there is a strong correlation between the results of the accelerometry system and the EndoViS simulator in 9 of 11 parameters, showing reliability of the accelerometry system as an objective evaluation method.

CONCLUSION

The accelerometry system was successfully validated. This method is potentially useful to complement the objective evaluation of surgeons during laparoscopic practice in training environments such as box-trainers and simulators.

Face, content, and construct validity of the virtual immersive operating room simulator for training laparoscopic procedures

BACKGROUND

The aim of this work is to present the face, content, and construct validation of the virtual immersive operating room simulator (VIORS) for procedural training of surgeons' laparoscopic psychomotor skills and evaluate the immersive training experience.

METHODS

The VIORS simulator consists of an HMD Oculus Rift 2016 with a visor on a 1080 × 1200 pixel OLED screen, two positioning sensors with two adapted controls to simulate laparoscopic instruments, and an acrylic base to simulate the conventional laparoscopic setup. The immersion consists of a 360° virtual operating room environment, based on the EndoSuite at Hospital Infantil de Mexico Federico Gomez, which reproduces a configuration of equipment, instruments, and common distractions in the operating room during a laparoscopic cholecystectomy procedure. Forty-five surgeons, residents, and medicine students participated in this study: 27 novices, 13 intermediates, and 5 experts. They completed a questionnaire on the realism and operating room immersion, as well as their capabilities for laparoscopic procedural training, scored in the 5-point Likert scale. The data of instrument movement were recorded and analyzed using 13 movement analysis parameters (MAPs). The experience during training with VIORS was evaluated through NASA-TLX.

RESULTS

The participants were enthusiastic about the immersion and sensation levels of the VIORS simulator, with positive scores on the realism and its capabilities for procedural training using VIORS. The results proved that the VIORS simulator was able to differentiate between surgeons with different skill levels. Statistically significant differences were found in nine MAPs, demonstrating their construct validity for the objective assessment of the procedural laparoscopic performance. At cognitive level, the inversion experience proves a moderate mental workload when the laparoscopic procedure is carried out.

CONCLUSION

The VIORS simulator has been successfully presented and validated. The VIORS simulator is a useful and effective device for the training of procedural laparoscopic psychomotor skills.

Automatic assessment of laparoscopic surgical skill competence based on motion metrics

The purpose of this study was to characterize the motion features of surgical devices associated with laparoscopic surgical competency and build an automatic skill-credential system in porcine cadaver organ simulation training. Participants performed tissue dissection around the aorta, dividing vascular pedicles after applying Hem-o-lok (tissue dissection task) and parenchymal closure of the kidney (suturing task). Movements of surgical devices were tracked by a motion capture (Mocap) system, and Mocap-metrics were compared according to the level of surgical experience (experts: ≥50 laparoscopic surgeries, intermediates: 10-49, novices: 0-9), using the Kruskal-Wallis test and principal component analysis (PCA). Three machine-learning algorithms: support vector machine (SVM), PCA-SVM, and gradient boosting decision tree (GBDT), were utilized for discrimination of the surgical experience level. The accuracy of each model was evaluated by nested and repeated k-fold cross-validation. A total of 32 experts, 18 intermediates, and 20 novices participated in the present study. PCA revealed that efficiency-related metrics (e.g., path length) significantly contributed to PC 1 in both tasks. Regarding PC 2, speed-related metrics (e.g., velocity, acceleration, jerk) of right-hand devices largely contributed to the tissue dissection task, while those of left-hand devices did in the suturing task. Regarding the three-group discrimination, in the tissue dissection task, the GBDT method was superior to the other methods (median accuracy: 68.6%). In the suturing task, SVM and PCA-SVM methods were superior to the GBDT method (57.4 and 58.4%, respectively). Regarding the two-group discrimination (experts vs. intermediates/novices), the GBDT method resulted in a median accuracy of 72.9% in the tissue dissection task, and, in the suturing task, the PCA-SVM method resulted in a median accuracy of 69.2%. Overall, the mocap-based credential system using machine-learning classifiers provides a correct judgment rate of around 70% (two-group discrimination). Together with motion analysis and wet-lab training, simulation training could be a practical method for objectively assessing the surgical competence of trainees.

Exploring the roles of artificial intelligence in surgical education: A scoping review

BACKGROUND

Technology-enhanced teaching and learning, including Artificial Intelligence (AI) applications, has started to evolve in surgical education. Hence, the purpose of this scoping review is to explore the current and future roles of AI in surgical education.

METHODS

Nine bibliographic databases were searched from January 2010 to January 2021. Full-text articles were included if they focused on AI in surgical education.

RESULTS

Out of 14,008 unique sources of evidence, 93 were included. Out of 93, 84 were conducted in the simulation setting, and 89 targeted technical skills. Fifty-six studies focused on skills assessment/classification, and 36 used multiple AI techniques. Also, increasing sample size, having balanced data, and using AI to provide feedback were major future directions mentioned by authors.

CONCLUSIONS

AI can help optimize the education of trainees and our results can help educators and researchers identify areas that need further investigation.

Development and Validation of a Novel Methodological Pipeline to Integrate Neuroimaging and Photogrammetry for Immersive 3D Cadaveric Neurosurgical Simulation

Background

Visualizing and comprehending 3-dimensional (3D) neuroanatomy is challenging. Cadaver dissection is limited by low availability, high cost, and the need for specialized facilities. New technologies, including 3D rendering of neuroimaging, 3D pictures, and 3D videos, are filling this gap and facilitating learning, but they also have limitations. This proof-of-concept study explored the feasibility of combining the spatial accuracy of 3D reconstructed neuroimaging data with realistic texture and fine anatomical details from 3D photogrammetry to create high-fidelity cadaveric neurosurgical simulations.

Methods

Four fixed and injected cadaver heads underwent neuroimaging. To create 3D virtual models, surfaces were rendered using magnetic resonance imaging (MRI) and computed tomography (CT) scans, and segmented anatomical structures were created. A stepwise pterional craniotomy procedure was performed with synchronous neuronavigation and photogrammetry data collection. All points acquired in 3D navigational space were imported and registered in a 3D virtual model space. A novel machine learning-assisted monocular-depth estimation tool was used to create 3D reconstructions of 2-dimensional (2D) photographs. Depth maps were converted into 3D mesh geometry, which was merged with the 3D virtual model’s brain surface anatomy to test its accuracy. Quantitative measurements were used to validate the spatial accuracy of 3D reconstructions of different techniques.

Results

Successful multilayered 3D virtual models were created using volumetric neuroimaging data. The monocular-depth estimation technique created qualitatively accurate 3D representations of photographs. When 2 models were merged, 63% of surface maps were perfectly matched (mean [SD] deviation 0.7 ± 1.9 mm; range −7 to 7 mm). Maximal distortions were observed at the epicenter and toward the edges of the imaged surfaces. Virtual 3D models provided accurate virtual measurements (margin of error <1.5 mm) as validated by cross-measurements performed in a real-world setting.

Conclusion

The novel technique of co-registering neuroimaging and photogrammetry-based 3D models can (1) substantially supplement anatomical knowledge by adding detail and texture to 3D virtual models, (2) meaningfully improve the spatial accuracy of 3D photogrammetry, (3) allow for accurate quantitative measurements without the need for actual dissection, (4) digitalize the complete surface anatomy of a cadaver, and (5) be used in realistic surgical simulations to improve neurosurgical education.

Objective evaluation of laparoscopic surgical skills in wet lab training based on motion analysis and machine learning

BACKGROUND

Our aim was to build a skill assessment system, providing objective feedback to trainees based on the motion metrics of laparoscopic surgical instruments.

METHODS

Participants performed tissue dissection around the aorta (tissue dissection task) and renal parenchymal closure (parenchymal-suturing task), using swine organs in a box trainer under a motion capture (Mocap) system. Two experts assessed the recorded movies, according to the formula of global operative assessment of laparoscopic skills (GOALS: score range, 5-25), and the mean scores were utilized as objective variables in the regression analyses. The correlations between mean GOALS scores and Mocap metrics were evaluated, and potential Mocap metrics with a Spearman's rank correlation coefficient value exceeding 0.4 were selected for each GOALS item estimation. Four regression algorithms, support vector regression (SVR), principal component analysis (PCA)-SVR, ridge regression, and partial least squares regression, were utilized for automatic GOALS estimation. Model validation was conducted by nested and repeated k-fold cross validation, and the mean absolute error (MAE) was calculated to evaluate the accuracy of each regression model.

RESULTS

Forty-five urologic, 9 gastroenterological, and 3 gynecologic surgeons, 4 junior residents, and 9 medical students participated in the training. In both tasks, a positive correlation was observed between the speed-related parameters (e.g., velocity, velocity range, acceleration, jerk) and mean GOALS scores, with a negative correlation between the efficiency-related parameters (e.g., task time, path length, number of opening/closing operations) and mean GOALS scores. Among the 4 algorithms, SVR showed the highest accuracy in the tissue dissection task ([Formula: see text]), and PCA-SVR in the parenchymal-suturing task ([Formula: see text]), based on 100 iterations of the validation process of automatic GOALS estimation.

CONCLUSION

We developed a machine learning-based GOALS scoring system in wet lab training, with an error of approximately 1-2 points for the total score, and motion metrics that were explainable to trainees. Our future challenges are the further improvement of onsite GOALS feedback, exploring the educational benefit of our model and building an efficient training program.

Machine learning for technical skill assessment in surgery: a systematic review

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.PROSPERO: CRD42020226071.

Artificial Intelligence and Surgical Education: A Systematic Scoping Review of Interventions

OBJECTIVE

To synthesize peer-reviewed evidence related to the use of artificial intelligence (AI) in surgical education DESIGN: We conducted and reported a scoping review according to the standards outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis with extension for Scoping Reviews guideline and the fourth edition of the Joanna Briggs Institute Reviewer's Manual. We systematically searched eight interdisciplinary databases including MEDLINE-Ovid, ERIC, EMBASE, CINAHL, Web of Science: Core Collection, Compendex, Scopus, and IEEE Xplore. Databases were searched from inception until the date of search on April 13, 2021.

SETTING/PARTICIPANTS

We only examined original, peer-reviewed interventional studies that self-described as AI interventions, focused on medical education, and were relevant to surgical trainees (defined as medical or dental students, postgraduate residents, or surgical fellows) within the title and abstract (see Table 2). Animal, cadaveric, and in vivo studies were not eligible for inclusion.

RESULTS

After systematically searching eight databases and 4255 citations, our scoping review identified 49 studies relevant to artificial intelligence in surgical education. We found diverse interventions related to the evaluation of surgical competency, personalization of surgical education, and improvement of surgical education materials across surgical specialties. Many studies used existing surgical education materials, such as the Objective Structured Assessment of Technical Skills framework or the JHU-ISI Gesture and Skill Assessment Working Set database. Though most studies did not provide outcomes related to the implementation in medical schools (such as cost-effective analyses or trainee feedback), there are numerous promising interventions. In particular, many studies noted high accuracy in the objective characterization of surgical skill sets. These interventions could be further used to identify at-risk surgical trainees or evaluate teaching methods.

CONCLUSIONS

There are promising applications for AI in surgical education, particularly for the assessment of surgical competencies, though further evidence is needed regarding implementation and applicability.

Years of experience is more effective in defining experts in the gaze analysis of laparoscopic suturing task than task duration

In this study, the relationship between gaze patterns, task duration, and years of experience, which are commonly used to define and evaluate expert surgeons in laparoscopic surgery, was identified. Ten surgeons with 1-28 years of experience and six inexperienced students were included. Subjects used forceps to repeat the task of suturing a suture pad. Each subject wore an eye-marking recorder, and gaze points were recorded and analyzed. No significant relationship between task duration and gaze pattern was observed. However, there was a significant relationship between a surgeon's years of experience and the percentage of time spent gazing at the forceps. Subjects with more years of experience operated without looking at the forceps and fixed their gaze on the operational target. Therefore, when analyzing laparoscopic gazing patterns, it may be more appropriate to define an "expert" based on the years of experience rather than task duration.

Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review

The assessment of surgical technical skills to be acquired by novice surgeons has been traditionally done by an expert surgeon and is therefore of a subjective nature. Nevertheless, the recent advances on IoT (Internet of Things), the possibility of incorporating sensors into objects and environments in order to collect large amounts of data, and the progress on machine learning are facilitating a more objective and automated assessment of surgical technical skills. This paper presents a systematic literature review of papers published after 2013 discussing the objective and automated assessment of surgical technical skills. 101 out of an initial list of 537 papers were analyzed to identify: 1) the sensors used; 2) the data collected by these sensors and the relationship between these data, surgical technical skills and surgeons' levels of expertise; 3) the statistical methods and algorithms used to process these data; and 4) the feedback provided based on the outputs of these statistical methods and algorithms. Particularly, 1) mechanical and electromagnetic sensors are widely used for tool tracking, while inertial measurement units are widely used for body tracking; 2) path length, number of sub-movements, smoothness, fixation, saccade and total time are the main indicators obtained from raw data and serve to assess surgical technical skills such as economy, efficiency, hand tremor, or mind control, and distinguish between two or three levels of expertise (novice/intermediate/advanced surgeons); 3) SVM (Support Vector Machines) and Neural Networks are the preferred statistical methods and algorithms for processing the data collected, while new opportunities are opened up to combine various algorithms and use deep learning; and 4) feedback is provided by matching performance indicators and a lexicon of words and visualizations, although there is considerable room for research in the context of feedback and visualizations, taking, for example, ideas from learning analytics.

Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches

BACKGROUND

The determination of surgeons' psychomotor skills in minimally invasive surgery techniques is one of the major concerns of the programs of surgical training in several hospitals. Therefore, it is important to assess and classify objectively the level of experience of surgeons and residents during their training process. The aim of this study was to investigate three classification methods for establishing automatically the level of surgical competence of the surgeons based on their psychomotor laparoscopic skills.

METHODS

A total of 43 participants, divided into an experienced surgeons group with ten experts (> 100 laparoscopic procedures performed) and non-experienced surgeons group with 24 residents and nine medical students (< 10 laparoscopic procedures performed), performed three tasks in the EndoViS training system. Motion data of the instruments were captured with a video-tracking system built into the EndoViS simulator and analyzed using 13 motion analysis parameters (MAPs). Radial basis function networks (RBFNets), K-star (K*), and random forest (RF) were used for classifying surgeons based on the MAPs' scores of all participants. The performance of the three classifiers was examined using hold-out and leave-one-out validation techniques.

RESULTS

For all three tasks, the K-star method was superior in terms of accuracy and AUC in both validation techniques. The mean accuracy of the classifiers was 93.33% for K-star, 87.58% for RBFNets, and 84.85% for RF in hold-out validation, and 91.47% for K-star, 89.92% for RBFNets, and 83.72% for RF in leave-one-out cross-validation.

CONCLUSIONS

The three proposed methods demonstrated high performance in the classification of laparoscopic surgeons, according to their level of psychomotor skills. Together with motion analysis and three laparoscopic tasks of the Fundamental Laparoscopic Surgery Program, these classifiers provide a means for objectively classifying surgical competence of the surgeons for existing laparoscopic box trainers.

The Effect of Training, Used-Hand, and Experience on Endoscopic Surgery Skills in an Educational Computer-Based Simulation Environment (ECE) for Endoneurosurgery Training

Today, virtual simulation environments create alternative hands-on practice opportunities for surgical training. In order to increase the potential benefits of such environments, it is critical to understand the factors that influence them. This study was conducted to determine the effects of training, used-hand, and experience, as well as the interactions between these variables, on endoscopic surgery skills in an educational computer-based surgical simulation environment. A 2-hour computer-based endoneurosurgery simulation training module was developed for this study. Thirty-one novice- and intermediate-level resident surgeons from the departments of neurosurgery and ear, nose, and throat participated in this experimental study. The results suggest that a 2-hour training during a 2-month period through computer-based simulation environment improves the surgical skills of the residents in both-hand tasks, which is necessary for endoscopic surgical procedures but not in dominant hand tasks. Based on the results of this study, it can be concluded that computer-based simulation environments potentially improve surgical skills; however, the scenarios for such training modules need to consider especially the bimanual coordination of hands and should be regularly adapted to the individual skill levels and progresses.

Machine learning methods for automated technical skills assessment with instructional feedback in ultrasound-guided interventions

OBJECTIVE

Currently, there is a worldwide shift toward competency-based medical education. This necessitates the use of automated skills assessment methods during self-guided interventions training. Making assessment methods that are transparent and configurable will allow assessment to be interpreted into instructional feedback. The purpose of this work is to develop and validate skills assessment methods in ultrasound-guided interventions that are transparent and configurable.

METHODS

We implemented a method based upon decision trees and a method based upon fuzzy inference systems for technical skills assessment. Subsequently, we validated these methods for their ability to predict scores of operators on a 25-point global rating scale in ultrasound-guided needle insertions and their ability to provide useful feedback for training.

RESULTS

Decision tree and fuzzy rule-based assessment performed comparably to state-of-the-art assessment methods. They produced median errors (on a 25-point scale) of 1.7 and 1.8 for in-plane insertions and 1.5 and 3.0 for out-of-plane insertions, respectively. In addition, these methods provided feedback that was useful for trainee learning. Decision tree assessment produced feedback with median usefulness 7 out of 7; fuzzy rule-based assessment produced feedback with median usefulness 6 out of 7.

CONCLUSION

Transparent and configurable assessment methods are comparable to the state of the art and, in addition, can provide useful feedback. This demonstrates their value in self-guided interventions training curricula.

Using Machine Learning to Assess Physician Competence: A Systematic Review

PURPOSE

To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties.

METHOD

In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students', residents', fellows', or attending physicians' competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted.

RESULTS

Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains.

CONCLUSIONS

A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.

Eye-Hand Coordination Patterns of Intermediate and Novice Surgeons in a Simulation-Based Endoscopic Surgery Training Environment

Endoscopic surgery procedures require specific skills, such as eye-hand coordination to be developed. Current education programs are facing with problems to provide appropriate skill improvement and assessment methods in this field. This study aims to propose objec-tive metrics for hand-movement skills and assess eye-hand coordination. An experimental study is conducted with 15 surgical residents to test the newly proposed measures. Two computer-based both-handed endoscopic surgery practice scenarios are developed in a simulation environment to gather the participants' eye-gaze data with the help of an eye tracker as well as the related hand movement data through haptic interfaces. Additionally, participants' eye-hand coordination skills are analyzed. The results indicate higher correla-tions in the intermediates' eye-hand movements compared to the novices. An increase in intermediates' visual concentration leads to smoother hand movements. Similarly, the novices' hand movements are shown to remain at a standstill. After the first round of practice, all participants' eye-hand coordination skills are improved on the specific task targeted in this study. According to these results, it can be concluded that the proposed metrics can potentially provide some additional insights about trainees' eye-hand coordi-nation skills and help instructional system designers to better address training requirements.

Force-based learning curve tracking in fundamental laparoscopic skills training

BACKGROUND

Within minimally invasive surgery (MIS), structural implementation of courses and structured assessment of skills are challenged by availability of trainers, time, and money. We aimed to establish and validate an objective measurement tool for preclinical skills acquisition in a basic laparoscopic at-home training program.

METHODS

A mobile laparoscopic simulator was equipped with a state-of-the-art force, motion, and time tracking system (ForceSense, MediShield B.V., Delft, the Netherlands). These performance parameters respectively representing tissue manipulation and instrument handling were continuously tracked during every trial. Proficiency levels were set by clinical experts for six different training tasks. Resident's acquisition and development of fundamental skills were evaluated by comparing pre- and post-course assessment measurements and OSATS forms. A questionnaire was distributed to determine face and content validity.

RESULTS

Out of 1842 captured attempts by novices, 1594 successful trials were evaluated. A decrease in maximum exerted absolute force was shown in comparison of four training tasks (p ≤ 0.023). Three of the six comparisons also showed lower mean forces during tissue manipulation (p ≤ 0.024). Lower instrument handling outcomes (i.e., time and motion parameters) were observed in five tasks (resp. (p ≤ 0.019) and (p ≤ 0.025)). Simultaneously, all OSATS scores increased (p ≤ 0.028). Proficiency levels for all tasks can be reached in 2 weeks of at home training.

CONCLUSIONS

Monitoring force, motion, and time parameters during training showed to be effective in determining acquisition and development of basic laparoscopic tissue manipulation and instrument handling skills. Therefore, we were able to gain insight into the amount of training needed to reach certain levels of competence. Skills improved after sufficient amount of training at home. Questionnaire outcomes indicated that skills and self-confidence improved and that this training should therefore be part of the regular residency training program.

Interpretation of motion analysis of laparoscopic instruments based on principal component analysis in box trainer settings

BACKGROUND

Motion analysis parameters (MAPs) have been extensively validated for assessment of minimally invasive surgical skills. However, there are discrepancies on how specific MAPs, tasks, and skills match with each other, reflecting that motion analysis cannot be generalized independently of the learning outcomes of a task. Additionally, there is a lack of knowledge on the meaning of motion analysis in terms of surgical skills, making difficult the provision of meaningful, didactic feedback. In this study, new higher significance MAPs (HSMAPs) are proposed, validated, and discussed for the assessment of technical skills in box trainers, based on principal component analysis (PCA).

METHODS

Motion analysis data were collected from 25 volunteers performing three box trainer tasks (peg grasping/PG, pattern cutting/PC, knot suturing/KS) using the EVA tracking system. PCA was applied on 10 MAPs for each task and hand. Principal components were trimmed to those accounting for an explained variance > 80% to define the HSMAPs. Individual contributions of MAPs to HSMAPs were obtained by loading analysis and varimax rotation. Construct validity of the new HSMAPs was carried out at two levels of experience based on number of surgeries.

RESULTS

Three new HSMAPs per hand were defined for PG and PC tasks, and two per hand for KS task. PG presented validity for HSMAPs related to insecurity and economy of space. PC showed validity for HSMAPs related to cutting efficacy, peripheral unawareness, and confidence. Finally, KS presented validity for HSMAPs related with economy of space and knotting security.

CONCLUSIONS

PCA-defined HSMAPs can be used for technical skills' assessment. Construct validation and expert knowledge can be combined to infer how competences are acquired in box trainer tasks. These findings can be exploited to provide residents with meaningful feedback on performance. Future works will compare the new HSMAPs with valid scoring systems such as GOALS.