How to objectively classify residents based on their psychomotor laparoscopic skills?

Objective psychomotor laparoscopic skills evaluation using a low-cost wearable device based on accelerometry: construct and concurrent validity study

BACKGROUND

Motion analysis of surgical maneuvers provides useful quantitative information for the objective evaluation of the surgeons. However, surgical simulation laboratories for laparoscopic training do not usually integrate devices that help quantify the level of skills of the surgeons due to their limited resources and the high costs of new technologies. The purpose of this study is to present the construct and concurrent validity of a low-cost motion tracking system, based on a wireless triaxial accelerometer, employed to objectively evaluate psychomotor skills of surgeons during laparoscopic training.

METHODS

An accelerometry system, a wireless three-axis accelerometer with appearance of wristwatch, was placed on the dominant hand of the surgeons to register the motion during the laparoscopy practice with the EndoViS simulator, which simultaneously recorded the motion of the laparoscopic needle driver. This study included the participation of 30 surgeons (6 experts, 14 intermediates and 10 novices) who performed the task of intracorporeal knot-tying suture. Using 11 motion analysis parameters (MAPs), the performance of each participant was assessed. Subsequently, the scores of the three groups of surgeons were statistically analyzed. In addition, a validity study was conducted comparing the metrics between the accelerometry-tracking system and the EndoViS hybrid simulator.

RESULTS

Construct validity was achieved for 8 of the 11 metrics examined with the accelerometry system. Concurrent validity demonstrated that there is a strong correlation between the results of the accelerometry system and the EndoViS simulator in 9 of 11 parameters, showing reliability of the accelerometry system as an objective evaluation method.

CONCLUSION

The accelerometry system was successfully validated. This method is potentially useful to complement the objective evaluation of surgeons during laparoscopic practice in training environments such as box-trainers and simulators.

Face, content, and construct validity of the virtual immersive operating room simulator for training laparoscopic procedures

BACKGROUND

The aim of this work is to present the face, content, and construct validation of the virtual immersive operating room simulator (VIORS) for procedural training of surgeons' laparoscopic psychomotor skills and evaluate the immersive training experience.

METHODS

The VIORS simulator consists of an HMD Oculus Rift 2016 with a visor on a 1080 × 1200 pixel OLED screen, two positioning sensors with two adapted controls to simulate laparoscopic instruments, and an acrylic base to simulate the conventional laparoscopic setup. The immersion consists of a 360° virtual operating room environment, based on the EndoSuite at Hospital Infantil de Mexico Federico Gomez, which reproduces a configuration of equipment, instruments, and common distractions in the operating room during a laparoscopic cholecystectomy procedure. Forty-five surgeons, residents, and medicine students participated in this study: 27 novices, 13 intermediates, and 5 experts. They completed a questionnaire on the realism and operating room immersion, as well as their capabilities for laparoscopic procedural training, scored in the 5-point Likert scale. The data of instrument movement were recorded and analyzed using 13 movement analysis parameters (MAPs). The experience during training with VIORS was evaluated through NASA-TLX.

RESULTS

The participants were enthusiastic about the immersion and sensation levels of the VIORS simulator, with positive scores on the realism and its capabilities for procedural training using VIORS. The results proved that the VIORS simulator was able to differentiate between surgeons with different skill levels. Statistically significant differences were found in nine MAPs, demonstrating their construct validity for the objective assessment of the procedural laparoscopic performance. At cognitive level, the inversion experience proves a moderate mental workload when the laparoscopic procedure is carried out.

CONCLUSION

The VIORS simulator has been successfully presented and validated. The VIORS simulator is a useful and effective device for the training of procedural laparoscopic psychomotor skills.

Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches

BACKGROUND

The determination of surgeons' psychomotor skills in minimally invasive surgery techniques is one of the major concerns of the programs of surgical training in several hospitals. Therefore, it is important to assess and classify objectively the level of experience of surgeons and residents during their training process. The aim of this study was to investigate three classification methods for establishing automatically the level of surgical competence of the surgeons based on their psychomotor laparoscopic skills.

METHODS

A total of 43 participants, divided into an experienced surgeons group with ten experts (> 100 laparoscopic procedures performed) and non-experienced surgeons group with 24 residents and nine medical students (< 10 laparoscopic procedures performed), performed three tasks in the EndoViS training system. Motion data of the instruments were captured with a video-tracking system built into the EndoViS simulator and analyzed using 13 motion analysis parameters (MAPs). Radial basis function networks (RBFNets), K-star (K*), and random forest (RF) were used for classifying surgeons based on the MAPs' scores of all participants. The performance of the three classifiers was examined using hold-out and leave-one-out validation techniques.

RESULTS

For all three tasks, the K-star method was superior in terms of accuracy and AUC in both validation techniques. The mean accuracy of the classifiers was 93.33% for K-star, 87.58% for RBFNets, and 84.85% for RF in hold-out validation, and 91.47% for K-star, 89.92% for RBFNets, and 83.72% for RF in leave-one-out cross-validation.

CONCLUSIONS

The three proposed methods demonstrated high performance in the classification of laparoscopic surgeons, according to their level of psychomotor skills. Together with motion analysis and three laparoscopic tasks of the Fundamental Laparoscopic Surgery Program, these classifiers provide a means for objectively classifying surgical competence of the surgeons for existing laparoscopic box trainers.

Measuring fatigue and stress in laparoscopic surgery: validity and reliability of the star-track test

BACKGROUND

The star-track test has been assessed as valid and reliable to measure manual dexterity in the context of open surgery. We aimed to determine the construct validity and test-retest reliability of the star-track test for manual dexterity in a laparoscopic setting.

MATERIAL AND METHODS

The star-track test was performed in a laparoscopic box trainer. To determine construct validity an open-label, randomized four-period crossover trial was conducted. Alongside a baseline (non-interventional) measurement, interventions involved: physical fatigue, mental stress and a combination of these. The test-retest trial involved two separate (non-interventional) measurements. The primary outcome measures were accuracy, speed and manual dexterity (the integrated measure of accuracy and speed).

RESULTS

Participants made significantly more errors when physically fatigued, whereas participants performed the test significantly slower when mentally stressed. Manual dexterity was significantly affected in the case of combined intervention. High test-retest reliability was found for errors (ICC = 0.90) and completion time (ICC = 0.64). Fair test-retest reliability for the integrated measure was found (ICC = 0.37).

CONCLUSION

The star-track test is a valid and reliable tool to evaluate the effect of physical fatigue and/or mental stress on the characteristics of manual dexterity in a laparoscopic setting.

Gesture segmentation and classification using affine speed and energy

The characterization and analysis of hand gestures are challenging tasks with an important number of applications in human-computer interaction, machine vision and control, and medical gesture recognition. Specifically, several researchers have tried to develop objective evaluation methods of surgical skills for medical training. As a result, the adequate selection and extraction of similarities and differences between experts and novices have become an important challenge in this area. In particular, some of these works have shown that human movements performed during surgery can be described as a sequence of constant affine-speed trajectories. In this article, we will show that affine speed can be used to segment medical hand movements and present how the mechanical energy computed in the segment is analyzed to compare surgical skills. The position and orientation of the instrument end effectors are determined by six video photographic cameras. In addition, two laparoscopic instruments are capable of measuring simultaneously the forces and torques applied to the tool. Finally, we will report the results of these experiments and present a correlation between the mechanical energy values, dissipated during a procedure, and the surgical skills.

Optimizing the positional relationships between instruments used in laparoscopic simulation using a simple trigonometric method

BACKGROUND

Various methods for evaluating laparoscopic skill have been reported, but without detailed information on the configuration used they are difficult to reproduce. Here we present a method based on the trigonometric relationships between the instruments used in a laparoscopic training platform in order to provide a tool to aid in the reproducible assessment of surgical laparoscopic technique.

MATERIALS AND METHODS

The positions of the instruments were represented using triangles. Basic trigonometry was used to objectively establish the distances among the working ports RL, the placement of the optical port h', and the placement of the surgical target OT.

RESULTS

The optimal configuration of a training platform depends on the selected working angles, the intracorporeal/extracorporeal lengths of the instrument, and the depth of the surgical target. We demonstrate that some distances, angles, and positions of the instruments are inappropriate for satisfactory laparoscopy.

CONCLUSIONS

By applying basic trigonometric principles we can determine the ideal placement of the working ports and the optics in a simple, precise, and objective way. In addition, because the method is based on parameters known to be important in both the performance and quantitative quality of laparoscopy, the results are generalizable to different training platforms and types of laparoscopic surgery.

Construct and concurrent validity of a Nintendo Wii video game made for training basic laparoscopic skills

BACKGROUND

Virtual reality (VR) laparoscopic simulators have been around for more than 10 years and have proven to be cost- and time-effective in laparoscopic skills training. However, most simulators are, in our experience, considered less interesting by residents and are often poorly accessible. Consequently, these devices are rarely used in actual training. In an effort to make a low-cost and more attractive simulator, a custom-made Nintendo Wii game was developed. This game could ultimately be used to train the same basic skills as VR laparoscopic simulators ought to. Before such a video game can be implemented into a surgical training program, it has to be validated according to international standards.

METHODS

The main goal of this study was to test construct and concurrent validity of the controls of a prototype of the game. In this study, the basic laparoscopic skills of experts (surgeons, urologists, and gynecologists, n = 15) were compared to those of complete novices (internists, n = 15) using the Wii Laparoscopy (construct validity). Scores were also compared to the Fundamentals of Laparoscopy (FLS) Peg Transfer test, an already established assessment method for measuring basic laparoscopic skills (concurrent validity).

RESULTS

Results showed that experts were 111 % faster (P = 0.001) on the Wii Laparoscopy task than novices. Also, scores of the FLS Peg Transfer test and the Wii Laparoscopy showed a significant, high correlation (r = 0.812, P < 0.001).

CONCLUSIONS

The prototype setup of the Wii Laparoscopy possesses solid construct and concurrent validity.

Relevance of motion-related assessment metrics in laparoscopic surgery

INTRODUCTION

Motion metrics have become an important source of information when addressing the assessment of surgical expertise. However, their direct relationship with the different surgical skills has not been fully explored. The purpose of this study is to investigate the relevance of motion-related metrics in the evaluation processes of basic psychomotor laparoscopic skills and their correlation with the different abilities sought to measure.

METHODS

A framework for task definition and metric analysis is proposed. An explorative survey was first conducted with a board of experts to identify metrics to assess basic psychomotor skills. Based on the output of that survey, 3 novel tasks for surgical assessment were designed. Face and construct validation was performed, with focus on motion-related metrics. Tasks were performed by 42 participants (16 novices, 22 residents, and 4 experts). Movements of the laparoscopic instruments were registered with the TrEndo tracking system and analyzed.

RESULTS

Time, path length, and depth showed construct validity for all 3 tasks. Motion smoothness and idle time also showed validity for tasks involving bimanual coordination and tasks requiring a more tactical approach, respectively. Additionally, motion smoothness and average speed showed a high internal consistency, proving them to be the most task-independent of all the metrics analyzed.

CONCLUSION

Motion metrics are complementary and valid for assessing basic psychomotor skills, and their relevance depends on the skill being evaluated. A larger clinical implementation, combined with quality performance information, will give more insight on the relevance of the results shown in this study.

Is the digitization of laparoscopic movement using accessible alternative technologies possible?

It is widely documented that laparoscopic surgeons require training, and an objective evaluation of the training that they receive. The most advanced evaluation systems integrate the digitization of the movement of laparoscopic tools. A great number of these systems, however, do not permit the use of real tools and their high cost limits their academic impact. Likewise, it is documented that new and accessible systems need to be developed. The aim of this article is to explore the possibility of digitizing the movement of laparoscopic tools in a three-dimensional workspace, using accessible alternative technology. Our proposal uses a commercial Wii video game control in conjunction with a program for determining kinematic variables during the execution of a recognition task.