Direct Observation vs. Video-Based Assessment in Flexible Cystoscopy

Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

BACKGROUND

Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP).

METHODS

We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS).

RESULTS

One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034).

CONCLUSION

We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.

New generation evaluations: video-based surgical assessments : A technology update

BACKGROUND

Surgical skill training, assessment, and feedback are the backbone of surgical training. High-quality skills require expert supervision and evaluation throughout a resource-intensive multi-year training process. As technological barriers to internet access and the ability to save and upload surgical videos continue to improve, video-based assessment technology is emerging as a tool that could reshape surgical training for the next generation of surgeons. Video-based assessment platforms have the potential to allow surgeons from across the globe to upload their surgical videos online and receive high-quality, standardized, and unbiased feedback. They combine visual recordings of a surgeon's operative technique, with standardized grading tools that have the potential to significantly impact surgical training and technical skill acquisition across the world.

METHOD

The platforms included in this review are in various stages of development after a thorough discussion with national experts on the SAGES TAVAC (Technology and Value Assessments) Committee. For each VBA program, a description of its platform was given and a literature review was obtained using a PubMed search performed from inception until December 2021.

RESULTS

The study reviewed all video-based assessment programs currently available in the market, identified their strengths and weaknesses, and how they can be optimized in future.

CONCLUSION

The technological platforms will play a key role in the training and technical skill acquisition of the next generation of surgeons and can have an immense impact on patient care across the world. There is immense potential for all these platforms to grow and become incorporated within the framework of an effective surgical training program.

A Plea for the Use of Video-recorded Cystoscopy for Blinded Independent Central Review of Tumor Response in Open-label Non-muscle invasive Bladder Cancer Trials

Given that most of non-muscle invasive bladder cancer trials available in the literature share the same limitation of an open-label design, they are likely prone to detection bias. Indeed, tumor response is currently assessed by unblinded investigators, mainly via follow-up cystoscopy, which has well-known intraobserver and interobserver variability. Thus, we believe that the use of video-recorded cystoscopy for blinded independent central review of tumor response could help to minimize detection bias in open-label NMIBC trials.

Video-Based versus On-Site Neonatal Pain Assessment in Neonatal Intensive Care Units: The Impact of Video-Based Neonatal Pain Assessment in Real-World Scenario on Pain Diagnosis and Its Artificial Intelligence Application

BACKGROUND

Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management.

PURPOSE

To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications.

SETTING

A total of 598 neonates were recruited from a pediatric hospital in China.

METHODS

This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively.

RESULTS

The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods.

CONCLUSION

VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

Spatiotemporal Modeling of Grip Forces Captures Proficiency in Manual Robot Control

New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).

National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients

Background

Objective

Design, setting, and participants

Outcome measurements and statistical analyses

Results and limitations

Conclusions

Patient summary

Video-based observation in impact evaluation

This paper reviews the literature on the use of video-based observation (VBO) with particular attention to monitoring and evaluation (M&E) on development projects. While the use of video both as a research tool and as a strategy for supporting professional development is well-documented across several disciplines, the extent to which VBO has been utilized in M&E contexts is less clearly defined. In order to provide theoretically-grounded recommendations for the development and implementation of one organisation's innovative VBO impact evaluation scheme, this review examines recent evidence of VBO in M&E contexts and draws on VBO literature more generally to identify its advantages and challenges together with advice for enhancing its effectiveness. Based on this analysis, the paper highlights a number of practical issues that should be considered when VBO is being developed for M&E in development contexts, particularly where videos are being made by participants themselves. The value of VBO in responding to COVID-19 and reducing carbon emissions is also noted.

The Impact of Simulation Training on Operative Performance in General Surgery: Lessons Learned from a Prospective Randomized Trial

BACKGROUND

Practice in the simulated environment can improve surgical skills. However, the transfer of open complex surgical skills to the operating room is unclear. This study evaluated the effect of resident operative performance following a simulation experience on a hand-sewn small bowel anastomosis and determined the impact of utilizing proficiency-based training.

METHODS

Nine categorical interns performed a hand-sewn small bowel anastomosis in the operating room prior to (pre-test) and following (post-test) a 3-h simulation training session with an assessment at the end. Participants were randomly assigned to 1of 2 simulation training groups: proficiency-based or standard. Operative performance was videotaped. 2 independent, blinded faculty surgeons assessed performances by a global rating scale. Pre- and post-confidence levels were obtained on a 5-point Likert scale.

RESULTS

Overall, pre-test and post-test operative performance was similar (3 [IQR, 2.5 -3.5] versus 3 [IQR, 3 -3], P = 0.59). Furthermore, no difference was observed in the post-test performance with proficiency-based or standard training (3 [IQR, 3 -3] versus 3 [IQR, 3 -3], P = 0.73). Self-reported confidence with the skills, however, significantly improved (median 1 versus 4, P = 0.007).

CONCLUSIONS

In this prospective, randomized study, we did not observe an improvement in operative performance following simulation instruction and assessment, with both training groups. Overcoming barriers to skills transfer will be paramount in the future to optimize simulation training in general surgery. These findings highlight the importance of continued study for the ideal conditions and timing of technical skills training.

Validation of a novel assessment tool identifying proficiency in Transurethral Bladder Tumour Resection: The OSATURBS assessment tool

BACKGROUND

Competence in transurethral bladder tumour resection (TURB) is critical in bladder cancer management and should be ensured before independent practice.

OBJECTIVE

Develop an assessment tool for TURB and explore validity evidence in a clinical context.

DESIGN, SETTING, AND PARTICIPANTS

July 2019-March 2021, a total of 33 volunteer doctors from three hospitals were included. Participants performed two TURB procedures on patients with bladder tumours. A newly developed assessment tool (OSATURBS) was used for direct observation assessment, self-assessment, and blinded video-assessment. Outcome measurements and statistical analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t-test and independent samples t-test.

RESULTS AND LIMITATIONS

The internal consistency reliability across items Cronbach's alpha was 0.94 (n = 260, p < 0.001). Inter-rater reliability = 0.80 (n = 64, p < 0.001). Test-retest correlation was high, r = 0.71 (n = 32, p < 0.001). Relation to TURB experience was high, r = 0.71 (n = 32, p < 0.001). Pass/fail score = 19 points. Direct observation assessments were strongly correlated with video ratings (r = 0.85, p < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances.

CONCLUSIONS

OSATURBS assessment tool for TURB can be used for assessment of surgical proficiency in the clinical setting. Direct observation assessment and self-assessment are biased, and blinded video-assessment of TURB performances is advised.

Virtual reality simulator metrics cannot be used to assess competence in ureteronephroscopy and stone removal - a validation study

OBJECTIVE

The growing use of simulation-based training makes it necessary to develop efficient training programs in order to ensure optimal use of time and resources. Our aim was to develop and gather validity evidence for a simulation-based test in ureteronephroscopy and set a pass/fail standard for the test that will allow future mastery learning.

DESIGN

This study is a validation study. A test in ureteronephroscopy and stone removal on the URO Mentor™ virtual reality simulator (3D Systems, USA) was developed by two experienced urologists in order to ensure content. Participants with different experience completed three standardized tasks on the simulator and simulator-generated metrics were used as outcome parameters to minimize bias and ensure a fair response process.

RESULTS

Twenty novices, 15 intermediates, and 8 experienced urologists were included in the study. Validity evidence for internal structure and relationship to other variables was questionable with weak and mostly insignificant correlations across all four metrics (Cronbach's alpha = 0.14, p = 0.15) and across the three modules (Cronbach's alpha = 0.41 (p = 0.02), 0.35 (p = 0.06), 0.10 (p = 0.35), and 0.30 (p = 0.09) for each metric, respectively). It was not possible to establish a pass/fail score for the simulation test with meaningful consequences.

CONCLUSION

Our study showed that automatically generated simulator metrics cannot be used as a valid way of assessing competence in ureteronephroscopy. Virtual-reality simulator training could still be a valuable and patient-safe way to practice these skills, but an experienced supervisor is needed to determine when the trainee is ready to continue to supervised practice on patients.

Analysis of Kinematic Differences in Hand Motion between Novice and Experienced Operators in IR: A Pilot Study

PURPOSE

To prospectively validate electromagnetic hand motion tracking in interventional radiology to detect differences in operator experience using simulation.

METHODS

Sheath task: Six attending interventional radiologists (experts) and 6 radiology trainees (trainees) placed a wire through a sheath and performed a "pin-pull" maneuver, while an electromagnetic motion detection system recorded the hand motion. Radial task: Eight experts and 12 trainees performed palpatory radial artery access task on a radial access simulator. The trainees repeated the task with the nondominant hand. The experts were classified by their most frequent radial artery access technique as having either palpatory, ultrasound, or overall limited experience. The time, path length, and number of movements were calculated. Mann-Whitney U tests were used to compare the groups, and P < .05 was considered significant.

RESULTS

Sheath task: The experts took less time, had shorter path lengths, and used fewer movements than the trainees (11.7 seconds ± 3.3 vs 19.7 seconds ± 6.5, P < .01; 1.1 m ± 0.3 vs 1.4 m ± 0.4, P < .01; and 19.5 movements ± 8.5 vs 31.0 movements ± 8.0, P < .01, respectively). Radial task: The experts took less time, had shorter path lengths, and used fewer movements than the trainees (24.2 seconds ± 10.6 vs 33.1 seconds ± 16.9, P < .01; 2.0 m ± 0.5 vs 3.0 m ± 1.9, P < .001; and 36.5 movements ± 15.0 vs 54.5 movements ± 28.0, P < .001, respectively). The trainees had a shorter path length for their dominant hand than their nondominant hand (3.0 m ± 1.9 vs 3.5 m ± 1.9, P < .05). The expert palpatory group had a shorter path length than the ultrasound and limited experience groups (1.8 m ± 0.4 vs 2.0 m ± 0.4 and 2.3 m ± 1.2, respectively, P < .05).

CONCLUSIONS

Electromagnetic hand motion tracking can differentiate between the expert and trainee operators for simulated interventional tasks.

Development of a photographic handbook to improve cystoscopy findings during resident's training: A randomised prospective study

Objectives: To evaluate if the use of a photographic handbook (PH) can be a useful tool to improve the detection of disorders during cystoscopy training, as several hands-on tools have been proposed to improve technical skills but very few aim to improve specificity and sensitivity.

Subjects and methods: Eight junior residents (JRs) were divided into two groups: Group A, comprised four JRs with previous limited experience of performing cystoscopies; and Group B, including four inexperienced JRs who were asked to study a specific PH before performing cystoscopies. The findings of the two groups were compared using the chi-squared test.

Results: A total of 401 consecutive cystoscopies, of which 214 (53.4%) were performed by Group A and 187 (46.6%) by Group B, were considered. Group B showed superior ability in detecting uncommon findings (i.e., carcinoma in situ, bullous oedema, interstitial cystitis, etc.) with 24/46 (52.2%) detected vs eight of 32 (25%) in Group A (P = 0.016).

Conclusions: The PH was a useful tool for improving identification of pathological conditions, which could be used to enhance hands-on simulator and practical tutored training.

Abbreviations: CIS: carcinoma in situ; JR: junior resident; PH: photographic handbook; VR: virtual reality Classification: Stones/Endourology