Automatic and unbiased assessment of competence in colonoscopy: exploring validity of the Colonoscopy Progression Score (CoPS)

A multi-modal training environment for colonoscopy with pressure feedback

BACKGROUND

Colonoscopy is a complex procedure with a 3-5% failure rate even for experts. Mastering endoscopy skills and reducing complication rates extend well beyond the initial training phase for most endoscopists, and continues into their clinical experience. Thus, continuing efforts should focus on optimizing training methods to make them individualized with clear objective learning goals for trainees.

METHODS

A hybrid (physical and computer) colonoscopy training simulator was developed using a novel pressure-sensing sleeve covering the full length of a colonoscope, and a physical colon simulator (Kyoto Kagaku) along with custom-designed training software to visualize the color-mapped 3D pressure profile of the colonoscope during the simulated procedure and provide a visual and quantitative evaluation of the endoscopist's skills post-procedure. A system usability questionnaire and objective evaluation metrics were used to determine the model's effectiveness as a training tool.

RESULTS

Thirty-three participants were enrolled in the study, among which 8 were experts and 25 trainees. The interactive maximum and average pressures applied by the trainees were generally higher than those applied by experts, however, this difference was only statistically significant in the recto-sigmoid region. The mean average pressure applied in the rectum and the rectosigmoid region was 6.5 kPa for the experts compared to 13.7 kPa for the trainees, with a p-value of 0.011. Both groups agreed that the system is easy to understand and use, and would be helpful as a learning aid in training programs for colonoscopy skills.

CONCLUSIONS

The proposed system is expected to enhance the quality of colonoscopy procedures by enabling endoscopists to adopt safer and more efficient navigational skills. The evaluation metrics discussed in this research offer useful insights into the performance of endoscopists, and the ability of trainees to compare their performance against expert benchmarks will enable them to establish personalized objective training goals.

Colonoscopy Training on Virtual-Reality Simulators or Physical Model Simulators: A Randomized Controlled Trial

OBJECTIVE

This study employed a randomized controlled trial to assess the efficacy of virtual-reality (VR) simulators and physical model simulators on colonoscopy training to explore the optimal and evidence-based simulation training.

DESIGN

Forty participants were divided into 2 groups and randomized as dyads: the VR simulator group and the physical model simulator group. All the participants performed a baseline test through porcine colonoscopy. After a 6 h simulation training, each participant underwent a post-test on a pig after bowel preparation, and the procedures were video-recorded. Both the baseline test and the post-test were blindly assessed by 2 experienced assistant director physicians based on the GAGES-C scoring system.

SETTING

Simulation center, Renji Hospital, Shanghai Jiaotong University School of Medicine, Shanghai.

PARTICIPANTS

Forty surgical residents without colonoscopy experience.

RESULTS

Both the VR simulator group and the physical model simulator group improved significantly over the baseline test. The VR simulator group performed significantly better than the physical model simulator group, p=0.042. The participants in both groups expressed a high level of simulator satisfaction.

CONCLUSIONS

Novice residents can benefit from both VR simulators and physical model simulators. The VR simulator was shown to be more effective for colonoscopy training. VR simulators were more recommended for novices conducting basic colonoscopy training.

Computer-aided quality assessment of endoscopist competence during colonoscopy: a systematic review

BACKGROUND AND AIMS

Endoscopists' competence can vary widely, as shown in the variation in the adenoma detection rate (ADR). Computer-aided quality assessment (CAQ) can automatically assess performance during individual procedures. In this review we identified and described different CAQ systems for colonoscopy.

METHODS

A systematic review of the literature was done using MEDLINE, EMBASE, and Scopus based on 3 blocks of terms according to the inclusion criteria: colonoscopy, competence assessment, and automatic evaluation. Articles were systematically reviewed by 2 reviewers, first by abstract and then in full text. The methodological quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

Of 12,575 identified studies, 6831 remained after removal of duplicates and 6806 did not pass the eligibility criteria and were excluded, leaving 25 studies, of which 13 studies were included in the final analysis. Five categories of CAQ systems were identified: withdrawal speedometer (7 studies), endoscope movement analysis (3 studies), effective withdrawal time (1 study), fold examination quality (1 study), and visual gaze pattern (1 study). The withdrawal speedometer was the only CAQ system that tested its feedback by examining changes in ADR. Three studies observed an improvement in ADR, and 2 studies did not. The methodological quality of the studies was high (mean MERSQI, 15.2 points; maximum, 18 points).

CONCLUSIONS

Thirteen studies developed or tested CAQ systems, most frequently by correlating it to the ADR. Only 5 studies tested feedback by implementing the CAQ system. A meta-analysis was impossible because of the heterogeneous study designs, and more studies are warranted.

Colonoscope retraction technique and predicting adenoma detection rate: a multicenter study

BACKGROUND AND AIMS

The success of preventing colorectal cancer relies on the expertise of the colonoscopists. Studies suggest that the retraction technique is a powerful indicator of expertise in distinguishing endoscopists with various adenoma detection rates (ADRs). We aimed to develop a retraction technique score and explore the correlation between endoscopists' retraction technique and their ADRs.

METHODS

In a prospective, multicenter study, 8 colonoscopist nurses and physicians with various ADRs were included. Data from patients admitted for a colonoscopy, as part of the Danish nationwide screening program, were gathered directly from the Olympus ScopeGuide system (UPD-3; Olympus Optical, Tokyo, Japan) providing XYZ-coordinates from the coils along the length of the colonoscope. Motor skill measures were developed based on tip retraction, retraction efficiency, and retraction distance. The principal component analysis was used to study the association among the 3 measures and the historical ADR to create a combined score, the colonoscopy retraction score (CoRS).

RESULTS

Three hundred thirty-three recordings were analyzed. We demonstrated a significant and strong correlation between CoRS and ADR (.90, P < .01). Conversely, withdrawal time did not correlate significantly with ADR (.33, P = .42). In procedures without polypectomies or biopsy sampling, a significant and strong correlation was found between CoRS and ADR (.88, P < .01) and between withdrawal time and ADR (.75, P = .03).

CONCLUSIONS

This study presents a novel, real-time computerized and unbiased assessment tool for colonoscopy withdrawal. CoRS strongly correlated with ADR with and without therapeutic interventions during withdrawal and could be used to ensure quality instead of minimal withdrawal time. (Clinical trial registration number: NCT03587935.).

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Validation of an assessment tool for estimation of abdominal aortic aneurysm compression in diagnostic ultrasound

The study investigated ultrasound (US) transducer push, tantamount to applied transducer pressure, during abdominal aortic aneurysm (AAA) US scanning in a simulated non-clinical setup. During an assessment of maximal AAA diameter on a three-dimensional print-based AAA phantom, US transducer push varied as much as 2000% (range: 0.52-12.45 kPa) amongst 16 experienced sonographers. The mean transducer push was 5.54 ± 3.91 kPa (CV = 0.71). Deformation of a standardized gel-pad allowed for transducer push calculation based on US images; Young's modulus of the gel-pad was estimated to 44,26 N/m2. The method is theoretically validated in a safe and non-clinical environment. Future investigations with the aim of clinical validation of the gel-pad principle on AAA patients are suggested, including the objectification of the magnitude of an eventual transducer push-related error during US AAA diameter measurement.

Computerized feedback during colonoscopy training leads to improved performance: a randomized trial

BACKGROUND AND AIMS

Simulation-based training in colonoscopy is increasingly replacing the traditional apprenticeship method to avoid patient-related risk. Mentoring during simulation is necessary to provide feedback and to motivate, but expert supervisors are a scarce resource. We aimed to determine whether computerized feedback in simulated colonoscopy would improve performance, optimize time spent practicing, and optimize the pattern of training.

METHODS

Forty-four participants were recruited and randomized to either a feedback group (FG) or a control group (CG). Participants were allowed 2 hours of self-practice during which they could practice as they saw fit on 2 different cases: 1 easy and 1 difficult. The CG practiced without feedback, but the participants in the FG were given a score of progression every time they reached the cecum. All participants were tested on a different case after end of training. The primary outcome was the progression score in the final case, and secondary outcomes were time spent practicing and the training pattern.

RESULTS

Regression analysis adjusting for sex was done because of an uneven sex distribution between groups (P = .026) and significantly higher performance scores by men (37.6, standard deviation [SD] 25.9) compared with women (19.7, SD 18.7); P = .012. The FG outperformed the CG in the final case, FG scoring 14.4 points (95% confidence interval [CI], 1.2-27.6) more than the CG; P = .033, and they spent more time practicing, FG practicing 25.8 minutes (95% CI, 11.6-39.9) more than the CG; P = .001. The FG practiced more on the easy case and reached the cecum 3.2 times more (95% CI, 2-4.5) during practice (P < .001).

CONCLUSIONS

Our findings of this study revealed that an automatic, computerized score of progression during simulated colonoscopy motivates the novices to improve performance, optimizes time spent practicing, and optimizes their pattern of training. (Clinical trial registration number: NCT03248453.).

Difference in real-time magnetic image analysis of colonic looping patterns between males and females undergoing diagnostic colonoscopy

Background and study aim Magnetic imaging technology is of proven benefit to trainees in colonoscopy, but few studies have examined its benefits in experienced hands. There is evidence that colonoscopy is more difficult in women. We set out to investigate (i) associations between the looping configurations in the proximal and distal colon and (ii) differences in the looping prevalence between the sexes. We have examined their significance in terms of segmental intubation times and position changes required for the completion of colonoscopy.

Patients and methods We analyzed 103 consecutive synchronized luminal and magnetic image videos of diagnostic colonoscopies with normal anatomy undertaken by a single experienced operator.

Results Deep transverse loops and sigmoid N-loops were more common in females. A deep transverse loop was more likely to be present if a sigmoid alpha-loop or N-loop had formed previously. Patients with sigmoid N-loops were turned more frequently from left lateral to supine before the sigmoid-descending junction was reached, but there was no statistical correlation between completion time and looping pattern.

Conclusions This study has reexamined the prevalence of the common looping patterns encountered during colonoscopy and has identified differences between the sexes. This finding may offer an explanation as to why colonoscopy has been shown to be more difficult in females. Although a deep transverse loop following a resolved sigmoid alpha-loop was the most commonly encountered pattern, no statistical correlation between completion time and looping pattern could be shown. It is the first study to examine segmental completion times using a magnetic imager in expert hands.

Combining different methods improves assessment of competence in colonoscopy

OBJECTIVES

To develop a reliable method of assessing competence in colonoscopy based on multiple sources.

MATERIALS AND METHODS

Physicians with varying degrees of experience in colonoscopy performed two colonoscopies each in a standardized simulated environment. Their performances were assessed under direct observation by an expert rater and by automatic computerized analysis of operator movements and scope movements, respectively. Reliability (Cronbach's alpha) for subjective assessment, time to cecum, analysis of operator movement and analysis of scope movements were calculated. Composite score calculations were used to explore different combinations of the measures.

RESULTS

Twenty physicians were included in the study. The reliability (Cronbach's alpha) were 0.92, 0.57, 0.87 and 0.55 for the subjective score assessed under direct observation, time to cecum, distance between operator's hands and colonoscopy progression score, respectively. Equal weight (=25%) to all four methods resulted in a reliability of 0.91 and optimal weighting of the methods (55%, 10%, 25% and 10%, respectively) resulted in a maximum reliability of 0.95.

CONCLUSION

Combining subjective expert ratings with automated objective assessments results in a less biased and more reliable assessment of competence in colonoscopy.