The right tool for the right job: the importance of CUSUM in self-assessment

Learning curves in ERCP during advanced endoscopy training: a Canadian multicenter prospective study

Background and study aims  Growing emphasis on quality and patient safety has supported the shift toward competency-based medical education for advanced endoscopy trainees (AETs). In this study, we aimed to examine Canadian AETs learning curves and achievement of competence using an ERCP assessment tool with strong evidence of validity. Methods  This prospective study was conducted at five institutions across Canada from 2017-2018. Data on every fifth procedure performed by trainees were collected using the United Kingdom Joint Advisory Joint Advisory Group of Gastrointestinal Endoscopy (JAG) ERCP Direct Observation of Procedural Skills (DOPS) tool, which includes a four-point rating scale for 27 items. Cumulative sum (CUSUM) analysis was used to create learning curves for overall supervision ratings and ERCP DOPS items by plotting scores for procedures performed during training. Results  Eleven trainees who were evaluated for 261 procedures comprised our sample. The median number of evaluations by site was 49 (Interquartile range (IQR) 31-76) and by trainee was 15 (IQR 11-45). The overall cannulation rate by trainees was 82 % (241/261), and the native papilla cannulation rate was 78 % (149/191). All trainees achieved competence in the "overall supervision" domain of the ERCP DOPS by the end of their fellowship. Trainees achieved competency in all individual domains, except for tissue sampling and sphincteroplasty. Conclusions  Canadian AETs are graduating from fellowship programs with acceptable levels of competence for overall ERCP performance and for the most specific tasks. Learning curves may help identify areas of deficiency that may require supplementary training, such as tissue sampling.

Anesthesia assessment in the era of competences: state of the art

Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature  shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness. Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia. Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical  framework. Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.

Setting minimum standards for training in EUS and ERCP: results from a prospective multicenter study evaluating learning curves and competence among advanced endoscopy trainees

BACKGROUND AND AIMS

Minimum EUS and ERCP volumes that should be offered per trainee in "high quality" advanced endoscopy training programs (AETPs) are not established. We aimed to define the number of procedures required by an "average" advanced endoscopy trainee (AET) to achieve competence in technical and cognitive EUS and ERCP tasks to help structure AETPs.

METHODS

American Society for Gastrointestinal Endoscopy (ASGE)-recognized AETPs were invited to participate; AETs were graded on every fifth EUS and ERCP examination using a validated tool. Grading for each skill was done using a 4-point scoring system, and learning curves using cumulative sum analysis for overall, technical, and cognitive components of EUS and ERCP were shared with AETs and trainers quarterly. Generalized linear mixed-effects models with a random intercept for each AET were used to generate aggregate learning curves, allowing us to use data from all AETs to estimate the average learning experience for trainees.

RESULTS

Among 62 invited AETPs, 37 AETs from 32 AETPs participated. Most AETs reported hands-on EUS (52%, median 20 cases) and ERCP (68%, median 50 cases) experience before starting an AETP. The median number of EUS and ERCPs performed per AET was 400 (range, 200-750) and 361 (range, 250-650), respectively. Overall, 2616 examinations were graded (EUS, 1277; ERCP-biliary, 1143; pancreatic, 196). Most graded EUS examinations were performed for pancreatobiliary indications (69.9%) and ERCP examinations for ASGE biliary grade of difficulty 1 (72.1%). The average AET achieved competence in core EUS and ERCP skills at approximately 225 and 250 cases, respectively. However, overall technical competence was achieved for grade 2 ERCP at about 300 cases.

CONCLUSION

The thresholds provided for an average AET to achieve competence in EUS and ERCP may be used by the ASGE and AETPs in establishing the minimal standards for case volume exposure for AETs during their training. (Clinical trial registration number: NCT02509416.).

Competence in Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography, From Training Through Independent Practice

BACKGROUND & AIMS

It is unclear whether participation in competency-based fellowship programs for endoscopic ultrasound (EUS) and endoscopic retrograde cholangiopancreatography (ERCP) results in high-quality care in independent practice. We measured quality indicator (QI) adherence during the first year of independent practice among physicians who completed endoscopic training with a systematic assessment of competence.

METHODS

We performed a prospective multicenter cohort study of invited participants from 62 training programs. In phase 1, 24 advanced endoscopy trainees (AETs), from 20 programs, were assessed using a validated competence assessment tool. We used a comprehensive data collection and reporting system to create learning curves using cumulative sum analysis that were shared with AETs and trainers quarterly. In phase 2, participating AETs entered data into a database pertaining to every EUS and ERCP examination during their first year of independent practice, anchored by key QIs.

RESULTS

By the end of training, most AETs had achieved overall technical competence (EUS 91.7%, ERCP 73.9%) and cognitive competence (EUS 91.7%, ERCP 94.1%). In phase 2 of the study, 22 AETs (91.6%) participated and completed a median of 136 EUS examinations per AET and 116 ERCP examinations per AET. Most AETs met the performance thresholds for QIs in EUS (including 94.4% diagnostic rate of adequate samples and 83.8% diagnostic yield of malignancy in pancreatic masses) and ERCP (94.9% overall cannulation rate).

CONCLUSIONS

In this prospective multicenter study, we found that although competence cannot be confirmed for all AETs at the end of training, most meet QI thresholds for EUS and ERCP at the end of their first year of independent practice. This finding affirms the effectiveness of training programs. Clinicaltrials.gov ID NCT02509416.

A Prospective Multicenter Study Evaluating Learning Curves and Competence in Endoscopic Ultrasound and Endoscopic Retrograde Cholangiopancreatography Among Advanced Endoscopy Trainees: The Rapid Assessment of Trainee Endoscopy Skills Study

BACKGROUND & AIMS

On the basis of the Next Accreditation System, trainee assessment should occur on a continuous basis with individualized feedback. We aimed to validate endoscopic ultrasound (EUS) and endoscopic retrograde cholangiopancreatography (ERCP) learning curves among advanced endoscopy trainees (AETs) by using a large national sample of training programs and to develop a centralized database that allows assessment of performance in relation to peers.

METHODS

ASGE recognized training programs were invited to participate, and AETs were graded on ERCP and EUS exams by using a validated competency assessment tool that assesses technical and cognitive competence in a continuous fashion. Grading for each skill was done by using a 4-point scoring system, and a comprehensive data collection and reporting system was built to create learning curves by using cumulative sum analysis. Individual results and benchmarking to peers were shared with AETs and trainers quarterly.

RESULTS

Of the 62 programs invited, 20 programs and 22 AETs participated in this study. At the end of training, median number of EUS and ERCP performed/AET was 300 (range, 155-650) and 350 (125-500), respectively. Overall, 3786 exams were graded (EUS, 1137; ERCP-biliary, 2280; ERCP-pancreatic, 369). Learning curves for individual end points and overall technical/cognitive aspects in EUS and ERCP demonstrated substantial variability and were successfully shared with all programs. The majority of trainees achieved overall technical (EUS, 82%; ERCP, 60%) and cognitive (EUS, 76%; ERCP, 100%) competence at conclusion of training.

CONCLUSIONS

These results demonstrate the feasibility of establishing a centralized database to report individualized learning curves and confirm the substantial variability in time to achieve competence among AETs in EUS and ERCP. ClinicalTrials.gov: NCT02509416.

Quantitative assessment of the learning curve for cleft lip repair using LC-CUSUM

The first step in cleft lip repair is the precise positioning of anatomical landmarks and tracing of the incisions on the patient's lip at the beginning of the procedure. The aim of this study was to evaluate progress made in learning cleft lip repair tracing using a quantitative assessment of learning curves: LC-CUSUM (learning curve - cumulative sum). Eight surgical residents were enrolled and asked to trace lip repair incisions on five cases of unilateral left cleft lip over 5 consecutive weeks. Results were compared to a reference tracing based on the positioning of nine anatomical landmarks and assessed using LC-CUSUM. Competence was defined as the accurate positioning of the nine landmarks (less than 1.4mm deviation from the reference positions, with an accepted 15% failure rate). After five tracing sessions, competence was not achieved evenly for all trainees, or for all landmarks, underlining differences in inter-individual learning ability even with similar training. However, despite an initial marked lack of theoretical and practical training in lip repair techniques, repeated drawings of cleft lip incisions allowed a satisfactory level of competence to be reached for most landmarks and most trainees. Nevertheless it was found that not all landmarks are understood by students with similar ease, and that landmark positioning reveals significant inter-individual differences. This approach allowed a global assessment of the teaching of cleft repair and will help to focus training on specific problematic points for which competence was not obtained according to the LC-CUSUM test.

Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis

BACKGROUND AND AIMS

There are limited data on learning curves and competence in ERCP. By using a standardized data collection tool, we aimed to prospectively define learning curves and measure competence among advanced endoscopy trainees (AETs) by using cumulative sum (CUSUM) analysis.

METHODS

AETs were evaluated by attending endoscopists starting with the 26th hands-on ERCP examination and then every ERCP examination during the 12-month training period. A standardized ERCP competency assessment tool (using a 4-point scoring system) was used to grade the examination. CUSUM analysis was applied to produce learning curves for individual technical and cognitive components of ERCP performance (success defined as a score of 1, acceptable and unacceptable failures [p1] of 10% and 20%, respectively). Sensitivity analyses varying p1 and by using a less-stringent definition of success were performed.

RESULTS

Five AETs were included with a total of 1049 graded ERCPs (mean ± SD, 209.8 ± 91.6/AET). The majority of cases were performed for a biliary indication (80%). The overall and native papilla allowed cannulation times were 3.1 ± 3.6 and 5.7 ± 4, respectively. Overall learning curves demonstrated substantial variability for individual technical and cognitive endpoints. Although nearly all AETs achieved competence in overall cannulation, none achieved competence for cannulation in cases with a native papilla. Sensitivity analyses increased the proportion of AETs who achieved competence.

CONCLUSION

This study demonstrates that there is substantial variability in ERCP learning curves among AETs. A specific case volume does not ensure competence, especially for native papilla cannulation.

Real-Time Characterization of Diminutive Colorectal Polyp Histology Using Narrow-Band Imaging: Implications for the Resect and Discard Strategy

BACKGROUND & AIMS

Narrow-band imaging (NBI) allows real-time histologic classification of colorectal polyps. We investigated whether endoscopists without prior training in NBI can achieve the following thresholds recommended by the American Society for Gastrointestinal Endoscopy: for diminutive colorectal polyps characterized with high confidence, a ≥90% negative predictive value for adenomas in the rectosigmoid and a ≥90% agreement in surveillance intervals.

METHODS

Twenty-six endoscopists from 2 tertiary care centers underwent standardized training in NBI interpretation. Endoscopists made real-time predictions of diminutive colorectal polyp histology and surveillance interval predictions based on NBI. Their performance was evaluated by comparing predicted with actual findings from histologic analysis. Multilevel logistic regression was used to assess predictors of performance. Cumulative summation analysis was used to characterize learning curves.

RESULTS

The endoscopists performed 1451 colonoscopies and made 3012 diminutive polyp predictions (74.3% high confidence) using NBI. They made 898 immediate post-procedure surveillance interval predictions. An additional 505 surveillance intervals were determined with histology input. The overall negative predictive value for high-confidence characterizations in the rectosigmoid was 94.7% (95% confidence interval: 92.6%-96.8%) and the surveillance interval agreement was 91.2% (95% confidence interval: 89.7%-92.7%). Overall, 97.0% of surveillance interval predictions would have brought patients back on time or early. High-confidence characterization was the strongest predictor of accuracy (odds ratio = 3.42; 95% confidence interval: 2.72-4.29; P < .001). Performance improved over time, however, according to cumulative summation analysis, only 7 participants (26.9%) identified adenomas with sufficient sensitivity such that further auditing is not required.

CONCLUSIONS

With standardized training, gastroenterologists without prior expertise in NBI were able to meet the negative predictive value and surveillance interval thresholds set forth by the American Society for Gastrointestinal Endoscopy. The majority of disagreement in surveillance interval brought patients back early. Performance improves with time, but most endoscopists will require ongoing auditing of performance. ClinicalTrials.gov ID NCT02441998.

Gastroenterology trainees can easily learn histologic characterization of diminutive colorectal polyps with narrow band imaging

BACKGROUND & AIMS

Little is known about how teaching gastroenterology trainees polyp patterns by using narrow band imaging (NBI) affects their ability to characterize the histology of diminutive colorectal polyps. We developed and tested a tool to teach trainees to characterize the histology of diminutive polyps by using NBI.

METHODS

Twelve gastroenterology trainees with varying levels of colonoscopy experience watched a teaching tool that described the NBI criteria to distinguish polyp histology. The trainees then watched 80 videos of NBI examination of diminutive polyps, recording their predictions of polyp histology and their degree of confidence. After each video, an expert provided feedback about actual polyp histology and the NBI criteria that supported each diagnosis. Twelve weeks later, without training or feedback during the interval, the trainees watched the same videos and predicted histologies of the polyps. Performance was evaluated by comparing predicted classification with actual histologic findings. Cumulative sum analysis was used to determine the learning curve for each trainee.

RESULTS

Trainees made significant improvements in accuracy and the proportion of high-confidence predictions as they progressed through video blocks during the first session (P < .001). With active feedback, all trainees predicted polyp histologies with >90% accuracy, with a negative predictive value >90% for adenomatous histology. A median of 49 videos was required to achieve competency. For diagnoses made with high confidence, trainee performance exceeded 90% during the first and second sessions. Interobserver agreement was substantial (session 1, κ = 0.71; session 2, κ = 0.70).

CONCLUSIONS

We developed a computer-based tool, combined with short videos and active feedback, to train gastroenterologists to identify polyp histology by using NBI. After training, gastroenterology trainees characterized the histology of diminutive polyps with ≥ 90% accuracy.