A prospective comparison of live and video-based assessments of colonoscopy performance

Mixed Reality for Remote Procedural Training and Assessment: A Feasibility Study

OBJECTIVE

Mixed reality (MR) enables real-time telecommunication with verbal and virtual information. We aimed to build a proof of concept and evaluate MR's feasibility for remote personalized technical skill training for chest tube insertion technique.

DESIGN

Nonexperimental correlational design.

SETTING

A commercial MR software was used to communicate between the participant's head-mounted display and the instructor's laptop. Study participants followed the instructions of an off-site instructor delivered by means of virtual annotations and verbal guidance while performing chest tube insertion on a mannequin. Expert-participants assessed the educational applicability of this teaching approach by a 5-point Likert scale. Non-expert-participants then performed the procedure without the instructions, while technical skills were assessed using a modified Objective Structured Assessment of Technical Skills by 2 experts, 1 rater present on-site and the other observing the performance remotely through the MR system. Non-expert-participants reported the usability of the system using the System Usability Scale (SUS). Inter-rater reliability (remote vs in-person) was calculated using the interclass correlation coefficient (ICC). Data are reported as median (interquartile range).

PARTICIPANTS

Five experts and 17 nonexpert individuals (10 medical trainees, 3 nurses, and 4 engineering students).

RESULTS

Experts evaluated the teaching with the MR system for remote procedural training useful (4.8 out of 5). The overall SUS score was excellent at 87.5 (77.5-95.0). Scores of in-person and remote technical skill assessment were 45 (39-47), and 44 (37-49) out of 50, respectively. ICC was 0.94.

CONCLUSIONS

We successfully established a proof of concept of an MR system for remote procedural training. The system was well-received both by experts and nonexperts, and this teaching approach led to sufficient proficiency in technical skill among non-expert-participants. Reliability of technical skill assessment between via remote and direct observation was excellent. MR is a promising technology for remote instruction and assessment in procedural training.

Validation of the GPAT - the Global Polypectomy Assessment Tool: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement

BACKGROUND : Colorectal polypectomy is operator dependent, with variable rates of complete resection. The currently available assessment tools do not provide specific competency-based evaluation of provider technique. We aimed to validate the Global Polypectomy Assessment Tool (GPAT), a novel competency assessment tool for colorectal polypectomy. METHODS : GPAT was derived from the ESGE Curriculum for Training in endoscopic mucosal resection in the colon. Members of the curriculum taskforce plus three invited trainees and three medical students (collectively: the assessors) anonymously assessed nine endoscopic-view only polypectomy videos. The primary end point was the correlation of the assessors' GPAT scores with a consensus-derived reference GPAT score per video. Secondary end points were the assessors' subjective impression versus their GPAT score and interobserver agreement among assessors' GPAT scores. RESULTS : 171 GPAT assessments by 19 assessors (consultant gastroenterologists [n = 10], trainee gastroenterologists [n = 4], consultant surgeons [n = 2], and medical students [n = 3]) were analyzed. Reference GPAT scores did not differ significantly from those of the assessors (73.1 % [95 %CI 64.6 %-81.6 %] vs. 69.3 % [95 %CI 64.9 %-81.2 %]; P = 0.47). There was moderate IOA in GPAT scores among gastroenterologists (intraclass correlation coefficient [ICC], 0.52 [moderate]) but not among nongastroenterologists (ICC 0.32 [poor]). GPAT correlated with assessors' subjective impression of polypectomy quality (correlation coefficient 0.98 [95 %CI 0.90-1.00]; P < 0.001). Overall assessors' qualitative usability scoring of GPAT was positive. CONCLUSIONS : GPAT allows standardized scoring of polypectomies, with moderate IOA among gastroenterologists and correlation with subjective impressions of polypectomy quality. GPAT could standardize assessment of trainee polypectomy competency offering structured feedback on performance.

Video recording in GI endoscopy

The current approach to procedure reporting in endoscopy aims to capture essential findings and interventions but inherently sacrifices the rich detail and nuance of the entire endoscopic experience. Endoscopic video recording (EVR) provides a complete archive of the procedure, extending the utility of the encounter beyond diagnosis and intervention, and potentially adding significant value to the care of the patient and the field in general. This white paper outlines the potential of EVR in clinical care, quality improvement, education, and artificial intelligence-driven innovation, and addresses critical considerations surrounding technology, regulation, ethics, and privacy. As with other medical imaging modalities, growing adoption of EVR is inevitable, and proactive engagement of professional societies and practitioners is essential to harness the full potential of this technology toward improving clinical care, education, and research.

Nonphysician Evaluators and Recording-Based Tools in Surgical Skill Assessment: A Feasibility Study

OBJECTIVES

To investigate the feasibility of nonmedically trained evaluators and image- and video-based tools in the assessment of surgical skills in a key orthopedic procedure.

DESIGN

Orthopedic surgeons at varying skill levels were evaluated by their ability to repair a cadaveric bi-malleolar ankle fracture. Nonphysician viewers and expert orthopedic surgeons independently scored video recordings and fluoroscopy images of the procedure through Global Rating Scales (GRS) and procedure-specific checklist tools. Statistical analysis was used to determine if the evaluators and assessment tools were able to differentiate skill level.

SETTING

An academic tertiary care hospital.

PARTICIPANTS

The surgical procedure was completed by 3 orthopedic residents, 3 orthopedic trauma fellows, and 4 orthopedic trauma attending surgeons. The procedure was independently evaluated by 2 orthopedic surgeons and 2 nonphysicians.

RESULTS

Operating participants were stratified by ≤ or >10 bimalleolar ankle fracture cases performed alone (inexperienced, n = 5 vs experienced, n = 5). Expert surgeon viewers could effectively stratify skill group through the GRS for video and fluoroscopy analysis (p < 0.05), and the video procedure-specific checklist (p < 0.05), but not the fluoroscopy procedure-specific checklist. Nonphysician viewers generally recognized skill groupings, although with less separation than surgeon viewers. These evaluators performed the best when aided by video and fluoroscopy procedure-specific checklists. Meanwhile, breakdowns of each tool into critical zones for improvement and evaluator-independent metrics such as case experience, self-reported confidence, and surgical time also indicated some skill differentiation.

CONCLUSIONS

The feasibility of using video recordings and fluoroscopic imaging based surgical skills assessment tools in orthopedic trauma was demonstrated. The tools highlighted in this study are applicable to both cadaver laboratory settings and live surgeries. The degree of training that is required by the evaluators and the utility of measuring surgical times of specific tasks should be the subject of future studies.

Supporting self-regulated-learning in colonoscopy training-A comparison cohort trial

Aim

This study aimed to demonstrate that using a self-regulated learning (SRL) approach can improve colonoscopy performance skills.

Background

Colonoscopy is the gold standard for detecting colorectal cancer and removing its precursors: polyps. Acquiring proficiency in colonoscopy is challenging, requiring completion of several hundred procedures. SRL seems to be beneficial to help trainees acquire competencies in regulating their future learning processes and enhance the outcomes of current learning situations. SRL is a learner-centred approach that refers to a trainee's ability to understand and control their learning environment, including cognitions, motivations and emotions. The key abilities include self- and situational awareness, task analysis, and strategic planning. This study is the first to use an SRL approach for workplace-based colonoscopy training.

Methods

In this comparison cohort trial, participants used two SRL supports: a self-review of videotaped performance, and an online learning platform with procedural and conceptual knowledge about colonoscopy. In the control cohort, participants performed patient-based colonoscopy as usual in their departments. Improvement was monitored via three video-based ratings (study start, end of the study period, and follow-up) using the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). Outcomes were analysed using two-way analysis of variance with repeated measurements.

Results

This study recruited 21 participants (12, intervention cohort; nine, control cohort); 58 videos were recorded. The intraclass correlation coefficient was 0.88 (95% CI 0.61-0.98; p < 0.001). The global rating scale (GRS) and checklist (CL) in GiECAT were analysed separately. No statistically significant main effects of cohort (GRS: F(1,16) = 2.84, p = 0.11; CL: F(1,16) = 1.06, p = 0.32), test (GRS: F(2,32) = 2.56, p = 0.09; CL: F(2,32) = 0.76, p = 0.48), or interactions between cohort and test were observed (GRS: F(2,32) = 1.16, p = 0.33; CL: F(2,32) = 1.01, p = 0.37).

Conclusions

SRL in patient-based colonoscopy is feasible; however, no clear effect on performance scores was observed.

Pediatric endoscopy: how can we improve patient outcomes and ensure best practices?

INTRODUCTION

Strategies to promote high-quality endoscopy in children require consensus around pediatric-specific quality standards and indicators. Using a rigorous guideline development process, the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN) was developed to support continuous quality improvement efforts within and across pediatric endoscopy services.

AREAS COVERED

This review presents a framework, informed by the PEnQuIN guidelines, for assessing endoscopist competence, granting procedural privileges, audit and feedback, and for skill remediation, when required. As is critical for promoting quality, PEnQuIN indicators can be benchmarked at the individual endoscopist, endoscopy facility, and endoscopy community levels. Furthermore, efforts to incorporate technologies, including electronic medical records and artificial intelligence, into endoscopic quality improvement processes can aid in creation of large-scale networks to facilitate comparison and standardization of quality indicator reporting across sites.

EXPERT OPINION

PEnQuIN quality standards and indicators provide a framework for continuous quality improvement in pediatric endoscopy, benefiting individual endoscopists, endoscopy facilities, and the broader endoscopy community. Routine and reliable measurement of data, facilitated by technology, is required to identify and drive improvements in care. Engaging all stakeholders in endoscopy quality improvement processes is crucial to enhancing patient outcomes and establishing best practices for safe, efficient, and effective pediatric endoscopic care.

Promoting Research that Supports High-Quality Gastrointestinal Endoscopy in Children

PURPOSE OF REVIEW

Defining and measuring the quality of endoscopic care is a key component of performing gastrointestinal endoscopy in children. The purpose of this review is to discuss quality metrics for pediatric gastrointestinal endoscopy and identify where additional research is needed.

RECENT FINDINGS

Pediatric-specific standards and indicators were recently defined by the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN) working group through a rigorous guideline consensus process. Although the aim of these guidelines is to facilitate best practices for safe and high-quality gastrointestinal endoscopy in children, they highlight the pressing need to expand upon the body of evidence supporting these standards and indicators as predictors of clinically relevant outcomes. In this review, we propose and discuss ideas for several high-yield research topics to engage pediatric endoscopists and promote best practices in pediatric endoscopy.

Video-Based versus On-Site Neonatal Pain Assessment in Neonatal Intensive Care Units: The Impact of Video-Based Neonatal Pain Assessment in Real-World Scenario on Pain Diagnosis and Its Artificial Intelligence Application

BACKGROUND

Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management.

PURPOSE

To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications.

SETTING

A total of 598 neonates were recruited from a pediatric hospital in China.

METHODS

This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively.

RESULTS

The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods.

CONCLUSION

VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

The Toronto Upper Gastrointestinal Cleaning Score: a prospective validation study

BACKGROUND : Assessment of mucosal visualization during esophagogastroduodenoscopy (EGD) can be improved with a standardized scoring system. To address this need, we created the Toronto Upper Gastrointestinal Cleaning Score (TUGCS). METHODS : We developed the TUGCS using Delphi methodology, whereby an international group of endoscopy experts iteratively rated their agreement with proposed TUGCS items and anchors on a 5-point Likert scale. After each Delphi round, we analyzed responses and refined the TUGCS using an 80 % agreement threshold for consensus. We used the intraclass correlation coefficient (ICC) to assess inter-rater and test-retest reliability. We assessed internal consistency with Cronbach's alpha and item-total and inter-item correlations with Pearson's correlation coefficient. We compared TUGCS ratings with an independent endoscopist's global rating of mucosal visualization using Spearman's ρ. RESULTS : We achieved consensus with 14 invited participants after three Delphi rounds. Inter-rater reliability was high at 0.79 (95 %CI 0.64-0.88). Test-retest reliability was excellent at 0.83 (95 %CI 0.77-0.87). Cronbach's α was 0.81, item-total correlation range was 0.52-0.70, and inter-item correlation range was 0.38-0.74. There was a positive correlation between TUGCS ratings and a global rating of visualization (r = 0.41, P = 0.002). TUGCS ratings for EGDs with global ratings of excellent were significantly higher than those for EGDs with global ratings of fair (P = 0.01). CONCLUSION : The TUGCS had strong evidence of validity in the clinical setting. The international group of assessors, broad variety of EGD indications, and minimal assessor training improves the potential for dissemination.

JAG consensus statements for training and certification in flexible sigmoidoscopy

INTRODUCTION

Joint Advisory Group (JAG) certification in endoscopy is awarded when trainees attain minimum competency standards for independent practice. A national evidence-based review was undertaken to update standards for training and certification in flexible sigmoidoscopy (FS).

METHODS

A modified Delphi process was conducted between 2019 and 2020 with multisociety representation from experts and trainees. Following literature review and Grading of Recommendations, Assessment, Development and Evaluations appraisal, recommendation statements on FS training and certification were formulated and subjected to anonymous voting to obtain consensus. Accepted statements were peer-reviewed by national stakeholders for incorporation into the JAG FS certification pathway.

RESULTS

In total, 41 recommendation statements were generated under the domains of: definition of competence (13), acquisition of competence (17), assessment of competence (7) and postcertification support (4). The consensus process led to revised criteria for colonoscopy certification, comprising: (A) achieving key performance indicators defined within British Society of Gastroenterology standards (ie, rectal retroversion >90%, polyp retrieval rate >90%, patient comfort <10% with moderate-severe discomfort); (B) minimum procedure count ≥175; (C) performing 15+ procedures over the preceding 3 months; (D) attendance of the JAG Basic Skills in Lower gastrointestinal Endoscopy course; (E) satisfying requirements for formative direct observation of procedural skill (DOPS) and direct observation of polypectomy skill (SMSA level 1); (F) evidence of reflective practice as documented on the JAG Endoscopy Training System reflection tool and (G) successful performance in summative DOPS.

CONCLUSION

The UK standards for training and certification in FS have been updated to support training, uphold standards in FS and polypectomy, and provide support to the newly independent practitioner.

JAG consensus statements for training and certification in colonoscopy

INTRODUCTION

In the UK, endoscopy certification is awarded when trainees attain minimum competency standards for independent practice. A national evidence-based review was undertaken to update and develop standards and recommendations for colonoscopy training and certification.

METHODS

Under the oversight of the Joint Advisory Group (JAG), a modified Delphi process was conducted between 2019 and 2020 with multisociety expert representation. Following literature review and Grading of Recommendations, Assessment, Development and Evaluations appraisal, recommendation statements on colonoscopy training and certification were formulated and subjected to anonymous voting to obtain consensus. Accepted statements were peer reviewed by JAG and relevant stakeholders for incorporation into the updated colonoscopy certification pathway.

RESULTS

In total, 45 recommendation statements were generated under the domains of: definition of competence (13), acquisition of competence (20), assessment of competence (8) and postcertification support (4). The consensus process led to revised criteria for colonoscopy certification, comprising: (1) achieving key performance indicators defined within British Society of Gastroenterology standards (ie, unassisted caecal intubation rate >90%, rectal retroversion >90%, polyp detection rate >15%+, polyp retrieval rate >90%, patient comfort <10% with moderate-severe discomfort); (2) minimum procedure count 280+; (3) performing 15+ procedures over the preceding 3 months; (4) attendance of the JAG Basic Skills in Colonoscopy course; (5) terminal ileal intubation rates of 60%+ in inflammatory bowel disease; (6) satisfying requirements for formative direct observation of procedure skills (DOPS) and direct observation of polypectomy skills (Size, Morphology, Site, Access (SMSA) level 2); (7) evidence of reflective practice as documented on the JAG Endoscopy Training System reflection tool; (8) successful performance in summative DOPS.

CONCLUSION

The UK standards for training and certification in colonoscopy have been updated, culminating in a single-stage certification process with emphasis on polypectomy competency (SMSA Level 2+). These standards are intended to support training, improve standards of colonoscopy and polypectomy, and provide support to the newly independent practitioner.

Endoscopy training in Australia during COVID-19: Efficacy and knowledge assessment of gastroenterology and general surgery trainees

BACKGROUND AND AIM

During COVID-19, restrictions to elective endoscopy were introduced worldwide. A reduction in procedures may impact trainees' endoscopy learning. This study aims to assess Australian advanced gastroenterology and general surgery trainees' self-perceived efficacy and knowledge in endoscopy during the pandemic.

METHODS

All Australian gastroenterology and general surgery trainees in their last 2 years of accredited training were invited to participate through email (2020-2021 and 2021-2022 training cycles). The primary outcome was to assess trainees' self-efficacy and knowledge regarding gastrointestinal endoscopy. Secondary outcomes included subgroup analysis between gastroenterology and general surgery trainees. Self-perceived efficacy was assessed with Likert-scale questions on 20 endoscopy procedures and knowledge was assessed through 21 endoscopy-related multiple choice questions.

RESULTS

Eighty-one trainees responded to a self-efficacy questionnaire and 77 responded to the knowledge questionnaire. Over 90% of the trainees were confident or extremely confident in diagnostic endoscopy, but only half demonstrated similar efficacy for therapeutic endoscopy. The efficacy for basic endoscopy procedures was higher for gastroenterology trainees (64.0% vs 51.1%, P < 0.001). Last-year trainee achievement of conjoint committee requirements for upper gastrointestinal endoscopy was achieved in 95.8% of gastroenterology trainees versus 22.2% of surgical trainees (P < 0.001). The median score on the knowledge questionnaire was also higher for the gastroenterology subset (90.5% vs 71.4%, P < 0.001).

CONCLUSION

During COVID-19, endoscopy trainees' self-efficacy in endoscopic diagnostic procedures was achieved for most trainees. The differences in self-perceived efficacy and knowledge between gastroenterology and surgical trainees may be reflective of the different opportunities for learning between the two groups.

The Role of Online Videos in Teaching Procedural Skills in Postgraduate Medical Education: A Scoping Review

OBJECTIVE

The purpose of this scoping review was to outline the extent of available literature including the prevalence of video quality appraisal tools, characterize how online videos were used, and identify the gaps in the literature with implications for future research.

DESIGN

The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for scoping reviews (PRISMA-ScR). Six databases were searched: (1) MEDLINE (Ovid), (2) EMBASE, (3) Cochrane, (4) ERIC, (5) CINAHL PLUS, and (6) Google Scholar for Medical Subject Headings terms online videos, postgraduate health education, and health professional/s.

RESULTS

A total of 6948 articles were identified, of which 78 were included in the review. The articles included were primarily either experimental or observational studies, with most being from North America. Twelve concepts were identified: (1) procedural skills teaching; (2) video assessment; (3) validation of an assessment tool; (4) video feedback; (5) coaching; (6) broadcasting; (7) learner characteristics; (8) video characteristics; (9) video quality; (10) a quality assessment tool; (11) platforms and video library; and (12) health information governance.

CONCLUSIONS

While there is a diverse and growing body of literature on this area, information is lacking about the quality appraisal of online videos.

Peer evaluation and feedback for invasive medical procedures: a systematic review

BACKGROUND

There is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures.

METHODS

A systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available.

RESULTS

Of 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review.

CONCLUSIONS

Peer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.

V-QBA vs. QBA—How Do Video and Live Analysis Compare for Qualitative Behaviour Assessment?

Animal welfare is an inextricable part of livestock production and sustainability. Assessing welfare, beyond physical indicators of health, is challenging and often relies on qualitative techniques. Behaviour is a key component of welfare to consider and Qualitative Behaviour Assessment (QBA) aims to achieve this by systematically scoring behaviour across specific terms. In recent years, numerous studies have conducted QBA by using video footage, however, the method was not originally developed using video and video QBA (V-QBA) requires validation. Forty live QBAs were conducted, by two assessors, on housed beef cattle to help fill this validation gap. Video was recorded over the assessment period and a second video assessment was conducted. Live and video scores for each term were compared for both correlation and significant difference. Principle component analysis (PCA) was then conducted and correlations and differences between QBA and V-QBA for the first two components were calculated. Of the 20 terms, three were removed due to an overwhelming majority of scores of zero. Of the remaining 17 terms, 12 correlated significantly, and a significant pairwise difference was found for one (“Bored”). QBA and V-QBA results correlated across both PC1 (defined as “arousal”) and PC2 (defined as “mood”). Whilst there was no significant difference between the techniques for PC1, there was for PC2, with V-QBA generally yielding lower scores than QBA. Furthermore, based on PC1 and PC2, corresponding QBA and V-QBA scores were significantly closer than would be expected at random. Results found broad agreement between QBA and V-QBA at both univariate and multivariate levels. However, the lack of absolute agreement and muted V-QBA results for PC2 mean that caution should be taken when implementing V-QBA and that it should ideally be treated independently from live QBA until further evidence is published. Future research should focus on a greater variety of animals, environments, and assessors to address further validation of the method.

Colonoscopy competence assessment tools: a systematic review of validity evidence

BACKGROUND

Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity.

METHODS

We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1-15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5-11.5 (maximum score 14.5).

CONCLUSIONS

The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

Self-assessment of Competence in Endoscopy: Challenges and Insights

BACKGROUND

Endoscopists use self-assessment to monitor the development and maintenance of their skills. The accuracy of these self-assessments, which reflects how closely one's own rating corresponds to an external rating, is unclear.

METHODS

In this narrative review, we critically examine the current literature on self-assessment in gastrointestinal endoscopy with the aim of informing training and practice and identifying opportunities to improve the methodological rigor of future studies.

RESULTS

In the seven included studies, the evidence regarding self-assessment accuracy was mixed. When stratified by experience level, however, novice endoscopists were least accurate in their self-assessments and tended to overestimate their performance. Studies examining the utility of video-based interventions using observation of expert benchmark performances show promise as a mechanism to improve self-assessment accuracy among novices.

CONCLUSIONS

Based on the results of this review, we highlight problematic areas, identify opportunities to improve the methodological rigor of future studies on endoscopic self-assessment and outline potential avenues for further exploration.

Development and initial validation of an instrument for video-based assessment of technical skill in ERCP

BACKGROUND AND AIMS

The accurate measurement of technical skill in ERCP is essential for endoscopic training, quality assurance, and coaching of this procedure. Hypothesizing that technical skill can be measured by analysis of ERCP videos, we aimed to develop and validate a video-based ERCP skill assessment tool.

METHODS

Based on review of procedural videos, the task of ERCP was deconstructed into its basic components by an expert panel that developed an initial version of the Bethesda ERCP Skill Assessment Tool (BESAT). Subsequently, 2 modified Delphi panels and 3 validation exercises were conducted with the goal of iteratively refining the tool. Fully crossed generalizability studies investigated the contributions of assessors, ERCP performance, and technical elements to reliability.

RESULTS

Twenty-nine technical elements were initially generated from task deconstruction. Ultimately, after iterative refinement, the tool comprised 6 technical elements and 11 subelements. The developmental process achieved consistent improvements in the performance characteristics of the tool with every iteration. For the most recent version of the tool, BESAT-v4, the generalizability coefficient (a reliability index) was .67. Most variance in BESAT scores (43.55%) was attributed to differences in endoscopists' skill, indicating that the tool can reliably differentiate between endoscopists based on video analysis.

CONCLUSIONS

Video-based assessment of ERCP skill appears to be feasible with a novel instrument that demonstrates favorable validity evidence. Future steps include determining whether the tool can discriminate between endoscopists of varying experience levels and predict important outcomes in clinical practice.

Interobserver and intraobserver reliability for 2 grading systems for gastric ulcer syndrome in horses

Background

Grading of equine gastric ulcer syndrome (EGUS) is undertaken in clinical and research settings, but the reliability of EGUS grading systems is poorly understood.

Hypothesis/Objectives

Investigate interobserver and intraobserver reliability of an established ordinal grading system and a novel visual analog scale (VAS), and assess the influence of observer experience.

Animals

Sixty deidentified gastroscopy videos.

Methods

Six observers (3 specialists and 3 residents) graded videos using the EGUS Council (EGUC) system and VAS. Observers graded the videos three 3 for each system, using a cross‐over design with at least 1 week between each phase. The order of videos was randomized for each phase.

Methods

Interobserver and intraobserver reliability were estimated using Gwet's agreement coefficient with ordinal weights applied (AC2) for the EGUC system and the intraclass correlation coefficient (ICC) for the VAS.

Results

Using the EGUC system, interobserver reliability was substantial for squamous (AC2 = 0.69; 95% confidence interval [CI], 0.57‐0.80) and glandular mucosa (AC2 = 0.72; 95% CI, 0.70‐0.75), and intraobserver reliability was substantial for squamous (AC2 = 0.80; 95% CI, 0.71‐0.90) and glandular mucosa (AC2 = 0.80; 95% CI, 0.74‐0.86). Interobserver reliability using the VAS was moderate for squamous (ICC = 0.64; 95% CI, 0.31‐0.96) and poor for glandular mucosa (ICC = 0.35; 95% CI, 0.06‐0.64), and intraobserver reliability was moderate for squamous (ICC = 0.74; 95% CI, 0.62‐0.86) and glandular mucosa (ICC = 0.56; 95% CI, 0.39‐0.72).

Conclusions and Clinical Importance

The EGUC system had acceptable intraobserver and interobserver reliability and performed well regardless of observer experience. Familiarity and observer experience improved reliability of the VAS.

The status of training in new technologies in advanced endoscopy: from defining competence to credentialing and privileging

The landscape of advanced endoscopy continues to evolve as new technologies and techniques become available. Although postgraduate advanced endoscopy fellowships have traditionally centered on ERCP and diagnostic EUS, the breadth of training has increased over the years in response to the ever-growing demand for therapeutic endoscopy. The increasing diversity and complexity of emerging endoscopic techniques accompanied by the shift in focus toward competency-based medical education requires innovative changes to the curriculum that will ensure adequate training yet without compromising best patient practices. The purpose of this review is to highlight the expansive array of advanced endoscopic procedures and the challenges of both defining and measuring competence during training. All authors are interventional endoscopists at their respective institutions performing these complex procedures, as well as training fellows in these techniques. We share our perspectives based on our experience navigating through these issues at our institutions and discuss strategies to standardize training and how to potentially incorporate these measures in the process of credentialing and privileging in endoscopy.

Impact of a simulation-based ergonomics training curriculum on work-related musculoskeletal injury risk in colonoscopy

BACKGROUND AND AIMS

Endoscopists are at risk of developing musculoskeletal injuries (MSIs), and few receive training on ergonomics. The aim of this study was to determine the impact of a simulation-based ergonomics training curriculum (ETC) on work-related MSI risk during clinical colonoscopy.

METHODS

Novice endoscopists underwent a simulation-based ETC and were compared with an historical control group who received simulation-based training without ergonomics training. The ETC included a didactic lecture and video on ergonomics in colonoscopy, feedback from supervisors on ergonomics, and an ergonomics checklist to augment feedback and promote self-reflection. Participants were assessed using the rapid entire body assessment (REBA) and rapid upper limb assessment (RULA). The primary outcome was participants' REBA scores during 2 clinical colonoscopies 4 to 6 weeks after training.

RESULTS

In clinical colonoscopy, the ETC group had superior REBA scores (clinical procedure 1: median score, 6 vs 11; P < .001; clinical procedure 2: median score, 6 vs 10; P < .001). In a simulated colonoscopy, the ETC group did not have significantly different REBA or RULA scores between baseline, immediately after training, and 4 to 6 weeks after (REBA: median scores of 5, 5, and 5, respectively; P > .05; RULA: median scores of 6, 6, and 6, respectively; P > .05). The control group had worsening REBA and RULA scores during the study timeline (REBA: median scores of 5 at baseline, 9 immediately after training, and 9 at 4-6 weeks after training; P < .001; RULA: median scores of 6, 7, and 7, respectively; P = .04) during simulated procedures.

CONCLUSIONS

A simulation-based ETC is associated with reduced risk of MSI during endoscopy. Although the REBA score was improved, the intervention group was still within the medium-risk range.

Understanding and developing procedures for video-based assessment in medical education

INTRODUCTION

Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use.

METHODS

Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods.

RESULTS

Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified.

DISCUSSION

Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.

Educational interventions are associated with improvements in colonoscopy quality indicators: a systematic review and meta-analysis

Background and study aims  The quality of screening-related colonoscopy depends on several physician- and patient-related factors. Adenoma detection rate (ADR) varies considerably between endoscopists. Educational interventions aim to improve endoscopists' ADRs, but their overall impact is uncertain. We aimed to assess whether there is an association between educational interventions and colonoscopy quality indicators. Methods  A comprehensive search was performed through August 2019 for studies reporting any associations between educational interventions and any colonoscopy quality indicators. Our primary outcome of interest was ADR. Two authors assessed eligibility criteria and extracted data independently. Risk of bias was also assessed for included studies. Pooled rate ratios (RR) with 95 % confidence intervals (CI) were reported using DerSimonian and Laird random effects models. Results  From 2,253 initial studies, eight were included in the meta-analysis for ADR, representing 86,008 colonoscopies. Educational interventions were associated with improvements in overall ADR (RR 1.29, 95 % CI 1.25 to 1.42, 95 % prediction interval 1.09 to 1.53) and proximal ADR (RR 1.39, 95 % CI 1.29 to 1.48), with borderline increases in withdrawal time, ([WT], mean difference 0.29 minutes, 95 % CI - 0.12 to 0.70 minutes). Educational interventions did not affect cecal intubation rate ([CIR], RR 1.01, 95 % CI 1.00 to 1.01). Heterogeneity was considerable across many of the analyses. Conclusions  Educational interventions are associated with significant improvements in ADR, in particular, proximal ADR, and are not associated with improvements in WT or CIR. Educational interventions should be considered an important option in quality improvement programs aiming to optimize the performance of screening-related colonoscopy.

Non-technical skills curriculum incorporating simulation-based training improves performance in colonoscopy among novice endoscopists: Randomized controlled trial

BACKGROUND AND AIMS

Non-technical skills (NTS), involving cognitive, social and interpersonal skills that complement technical skills, are important for the completion of safe and efficient procedures. We investigated the impact of a simulation-based curriculum with dedicated NTS training on novice endoscopists' performance of clinical colonoscopies.

METHODS

A single-blinded randomized controlled trial was conducted at a single center. Novice endoscopists were randomized to a control curriculum or a NTS curriculum. The control curriculum involved a didactic session, virtual reality (VR) simulator colonoscopy training, and integrated scenario practice using a VR simulator, a standardized patient, and endoscopy nurse. Feedback and training were provided by experienced endoscopists. The NTS curriculum group received similar training that included a small-group session on NTS, feedback targeting NTS, and access to a self-reflective NTS checklist. The primary outcome was performance during two clinical colonoscopies, assessed using the Joint Advisory Group Direct Observation of Procedural Skills (JAG DOPS) tool.

RESULTS

Thirty-nine participants completed the study. The NTS group (n = 21) had superior clinical performance during their first (P < 0.001) and second clinical colonoscopies (P < .0.001), compared to the control group (n = 18). The NTS group performed significantly better on the VR simulator (P < 0.05) and in the integrated scenario (P < 0.05).

CONCLUSION

Our findings demonstrate that dedicated NTS training led to improved performance of clinical colonoscopies among novices.

Video-Based Assessment in Surgical Education: A Scoping Review

BACKGROUND AND OBJECTIVE

Video-based assessment of residents' surgical skills may offer several advantages over direct observations of clinical performance in terms of objectivity, time-efficiency, and feasibility. Although video-based assessment is becoming more common in surgical training, a broad understanding of its utility is lacking. This scoping review explores video-based assessment in surgical training and presents the evidence supporting its use.

DESIGN

A literature search was conducted using the Web of Science database with key words related to video-based assessment and surgical training. Exclusion criteria included articles not published in English and articles on undergraduate medical education, continuing professional development, or non-surgical disciplines. Initially, 702 articles were identified; after title, abstract, and full-text screening by two independent reviewers (SM and VM), 199 articles remained.

RESULTS

We present the benefits of video-based assessment, including the ability to capture clinical ability in the operating room without decreasing intraoperative efficiency, as well as the potential to improve formative assessment and feedback practices. We describe the validity, reliability, and challenges of video-based assessment, as well as the use of video-based methods in clinical and simulated settings. We conclude by discussing questions that remain to be addressed.

CONCLUSIONS

Although further research and cost-benefit analyses are required, greater adoption of video-based assessment into surgical training may help meet increased assessment demands in an era of competency-based medical education.

Digital recording and documentation of endoscopic procedures: physicians' practice and perspectives

Background

In recent years, it has become increasingly prevalent internationally to record and archive digital recordings of endoscopic procedures. This emerging documentation tool raises weighty educational, ethical and legal issues – which are viewed as both deterrents and incentives to its adoption. We conducted a survey study aimed at evaluating the use of DRD in endoscopic procedures, to examine physicians’ support of this practice and to map the considerations weighed by physicians when deciding whether or not to support a more extensive use of DRD.

Methods

Israeli physicians from specialties that employ endoscopic technics were surveyed anonymously for demographic background, existence and use of recording equipment, existence of institutional guidelines regarding DRD, and self-ranking (on a scale from 1 to 7) of personal attitudes regarding DRD.

Results

322 physicians were surveyed. 84% reported performing routine endoscopic procedures, 78% had the required equipment for digital recording, and 64% of them stated that they never or only rarely actually recorded the procedure. General surgeons had the second highest rate of DRD equipment (96.5%) but the lowest rates of DRD practice (17.5%). The average ranking of support of DRD by all participants was 5.07 ± 1.9, indicating a moderately high level of support. Significant positive correlation exists between actual DRD rates and average support of DRD (p < 0.001). Based on mediation models, for all specialties and with no exceptions, having routine recording guidelines and positive support of DRD were found to increase the probability of actual recording. Being a surgeon or an urologist negatively correlated with support of DRD, and decreased actual recording rates. The argument “Recording might cause more lawsuits” was ranked significantly higher than all other arguments against DRD (p < 0.001), and “Recording could aid teaching of interns” was ranked higher than all other arguments in favor of DRD (p < 0.001).

Conclusions

While DRD facilities and equipment are fairly widespread in Israel, the actual recording rate is generally low and varies among specialties. Having institutional guidelines requiring routine recording and a positive personal support of DRD correlated with actual DRD rates, with general surgeons being markedly less supportive of DRD and having the lowest actual recording rates. Physicians in all specialties were very much concerned about DRD’s potential to enhance lawsuits, and this greatly influenced their use of DRD.

These findings should be addressed by educational efforts, centering on professionals from reluctant specialties, as well as by the issuing of both professional and institutional guidelines endorsing DRD as well as requiring it where applicable.

Electronic supplementary material

The online version of this article (10.1186/s13584-019-0332-6) contains supplementary material, which is available to authorized users.

Development and validation of a video-based cold snare polypectomy assessment tool (with videos)

BACKGROUND AND AIMS

Polyps <1 cm are the most commonly found polyps during colonoscopy. Cold snare removal is preferred given the significantly higher rate of incomplete resection associated with piecemeal biopsy forceps resection. There are currently no standardized tools to assess competence in cold snare polypectomy. This study aimed to develop and validate a cold snare polypectomy assessment tool (CSPAT).

METHODS

Experts in cold snare polypectomy used a Delphi method to develop the CSPAT. Metrics with a greater than 85% agreement as being "important" or "very important" were included in the CSPAT. The tool included evaluation of polyp inspection, positioning, appropriate ensnarement of tissue to ensure a rim of normal tissue, tissue retrieval, and postpolypectomy site inspection. Experts in cold snare polypectomy used the CSPAT to evaluate preselected videos that were previously evaluated using the Direct Observation of Polypectomy Skills (DOPyS) tool. Interobserver agreement was evaluated. CSPAT scores were compared with DOPyS scores to assess content validity.

RESULTS

Sixteen experts developed the 12-item CSPAT, and 13 experts reviewed 55 videos. There was a moderate degree of agreement in 10 metrics (κ = .52-.59) and a substantial degree of agreement (κ = .61-.63) in the other 2. There was a strong correlation between the mean of individual metrics 1 to 12 compared with the global competence assessment (ρ = .88, P < .001). There was a moderate correlation between the average overall DOPyS score and the overall CSPAT competence score (ρ = .56, P < .001).

CONCLUSIONS

We have developed and validated a customized CSPAT that can be easily applied to video-based assessments to assess competence in training and among practicing endoscopists.

Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial

Background and study aims  Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists' self-assessment accuracy of endoscopic competence. Materials and methods  Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS). Results  Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group ( P  = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time ( P  = .016). There were no significant effects of group or of time for the SVR group. Conclusions  Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

Protocol for a randomised trial evaluating the effect of applying gamification to simulation-based endoscopy training

BACKGROUND

Simulation-based training (SBT) provides a safe environment and effective means to enhance skills development. Simulation-based curricula have been developed for a number of procedures, including gastrointestinal endoscopy. Gamification, which is the application of game-design principles to non-game contexts, is an instructional strategy with potential to enhance learning. No studies have investigated the effects of a comprehensive gamification curriculum on the acquisition of endoscopic skills among novice endoscopists.

METHODS AND ANALYSIS

Thirty-six novice endoscopists will be randomised to one of two endoscopy SBT curricula: (1) the Conventional Curriculum Group, in which participants will receive 6 hours of one-on-one simulation training augmented with expert feedback and interlaced with 4 hours of small group teaching on the theory of colonoscopy or (2) the Gamified Curriculum Group, in which participants will receive the same curriculum with integration of the following game-design elements: a leaderboard summarising participants' performance, game narrative, achievement badges and rewards for top performance. In line with a progressive learning approach, simulation training for participants will progress from low to high complexity simulators, starting with a bench-top model and then moving to the EndoVR virtual reality simulator. Performance will be assessed at three points: pretraining, immediately post-training and 4-6 weeks after training. Assessments will take place on the simulator at all three time points and transfer of skills will be assessed during two clinical colonoscopies 4-6 weeks post-training. Mixed factorial ANOVAs will be used to determine if there is a performance difference between the two groups during simulated and clinical assessments.

ETHICS AND DISSEMINATION

Ethical approval was obtained at St. Michael's Hospital. Results of this trial will be submitted for presentation at academic meetings and for publication in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

NCT03176251.