Using structured progress to measure competence in flexible bronchoscopy

Artificial Intelligence-Guided Bronchoscopy is Superior to Human Expert Instruction for the Performance of Critical-Care Physicians: A Randomized Controlled Trial

OBJECTIVES

Bronchoscopy in the mechanically ventilated patient is an important skill for critical-care physicians. However, training opportunity is heterogenous and limited by infrequent caseload or inadequate instructor feedback for satisfactory competencies. A new artificial intelligence (AI) navigational system using augmented reality - the Ambu Broncho Simulator - can guide bronchoscopy training. Is training with the AI system comparable to bedside, expert tutor instruction in improving bronchoscopy performance?

DESIGN

A nonblinded, parallel group randomized controlled trial was conducted.

SETTING

The study was conducted in a simulated setting at an academic university hospital.

SUBJECTS

Critical-care physicians were invited to take part in the study.

INTERVENTIONS

Forty participants received 30 minutes of bronchoscopy training, either guided by AI only (artificial intelligence group [AIG]) or by expert tutor feedback (expert tutor group [ETG]). All participants performed a final full navigation bronchoscopy performance test and completed a cognitive load questionnaire, the NASA Task Load Index .

MEASUREMENTS AND MAIN RESULTS

Mean intersegmental time (MIT = PT/DC), diagnostic completeness (DC), procedure time (PT), structured progress (SP), and number of segments revisited (SR) were measured. The primary outcome measure assessed was MIT, a measure of bronchoscopic performance efficiency. The secondary outcome measures were DC, PT, SP, and SR. Nineteen participants were randomized to the AIG and 21 participants to the ETG. MIT, PT, and SR were significantly better in the AIG compared to the ETG (median difference, p ): MIT (-7.9 s, 0.027), PT (-77 s, 0.022), SR (-7 segments, 0.019); all showing moderate effect sizes (0.35, 0.36, and 0.37, respectively) as per Cohen's classification.There was no significant difference between the groups for all other final test measures.

CONCLUSIONS

Training using an AI system resulted in faster and more efficient bronchoscopy performance by critical-care physicians when compared to expert human tutor instruction. This could change the future of bronchoscopy training in critical care and warrants validation in patients through clinical studies.

Artificial intelligence improves bronchoscopy performance: a randomised crossover trial

Rationale

Flexible bronchoscopy is an operator-dependent procedure. An automatic bronchial identification system based on artificial intelligence (AI) could help bronchoscopists to perform more complete and structured procedures through automatic guidance.

Methods

101 participants were included from six different continents at the European Respiratory Society annual conference in Milan, 9-13 September 2023. Participants were split into three groups based on experience: novices (0 bronchoscopies), intermediates (1-249 bronchoscopies) and experienced (≥250 bronchoscopies). The participants performed two bronchoscopies on a realistic physical phantom, one with AI (AmbuBronchoSimulatorTrainingGUIDEv.0.0.1, Prototype version, Ambu) and one Standard procedure. The F1-group received AI guidance for their first procedure, the F2-group for their second. A crossover randomisation controlled for learning by testing. All procedures were automatically rated according to the outcome measures: inspected segments, structured progressions and procedure time.

Results

AI guidance caused the participants to inspect more segments (mean difference, paired t-test: +6.0 segments, p<0.001), perform more structured progressions (+5.2 progressions, p<0.001) and spend more time on the procedure (+72 s, p<0.001) compared to their standard procedures. The effects of AI guidance on inspected segments and structured progression were highest for novices but significant for all experience groups: novices (+8.2 segments, p=0.012 and +6.6 progressions, p<0.001), intermediates (+5.7 segments, p=0.006 and +5.1 progressions, p<0.001) and experienced (+4.3 segments, p=0.006 and +3.8 progressions, p<0.016).

Conclusions

AI guidance helped bronchoscopists of all experience levels to inspect more segments in a more structured order. Clinical implementation of AI guidance could help ensure and document more complete bronchoscopy procedures in the future.

Artificial intelligence augmentation raises questions about the future of bronchoscopy

This editorial discusses the article by Coldet al. demonstrating improvements in bronchoscopy on a model when aided by artificial intelligence (AI) software. It explores hypothetical benefits and concerns stemming from AI-enhanced bronchoscopy. https://bit.ly/3BAExJs.

Mastery Learning Guided by Artificial Intelligence Is Superior to Directed Self-Regulated Learning in Flexible Bronchoscopy Training: An RCT

INTRODUCTION

Simulation-based training has proven effective for learning flexible bronchoscopy. However, no studies have tested the efficacy of training toward established proficiency criteria, i.e., mastery learning (ML). We wish to test the effectiveness of ML compared to directed self-regulated learning (DSRL) on novice bronchoscopists' end-of-training performance.

METHODS

In a standardized simulated setting, novices without prior bronchoscopy experience were trained using an artificial intelligence (AI) guidance system that automatically recognizes the bronchial segments. They were randomized into two groups: the ML group and the DSRL group. The ML group was trained until they completed two procedures meeting the proficiency targets: 18 inspected segments, 18 structured progressions, <120-s procedure time. The DSRL group was trained until they no longer perceived any additional benefits from training. Both groups then did a finalizing test, without the AI guidance enabled.

RESULTS

A total of 24 participants completed the study, with 12 in each group. Both groups had a high mean number of inspected segments (ML = 17.2 segments, DSRL = 17.3 segments, p = 0.85) and structured progressions (ML = 15.5 progressions, DSRL = 14.8 progressions, p = 0.58), but the ML group performed the test procedure significantly faster (ML = 107 s, DSRL = 180 s, p < 0.001). The ML did not spend significantly longer time training (ML = 114 min, DSRL = 109 min, p = 0.84).

CONCLUSIONS

ML is a very efficient training form allowing novice trainees to learn how to perform a thorough, systematic, and quick flexible bronchoscopy. ML does not require longer time spent training compared to DSRL, and we therefore recommend training of future bronchoscopists by this method.

Artificial intelligence for automatic and objective assessment of competencies in flexible bronchoscopy

Background

Bronchoscopy is a challenging technical procedure, and assessment of competence currently relies on expert raters. Human rating is time consuming and prone to rater bias. The aim of this study was to evaluate if a bronchial segment identification system based on artificial intelligence (AI) could automatically, instantly, and objectively assess competencies in flexible bronchoscopy in a valid way.

Methods

Participants were recruited at the Clinical Skills Zone of the European Respiratory Society Annual Conference in Milan, 9th-13th September 2023. The participants performed one full diagnostic bronchoscopy in a simulated setting and were rated immediately by the AI according to its four outcome measures: diagnostic completeness (DC), structured progress (SP), procedure time (PT), and mean intersegmental time (MIT). The procedures were video-recorded and rated after the conference by two blinded, expert raters using a previously validated assessment tool with nine items regarding anatomy and dexterity.

Results

Fifty-two participants from six different continents were included. All four outcome measures of the AI correlated significantly with the experts' anatomy-ratings (Pearson's correlation coefficient, P value): DC (r=0.47, P<0.001), SP (r=0.57, P<0.001), PT (r=-0.32, P=0.02), and MIT (r=-0.55, P<0.001) and also with the experts' dexterity-ratings: DC (r=0.38, P=0.006), SP (r=0.53, P<0.001), PT (r=-0.34, P=0.014), and MIT (r=-0.47, P<0.001).

Conclusions

The study provides initial validity evidence for AI-based immediate and automatic assessment of anatomical and navigational competencies in flexible bronchoscopy. SP provided stronger correlations with human experts' ratings than the traditional DC.

Artificial Intelligence Improves Novices' Bronchoscopy Performance: A Randomized Controlled Trial in a Simulated Setting

BACKGROUND

Navigating through the bronchial tree and visualizing all bronchial segments is the initial step toward learning flexible bronchoscopy. A novel bronchial segment identification system based on artificial intelligence (AI) has been developed to help guide trainees toward more effective training.

RESEARCH QUESTION

Does feedback from an AI-based automatic bronchial segment identification system improve novice bronchoscopists' end-of-training performance?

STUDY DESIGN AND METHODS

The study was conducted as a randomized controlled trial in a standardized simulated setting. Novices without former bronchoscopy experience practiced on a mannequin. The feedback group (n = 10) received feedback from the AI, and the control group (n = 10) trained according to written instructions. Each participant decided when to end training and proceed to performing a full bronchoscopy without any aids.

RESULTS

The feedback group performed significantly better on all three outcome measures (median difference, P value): diagnostic completeness (3.5 segments, P < .001), structured progress (13.5 correct progressions, P < .001), and procedure time (-214 seconds, P = .002).

INTERPRETATION

Training guided by this novel AI makes novices perform more complete, more systematic, and faster bronchoscopies. Future studies should examine its use in a clinical setting and its effects on more advanced learners.

Effectiveness of Flexible Bronchoscopy Simulation-Based Training: A Systematic Review

BACKGROUND

The implementation of simulation-based training (SBT) to teach flexible bronchoscopy (FB) skills to novice trainees has increased during the last decade. However, it is unknown whether SBT is effective to teach FB to novices and which instructional features contribute to training effectiveness.

RESEARCH QUESTION

How effective is FB SBT and which instructional features contribute to training effectiveness?

STUDY DESIGN AND METHODS

We searched Embase, PubMed, Scopus, and Web of Science for articles on FB SBT for novice trainees, considering all available literature until November 10, 2022. We assessed methodological quality of included studies using a modified version of the Medical Education Research Study Quality Instrument, evaluated risk of bias with relevant tools depending on study design, assessed instructional features, and intended to correlate instructional features to outcome measures.

RESULTS

We identified 14 studies from an initial pool of 544 studies. Eleven studies reported positive effects of FB SBT on most of their outcome measures. However, risk of bias was moderate or high in eight studies, and only six studies were of high quality (modified Medical Education Research Study Quality Instrument score ≥ 12.5). Moreover, instructional features and outcome measures varied highly across studies, and only four studies evaluated intervention effects on behavioral outcome measures in the patient setting. All of the simulation training programs in studies with the highest methodological quality and most relevant outcome measures included curriculum integration and a range in task difficulty.

INTERPRETATION

Although most studies reported positive effects of simulation training programs on their outcome measures, definitive conclusions regarding training effectiveness on actual bronchoscopy performance in patients could not be made because of heterogeneity of training features and the sparse evidence of training effectiveness on validated behavioral outcome measures in a patient setting.

TRIAL REGISTRATION

PROSPERO; No.: CRD42021262853; URL: https://www.crd.york.ac.uk/prospero/.

Assessment of esophagogastroduodenoscopy skills on simulators before real-life performance

Background and study aims  Operator competency is essential for esophagogastroduodenoscopy (EGD) quality, which makes appropriate training with a final test important. The aims of this study were to develop a test for assessing skills in performing EGD, gather validity evidence for the test, and establish a credible pass/fail score. Methods  An expert panel developed a practical test using the Simbionix GI Mentor II simulator (3 D Systems) and an EGD phantom (OGI 4, CLA Medical) with a diagnostic (DP) and a technical skills part (TSP) for a prospective validation study. During the test a supervisor measured: 1) total time; 2) degree of mucosal visualization; and 3) landmarks and pathology identification. The contrasting groups standard setting method was used to establish a pass/fail score. Results  We included 15 novices (N), 10 intermediates (I), and 10 experienced endoscopists (E). The internal structure was high with a Cronbach's alpha of 0.76 for TSP time consumption and 0.74 for the identification of landmarks. Mean total times, in minutes, for the DP were N 15.7, I 11.3, and E 7.0, and for TSP., they were N 7.9, I 8.9, and E 2.9. The total numbers of identified landmarks were N 26, I 41, and E 48. Mean visualization percentages were N 80, I 71, and E 71. A pass/fail standard was established requiring identification of all landmarks and performance of the TSP in < 5 minutes. All experienced endoscopists passed, while none of the endoscopists in the other categories did. Conclusions  We established a test that can distinguish between participants with different competencies. This enables an objective and evidence-based approach to assessment of competencies in EGD.