Prospective evaluation of a teaching model to determine competency in performing flexible sigmoidoscopies

Validation of an Educational Tool for Skin Abscess Incision and Drainage by Delphi and Angoff Methods

BACKGROUND

Bedside incision and drainage (I&D) of skin abscesses is a common medical procedure performed in a variety of medical settings. Yet, there is a paucity of published validated educational tools to teach and assess competency for this procedure.

OBJECTIVE

To validate an educational tool to teach and assess competency for bedside I&D of skin abscesses via the Delphi consensus and Angoff standard setting methods.

DESIGN

Expert consensus on the importance of each procedural step in the educational tool was obtained using the Delphi method, consisting of four rounds of iterative revisions based on input from a panel of experts. The passing cut-off score for a proficient provider was determined using the modified dichotomous Angoff method.

PARTICIPANTS

All participants met the minimum criteria of active involvement in resident education and performance of at least 20 skin abscess I&D's within the past 5 years. Participant specialties included general surgery, emergency medicine, and internal medicine.

MAIN MEASURES

The primary outcome was consensus on procedural steps and errors, defined as an interquartile range ≤ 2 on a 9-point Likert scale. A cut-off score was determined by the average across all respondents for the anticipated number of errors that would be committed by a provider with the level of proficiency defined in the survey. Qualitative input was incorporated into the educational tool.

KEY RESULTS

At the end of four rounds of review via the Delphi process, participants achieved consensus on 93% of items on the clinical checklist and 85% of errors on the assessment checklist. Via the modified dichotomous Angoff method, the determined passing cut-off for competency was 6 out of 22 errors.

CONCLUSION

An educational and evaluation tool for bedside I&D of skin abscesses was validated via the Delphi and Angoff methods.

Training and competence assessment in GI endoscopy: a systematic review

INTRODUCTION

Training procedural skills in GI endoscopy once focused on threshold numbers. As threshold numbers poorly reflect individual competence, the focus gradually shifts towards a more individual approach. Tools to assess and document individual learning progress are being developed and incorporated in dedicated training curricula. However, there is a lack of consensus and training guidelines differ worldwide, which reflects uncertainties on optimal set-up of a training programme.

AIMS

The primary aim of this systematic review was to evaluate the currently available literature for the use of training and assessment methods in GI endoscopy. Second, we aimed to identify the role of simulator-based training as well as the value of continuous competence assessment in patient-based training. Third, we aimed to propose a structured training curriculum based on the presented evidence.

METHODS

A literature search was carried out in the available medical and educational literature databases. The results were systematically reviewed and studies were included using a predefined protocol with independent assessment by two reviewers and a final consensus round.

RESULTS

The literature search yielded 5846 studies. Ninety-four relevant studies on simulators, assessment methods, learning curves and training programmes for GI endoscopy met the inclusion criteria. Twenty-seven studies on simulator validation were included. Good validity was demonstrated for four simulators. Twenty-three studies reported on simulator training and learning curves, including 17 randomised control trials. Increased performance on a virtual reality (VR) simulator was shown in all studies. Improved performance in patient-based assessment was demonstrated in 14 studies. Four studies reported on the use of simulators for assessment of competence levels. Current simulators lack the discriminative power to determine competence levels in patient-based endoscopy. Eight out of 14 studies on colonoscopy, endoscopic retrograde cholangiopancreatography and endosonography reported on learning curves in patient-based endoscopy and proved the value of this approach for measuring performance. Ten studies explored the numbers needed to gain competence, but the proposed thresholds varied widely between them. Five out of nine studies describing the development and evaluation of assessment tools for GI endoscopy provided insight into the performance of endoscopists. Five out of seven studies proved that intense training programmes result in good performance.

CONCLUSIONS

The use of validated VR simulators in the early training setting accelerates the learning of practical skills. Learning curves are valuable for the continuous assessment of performance and are more relevant than threshold numbers. Future research will strengthen these conclusions by evaluating simulation-based as well as patient-based training in GI endoscopy. A complete curriculum with the assessment of competence throughout training needs to be developed for all GI endoscopy procedures.

Computer simulator among experts involved in screening colonoscopy

BACKGROUND

In the era of screening colonoscopy, assessment of operator competence is warranted.

AIM

To evaluate feasibility of a computer simulator (CS) use for assessment of competence in colonoscopy by investigating performance of expert endoscopists at CS.

SUBJECTS

Twenty expert endoscopists involved in screening colonoscopy.

METHODS

Experts returned a questionnaire regarding personal practice (duration of activity, number of colonoscopies in the last year and assistance by a nurse) and performances (percentage of caecal intubation and polyp detection rate). One easy and one difficult colonoscopy were proposed at CS in randomized order.

RESULTS

Participation rate was 75%. Caecal intubation rate in clinical practice was more than 90% for all experts. At CS, time to caecal intubation and number of attempts for ileal intubation were significantly lower during easy versus difficult colonoscopy (P<0.01 for both items); interestingly, percentage of mucosa explored was higher (P<0.05) during the difficult simulation. Withdrawal time >/=6 min was achieved by 40 and 33% of experts during the easy and difficult simulation, respectively. Independent of simulation difficulty, time with loop was lower (P<0.05) for experts using hands-free insertion (n = 8) compared with those using nurse assistance in their clinical practice (n = 7). No correlation was found between scores at CS and performance in clinical practice.

CONCLUSION

Scores at CS are sensitive to the rate of technical difficulty and nurse assistance during daily practice. Withdrawal time is often shorter than required for high accuracy in polyp detection. CS could be a well-accepted tool for assessment of competence.

Proficiency of pediatric residents in performing neonatal endotracheal intubation

OBJECTIVE

Current guidelines of the Accreditation Council for Graduate Medical Education have restricted the amount of intensive care experience obtained during pediatric residency. The impact on performing procedures has not been evaluated. To determine the current level of competency in 1 common procedure, we investigated the proficiency of pediatric residents in performing neonatal endotracheal intubation during the academic years 1998-1999 and 2000-2001.

METHODS

Indication for intubation, number of attempts, and achievement of success were recorded by the respiratory therapist present for the procedure. Each intubation was scored according to the attempt on which intubation was successful. Indications for intubation were categorized as respiratory failure, delivery room resuscitation, and meconium-stained amniotic fluid. Competency was defined as a successful intubation occurring on the first or second attempt >or=80% of the time. Intubation scores were compared between residents at various stages of training and analyzed by multivariate logistic regression analysis for significance. Comparisons were then performed to determine percentage success with confidence intervals. We also surveyed previous graduates of the training program not included in the observations for this study and asked them to indicate how frequently they perform intubation in current practice and to assess their own competence in the procedure.

RESULTS

A total of 449 resident procedures were observed during the study periods: 192 by postgraduate year 1 (PGY-1) residents, 126 by PGY-2 residents, and 131 by PGY-3 residents. A total of 35% (160 of 449) of intubation procedures were never successful by pediatric house officers. Intubation was successful on the first or second attempt for 50% of PGY-1 residents (95% confidence interval [CI]: 42.6-56.8), 55% of PGY-2 residents (95% CI: 46-63.5), and 62% of PGY-3 residents (95% CI: 53.9-70.7). The third-year residents exhibited a significantly higher likelihood of performing a successful intubation compared with first-year residents. The first-year residents in 1998-1999 showed no improvement by their third year in 2000-2001. Surveys were sent to 56 graduates of our residency program (1998-2000). Completed surveys were received from 31 (66%) of 47. A total of 71% of the respondents are practicing general pediatrics, and 36% attend deliveries or perform intubations. A total of 87% reported that their level of confidence with endotracheal intubation was good or excellent after completion of residency training.

CONCLUSIONS

We provide objective and subjective data concerning the proficiency of pediatric residents in performing neonatal endotracheal intubation. None of our resident groups met the specified definition of technical competence, although there was improvement with advancing training level in bivariate analyses. However, graduates of our training program felt confident with their intubation skills in contrast to our objective findings. As exposure to these important skills becomes limited, methods to ensure attainment of technical competency during training may need to be redefined.

Achieving quality in flexible sigmoidoscopy screening for colorectal cancer

BACKGROUND

Sigmoidoscopy screening, which can dramatically reduce colorectal cancer mortality, is supported increasingly by physicians and payers, and is likely to be performed more frequently in the future. As more physicians and nonphysician medical personnel learn how to perform this procedure, and with attention to quality standards, the overall impact of sigmoidoscopy screening may improve. This review describes elements that characterize high-quality examinations and identifies resources for in-depth information on performing flexible sigmoidoscopy.

METHODS

The domains of quality were identified from textbooks, articles, and the professional opinions of gastroenterologists and primary care physicians. Information was obtained from MEDLINE, bibliographies in recent articles, medical professional organizations, equipment manufacturers' representatives, and focus groups of primary care physicians.

RESULTS

Nine domains of quality are identified and discussed: training, logistical start-up, patient interaction, bowel preparation, examination technique, lesion recognition, complications, reporting, and processing (equipment cleaning and disinfection).

CONCLUSIONS

Persons learning how to perform and to implement flexible sigmoidoscopy may use this information to help ensure the quality of screening examinations.