The Joint Council on Thoracic Surgery Education Coronary Artery Assessment Tool Has High Interrater Reliability

The use of objective assessments in the evaluation of technical skills in cardiothoracic surgery: a systematic review

OBJECTIVES

With reductions in training time and intraoperative exposure, there is a need for objective assessments to measure trainee progression. This systematic review focuses on the evaluation of trainee technical skill performance using objective assessments in cardiothoracic surgery and its incorporation into training curricula.

METHODS

Databases (EBSCOHOST, Scopus and Web of Science) and reference lists of relevant articles for studies that incorporated objective assessment of technical skills of trainees/residents in cardiothoracic surgery were included. Data extraction included task performed; assessment setting and tool used; number/level of assessors; study outcome and whether the assessments were incorporated into training curricula. The methodological rigour of the studies was scored using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

Fifty-four studies were included for quantitative synthesis. Six were randomized-controlled trials. Cardiac surgery was the most common speciality utilizing objective assessment methods with coronary anastomosis the most frequently tested task. Likert-based assessment tools were most commonly used (61%). Eighty-five per cent of studies were simulation-based with the rest being intraoperative. Expert surgeons were primarily used for objective assessments (78%) with 46% using blinding. Thirty (56%) studies explored objective changes in technical performance with 97% demonstrating improvement. The other studies were primarily validating assessment tools. Thirty-nine per cent of studies had established these assessment tools into training curricula. The mean ± standard deviation MERSQI score for all studies was 13.6 ± 1.5 demonstrating high validity.

CONCLUSIONS

Despite validated technical skill assessment tools being available and demonstrating trainee improvement, their regular adoption into training curricula is lacking. There is a need to incorporate these assessments to increase the efficiency and transparency of training programmes for cardiothoracic surgeons.

Validity Evidence for Procedure-Specific Competency Assessment Tools in Cardiovascular and Thoracic Surgery: A Scoping Review

OBJECTIVE

Surgical education has shifted from a time-based approach to the achievement and demonstration of procedural competency. High quality, objective assessment instruments are required to support this new approach. This study comprehensively reviewed the literature to identify and evaluate available procedure-specific assessment instruments in cardiothoracic and vascular surgery.

DESIGN

A systematic search of 8 databases identified studies containing procedure-specific operative assessment instruments in cardiothoracic and vascular surgery. Generic global rating scales were excluded, unless modified to be procedure-specific. Two reviewers independently evaluated the validity evidence, methodological rigour and educational utility of each instrument using objective scoring criteria. Validity evidence was evaluated with a scoring tool aligned with the contemporary framework of validity. Methodological rigour was evaluated using the Medical Education Research Study Quality Instrument. Educational utility was evaluated according to the Accreditation Council for Graduate Medical Education (ACGME) framework.

RESULTS

There were 2130 unique studies describing procedure-specific assessment in surgery. Of these, 9 studies evaluating 8 procedure-specific assessment instruments met inclusion criteria for cardiothoracic and vascular surgery. Four instruments were identified in thoracic surgery, 2 in cardiac surgery, and 2 in vascular surgery. Only 1 instrument was designed to evaluate surgeon performance, with the remainder designed to evaluate residents. No single instrument scored the maximum score of 15 for validity evidence. The highest score was 11, with 62.5% (n = 5) of instruments scoring greater than 10. All tools attained high scores in content validity, with minimal evidence generally presented regarding the consequences of assessment using a particular instrument. All but 1 instrument scored greater than 11 out of a maximum 16.5 points for methodological rigour. Very few studies reported on the ACGME domains of educational utility.

CONCLUSIONS

In an era where surgical education is shifting towards the demonstration of procedural competency, objective procedure-specific assessment is critical. This review identified that few procedure-specific assessment instruments in cardiothoracic and vascular surgery exist, emphasizing the need for such instruments to ensure the success of competency-based education models.

Obtaining Meaningful Assessment in Thoracic Surgery Education

Training in thoracic surgery has evolved immensely over the past decade due to the advent of integrated programs, technological innovations, and regulations on resident duty hours, decreasing the time trainees have to learn. These changes have made assessment of thoracic surgical trainees even more important. Shifts in medical education have increasingly emphasized competency, which has led to novel competency-based assessment tools for clinical and operative assessment. These novel tools take advantage of simulation and modern technology to provide more frequent and comprehensive assessment of the surgical trainee to ensure competence.

Prospective Trial of Low-Fidelity Deliberate Practice of Aortic and Coronary Anastomoses (TECoG 002)

OBJECTIVE

We sought to examine the feasibility of a home practice curriculum of vascular anastomosis in cardiovascular surgery using a low-fidelity simulation platform and to examine its effectiveness in skill acquisition in senior surgical trainees.

DESIGN

We organized a multicenter prospective randomized study of senior residents and fellows, who were oriented to a low-fidelity cardiac simulator and an 8-week curriculum of independent practice of aortic and coronary anastomosis. "Treatment" trainees received a simulator and the curriculum. Control trainees received only their usual operative experience. The groups then crossed over; all were studied for 16 weeks in total. Video skill assessments were captured at 0, 8, and 16 weeks and were scored by one blinded investigator using the Joint Council on Thoracic Surgery Education Assessment tool. A post-hoc survey was distributed to invited participants following study completion.

SETTING

University of Minnesota Department of Surgery, Mayo Clinic Department of Cardiovascular Surgery, and the University of Texas Health Science Center at Houston. Participants used the simulator in offices, call rooms, and their homes.

PARTICIPANTS

Program participation in the study was solicited through the Thoracic Education Cooperative Group. Four institutions expressed interest and a total of 29 trainees were invited to the study and randomized. Of these, 12 (38%) completed the curriculum and submitted the requisite 3 sets of videos (6 treatment, 6 control). All were senior residents and fellows in general and cardiothoracic surgery.

RESULTS

No significant differences were detected in assessment scores before and after the curriculum nor before or after the control period in the overall or postgraduate year-stratified populations. Participant case numbers during the study did not have a significant effect on assessment scores. Randomized participants reported strong interest in deliberate practice of technical skills but identified competing clinical and personal obligations and significant barriers to simulation.

CONCLUSIONS

Considerable variability in performance existed among participants who completed the study, but overall, the curriculum alone was insufficient to improve simulator Joint Council on Thoracic Surgery Education scores compared to those not undergoing the curriculum. Among senior residents and fellows, provision of a practice curriculum and simulator for repeated practice is feasible but clinical and personal responsibilities were barriers to repetitive practice.

Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy

BACKGROUND

The societies of thoracic surgery are working to incorporate simulation and competency-based assessment into specialty training. One challenge is the development of a simulation-based test, which can be used as an assessment tool. The study objective was to establish validity evidence for a virtual reality simulator test of a video-assisted thoracoscopic surgery (VATS) lobectomy of a right upper lobe.

METHODS

Participants with varying experience in VATS lobectomy were included. They were familiarized with a virtual reality simulator (LapSim^®) and introduced to the steps of the procedure for a VATS right upper lobe lobectomy. The participants performed two VATS lobectomies on the simulator with a 5-min break between attempts. Nineteen pre-defined simulator metrics were recorded.

RESULTS

Fifty-three participants from nine different countries were included. High internal consistency was found for the metrics with Cronbach's alpha coefficient for standardized items of 0.91. Significant test-retest reliability was found for 15 of the metrics (p-values <0.05). Significant correlations between the metrics and the participants VATS lobectomy experience were identified for seven metrics (p-values <0.001), and 10 metrics showed significant differences between novices (0 VATS lobectomies performed) and experienced surgeons (>50 VATS lobectomies performed). A pass/fail level defined as approximately one standard deviation from the mean metric scores for experienced surgeons passed none of the novices (0 % false positives) and failed four of the experienced surgeons (29 % false negatives).

CONCLUSION

This study is the first to establish validity evidence for a VATS right upper lobe lobectomy virtual reality simulator test. Several simulator metrics demonstrated significant differences between novices and experienced surgeons and pass/fail criteria for the test were set with acceptable consequences. This test can be used as a first step in assessing thoracic surgery trainees' VATS lobectomy competency.

Development and Evaluation of a Three-Dimensional Multistation Cardiovascular Simulator

BACKGROUND

To facilitate deliberate practice, we developed and evaluated a three-dimensional multistation cardiovascular simulator with low-fidelity, anatomically correct materials.

METHODS

Surgical educators and senior fellows at several national institutions who are a part of the Thoracic Education Cooperative Group completed expert validation surveys that feature a self-report rating scale, ranging from 0 (don't know) to 4 (highly realistic, no changes needed), and a global recommendation rating scale, ranging from 0 to 3. To analyze the survey results, we used paired Student t tests and a many-facet Rasch model.

RESULTS

We analyzed the results of 18 expert validation surveys (completed by 9 senior fellows and by 9 staff surgeons). The overall realism-of-experience rating for the simulator was 3.23 (of 4), suggesting adequate realism with room remaining for improvement. Senior fellows rated the aortic graft replacement station and overall physical attributes more favorably than staff. Staff surgeons favored the simulator for realism of the mitral valve annuloplasty station and the cannulation station and for its use as a training tool. The overall global recommendation rating for the simulator was 2.38 (of 3), suggesting that the simulator can be considered for use in cardiovascular surgery training programs with minor improvements.

CONCLUSIONS

This three-dimensional multistation cardiovascular simulator has adequate realism and can be considered for use in cardiovascular training programs. Platforms such as this may facilitate deliberate home practice and reinforcement of technical skills.

Gamification in thoracic surgical education: Using competition to fuel performance

OBJECTIVES

In an effort to stimulate residents and trainers to increase their use of simulation training and the Thoracic Surgery Curriculum, a gamification strategy was developed in a friendly but competitive environment.

METHODS

"Top Gun." Low-fidelity simulators distributed annually were used for the technical competition. Baseline and final video assessments were performed, and 5 finalists were invited to compete in a live setting from 2013 to 2015. "Jeopardy." A screening examination was devised to test knowledge contained in the Thoracic Surgery Curriculum. The top 6 2-member teams were invited to compete in a live setting structured around the popular game show Jeopardy.

RESULTS

"Top Gun." Over 3 years, there were 43 baseline and 34 final submissions. In all areas of assessment, there was demonstrable improvement. There was increasing evidence of simulation as seen by practice and ritualistic behavior. "Jeopardy." Sixty-eight individuals completed the screening examination, and 30 teams were formed. The largest representation came from the second-year residents in traditional programs. Contestants reported an average in-training examination percentile of 72.9. Finalists reported increased use of the Thoracic Surgery Curriculum by an average of 10 hours per week in preparation. The live competition was friendly, engaging, and spirited.

CONCLUSIONS

This gamification approach focused on technical and cognitive skills, has been successfully implemented, and has encouraged the use of simulators and the Thoracic Surgery Curriculum. This framework may capitalize on the competitive nature of our trainees and can provide recognition of their achievements.

Practice improves performance on a coronary anastomosis simulator, attending surgeon supervision does not

OBJECTIVES

Enthusiasm for simulation early in cardiothoracic surgery training is growing, yet evidence demonstrating its utility is limited. We examined the effect of supervised and unsupervised training on coronary anastomosis performance in a randomized trial among medical students.

METHODS

Forty-five medical students were recruited for this single-blinded, randomized controlled trial using a low-fidelity simulator. After viewing an instructional video, all participants attempted an anastomosis. Subsequently, the participants were randomized to 1 of 3 groups: control (n = 15), unsupervised training (n = 15), or supervised training with a cardiothoracic surgeon or fellow (n = 15). Both the supervised and unsupervised groups practiced for 1 hour per week. After 4 weeks, the participants repeated the anastomosis. All pre- and posttraining performances were videotaped and rated independently by 3 cardiothoracic surgeons blinded to the randomization. All raters scored 13 assessment items on a 1 to 5 (low-high) scale along with an overall pass/fail rating.

RESULTS

After the training period, all 3 groups showed significant improvements in composite scores (control: +0.52 ± 0.69 [P = .014], unsupervised: +1.05 ± 0.48 [P < .001], and supervised: +1.10 ± 0.84 [P < .001]). Compared with control group, both supervised (P = .005) and unsupervised trainees (P = .005) demonstrated a significant improvement. Between the supervised and unsupervised groups there were no statistically significant differences in composite scores.

CONCLUSIONS

Practice on low-fidelity simulators enabled trainees to improve on a broad range of skills; however, the additional effect of attending-level supervision is limited. In an era of increasing staff surgeon responsibilities, unsupervised practice may be sufficient for inexperienced trainees.