Development and validation of a theoretical test of proficiency for video-assisted thoracoscopic surgery (VATS) lobectomy

Structured and evidence-based training of technical skills in respiratory medicine and thoracic surgery

Within the last decade, the number of technical procedures in respiratory medicine and thoracic surgery has grown and created a higher demand for effective and evidence-based education. Today, trainees are often allowed to perform procedures unsupervised on patients after obtaining a course certificate from a theoretic course and having performed a fixed number of supervised procedures. However, these methods do not ensure adequate competence. Well-structured and effective educational programmes including validated tests are needed to reduce economic expenses, optimize time spend, and ensure patient safety. The aim of this article is to summarize current state of educational strategies for technical procedures within respiratory medicine and thoracic surgery. Thus, to discuss future recommendations for curriculum development and assessment of competences based on Kern’s framework. The approach by Kern consists of six topics, which needs to be considered and evaluated and in order to educate physicians and surgeons most effective and evidence-based. We present a practical guide contributing to future educators’ considerations on (I) problem identification and general needs assessment, (II) targeted needs assessment, (III) goals and objectives, (IV) educational strategies, (V) implementation, and finally (VI) evaluation and feedback.

Simulation of advanced cataract surgery - validation of a newly developed test

PURPOSE

To develop and investigate an Eyesi simulator-based test for the more experienced cataract surgeon for evidence of validity.

METHODS

The study was a prospective interventional cohort study and carried out at the Copenhagen Academy for Medical Education and Simulation. The Eyesi Simulator was used for the test which was developed by three expert cataract surgeons. Ten cataract surgeons (>250 surgeries performed) and ten ophthalmic residents performed two repetitions of the test. The test consisted of four modules: Iris Expansion Ring insertion - level 1, Iris Expansion Ring extraction - level 2, Capsulorhexis - level 3 and Anterior Vitrectomy - level 6.

RESULTS

Internal consistency reliability showed Cronbach's alpha of 0.63. Test-retest reliabilities were significant for Iris Expansion Ring extraction - level 2 (p = 0.012) and Capsulorhexis - level 3 (p = 0.018). Differences between the two groups were only significant in both repetitions for the Iris Expansion Ring extraction - level 2 (p < 0.001 and p = 0.041, respectively). Furthermore, we found a statistically significant difference between the mean module scores for novices and the more experienced surgeons for Iris Expansion Ring insertion - level 1 (p = 0.021) and Capsulorhexis - level 3 (p = 0.019) in the first repetition.

CONCLUSION

The investigated modules show evidence of validity within several aspects of Messick's framework. However, the evidence is not strong enough to apply the test for certification purposes of cataract surgeons, but the modules may still be relevant in the training of advanced cataract surgical procedures.

Reliable test of clinicians' mastery in skin cancer diagnostics

Differentiating between benign and malignant skin lesions can be very difficult and should only be done by sufficiently trained and skilled clinicians. To our knowledge there are no validated tests for reliable assessments of clinicians' ability to perform skin cancer diagnostics. To develop and gather validity evidence for a test in skin cancer diagnostics, a multiple-choice questionnaire (MCQ) was developed based on informal interviews with seven content experts from five skin cancer centers in Denmark. Validity evidence for the test was gathered from May until July 2019 using Messick's validity framework (content, response process, internal structure, relationship to other variables and consequences). Item content was revised through a Delphi-like review process and then piloted on 36 medical students and 136 doctors using a standardized response process. Results enabled an analysis of the internal structure and relationship to other variables of the test. Finally, the contrasting groups method was used to investigate the test's consequences (pass-fail standard). The initial 90-item MCQ was reduced to 40 items during the Delphi-like review process. Item analysis revealed that 25 of the 40 selected items were level I-III quality items with a high internal consistency (Cronbach's α = 0.83) and highly significant (P ≤ 0.0001) differences in test scores between participants with different occupations or levels of experience. A pass-fail standard of 12 (48%) correct answers was established using the contrasting groups' method. The skin cancer diagnostics MCQ developed in this study can be used for reliable assessments of clinicians' competencies.

Development and validation of a multiple-choice questionnaire-based theoretical test in direct ophthalmoscopy

PURPOSE

Direct ophthalmoscopy can reveal systemic, neurologic and ophthalmic conditions, but is poorly mastered among young physicians. A theoretical test is needed to measure effect of educational interventions. We developed and gathered validity evidence for a multiple-choice questionnaire (MCQ)-based theoretical test in direct ophthalmoscopy.

METHODS

The MCQ was developed by interviewing experts. Then, validity evidence was evaluated using Messick's validity framework. Content was ensured by inviting the experts to contribute in a Delphi-like process. Response process was ensured by piloting and by streamlining all instructions. Then, the test was taken by ophthalmologists and by medical students without experience in direct ophthalmoscopy. Results were used to evaluate internal structure (item quality analysis and internal consistency), relations to other variables (correlation of test scores to experience level) and consequences (establishment of pass-fail score and the consequences of its use).

RESULTS

The first phase of the study yielded 100 MCQs. In second phase, we identified that 60 items fulfilled predefined relevance and item quality requirements. These items demonstrated very high internal consistency (Cronbach's alpha = 0.95), significantly discriminated medical students from specialists (p < 0.001, independent samples t-test) and the established pass-fail score of 50 (83%) correct answers resulted in no false positives (students passing) and no false negatives (specialists failing). A Decision study identified that sampling 15 items suffice for certification.

CONCLUSION

We developed and validated an MCQ-based theoretical test in direct ophthalmoscopy that enables an evidence-based approach to measuring, evaluating and certifying the theoretical knowledge necessary for direct ophthalmoscopy.

Gathering Validity Evidence for Surgical Simulation: A Systematic Review

OBJECTIVE

To identify current trends in the use of validity frameworks in surgical simulation, to provide an overview of the evidence behind the assessment of technical skills in all surgical specialties, and to present recommendations and guidelines for future validity studies.

SUMMARY OF BACKGROUND DATA

Validity evidence for assessment tools used in the evaluation of surgical performance is of paramount importance to ensure valid and reliable assessment of skills.

METHODS

We systematically reviewed the literature by searching 5 databases (PubMed, EMBASE, Web of Science, PsycINFO, and the Cochrane Library) for studies published from January 1, 2008, to July 10, 2017. We included original studies evaluating simulation-based assessments of health professionals in surgical specialties and extracted data on surgical specialty, simulator modality, participant characteristics, and the validity framework used. Data were synthesized qualitatively.

RESULTS

We identified 498 studies with a total of 18,312 participants. Publications involving validity assessments in surgical simulation more than doubled from 2008 to 2010 (∼30 studies/year) to 2014 to 2016 (∼70 to 90 studies/year). Only 6.6% of the studies used the recommended contemporary validity framework (Messick). The majority of studies used outdated frameworks such as face validity. Significant differences were identified across surgical specialties. The evaluated assessment tools were mostly inanimate or virtual reality simulation models.

CONCLUSION

An increasing number of studies have gathered validity evidence for simulation-based assessments in surgical specialties, but the use of outdated frameworks remains common. To address the current practice, this paper presents guidelines on how to use the contemporary validity framework when designing validity studies.

Development of and Gathering Validity Evidence for a Theoretical Test in Thoracic Ultrasound

BACKGROUND

Thoracic ultrasound (TUS) has a high diagnostic accuracy for many common pulmonary diseases, but theoretic knowledge in sonographic physics, thoracic anatomy and physiology, and sonopathologic patterns is required to develop competence.

OBJECTIVES

The aims of the study were to develop and gather validity evidence for a theoretical test in TUS and to establish a pass/fail standard.

METHODS

Content was provided based on expert interviews, leading to the creation of 113 initial multiple-choice question (MCQ) items. Consensus was reached on 92 proceeding items through a Delphi process, and items were presented to physicians with different knowledge and experience in TUS. Answers were used for item statistics in order to select the items with the most optimal item discrimination and difficulty (i.e., level I items) to be included in the final test. Mean scores of the novice, intermediate and experienced groups were compared, and a pass/fail score was established using the contrasting groups' standard setting method.

RESULTS

Item statistics revealed 38 level I items, of which 30 were selected to be included in the final test. The internal consistency was high (Cronbach's alpha = 0.88). Differences in mean scores were 8.6 points (p < 0.001), 6.3 points (p = 0.01), and 14.9 points (p < 0.001) between novices and intermediates, intermediates and experienced, and novices and experienced, respectively. A pass/fail standard of 20 points was established.

CONCLUSION

The established MCQ test can distinguish between physicians with different levels of competence in TUS and enables an objective, evidence-based approach for assessing the theoretical knowledge of trainees undergoing an educational programme in TUS.

Surgical simulation: Current practices and future perspectives for technical skills training

Simulation-based training (SBT) has become a standard component of modern surgical education, yet successful implementation of evidence-based training programs remains challenging. In this narrative review, we use Kern's framework for curriculum development to describe where we are now and what lies ahead for SBT within surgery with a focus on technical skills in operative procedures. Despite principles for optimal SBT (proficiency-based, distributed, and deliberate practice) having been identified, massed training with fixed time intervals or a fixed number of repetitions is still being extensively used, and simulators are generally underutilized. SBT should be part of surgical training curricula, including theoretical, technical, and non-technical skills, and be based on relevant needs assessments. Furthermore, training should follow evidence-based theoretical principles for optimal training, and the effect of training needs to be evaluated using relevant outcomes. There is a larger, still unrealized potential of surgical SBT, which may be realized in the near future as simulator technologies evolve, more evidence-based training programs are implemented, and cost-effectiveness and impact on patient safety is clearly demonstrated.

A novel assessment tool for evaluating competence in video-assisted thoracoscopic surgery lobectomy

BACKGROUND

Specific assessment tools can accelerate trainees' learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by achieving consensus within an international group of VATS experts.

METHOD

The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until consensus was reached.

RESULTS

Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue and structures, and (8) technical skills in general.

CONCLUSION

A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts' consensus. The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the important decision of when trainees have acquired VATS lobectomy competencies for independent performance.

Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy

BACKGROUND

The societies of thoracic surgery are working to incorporate simulation and competency-based assessment into specialty training. One challenge is the development of a simulation-based test, which can be used as an assessment tool. The study objective was to establish validity evidence for a virtual reality simulator test of a video-assisted thoracoscopic surgery (VATS) lobectomy of a right upper lobe.

METHODS

Participants with varying experience in VATS lobectomy were included. They were familiarized with a virtual reality simulator (LapSim^®) and introduced to the steps of the procedure for a VATS right upper lobe lobectomy. The participants performed two VATS lobectomies on the simulator with a 5-min break between attempts. Nineteen pre-defined simulator metrics were recorded.

RESULTS

Fifty-three participants from nine different countries were included. High internal consistency was found for the metrics with Cronbach's alpha coefficient for standardized items of 0.91. Significant test-retest reliability was found for 15 of the metrics (p-values <0.05). Significant correlations between the metrics and the participants VATS lobectomy experience were identified for seven metrics (p-values <0.001), and 10 metrics showed significant differences between novices (0 VATS lobectomies performed) and experienced surgeons (>50 VATS lobectomies performed). A pass/fail level defined as approximately one standard deviation from the mean metric scores for experienced surgeons passed none of the novices (0 % false positives) and failed four of the experienced surgeons (29 % false negatives).

CONCLUSION

This study is the first to establish validity evidence for a VATS right upper lobe lobectomy virtual reality simulator test. Several simulator metrics demonstrated significant differences between novices and experienced surgeons and pass/fail criteria for the test were set with acceptable consequences. This test can be used as a first step in assessing thoracic surgery trainees' VATS lobectomy competency.

Development and validation of a theoretical test in non-anaesthesiologist-administered propofol sedation for gastrointestinal endoscopy

OBJECTIVE

Safety with non-anaesthesiologist-administered propofol sedation (NAAP) during gastrointestinal (GI) endoscopy is related to theoretical knowledge. A summative testing of knowledge before attempting supervised nurse-administered propofol sedation (NAPS) in the clinic is advised. The aims of this study were to develop a theoretical test about propofol sedation, to gather validity evidence for the test and to measure the effect of a NAPS-specific training course.

MATERIAL AND METHODS

A three-phased psychometric study on multiple choice questionnaire (MCQ) test development, gathering of validity evidence and evaluation of the effect of a specific NAAP course on the test result. A MCQ containing 86 questions was developed and administered 113 times to 91 participants representing novices, intermediates and experienced.

RESULTS

Question difficulty analyses revealed 50 level I and II questions. The 50 MCQs showed mean (SD) intergroup differences (p = 0.001) between novices = 28.6 (4.82), intermediates = 36.8 (5.43) and experienced = 41.8 (4.65) and provided a pass score of 35.2. The course with pre-course test had significant effect on the knowledge of nurses (18% increase) and physicians (19% increase; p = 0.001 and 0.001, respectively).

CONCLUSIONS

Data supported the validity of the developed MCQ test. The NAPS-specific course with pre-course testing adds theoretical knowledge to already well-prepared participants.