Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery

Craniotomies at an overseas military treatment facility: Maintaining readiness for the unit and the surgeon

INTRODUCTION

Craniotomy and craniectomy are common neurosurgical procedures with wide applications in both civilian and military practice. Skill maintenance for these procedures is required for military providers in the event they are called to support forward deployed service members suffering from combat and non-combat injuries. The presents investigation details the performance of such procedures at a small, overseas military treatment facility (MTF).

MATERIALS AND METHODS

A retrospective review of craniotomy procedures performed at an overseas military treatment facility (MTF) over a 2-year period (2019-2021) was performed. Patient and procedural data were collected for all elective and emergent craniotomies including surgical indications, outcomes, complications, military rank, and impact on duty status and tour curtailment.

RESULTS

A total of 11 patients underwent a craniotomy or craniectomy procedure with an average follow-up of 496.8 days (range 103-797). Seven of the 11 patients were able to undergo surgery, recovery, and convalesce without transfer to a larger hospital network or MTF. Of the 6 patients that were active duty (AD), one returned to full duty while three separated and two remain in partial duty status at latest follow-up. There were four complications in four patients with one death.

CONCLUSIONS

In this series, we demonstrate that cranial neurosurgical procedures can be performed safely and effectively while at an overseas MTF. There are potential benefits to the AD service members, their unit, and family as well as to the hospital treatment team and surgeon as this represents a clinical capability requisite to maintain trauma readiness for future conflicts.

Data-Driven Residency Training: A Scoping Review of Educational Interventions for Neurosurgery Residency Programs

BACKGROUND

Neurosurgery residency programs are tasked with imparting large volumes of both clinical knowledge and technical skill to trainees in limited time. Many investigators have described local practices, which may offer evidence-based interventions in neurosurgical residency education, but this literature has not been systematically reviewed.

OBJECTIVE

To perform a scoping review of educational practices in neurosurgery, which are supported by quantitative, peer-reviewed research.

METHODS

A scoping review of the literature was performed. PubMed, Embase, and Web of Science databases were queried for articles describing educational interventions for neurosurgery residents, which included a quantitative assessment of the effect on resident performance.

RESULTS

From an initial set of 1785 unique articles, 29 studies were ultimately screened and included. Studies were into the following 6 topics: (1) didactics and curricula (n = 13), (2) nontechnical skills (n = 6), (3) wellness and burnout (n = 4), (4) assessment and feedback (n = 2), (5) mentorship and career development (n = 2), and (6) research (n = 2). Individual study results were described.

CONCLUSION

Several educational interventions in neurosurgical training are supported by quantitative evidence. Methodological shortcomings are prevalent among studies of education, particularly in the selection of meaningful outcome measures. A summary of evidence-based considerations is provided for current and future program directors.

Resident evaluations in the age of competency-based medical education: faculty perspectives on minimizing burdens

OBJECTIVE

Competency-based medical education (CBME), an outcomes-based approach to medical education, continues to be implemented across many postgraduate medical education programs worldwide, including a recent introduction into Canadian neurosurgical training programs (July 2019). The success of this educational paradigm shift requires frequent faculty observation and evaluation of residents performing defined tasks of the specialty. A main challenge involves providing residents with frequent performance evaluations and feedback that are feasible for faculty to complete. This study aims to define what is currently happening and what changes are needed to make CBME successful for the certification of neurosurgeons' competence.

METHODS

A 55-item questionnaire was emailed nationwide to survey Canadian neurosurgical faculty.

RESULTS

Fifty-two complete responses were received and achieved a distribution highly correlated with the number of faculty neurosurgeons practicing in each Canadian province (Pearson's r = 0.94). Two-thirds (35/52) of faculty reported currently taking a median of 10 minutes to complete evaluation forms at the end of a resident's rotation block. Regardless of the faculty's province of practice (p = 0.50) or years of experience (p = 0.06), they reported 3 minutes (minimum 1 minute, maximum 10 minutes, interquartile range [IQR] 3 minutes) as a feasible amount of time to spend completing an evaluation form following an observation of a resident's performance of an entrustable professional activity (EPA). If evaluation forms took 3 minutes to complete, 85% of respondents (44/52) would complete EPA evaluations weekly or daily. The faculty recommended 5 minutes as a feasible amount of time to provide oral feedback (minimum 1 minute, maximum 20 minutes, IQR 3.25 minutes), which was significantly higher (p = 0.00099) than their recommended amount of time for completing evaluation forms. The majority of faculty (71%) stated they would prefer to access resident evaluation forms through a mobile application compared to a paper form (12%), an evaluation website (8%), or through a URL link sent via email (10%; p = 0.0032).

CONCLUSIONS

To facilitate the successful implementation of CBME into a neurosurgical training curriculum, resident EPA assessment forms should take 3 minutes or less to complete and be accessible through a mobile application.

Feasibility of Conducting a Virtual Exit Exam in Neurosurgery During the SARS-COV19 Pandemic

Background

Skills assessment forms an integral part of the exit examination in neurosurgical training programs. The established method of evaluating trainees for their clinical knowledge and surgical proficiency in the operating room is not feasible in the current time of SARS-COV19 pandemic.

Objective

The feasibility of conducting such an assessment using case modules on an online meeting platform is discussed.

Methods

Six candidates were evaluated on two consecutive days with two internal examiners located at the examination site and two external examiners situated at their own institutions elsewhere in the country. Clinical details, including images and videos of patients managed at our institute were recorded and provided to the candidates as case modules. Four sessions were conducted in the form of long and short cases, operative neurosurgery, neuroradiology, and neuropathology and a general viva-voce using "Zoom" (Zoom-Video-Communications, Inc. USA) platform. Feedback from the examinee and the examiners were obtained for any modification in the current format.

Result

The online platform worked well without any interruption except for slight lag in the audio-visual system and occasional difficulty in using microphone and screen simultaneously. Trainees were able to interpret the clinical details and rated this format close to actual clinical evaluation. The examiners uniformly agreed that the online format for assessment was satisfactory and made some suggestions for improvement.

Conclusion

Clinical and surgical skill evaluation is feasible using case modules and online meeting platforms. Use of original patient's data, images, videos demonstrating clinical signs, and operative procedures makes this assessment more objective.

Hot Seat Diagnosis: Competency-Based Tool Is Superior to Time-Based Tool for the Formative In-Service Assessment of Pathology Trainees

CONTEXT.—

Competency-based medical education relies on frequent formative in-service assessments to ascertain trainee progression. Currently at our institution, trainees receive a summative end-of-rotation In-Training Evaluation Report based on feedback collected from staff pathologists. There is no method of simulating report sign-out.

OBJECTIVE.—

To develop a formative in-service assessment tool that is able to simulate report sign-out and provide case-by-case feedback to trainees. Further, to compare time- versus competency-based assessment models.

DESIGN.—

Twenty-one pathology trainees were assessed for 20 months. Hot Seat Diagnosis by trainees and trainee assessment by pathologists were recorded in the Laboratory Information System. In the first iteration, trainees were assessed by using a time-based assessment scale on their ability to diagnose, report, use ancillary testings, comment on clinical implications, provide intraoperative consultation and/or gross cases. The second iteration used a competency-based assessment scale. Trainees and pathologists completed surveys on the effectiveness of the In-Training Evaluation Report versus the Hot Seat Diagnosis tool.

RESULTS.—

Scores from both iterations correlated significantly with other assessment tools including the Resident In-Service Examination (r = 0.93, P = .04 and r = 0.87, P = .03). The competency-based model was better able to demonstrate improvement over time and stratify junior versus senior trainees than the time-based model. Trainees and pathologists rated Hot Seat Diagnosis as significantly more objective, detailed, and timely than the In-Training Evaluation Report, and effective at simulating report sign-out.

CONCLUSIONS.—

Hot Seat Diagnosis is an effective tool for the formative in-service assessment of pathology trainees and simulation of report sign-out, with the competency-based model outperforming the time-based model.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Peritonitis caused by Roseomonas mucosa after ventriculoperitoneal shunt revision: a case report

Ventriculoperitoneal shunt (VPS) is an adequate treatment for congenital hydrocephalus or chronic hydrocephalus in adults. Yet, it is a surgery associated with a significant rate of complications amongst neurosurgical procedures, with frequent shunt obstructions and infections. We report the first-ever case of peritonitis caused by Roseomonas mucosa, shortly after the revision of a VPS ventricular catheter. Hardware removal and proper antibiotic therapy led to the patient's recovery. Roseomonas mucosa is an opportunistic skin pathogen with an antibiotic resistance profile to many beta-lactamines and a tropism for indwelling catheters and post-operative period. Nowadays, it should be taken into account in case of infection of indwelling catheters and for some implantable medical devices.

The effect of simulation training on resident proficiency in thoracolumbar pedicle screw placement using computer-assisted navigation

OBJECTIVE

Residency work-hour restrictions necessitate efficient, reproducible training. Simulation training for spinal instrumentation placement shows significant benefit to learners' subjective and objective proficiency. Cadaveric laboratories are most effective but have high cost and low availability. The authors' goal was to create a low-cost, efficient, reproducible spinal instrumentation placement simulation curriculum for neurosurgery and orthopedic surgery residents using synthetic models and 3D computer-assisted navigation, assessing subjective and objective proficiency with placement of thoracolumbar pedicle screws.

METHODS

Fifteen neurosurgery and orthopedic surgery residents participated in a standardized curriculum with lecture followed by two separate sessions of thoracolumbar pedicle screw placement in a synthetic spine model utilizing 3D computer-assisted navigation. Data were collected on premodule experience, time and accuracy of screw placement, and both subjective and objective ratings of proficiency.

RESULTS

Fifteen of 15 residents demonstrated improvement in subjective (Physician Performance Diagnostic Inventory Scale [PPDIS]) and 14 in objective (Objective Structured Assessment of Technical Skills [OSATS]) measures of proficiency in navigated screw placement with utilization of this curriculum (p < 0.001 for both), regardless of the number of cases of previous experience using thoracolumbar spinal instrumentation. Fourteen of 15 residents demonstrated decreased time per screw placement from session 1 to session 2 (p = 0.006). There was no significant difference in pedicle screw accuracy between session 1 and session 2.

CONCLUSIONS

A standardized curriculum using synthetic simulation training for navigated thoracolumbar pedicle screw placement results in significantly improved resident subjective and objective proficiency. Development of a nationwide competency curriculum using simulation training for spinal instrumentation placement should be considered for safe, efficient resident training.

Core Competency Self-Assessment Enhances Critical Review of Complications and Entrustable Activities

BACKGROUND

The Accreditation Council for Graduate Medical Education has defined six core competencies (CCs) that every successful physician should possess. However, the assessment of CC achievement among trainees is difficult. This project was designed to prospectively evaluate the impact of resident identification of CC as a component of morbidity review on error identification and standard of care (SOC) assessments. The platform was assessed for its reliability as a measure of resident critical analysis of complication causality across postgraduate year (PGY).

MATERIALS AND METHODS

A total of 1945 general surgery cases with complications were assessed for error identification and SOC management between January 1, 2016, and December 31, 2018. CC identification was additionally assessed between January 1, 2019, and December 31, 2019, and included 708 general surgery cases. Data were evaluated for error assessments and overall SOC management. PGY4 and 5 residents were compared for number of cases and complications reviewed, severity, error causation, and CC relevance.

RESULTS

Study groups were equivalent by Clavien-Dindo scores. Error identification significantly increased in all categories: diagnostic (P < 0.001), technical (P < 0.05), judgment (P < 0.001), system (P < 0.001), and communication (P < 0.001). Overall SOC assessments validated by a supervising surgical quality officer were unchanged. An increased exposure to cases with severe complications, error causation, and CC relevance was noted across PGY.

CONCLUSIONS

The addition of CC assessment into morbidity review appears to improve the critical thinking of evaluating residents by increasing the identification of management errors. Used as an element of prospective self-assessment, teaching residents to identify CC principles in cases with complications may assist in learner progression toward clinical competence and critical thinking.

Development and Evaluation of a Surgical Direct Assessment Tool for Resident Training

In podiatric residency training, minimum activity volume numbers are used to assess surgical competency. The purpose of this study was to develop a standardized direct assessment form as a complement to minimum activity volume numbers. Sixteen attending physicians completed 121 direct assessment forms, evaluating six podiatric medicine and surgery residents. Evaluation scores were highly correlated with residency year. Resident feedback was positive, with the open-response portion identified as especially useful. Although further efforts may help refine this approach, the use of standardized, competency-based direct assessment has the potential to improve the training of podiatric medicine and surgery residents.

Development of a Modular Cadaveric Endoscopic Orbital Surgery Model

Introduction

Endoscopic orbital surgery requires the acquisition of unique skill set including endoscopic bimanual dissection of intra/extraconal lesions adherent to orbital fat and neurovascular structures. Our goal was to develop a modular cadaveric model used to train surgeons to resect orbital pathology within any desired orbital compartment.

Methods

Expansile superabsorbent polymer (SAP) beads (2 mm) were soaked in Omnipaque™ (Iohexol) solution for 15 minutes prior to transcaruncular orbital implantation using 10-gauge angiocath. Insertion depth was designed to implant beads in predetermined intraconal compartments corresponding to established orbital tumor stages. Beads were left to expand in situ over a period of 1 to 5 hours. Computed tomography scans were performed using the FUSION image guidance protocol. Model utility and learning curves were assessed by quantifying resection time over 8 sequential attempts.

Results

All 24 beads were successfully implanted in 8 orbits corresponding to CHEER stage II to IV lesions (n = 3 per orbit). Beads expanded from 2 mm to an average of 5.2 mm within 1 hour. During expansion, the beads interpolated into the adjacent fascia similar to in vivo tumors. Average insertion time was 5:53 minutes per orbit (range, 3:24–10:33 min) and average time to bead identification was 10:47 minutes. Across all beads, dissection times decreased in a nonsignificant manner over 8 consecutive attempts.

Conclusion

The directed implantation of expansile SAP beads in this cadaveric model accurately replicates the approach, identification, and resection of isolated orbital lesions. This orbital model can assist the endoscopic surgical team to develop further knowledge and technical skill sets to approach orbital lesions. Further ongoing studies to validate this model are currently underway.

A Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study

BACKGROUND

Adequate assessment and feedback remains a cornerstone of psychomotor skills acquisition, particularly within neurosurgery where the consequence of adverse operative events is significant. However, a critical appraisal of the reliability of visual rating scales in neurosurgery is lacking. Therefore, we sought to design a study to compare visual rating scales with simulated metrics in a neurosurgical virtual reality task.

METHODS

Neurosurgical faculty rated anonymized participant video recordings of the removal of simulated brain tumors using a visual rating scale made up of seven composite elements. Scale reliability was evaluated using generalizability theory, and scale subcomponents were compared with simulated metrics using Pearson correlation analysis.

RESULTS

Four staff neurosurgeons evaluated 16 medical student neurosurgery applicants. Overall scale reliability and internal consistency were 0.73 and 0.90, respectively. Reliability of 0.71 was achieved with two raters. Individual participants, raters, and scale items accounted for 27%, 11%, and 0.6% of the data variability. The hemostasis scale component related to the greatest number of simulated metrics, whereas respect for no-go zones and tissue was correlated with none. Metrics relating to instrument force and patient safety (brain volume removed and blood loss) were captured by the fewest number of rating scale components.

CONCLUSIONS

To our knowledge, this is the first study comparing participant's ratings with simulated performance. Given rating scales capture less well instrument force, quantity of brain volume removed, and blood loss, we suggest adopting a hybrid educational approach using visual rating scales in an operative environment, supplemented by simulated sessions to uncover potentially problematic surgical technique.

The Effectiveness of Bioskills Training for Simulated Lumbar Pedicle Screw Placement

STUDY DESIGN

Prospective randomized study.

OBJECTIVES

To define the impact of an inexpensive, user-friendly, and reproducible lumbar pedicle screw instrumentation bioskills training module and evaluation protocol.

METHODS

Participants were randomized to control (n = 9) or intervention (n = 10) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 20-minute bioskills training module while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Pedicle Instrumentation Score metrics. In addition, identification of pedicle breach and breach anatomic location was measured pre- and posttest in lumbar spine models.

RESULTS

The intervention group showed a 30.8% improvement in PPDIS scores, compared with 13.4% for the control group (P = .01). The intervention group demonstrated statistically significant 66% decrease in breaches (P = .001) compared with 28% decrease in the control group (P = .06). Breach identification demonstrated no change in accuracy of the control group (incorrect identification from 32.2% pre- to posttest 35%; P = .71), whereas the intervention group's improvement was statistically significant (42% pre- to posttest 36.5%; P = .0047).

CONCLUSIONS

We conclude that a concise lumbar pedicle screw instrumentation bioskills training session can be a useful educational tool to augment clinical education.

The Effectiveness of Bioskills Training for Simulated Open Lumbar Laminectomy

STUDY DESIGN

Randomized, prospective study within an orthopedic surgery resident program at a large urban academic medical center.

OBJECTIVES

To develop an inexpensive, user-friendly, and reproducible lumbar laminectomy bioskills training module and evaluation protocol that can be readily implemented into residency training programs to augment the clinical education of orthopedic and neurosurgical physicians-in-training.

METHODS

Twenty participants comprising senior medical students and orthopedic surgical residents. Participants were randomized to control (n = 9) or intervention (n = 11) groups controlling for level of experience (medical students, junior resident, or senior resident). The intervention group underwent a 40-minute bioskills training module, while the control group spent the same time with self-directed study. Pre- and posttest performance was self-reported by each participant (Physician Performance Diagnostic Inventory Scale [PPDIS]). Objective outcome scores were obtained from a blinded fellowship-trained attending orthopedic spine surgeon using Objective Structured Assessment of Technical Skills (OSATS) and Objective Decompression Score metrics.

RESULTS

When compared with the control group, the intervention group yielded a significant mean improvement in OSATS (P = .022) and PPDIS (P = .0001) scores. The Objective Decompression Scores improved in the intervention group with a trend toward significance (P = .058).

CONCLUSIONS

We conclude that a concise lumbar laminectomy bioskills training session can be a useful educational tool for to augment clinical education. Although no direct clinical correlation can be concluded from this study, the improvement in trainee's technical and procedural skills suggests that Sawbones training modules can be an efficient and effective tool for teaching fundamental spine surgical skills outside of the operating room.

Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery

OBJECTIVE The authors previously demonstrated the use of a validated Objective Structured Assessment of Technical Skills (OSATS) tool for evaluating residents' operative skills in pediatric neurosurgery. However, no benchmarks have been established for specific pediatric procedures despite an increased need for meaningful assessments that can either allow for early intervention for underperforming trainees or allow for proficient residents to progress to conducting operations independently with more passive supervision. This validated methodology and tool for assessment of operative skills for common pediatric neurosurgical procedures-external ventricular drain (EVD) placement and shunt surgery- was applied to establish its procedure-based feasibility and reliability, and to document the effect of repetition on achieving surgical skill proficiency in pediatric EVD placement and shunt surgery. METHODS A procedure-based technical skills assessment for EVD placements and shunt surgeries in pediatric neurosurgery was established through the use of task analysis. The authors enrolled all residents from 3 training programs (Baylor College of Medicine, Houston Methodist Hospital, and University of Texas-Medical Branch) who rotated through pediatric neurosurgery at Texas Children's Hospital over a 26-month period. For each EVD placement or shunt procedure performed with a resident, the faculty and resident (for self-assessment) completed an evaluation form (OSATS) based on a 5-point Likert scale with 7 categories. Data forms were then grouped according to faculty versus resident (self) assessment, length of pediatric neurosurgery rotation, postgraduate year level, and date of evaluation ("beginning of rotation," within 1 month of start date; "end of rotation," within 1 month of completion date; or "middle of rotation"). Descriptive statistical analyses were performed with the commercially available SPSS statistical software package. A p value < 0.05 was considered statistically significant. RESULTS Five attending evaluators (including 2 fellows who acted as attending surgeons) completed 260 evaluations. Twenty house staff completed 269 evaluations for self-assessment. Evaluations were completed in 562 EVD and shunt procedures before the surgeons left the operating room. There were statistically significant differences (p < 0.05) between overall attending (mean 4.3) and junior resident (self; mean 3.6) assessments, and between overall attending (mean 4.8) and senior resident (self; mean 4.6) assessment scores on general performance and technical skills. The learning curves produced for the residents demonstrate a stereotypical U- or V-shaped curve for acquiring skills, with a significant improvement in overall scores at the end of the rotation compared with the beginning. The improvement for junior residents (Δ score = 0.5; p = 0.002) was larger than for senior residents (Δ score = 0.2; p = 0.018). CONCLUSIONS The OSATS is an effective assessment tool as part of a comprehensive evaluation of neurosurgery residents' performance for specific pediatric procedures. The authors observed a U-shaped learning curve, contradicting the idea that developing one's surgical technique and learning a procedure represents a monotonic, cumulative process of repetitions and improvement.

Endoscopic Management of Cavernous Carotid Surgical Complications: Evaluation of a Simulated Perfusion Model

OBJECTIVE

Endoscopic surgical treatment of pituitary tumors, lateral invading tumors, or aneurysms requires surgeons to operate adjacent to the cavernous sinus. During these endoscopic endonasal procedures, the carotid artery is vulnerable to surgical injury at its genu. The objective of this simulation model was to evaluate trainees regarding management of a potentially life-threatening vascular injury.

METHODS

Cadaveric heads were prepared in accordance with the Oregon Health & Science University body donation program. An endoscopic endonasal approach was used, and a perfusion pump with a catheter was placed in the ipsilateral common carotid artery at its origin in the neck. Learners used a muscle graft to establish vascular control and were evaluated over 3 training sessions. Simulation assessment, blood loss during sessions, and performance metric data were collected for learners.

RESULTS

Vascular control was obtained at a mean arterial pressure of 65 mm Hg using a muscle graft correctly positioned at the arteriotomy site. Learners improved over the course of training, with senior residents (n = 4) performing better across all simulation categories (situation awareness, decision making, communications and teamwork, and leadership); the largest mean difference was in communication and teamwork. Additionally, learner performance concerning blood loss improved between sessions (t = 3.667, P < 0.01).

CONCLUSIONS

In this pilot endoscopic endonasal simulation study, we successfully demonstrate a vascular complication perfusion model. Learners were able to gain direct applicable expertise in endoscopic endonasal techniques, instrumentation use, and teamwork required to optimize the technique. Learners gained skills of vascular complication management that transcend this model.