Assessment of competence in surgical skills using functional magnetic resonance imaging: a feasibility study

Physical and biophysical markers of assessment in medical training: A scoping review of the literature

PURPOSE

Assessment in medical education has changed over time to measure the evolving skills required of current medical practice. Physical and biophysical markers of assessment attempt to use technology to gain insight into medical trainees' knowledge, skills, and attitudes. The authors conducted a scoping review to map the literature on the use of physical and biophysical markers of assessment in medical training.

MATERIALS AND METHODS

The authors searched seven databases on 1 August 2022, for publications that utilized physical or biophysical markers in the assessment of medical trainees (medical students, residents, fellows, and synonymous terms used in other countries). Physical or biophysical markers included: heart rate and heart rate variability, visual tracking and attention, pupillometry, hand motion analysis, skin conductivity, salivary cortisol, functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The authors mapped the relevant literature using Bloom's taxonomy of knowledge, skills, and attitudes and extracted additional data including study design, study environment, and novice vs. expert differentiation from February to June 2023.

RESULTS

Of 6,069 unique articles, 443 met inclusion criteria. The majority of studies assessed trainees using heart rate variability (n = 160, 36%) followed by visual attention (n = 143, 32%), hand motion analysis (n = 67, 15%), salivary cortisol (n = 67, 15%), fMRI (n = 29, 7%), skin conductivity (n = 26, 6%), fNIRs (n = 19, 4%), and pupillometry (n = 16, 4%). The majority of studies (n = 167, 38%) analyzed non-technical skills, followed by studies that analyzed technical skills (n = 155, 35%), knowledge (n = 114, 26%), and attitudinal skills (n = 61, 14%). 169 studies (38%) attempted to use physical or biophysical markers to differentiate between novice and expert.

CONCLUSION

This review provides a comprehensive description of the current use of physical and biophysical markers in medical education training, including the current technology and skills assessed. Additionally, while physical and biophysical markers have the potential to augment current assessment in medical education, there remains significant gaps in research surrounding reliability, validity, cost, practicality, and educational impact of implementing these markers of assessment.

Investigating the impact of mental rehearsal on prefrontal and motor cortical haemodynamic responses in surgeons using optical neuroimaging

Introduction

Inadequate exposure to real-life operating can impede timely acquisition of technical competence among surgical residents, and is a major challenge faced in the current training climate. Mental rehearsal (MR)-the cognitive rehearsal of a motor task without overt physical movement-has been shown to accelerate surgical skills learning. However, the neuroplastic effect of MR of a complex bimanual surgical task is unknown. The aim of this study is to use functional near-infrared spectroscopy (fNIRS) to assess the impact of MR on prefrontal and motor cortical activation during a laparoscopic knot tying task.

Methods

Twelve surgical residents performed a laparoscopic knot tying task before and after either mental rehearsal (MR, intervention group) or textbook reading (TR, control group). In both groups, fNIRS was used to measure changes in oxygenated hemoglobin concentration (HbO2) in the prefrontal (24 channels) and motor cortices (22 channels). Technical performance was measured using leak volume, objective performance score and task progression score.

Results

MR led to a decrease in HbO2 (reduced activation) in the bilateral prefrontal cortex (PFC), and an increase in HbO2 (increased activation) in the left middle frontal gyrus, left precentral gyrus, and left postcentral gyrus. No discernible changes in activation were observed after TR in either the PFC or motor cortex. Moreover, smaller ΔHbO2 responses in the right PFC and greater ΔHbO2 responses in the left motor cortex were observed in the MR group compared with the TR group. Leak volume was significantly less following MR (p = 0.019), but not after TR (p = 0.347). Mean objective performance score was significantly higher following MR compared with TR (p = 0.043).

Conclusion

Mental rehearsal may enhance surgical skill acquisition and technical proficiency by reducing utilization of attentional resources in the prefrontal cortex and improving neural efficiency in motor areas during a laparoscopic surgical task.

Using neuroimaging to assess brain activity and areas associated with surgical skills: a systematic review

BACKGROUND

Surgical skills acquisition is under continuous development due to the emergence of new technologies, and there is a need for assessment tools to develop along with these. A range of neuroimaging modalities has been used to map the functional activation of brain networks while surgeons acquire novel surgical skills. These have been proposed as a method to provide a deeper understanding of surgical expertise and offer new possibilities for the personalized training of future surgeons. With studies differing in modalities, outcomes, and surgical skills there is a need for a systematic review of the evidence. This systematic review aims to summarize the current knowledge on the topic and evaluate the potential use of neuroimaging in surgical education.

METHODS

We conducted a systematic review of neuroimaging studies that mapped functional brain activation while surgeons with different levels of expertise learned and performed technical and non-technical surgical tasks. We included all studies published before July 1st, 2023, in MEDLINE, EMBASE and WEB OF SCIENCE.

RESULTS

38 task-based brain mapping studies were identified, consisting of randomized controlled trials, case-control studies, and observational cohort or cross-sectional studies. The studies employed a wide range of brain mapping modalities, including electroencephalography, functional magnetic resonance imaging, positron emission tomography, and functional near-infrared spectroscopy, activating brain areas involved in the execution and sensorimotor or cognitive control of surgical skills, especially the prefrontal cortex, supplementary motor area, and primary motor area, showing significant changes between novices and experts.

CONCLUSION

Functional neuroimaging can reveal how task-related brain activity reflects technical and non-technical surgical skills. The existing body of work highlights the potential of neuroimaging to link task-related brain activity patterns with the individual level of competency or improvement in performance after training surgical skills. More research is needed to establish its validity and usefulness as an assessment tool.

Use of neuroimaging to measure neurocognitive engagement in health professions education: a scoping review

PURPOSE

To map the current literature on functional neuroimaging use in medical education research as a novel measurement modality for neurocognitive engagement, learning, and expertise development.

METHOD

We searched PubMed, Embase, Cochrane, ERIC, and Web of Science, and hand-searched reference lists of relevant articles on April 4, 2019, and updated the search on July 7, 2020. Two authors screened the abstracts and then full-text articles for eligibility based on inclusion criteria. The data were then charted, synthesized, and analyzed descriptively.

RESULTS

Sixty-seven articles published between 2007 and 2020 were included in this scoping review. These studies used three main neuroimaging modalities: functional magnetic resonance imaging, functional near-infrared spectroscopy, and electroencephalography. Most of the publications (90%, n = 60) were from the last 10 years (2011-2020). Although these studies were conducted in 16 countries, 68.7% (n = 46) were from three countries: the USA (n = 21), UK (n = 15), and Canada (n = 10). These studies were mainly non-experimental (74.6%, n = 50). Most used neuroimaging techniques to examine psychomotor skill development (57%, n = 38), but several investigated neurocognitive correlates of clinical reasoning skills (22%, n = 15).

CONCLUSION

This scoping review maps the available literature on functional neuroimaging use in medical education. Despite the heterogeneity in research questions, study designs, and outcome measures, we identified a few common themes. Included studies are encouraging of the potential for neuroimaging to complement commonly used measures in education research and may help validate/challenge established theoretical assumptions and provide insight into training methods. This review highlighted several areas for further research. The use of these emerging technologies appears ripe for developing precision education, establishing viable study protocols for realistic operational settings, examining team dynamics, and exploring applications for real-time monitoring/intervention during critical clinical tasks.

Neuromonitoring Correlates of Expertise Level in Surgical Performers: A Systematic Review

Surgical expertise does not have a clear definition and is often culturally associated with power, authority, prestige, and case number rather than more objective proxies of excellence. Multiple models of expertise progression have been proposed including the Dreyfus model, however, they all currently require subjective evaluation of skill. Recently, efforts have been made to improve the ways in which surgical excellence is measured and expertise is defined using artificial intelligence, video recordings, and accelerometers. However, these aforementioned methods of assessment are still subjective or indirect proxies of expertise, thus uncovering the neural mechanisms that differentiate expert surgeons from trainees may enhance the objectivity of surgical expertise validation. In fact, some researchers have already suggested that their neural imaging-based expertise classification methods outperform currently used methods of surgical skill certification such as the Fundamentals of Laparoscopic Surgery (FLS) scores. Such imaging biomarkers would not only help better identify the highest performing surgeons, but could also improve residency programs by providing more objective, evidence-based feedback and developmental milestones for those in training and perhaps act as a marker of surgical potential in medical students. Despite the potential advantages of using neural imaging in the assessment of surgical expertise, this field of research remains in its infancy. This systematic review identifies studies that have applied neuromonitoring in assessing surgical skill across levels of expertise. The goals of this review are to identify (1) the strongest neural indicators of surgical expertise, (2) the limitations of the current literature on this subject, (3) the most sensible future directions for further study. We found substantial evidence that surgical expertise can be delineated by differential activation and connectivity in the prefrontal cortex (PFC) across multiple task and neuroimaging modalities. Specifically, novices tend to have greater PFC activation than experts under standard conditions in bimanual and decision-making tasks. However, under high temporal demand tasks, experts had increased PFC activation whereas novices had decreased PFC activation. Common limitations uncovered in this review were that task difficulty was often insufficient to delineate between residents and attending. Moreover, attending level involvement was also low in multiple studies which may also have contributed to this issue. Most studies did not analyze the ability of their neuromonitoring findings to accurately classify subjects by level of expertise. Finally, the predominance of fNIRS as the neuromonitoring modality limits our ability to uncover the neural correlates of surgical expertise in non-cortical brain regions. Future studies should first strive to address these limitations. In the longer term, longitudinal within-subjects design over the course of a residency or even a career will also advance the field. Although logistically arduous, such studies would likely be most beneficial in demonstrating effects of increasing surgical expertise on regional brain activation and inter-region connectivity.

Frontal theta brain activity varies as a function of surgical experience and task error

OBJECTIVE

Investigations into surgical expertise have almost exclusively focused on overt behavioral characteristics with little consideration of the underlying neural processes. Recent advances in neuroimaging technologies, for example, wireless, wearable scalp-recorded electroencephalography (EEG), allow an insight into the neural processes governing performance. We used scalp-recorded EEG to examine whether surgical expertise and task performance could be differentiated according to an oscillatory brain activity signal known as frontal theta-a putative biomarker for cognitive control processes.

DESIGN SETTING AND PARTICIPANTS

Behavioral and EEG data were acquired from dental surgery trainees with 1 year (n=25) and 4 years of experience (n=20) while they performed low and high difficulty drilling tasks on a virtual reality surgical simulator. EEG power in the 4-7 Hz range in frontal electrodes (indexing frontal theta) was examined as a function of experience, task difficulty and error rate.

RESULTS

Frontal theta power was greater for novices relative to experts (p=0.001), but did not vary according to task difficulty (p=0.15) and there was no Experience × Difficulty interaction (p=0.87). Brain-behavior correlations revealed a significant negative relationship between frontal theta and error in the experienced group for the difficult task (r=-0.594, p=0.0058), but no such relationship emerged for novices.

CONCLUSION

We find frontal theta power differentiates between surgical experiences but correlates only with error rates for experienced surgeons while performing difficult tasks. These results provide a novel perspective on the relationship between expertise and surgical performance.

Brain activation during laparoscopic tasks in high- and low-performing medical students: a pilot fMRI study

BACKGROUND

Up to 20% of medical students are unable to reach competency in laparoscopic surgery. It is unknown whether these difficulties arise from heterogeneity in neurological functioning across individuals. We sought to examine the differences in neurological functioning during laparoscopic tasks between high- and low-performing medical students using functional magnetic resonance imaging (fMRI).

METHODS

This prospective cohort study enrolled North American medical students who were within the top 20% and bottom 20% of laparoscopic performers from a previous study. Brain activation was recorded using fMRI while participants performed peg-pointing, intracorporeal knot tying (IKT), and the Pictorial Surface Orientation (PicSOr) test. Brain activation maps were created and areas of activation were compared between groups.

RESULTS

In total, 9/12 high and 9/13 low performers completed the study. High performers completed IKT faster and made more successful knot ties than low performers [standing: 23.5 (5.0) sec vs. 37.6 (18.4) sec, p = 0.03; supine: 23.2 (2.5) sec vs. 72.7 (62.8) sec, p = 0.02; number of successful ties supine, 3 ties vs. 1 tie, p = 0.01]. Low performers showed more brain activation than high performers in the peg-pointing task (q < 0.01), with no activation differences in the IKT task. There were no behavioral differences in the PiCSOr task.

CONCLUSIONS

This study is the first to show differences between low and high performers of laparoscopic tasks at the brain level. This pilot study has shown the feasibility of using fMRI to examine laparoscopic surgical skills. Future studies are needed for further exploration of our initial findings.

The Neural Implementation of Surgical Expertise Within the Mirror-Neuron System: An fMRI Study

Motor expertise is an important aspect of high-level performance in professional tasks such as surgery. While recently it has been shown that brain activation as measured by functional magnetic resonance imaging (fMRI) within the mirror-neuron system (MNS) is modulated by expertise in sports and music, little is known about the neural underpinnings of professional, e.g., surgical expertise. Here, we investigated whether and (if so) how surgical expertise is implemented in the MNS in medical professionals across three levels of surgical qualification. In order to answer the more specific research question, namely, if the neural implementation of motor expertise develops in a linear or non-linear fashion, the study compares not only brain activation within the MNS related to action observation of novices and experts, but also intermediates. Ten novices (medical students), ten intermediates (residents in orthopedic surgery) and ten experts (orthopedic surgeons) watched 60 video clips (5 s each) of daily-life activities and surgical procedures each while their brain activation was measured using a 3-T fMRI scanner. An established localization procedure was followed to functionally define the MNS for each participant individually. A 2 (video type: daily-life activities, surgical procedures) × 3 (expertise level: novice, intermediate, expert) ANOVA yielded a non-significant interaction. Furthermore, separate analyses of the precentral and parietal part of the MNS also yielded non-significant interactions. However, post hoc comparisons showed that intermediates displayed marginally significantly lower brain activation in response to surgery-related videos within the MNS than novices. No other significant differences were found. We did not find evidence for the hypothesis that the brain-activation level in the MNS evoked by observing surgical videos reflects the level of surgical expertise in the professional task of (orthopedic) surgery. However, the results suggest a potential non-linear relationship between expertise level and MNS-activation level.

Move faster, think later: Women who play action video games have quicker visually-guided responses with later onset visuomotor-related brain activity

A history of action video game (AVG) playing is associated with improvements in several visuospatial and attention-related skills and these improvements may be transferable to unrelated tasks. These facts make video games a potential medium for skill-training and rehabilitation. However, examinations of the neural correlates underlying these observations are almost non-existent in the visuomotor system. Further, the vast majority of studies on the effects of a history of AVG play have been done using almost exclusively male participants. Therefore, to begin to fill these gaps in the literature, we present findings from two experiments. In the first, we use functional MRI to examine brain activity in experienced, female AVG players during visually-guided reaching. In the second, we examine the kinematics of visually-guided reaching in this population. Imaging data demonstrate that relative to women who do not play, AVG players have less motor-related preparatory activity in the cuneus, middle occipital gyrus, and cerebellum. This decrease is correlated with estimates of time spent playing. Further, these correlations are strongest during the performance of a visuomotor mapping that spatially dissociates eye and arm movements. However, further examinations of the full time-course of visuomotor-related activity in the AVG players revealed that the decreased activity during motor preparation likely results from a later onset of activity in AVG players, which occurs closer to beginning motor execution relative to the non-playing group. Further, the data presented here suggest that this later onset of preparatory activity represents greater neural efficiency that is associated with faster visually-guided responses.

A decade of imaging surgeons' brain function (part II): A systematic review of applications for technical and nontechnical skills assessment

BACKGROUND

Functional neuroimaging technologies enable assessment of operator brain function and can deepen our understanding of skills learning, ergonomic optima, and cognitive processes in surgeons. Although there has been a critical mass of data detailing surgeons' brain function, this literature has not been reviewed systematically.

METHODS

A systematic search of original neuroimaging studies assessing surgeons' brain function and published up until November 2016 was conducted using Medline, Embase, and PsycINFO databases.

RESULTS

Twenty-seven studies fulfilled the inclusion criteria, including 3 feasibility studies, 14 studies exploring the neural correlates of technical skill acquisition, and the remainder investigating brain function in the context of intraoperative decision-making (n = 1), neurofeedback training (n = 1), robot-assisted technology (n = 5), and surgical teaching (n = 3). Early stages of learning open surgical tasks (knot-tying) are characterized by prefrontal cortical activation, which subsequently attenuates with deliberate practice. However, with complex laparoscopic skills (intracorporeal suturing), prefrontal cortical engagement requires substantial training, and attenuation occurs over a longer time course, after years of refinement. Neurofeedback and interventions that improve neural efficiency may enhance technical performance and skills learning.

CONCLUSION

Imaging surgeons' brain function has identified neural signatures of expertise that might help inform objective assessment and selection processes. Interventions that improve neural efficiency may target skill-specific brain regions and augment surgical performance.

A decade of imaging surgeons' brain function (part I): Terminology, techniques, and clinical translation

BACKGROUND

Functional neuroimaging has the potential to deepen our understanding of technical and nontechnical skill acquisition in surgeons, particularly as established assessment tools leave unanswered questions about inter-operator differences in ability that seem independent of experience.

METHODS

In this first of a 2-part article, we aim to utilize our experience in neuroimaging surgeons to orientate the nonspecialist reader to the principles of brain imaging. Terminology commonly used in brain imaging research is explained, placing emphasis on the "activation response" to an surgical task and its effect on local cortical hemodynamic parameters (neurovascular coupling).

RESULTS

Skills learning and subsequent consolidation and refinement through practice lead to reorganization of the functional architecture of the brain (known as "neuroplasticity"), evidenced by changes in the strength of regional activation as well as alterations in connectivity between brain regions, culminating in more efficient use of neural resources during task performance.

CONCLUSION

Currently available neuroimaging techniques that either directly (ie, measure electrical activity) or indirectly (ie, measure tissue hemodynamics) assess brain function are discussed. Finally, we highlight the important practical considerations when conducting brain imaging research in surgeons.

Objective Measurement of Clinical Competency in Surgical Education Using Electrodermal Activity

OBJECTIVE

Within the realm of surgical education, there is a need for objective means to determine surgical competence and resident readiness to operate independently. We propose a novel, objective method of assessing resident confidence and clinical competence based on measurement of electrodermal activity (EDA) during live surgical procedures. We hypothesized that with progressive training, EDA responses to the stress of performing surgery would exhibit decline, elucidating an objective correlate of clinical competence.

DESIGN

EDA was measured using galvanic skin response sensors worn by residents performing laparoscopic cholecystectomy on sequential live human patients over an 8-month period. Baseline, phasic (peak) and tonic EDA responses were measured as a fractional change from baseline.

SETTING

University of Missouri, Columbia, Missouri, an academic tertiary care facility.

PARTICIPANTS

Fourteen categorical general surgery residents and 5 faculty surgeons were voluntarily enrolled and participated through completion.

RESULTS

Tonic fractional change (FC_TONIC) was highest in PGY3 residents compared with postgraduate year (PGY) 1 and 2 residents (7.199 vs. 2.100, p = 0.004, 95% CI: 8.58-1.61 and PGY4 and 5 residents (7.199 vs. 2.079, p = 0.002, 95% CI: 8.38-0.29). Phasic fractional change in EDA (FC_PHASIC) exhibited a progressive decline across resident training levels, with PGY1 and 2 residents having the highest response, and faculty displaying the lowest FC_PHASIC responses. Statistical differences were seen between FC_PHASIC faculty and PGY4 and 5 (3.596 vs. 6.180, p = 0.004, 95% CI: 0.80-4.36), PGY4 and 5, and PGY3 (6.180 vs. 15.998, p = 0.003, 95% CI: 3.33-16.3), as well as among all residents and faculty (13.057 vs. 3.596, p = 0.004, 95% CI: 15.8-3.1).

CONCLUSION

Phasic EDA changes decrease with increasing clinical competence. For those participants with the lowest and highest levels of competence, tonic EDA changes are minimal. Tonic EDA changes follow an inverse-U shape with differing levels of clinical competence.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Technical mentorship during robot-assisted surgery: a cognitive analysis

OBJECTIVE

To investigate cognitive and mental workload assessments, which may play a critical role in defining successful mentorship.

MATERIALS AND METHODS

The 'Mind Maps' project aimed at evaluating cognitive function with regard to surgeon's expertise and trainee's skills. The study included electroencephalogram (EEG) recordings of a mentor observing trainee surgeons in 20 procedures involving extended lymph node dissection (eLND) or urethrovesical anastomosis (UVA), with simultaneous assessment of trainees using the National Aeronautics and Space Administration Task Load index (NASA-TLX) questionnaire. We also compared the brain activity of the mentor during this study with his own brain activity while actually performing the same surgical steps from previous procedures populated in the 'Mind Maps' project.

RESULTS

During eLND and UVA, when the mentor thought the trainee's mental demand and effort were low based on his NASA-TLX questionnaire (not satisfied with his performance), his EEG-based mental workload increased (reflecting more concern and attention). The mentor was mentally engaged and concerned while he was engrossed in observing the surgery. This was further supported by the finding that there was no significant difference in the mental demands and workload between observing and operating for the expert surgeon.

CONCLUSIONS

This study objectively evaluated the cognitive engagement of a surgical mentor teaching technical skills during surgery. The study provides a deeper understanding of how surgical teaching actually works and opens new horizons for assessment and teaching of surgery. Further research is needed to study the feasibility of this novel concept in assessment and guidance of surgical performance.

When should surgeons retire?

BACKGROUND

Retirement policies for surgeons differ worldwide. A range of normal human functional abilities decline as part of the ageing process. As life expectancy and their population increases, the performance ability of ageing surgeons is now a growing concern in relation to patient care. The aim was to explore the effects of ageing on surgeons' performance, and to identify current practical methods for transitioning surgeons out of practice at the appropriate time and age.

METHODS

A narrative review was performed in MEDLINE using the terms 'ageing' and 'surgeon'. Additional articles were hand-picked. Modified PRISMA guidelines informed the selection of articles for inclusion. Articles were included only if they explored age-related changes in brain biology and the effect of ageing on surgeons' performance.

RESULTS

The literature search yielded 1811 articles; of these, 36 articles were included in the final review. Wide variation in ability was observed across ageing individuals (both surgical and lay). Considerable variation in the effects of the surgeon's age on patient mortality and postoperative complications was noted. A lack of neuroimaging research exploring the ageing of surgeons' brains specifically, and lack of real markers available for measuring surgical performance, both hinder further investigation. Standard retirement policies in accordance with age-related surgical ability are lacking in most countries around the world.

CONCLUSION

Competence should be assessed at an individual level, focusing on functional ability over chronological age; this should inform retirement policies for surgeons.