Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy

How well do faculty do in providing general surgery EPA feedback?

INTRODUCTION

Entrustable Professional Activities (EPAs) provide a framework for competency-based assessment in surgery. EPA descriptions include observable behaviors by trainees at progressive levels of autonomy. The American Board of Surgery (ABS) required all General Surgery (GS) residency programs to implement assessment of 18 EPAs at the beginning of academic year 2023-2024. Microassessments provide formative self-reflection by the resident and feedback by faculty upon completion of the EPA. These frequent assessments culminate in a resident performance profile utilized by the trainee for formative growth and the clinical competency committee for summative feedback. Assessor free text comments are an opportunity to provide meaningful, constructive feedback to residents. Our aim was to analyze comments provided by faculty to residents in terms of their alignment with EPA descriptors and provision of actionable feedback.

METHODS

A total of 540 ​GS EPA assessments for inguinal hernia, gallbladder disease, appendicitis, trauma, and surgical consultation were evaluated from 6/2021-12/2022. We assessed free text EPA comments from faculty compared to EPA behavior descriptions for alignment with the selected EPA level of entrustment. The comments were judged on a binary scale of "Align" vs "Not Align" by two independent evaluators, with a third evaluator to address discordance. Comments were then evaluated for resident behavioral descriptions, suggestions for improvement, and positive or negative feedback.

RESULTS

Approximately 77 ​% of EPA microassessments had alignment between level of autonomy and free text feedback. A common example of feedback discordant with level of autonomy was rating a trainee at an intraoperative level 4 (independent practice) with comments such as "required some guidance with retrocecal case and upsizing port." Based on behavior descriptions this would be a level 3 (indirect supervision). Approximately 88 ​% of feedback contained positive comments with minimal negative feedback (e.g., "this did not go well."). Actionable feedback including "work on optimization of retracting hand" or "continue to work clamp/tie technique and square off each knot" was present in 28.3 ​% of feedback.

CONCLUSIONS

The majority of faculty provide feedback that is aligned with the behavioral anchors of the EPAs assessed, but frequently did not provide actionable feedback to the resident regarding how to advance to the next level of entrustment. EPA entrustment behaviors provide a framework for the development of practice-ready behaviors, and if assessors anchor their feedback in the behaviors for a given entrustment level and project how a resident could proceed to the next level, they can provide a clear trajectory for skill development. Faculty development should focus on improving the frequency of actionable free text feedback, outlining how residents can advance in the future.

Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities

Importance

A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known.

Objective

To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions.

Design, Setting, and Participants

This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023.

Exposure

Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle.

Main Outcomes and Measures

The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence.

Results

Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association.

Conclusions and Relevance

In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.

Artificial Intelligence Applications for Thoracic Surgeons: "The Phenomenal Cosmic Powers of the Magic Lamp"

In the digital age, artificial intelligence (AI) is emerging as a transformative force in various sectors, including medicine. This article explores the potential of AI, which is akin to the magical genie of Aladdin's lamp, particularly within thoracic surgery and lung cancer management. It examines AI applications like machine learning and deep learning in achieving more precise diagnoses, preoperative risk assessment, and improved surgical outcomes. The challenges and advancements in AI integration, especially in computer vision and multi-modal models, are discussed alongside their impact on robotic surgery and operating room management. Despite its transformative potential, implementing AI in medicine faces challenges regarding data scarcity, interpretability issues, and ethical concerns. Collaboration between AI and medical communities is essential to address these challenges and unlock the full potential of AI in revolutionizing clinical practice. This article underscores the importance of further research and interdisciplinary collaboration to ensure the safe and effective deployment of AI in real-world clinical settings.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Large language models in physical therapy: time to adapt and adept

Healthcare is experiencing a transformative phase, with artificial intelligence (AI) and machine learning (ML). Physical therapists (PTs) stand on the brink of a paradigm shift in education, practice, and research. Rather than visualizing AI as a threat, it presents an opportunity to revolutionize. This paper examines how large language models (LLMs), such as ChatGPT and BioMedLM, driven by deep ML can offer human-like performance but face challenges in accuracy due to vast data in PT and rehabilitation practice. PTs can benefit by developing and training an LLM specifically for streamlining administrative tasks, connecting globally, and customizing treatments using LLMs. However, human touch and creativity remain invaluable. This paper urges PTs to engage in learning and shaping AI models by highlighting the need for ethical use and human supervision to address potential biases. Embracing AI as a contributor, and not just a user, is crucial by integrating AI, fostering collaboration for a future in which AI enriches the PT field provided data accuracy, and the challenges associated with feeding the AI model are sensitively addressed.

Artificial Intelligence in Urology: Current Status and Future Perspectives

Surgical fields, especially urology, have shifted increasingly toward the use of artificial intelligence (AI). Advancements in AI have created massive improvements in diagnostics, outcome predictions, and robotic surgery. For robotic surgery to progress from assisting surgeons to eventually reaching autonomous procedures, there must be advancements in machine learning, natural language processing, and computer vision. Moreover, barriers such as data availability, interpretability of autonomous decision-making, Internet connection and security, and ethical concerns must be overcome.

Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education

OBJECTIVE

The purpose of this paper is to describe a sentiment analysis program that aids in identifying pharmacy students at risk for progression issues by automatically scoring preceptor comments as positive or negative.

METHODS

An R-based program to analyze advanced pharmacy practice experiences and introductory pharmacy practice experiences midpoint evaluation of preceptor comments was piloted in phase 1 by comparing the sentiment analysis algorithm results to human coding. The algorithm was refined in phase 2. In phase 3, the validation phase, the final sentiment analysis algorithm analyzed all midpoint student evaluations (n = 1560). Sentiment scores were generated for each preceptor comment, and correlations were performed between sentiment scores and the quantitative scoring provided on the assessment.

RESULTS

In phase 1, agreement between faculty coders and sentiment analysis was 96%, and in phase 2, agreement between the final codes and sentiment analysis was 92.4% once keywords were added to the sentiment dictionary. In phase 3, a total of 3919 comments from 1560 evaluations were analyzed, and overall, the sentiment analysis results aligned with the quantitative data.

CONCLUSION

This sentiment analysis algorithm was accurate in capturing positive and negative comments corresponding to pharmacy student performance. Given the accuracy of this preliminary validation for flagging preceptor comments, there are numerous implications when considering the use of sentiment analysis in pharmacy education. Using a sentiment analysis program minimizes the number of qualitative preceptor comments needing review by experiential faculty, as this program can aid in identifying students at risk of progression issues.

The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship Performance: Lessons Learned

Background

Natural language processing is a promising technique that can be used to create efficiencies in the review of narrative feedback to learners. The Feinberg School of Medicine has implemented formal review of pre-clerkship narrative feedback since 2014 through its portfolio assessment system but this process requires considerable time and effort. This article describes how natural language processing was used to build a predictive model of pre-clerkship student performance that can be utilized to assist competency committee reviews.

Approach

The authors took an iterative and inductive approach to the analysis, which allowed them to identify characteristics of narrative feedback that are both predictive of performance and useful to faculty reviewers. Words and phrases were manually grouped into topics that represented concepts illustrating student performance. Topics were reviewed by experienced reviewers, tested for consistency across time, and checked to ensure they did not demonstrate bias.

Outcomes

Sixteen topic groups of words and phrases were found to be predictive of performance. The best-fitting model used a combination of topic groups, word counts, and categorical ratings. The model had an AUC value of 0.92 on the training data and 0.88 on the test data.

Reflection

A thoughtful, careful approach to using natural language processing was essential. Given the idiosyncrasies of narrative feedback in medical education, standard natural language processing packages were not adequate for predicting student outcomes. Rather, employing qualitative techniques including repeated member checking and iterative revision resulted in a useful and salient predictive model.

Multi-disciplinary assessment of the entrustable professional activities of surgery residents

Purpose

Medicine is practiced in a collaborative and interdisciplinary manner. However, medical training and assessment remain largely isolated in traditional departmental silos. Two Entrustable Professional Activities (EPAs) developed by the American Board of Surgery are multidisciplinary in nature and offer a unique opportunity to study interdisciplinary assessment.

Methods

EPA microassessments were collected from Surgery and Emergency Medicine (EM) faculty between July 2018 and May 2020. Differences in feedback provided by faculty were assessed using natural language processing (NLP) techniques, (1) automated algorithms; and (2) topic modeling. Summative content analysis was used to identify themes in text feedback. We developed automated coding algorithms for these themes using regular expressions. Topic modeling was performed using latent Dirichlet allocation.

Results

549 assessments were collected for two EPAs: 198 for GS Consultation and 351 for Trauma. 27 EM and 27 Surgery faculty provided assessments for 71 residents. EM faculty were significantly more likely than Surgery faculty to submit feedback coded as Communication, Demeanor, and Timeliness, (all chi-square test p-values < 0.01). No significant differences were found for Clinical Performance, Skill Level, or Areas for Improvement. Similarly, topic modeling indicated that assessments submitted by EM faculty focused on communication, timeliness, and interpersonal skills, while those submitted by Surgery faculty focused on the residents' abilities to effectively gather information and correctly diagnose the underlying pathology.

Conclusions

Feedback from EM and Surgery faculty differed significantly based on NLP analyses. EPA assessments should stem from multiple sources to avoid assessment gaps and represent a more holistic picture of performance.

Exploring how feedback reflects entrustment decisions using artificial intelligence

CONTEXT

Clinical supervisors make judgements about how much to trust learners with critical activities in patient care. Such decisions mediate trainees' opportunities for learning and competency development and thus are a critical component of education. As educators apply entrustment frameworks to assessment, it is important to determine how narrative feedback reflecting entrustment may also address learners' educational needs.

METHODS

In this study, we used artificial intelligence (AI) and natural language processing (NLP) to identify characteristics of feedback tied to supervisors' entrustment decisions during direct observation encounters of clerkship medical students (3328 unique observations). Supervisors conducted observations of students and collaborated with them to complete an entrustment-based assessment in which they documented narrative feedback and assigned an entrustment rating. We trained a deep neural network (DNN) to predict entrustment levels from the narrative data and developed an explainable AI protocol to uncover the latent thematic features the DNN used to make its prediction.

RESULTS

We found that entrustment levels were associated with level of detail (specific steps for performing clinical tasks), feedback type (constructive versus reinforcing) and task type (procedural versus cognitive). In justifying both high and low levels of entrustment, supervisors detailed concrete steps that trainees performed (or did not yet perform) competently.

CONCLUSIONS

Framing our results in the factors previously identified as influencing entrustment, we find a focus on performance details related to trainees' clinical competency as opposed to nonspecific feedback on trainee qualities. The entrustment framework reflected in feedback appeared to guide specific goal-setting, combined with details necessary to reach those goals. Our NLP methodology can also serve as a starting point for future work on entrustment and feedback as similar assessment datasets accumulate.

Gender Differences in Entrustable Professional Activity Evaluations of General Surgery Residents

OBJECTIVE

To determine differences in entrustable professional activity (EPA) assessments between male and female general surgery residents.

SUMMARY BACKGROUND DATA

Evaluations play a critical role in career advancement for physicians. However, female physicians in training receive lower evaluations and underrate their own performance. Competency-based assessment frameworks, such as EPAs, may help address gender bias in surgery by linking evaluations to specific, observable behaviors.

METHODS

In this cohort study, EPA assessments were collected from July 2018 to May 2020. The effect of resident sex on EPA entrustment levels was analyzed using multiple linear and ordered logistic regressions. Narrative comments were analyzed using latent dirichlet allocation to identify topics correlated with resident sex.

RESULTS

Of the 2480 EPAs, 1230 EPAs were submitted by faculty and 1250 were submitted by residents. After controlling for confounding factors, faculty evaluations of residents were not impacted by resident sex (estimate = 0.09, P = 0.08). However, female residents rated themselves lower by 0.29 (on a 0-4 scale) compared to their male counterparts (P < 0.001). Within narrative assessments, topics associated with resident sex demonstrated that female residents focus on the "guidance" and "supervision" they received while performing an EPA, while male residents were more likely to report "independent" action.

CONCLUSIONS

Faculty assessments showed no difference in EPA levels between male and female residents. Female residents rate themselves lower by nearly an entire post graduate year (PGY) level compared to male residents. Latent dirichlet allocation -identified topics suggest this difference in self-assessment is related to differences in perception of autonomy.

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

Most surgeons are skeptical as to the feasibility of autonomous actions in surgery. Interestingly, many examples of autonomous actions already exist and have been around for years. Since the beginning of this millennium, the field of artificial intelligence (AI) has grown exponentially with the development of machine learning (ML), deep learning (DL), computer vision (CV) and natural language processing (NLP). All of these facets of AI will be fundamental to the development of more autonomous actions in surgery, unfortunately, only a limited number of surgeons have or seek expertise in this rapidly evolving field. As opposed to AI in medicine, AI surgery (AIS) involves autonomous movements. Fortuitously, as the field of robotics in surgery has improved, more surgeons are becoming interested in technology and the potential of autonomous actions in procedures such as interventional radiology, endoscopy and surgery. The lack of haptics, or the sensation of touch, has hindered the wider adoption of robotics by many surgeons; however, now that the true potential of robotics can be comprehended, the embracing of AI by the surgical community is more important than ever before. Although current complete surgical systems are mainly only examples of tele-manipulation, for surgeons to get to more autonomously functioning robots, haptics is perhaps not the most important aspect. If the goal is for robots to ultimately become more and more independent, perhaps research should not focus on the concept of haptics as it is perceived by humans, and the focus should be on haptics as it is perceived by robots/computers. This article will discuss aspects of ML, DL, CV and NLP as they pertain to the modern practice of surgery, with a focus on current AI issues and advances that will enable us to get to more autonomous actions in surgery. Ultimately, there may be a paradigm shift that needs to occur in the surgical community as more surgeons with expertise in AI may be needed to fully unlock the potential of AIS in a safe, efficacious and timely manner.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.