Can Natural Language Processing and Artificial Intelligence Automate The Generation of Billing Codes From Operative Note Dictations?

Applications of natural language processing tools in the surgical journey

Background

Natural language processing tools are becoming increasingly adopted in multiple industries worldwide. They have shown promising results however their use in the field of surgery is under-recognised. Many trials have assessed these benefits in small settings with promising results before large scale adoption can be considered in surgery. This study aims to review the current research and insights into the potential for implementation of natural language processing tools into surgery.

Methods

A narrative review was conducted following a computer-assisted literature search on Medline, EMBASE and Google Scholar databases. Papers related to natural language processing tools and consideration into their use for surgery were considered.

Results

Current applications of natural language processing tools within surgery are limited. From the literature, there is evidence of potential improvement in surgical capability and service delivery, such as through the use of these technologies to streamline processes including surgical triaging, data collection and auditing, surgical communication and documentation. Additionally, there is potential to extend these capabilities to surgical academia to improve processes in surgical research and allow innovation in the development of educational resources. Despite these outcomes, the evidence to support these findings are challenged by small sample sizes with limited applicability to broader settings.

Conclusion

With the increasing adoption of natural language processing technology, such as in popular forms like ChatGPT, there has been increasing research in the use of these tools within surgery to improve surgical workflow and efficiency. This review highlights multifaceted applications of natural language processing within surgery, albeit with clear limitations due to the infancy of the infrastructure available to leverage these technologies. There remains room for more rigorous research into broader capability of natural language processing technology within the field of surgery and the need for cross-sectoral collaboration to understand the ways in which these algorithms can best be integrated.

Artificial intelligence in surgery

Artificial intelligence (AI) is rapidly emerging in healthcare, yet applications in surgery remain relatively nascent. Here we review the integration of AI in the field of surgery, centering our discussion on multifaceted improvements in surgical care in the preoperative, intraoperative and postoperative space. The emergence of foundation model architectures, wearable technologies and improving surgical data infrastructures is enabling rapid advances in AI interventions and utility. We discuss how maturing AI methods hold the potential to improve patient outcomes, facilitate surgical education and optimize surgical care. We review the current applications of deep learning approaches and outline a vision for future advances through multimodal foundation models.

Performance of a Large Language Model in the Generation of Clinical Guidelines for Antibiotic Prophylaxis in Spine Surgery

OBJECTIVE

Large language models, such as chat generative pre-trained transformer (ChatGPT), have great potential for streamlining medical processes and assisting physicians in clinical decision-making. This study aimed to assess the potential of ChatGPT's 2 models (GPT-3.5 and GPT-4.0) to support clinical decision-making by comparing its responses for antibiotic prophylaxis in spine surgery to accepted clinical guidelines.

METHODS

ChatGPT models were prompted with questions from the North American Spine Society (NASS) Evidence-based Clinical Guidelines for Multidisciplinary Spine Care for Antibiotic Prophylaxis in Spine Surgery (2013). Its responses were then compared and assessed for accuracy.

RESULTS

Of the 16 NASS guideline questions concerning antibiotic prophylaxis, 10 responses (62.5%) were accurate in ChatGPT's GPT-3.5 model and 13 (81%) were accurate in GPT-4.0. Twenty-five percent of GPT-3.5 answers were deemed as overly confident while 62.5% of GPT-4.0 answers directly used the NASS guideline as evidence for its response.

CONCLUSION

ChatGPT demonstrated an impressive ability to accurately answer clinical questions. GPT-3.5 model's performance was limited by its tendency to give overly confident responses and its inability to identify the most significant elements in its responses. GPT-4.0 model's responses had higher accuracy and cited the NASS guideline as direct evidence many times. While GPT-4.0 is still far from perfect, it has shown an exceptional ability to extract the most relevant research available compared to GPT-3.5. Thus, while ChatGPT has shown far-reaching potential, scrutiny should still be exercised regarding its clinical use at this time.

Can Publicly Available Artificial Intelligence Successfully Identify Current Procedural Terminology Codes for Common Procedures in Neurosurgery?

OBJECTIVE

Coding for neurosurgical procedures is a complex process that is dynamically changing year to year, through the annual introduction and removal of codes and modifiers. The authors hoped to elucidate if publicly available artificial intelligence (AI) could offer solutions for neurosurgeons with regard to coding.

METHODS

Multiple publicly available AI platforms were asked to provide Current Procedural Terminology (CPT) codes and Revenue Value Units (RVU) values for common neurosurgical procedures of the brain and spine with a given indication for the procedure. The responses of platforms were recorded and compared to the currently valid CPT codes used for the procedure and the amount of RVUs that would be gained.

RESULTS

Six platforms and Google were asked for the appropriate CPT codes for 10 endovascular, spinal, and cranial procedures each. The highest performing platforms were as follows: Perplexity.AI identified 70% of endovascular, BingAI identified 55% of spinal, and ChatGPT 4.0 with Bing identified 75% of cranial CPT codes. With regard to RVUs, the top performer gained 78% of endovascular, 42% of spinal, and 70% of cranial possible RVUs. With regard to accuracy, AI platforms on average outperformed Google (45% vs. 25%, P = 0.04236).

CONCLUSIONS

The ability of publicly available AIs to successfully code for neurosurgical procedures holds great promise in the future. Future development of AI should focus on improving accuracy with regard to CPT codes and providing supporting documentation for its decisions. Improvement on the existing capabilities of AI platforms can allow for increased operational efficiency and cost savings for practices.

Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives

Machine learning (ML) solutions are increasingly entering healthcare. They are complex, sociotechnical systems that include data inputs, ML models, technical infrastructure and human interactions. They have promise for improving care across a wide range of clinical applications but if poorly implemented, they may disrupt clinical workflows, exacerbate inequities in care and harm patients. Many aspects of ML solutions are similar to other digital technologies, which have well-established approaches to implementation. However, ML applications present distinct implementation challenges, given that their predictions are often complex and difficult to understand, they can be influenced by biases in the data sets used to develop them, and their impacts on human behaviour are poorly understood. This manuscript summarises the current state of knowledge about implementing ML solutions in clinical care and offers practical guidance for implementation. We propose three overarching questions for potential users to consider when deploying ML solutions in clinical care: (1) Is a clinical or operational problem likely to be addressed by an ML solution? (2) How can an ML solution be evaluated to determine its readiness for deployment? (3) How can an ML solution be deployed and maintained optimally? The Quality Improvement community has an essential role to play in ensuring that ML solutions are translated into clinical practice safely, effectively, and ethically.

Artificially Intelligent Billing in Spine Surgery: An Analysis of a Large Language Model

STUDY DESIGN

Retrospective cohort study.

OBJECTIVES

This study assessed the effectiveness of a popular large language model, ChatGPT-4, in predicting Current Procedural Terminology (CPT) codes from surgical operative notes. By employing a combination of prompt engineering, natural language processing (NLP), and machine learning techniques on standard operative notes, the study sought to enhance billing efficiency, optimize revenue collection, and reduce coding errors.

METHODS

The model was given 3 different types of prompts for 50 surgical operative notes from 2 spine surgeons. The first trial was simply asking the model to generate CPT codes for a given OP note. The second trial included 3 OP notes and associated CPT codes to, and the third trial included a list of every possible CPT code in the dataset to prime the model. CPT codes generated by the model were compared to those generated by the billing department. Model evaluation was performed in the form of calculating the area under the ROC (AUROC), and area under precision-recall curves (AUPRC).

RESULTS

The trial that involved priming ChatGPT with a list of every possible CPT code performed the best, with an AUROC of .87 and an AUPRC of .67, and an AUROC of .81 and AUPRC of .76 when examining only the most common CPT codes.

CONCLUSIONS

ChatGPT-4 can aid in automating CPT billing from orthopedic surgery operative notes, driving down healthcare expenditures and enhancing billing code precision as the model evolves and fine-tuning becomes available.

Natural language processing in radiology: Clinical applications and future directions

Natural language processing (NLP) is a wide range of techniques that allows computers to interact with human text. Applications of NLP in everyday life include language translation aids, chat bots, and text prediction. It has been increasingly utilized in the medical field with increased reliance on electronic health records. As findings in radiology are primarily communicated via text, the field is particularly suited to benefit from NLP based applications. Furthermore, rapidly increasing imaging volume will continue to increase burden on clinicians, emphasizing the need for improvements in workflow. In this article, we highlight the numerous non-clinical, provider focused, and patient focused applications of NLP in radiology. We also comment on challenges associated with development and incorporation of NLP based applications in radiology as well as potential future directions.

Can a Novel Natural Language Processing Model and Artificial Intelligence Automatically Generate Billing Codes From Spine Surgical Operative Notes?

STUDY DESIGN

Retrospective cohort.

OBJECTIVE

Billing and coding-related administrative tasks are a major source of healthcare expenditure in the United States. We aim to show that a second-iteration Natural Language Processing (NLP) machine learning algorithm, XLNet, can automate the generation of CPT codes from operative notes in ACDF, PCDF, and CDA procedures.

METHODS

We collected 922 operative notes from patients who underwent ACDF, PCDF, or CDA from 2015 to 2020 and included CPT codes generated by the billing code department. We trained XLNet, a generalized autoregressive pretraining method, on this dataset and tested its performance by calculating AUROC and AUPRC.

RESULTS

The performance of the model approached human accuracy. Trial 1 (ACDF) achieved an AUROC of .82 (range: .48-.93), an AUPRC of .81 (range: .45-.97), and class-by-class accuracy of 77% (range: 34%-91%); trial 2 (PCDF) achieved an AUROC of .83 (.44-.94), an AUPRC of .70 (.45-.96), and class-by-class accuracy of 71% (42%-93%); trial 3 (ACDF and CDA) achieved an AUROC of .95 (.68-.99), an AUPRC of .91 (.56-.98), and class-by-class accuracy of 87% (63%-99%); trial 4 (ACDF, PCDF, CDA) achieved an AUROC of .95 (.76-.99), an AUPRC of .84 (.49-.99), and class-by-class accuracy of 88% (70%-99%).

CONCLUSIONS

We show that the XLNet model can be successfully applied to orthopedic surgeon's operative notes to generate CPT billing codes. As NLP models as a whole continue to improve, billing can be greatly augmented with artificial intelligence assisted generation of CPT billing codes which will help minimize error and promote standardization in the process.

New Frontiers of Natural Language Processing in Surgery

The vast and ever-growing volume of electronic health records (EHR) have generated a wealth of information-rich data. Traditional, non-machine learning data extraction techniques are error-prone and laborious, hindering the analytical potential of these massive data sources. Equipped with natural language processing (NLP) tools, surgeons are better able to automate, and customize their review to investigate and implement surgical solutions. We identify current perioperative applications of NLP algorithms as well as research limitations and future avenues to outline the impact and potential of this technology for progressing surgical innovation.

Meaningless Applications and Misguided Methodologies in Artificial Intelligence-Related Orthopaedic Research Propagates Hype Over Hope

There exists great hope and hype in the literature surrounding applications of artificial intelligence (AI) to orthopaedic surgery. Between 2018 and 2021, a total of 178 AI-related articles were published in orthopaedics. However, for every 2 original research papers that apply AI to orthopaedics, a commentary or review is published (30.3%). AI-related research in orthopaedics frequently fails to provide use cases that offer the uninitiated an opportunity to appraise the importance of AI by studying meaningful questions, evaluating unknown hypotheses, or analyzing quality data. The hype perpetuates a feed-forward cycle that relegates AI to a meaningless buzzword by rewarding those with nascent understanding and rudimentary technical knowhow into committing several basic errors: (1) inappropriately conflating vernacular ("AI/machine learning"), (2) repackaging registry data, (3) prematurely releasing internally validated algorithms, (4) overstating the "black box phenomenon" by failing to provide weighted analysis, (5) claiming to evaluate AI rather than the data itself, and (6) withholding full model architecture code. Relevant AI-specific guidelines are forthcoming, but forced application of the original Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines designed for regression analyses is irrelevant and misleading. To safeguard meaningful use, AI-related research efforts in orthopaedics should be (1) directed toward administrative support over clinical evaluation and management, (2) require the use of the advanced model, and (3) answer a question that was previously unknown, unanswered, or unquantifiable.