Can generative artificial intelligence pass the orthopaedic board examination?

Background

Resident training programs in the US use the Orthopaedic In-Training Examination (OITE) developed by the American Academy of Orthopaedic Surgeons (AAOS) to assess the current knowledge of their residents and to identify the residents at risk of failing the Amerian Board of Orthopaedic Surgery (ABOS) examination. Optimal strategies for OITE preparation are constantly being explored. There may be a role for Large Language Models (LLMs) in orthopaedic resident education. ChatGPT, an LLM launched in late 2022 has demonstrated the ability to produce accurate, detailed answers, potentially enabling it to aid in medical education and clinical decision-making. The purpose of this study is to evaluate the performance of ChatGPT on Orthopaedic In-Training Examinations using Self-Assessment Exams from the AAOS database and approved literature as a proxy for the Orthopaedic Board Examination.

Methods

301 SAE questions from the AAOS database and associated AAOS literature were input into ChatGPT's interface in a question and multiple-choice format and the answers were then analyzed to determine which answer choice was selected. A new chat was used for every question. All answers were recorded, categorized, and compared to the answer given by the OITE and SAE exams, noting whether the answer was right or wrong.

Results

Of the 301 questions asked, ChatGPT was able to correctly answer 183 (60.8%) of them. The subjects with the highest percentage of correct questions were basic science (81%), oncology (72.7%, shoulder and elbow (71.9%), and sports (71.4%). The questions were further subdivided into 3 groups: those about management, diagnosis, or knowledge recall. There were 86 management questions and 47 were correct (54.7%), 45 diagnosis questions with 32 correct (71.7%), and 168 knowledge recall questions with 102 correct (60.7%).

Conclusions

ChatGPT has the potential to provide orthopedic educators and trainees with accurate clinical conclusions for the majority of board-style questions, although its reasoning should be carefully analyzed for accuracy and clinical validity. As such, its usefulness in a clinical educational context is currently limited but rapidly evolving.

Clinical relevance

ChatGPT can access a multitude of medical data and may help provide accurate answers to clinical questions.

Reliable Prediction of Discharge Disposition Following Cervical Spine Surgery With Ensemble Machine Learning and Validation on a National Cohort

STUDY DESIGN

A retrospective cohort study.

OBJECTIVE

The purpose of this study is to develop a machine learning algorithm to predict nonhome discharge after cervical spine surgery that is validated and usable on a national scale to ensure generalizability and elucidate candidate drivers for prediction.

SUMMARY OF BACKGROUND DATA

Excessive length of hospital stay can be attributed to delays in postoperative referrals to intermediate care rehabilitation centers or skilled nursing facilities. Accurate preoperative prediction of patients who may require access to these resources can facilitate a more efficient referral and discharge process, thereby reducing hospital and patient costs in addition to minimizing the risk of hospital-acquired complications.

METHODS

Electronic medical records were retrospectively reviewed from a single-center data warehouse (SCDW) to identify patients undergoing cervical spine surgeries between 2008 and 2019 for machine learning algorithm development and internal validation. The National Inpatient Sample (NIS) database was queried to identify cervical spine fusion surgeries between 2009 and 2017 for external validation of algorithm performance. Gradient-boosted trees were constructed to predict nonhome discharge across patient cohorts. The area under the receiver operating characteristic curve (AUROC) was used to measure model performance. SHAP values were used to identify nonlinear risk factors for nonhome discharge and to interpret algorithm predictions.

RESULTS

A total of 3523 cases of cervical spine fusion surgeries were included from the SCDW data set, and 311,582 cases were isolated from NIS. The model demonstrated robust prediction of nonhome discharge across all cohorts, achieving an area under the receiver operating characteristic curve of 0.87 (SD=0.01) on both the SCDW and nationwide NIS test sets. Anterior approach only, age, elective admission status, Medicare insurance status, and total Elixhauser Comorbidity Index score were the most important predictors of discharge destination.

CONCLUSIONS

Machine learning algorithms reliably predict nonhome discharge across single-center and national cohorts and identify preoperative features of importance following cervical spine fusion surgery.

Can Natural Language Processing and Artificial Intelligence Automate The Generation of Billing Codes From Operative Note Dictations?

STUDY DESIGN

Retrospective Cohort Study.

OBJECTIVES

Using natural language processing (NLP) in combination with machine learning on standard operative notes may allow for efficient billing, maximization of collections, and minimization of coder error. This study was conducted as a pilot study to determine if a machine learning algorithm can accurately identify billing Current Procedural Terminology (CPT) codes on patient operative notes.

METHODS

This was a retrospective analysis of operative notes from patients who underwent elective spine surgery by a single senior surgeon from 9/2015 to 1/2020. Algorithm performance was measured by performing receiver operating characteristic (ROC) analysis, calculating the area under the ROC curve (AUC) and the area under the precision-recall curve (AUPRC). A deep learning NLP algorithm and a Random Forest algorithm were both trained and tested on operative notes to predict CPT codes. CPT codes generated by the billing department were compared to those generated by our model.

RESULTS

The random forest machine learning model had an AUC of .94 and an AUPRC of .85. The deep learning model had a final AUC of .72 and an AUPRC of .44. The random forest model had a weighted average, class-by-class accuracy of 87%. The LSTM deep learning model had a weighted average, class-by-class accuracy 0f 59%.

CONCLUSIONS

Combining natural language processing with machine learning is a valid approach for automatic generation of CPT billing codes. The random forest machine learning model outperformed the LSTM deep learning model in this case. These models can be used by orthopedic or neurosurgery departments to allow for efficient billing.

Readmission and Associated Factors in Surgical Versus Non-Surgical Management of Spinal Epidural Abscess: A Nationwide Readmissions Database Analysis

STUDY DESIGN

Retrospective cohort study.

OBJECTIVES

Spinal epidural abscess (SEA) is a rare but potentially life-threatening infection treated with antimicrobials and, in most cases, immediate surgical decompression. Previous studies comparing medical and surgical management of SEA are low powered and limited to a single institution. As such, the present study compares readmission in surgical and non-surgical management using a large national dataset.

METHODS

We identified all hospital admissions for SEA using the Nationwide Readmissions Database (NRD), which is the largest collection of hospital admissions data. Patients were grouped into surgically and non-surgically managed cohorts using ICD-10 coding and compared using information retrieved from the NRD such as demographics, comorbidities, length of stay and cost of admission.

RESULTS

We identified 350 surgically managed and 350 non-surgically managed patients. The 90-day readmission rates for surgical and non-surgical management were 26.0% and 35.1%, respectively (P < .05). Expectedly, surgical management was associated with a significantly higher charge and length of stay at index hospital admission. Surgically managed patients had a significantly lower risk of readmission for osteomyelitis (P < .05). Finally, in patients with a low comorbidity burden, we observed a significantly lower 90-day readmission rate for surgically managed patients (surgical: 23.0%, non-surgical: 33.8%, P < .05).

CONCLUSION

In patients with a low comorbidity burden, we observed a significantly lower readmission rate for surgically managed patients than non-surgically managed patients. The results of this study suggest a lower readmission rate as an advantage to surgical management of SEA and emphasize the importance of SEA as a not-to-miss diagnosis.

Can a Novel Natural Language Processing Model and Artificial Intelligence Automatically Generate Billing Codes From Spine Surgical Operative Notes?

STUDY DESIGN

Retrospective cohort.

OBJECTIVE

Billing and coding-related administrative tasks are a major source of healthcare expenditure in the United States. We aim to show that a second-iteration Natural Language Processing (NLP) machine learning algorithm, XLNet, can automate the generation of CPT codes from operative notes in ACDF, PCDF, and CDA procedures.

METHODS

We collected 922 operative notes from patients who underwent ACDF, PCDF, or CDA from 2015 to 2020 and included CPT codes generated by the billing code department. We trained XLNet, a generalized autoregressive pretraining method, on this dataset and tested its performance by calculating AUROC and AUPRC.

RESULTS

The performance of the model approached human accuracy. Trial 1 (ACDF) achieved an AUROC of .82 (range: .48-.93), an AUPRC of .81 (range: .45-.97), and class-by-class accuracy of 77% (range: 34%-91%); trial 2 (PCDF) achieved an AUROC of .83 (.44-.94), an AUPRC of .70 (.45-.96), and class-by-class accuracy of 71% (42%-93%); trial 3 (ACDF and CDA) achieved an AUROC of .95 (.68-.99), an AUPRC of .91 (.56-.98), and class-by-class accuracy of 87% (63%-99%); trial 4 (ACDF, PCDF, CDA) achieved an AUROC of .95 (.76-.99), an AUPRC of .84 (.49-.99), and class-by-class accuracy of 88% (70%-99%).

CONCLUSIONS

We show that the XLNet model can be successfully applied to orthopedic surgeon's operative notes to generate CPT billing codes. As NLP models as a whole continue to improve, billing can be greatly augmented with artificial intelligence assisted generation of CPT billing codes which will help minimize error and promote standardization in the process.

Robust prediction of nonhome discharge following elective anterior cervical discectomy and fusion using explainable machine learning

PURPOSE

Predict nonhome discharge (NHD) following elective anterior cervical discectomy and fusion (ACDF) using an explainable machine learning model.

METHODS

2227 patients undergoing elective ACDF from 2008 to 2019 were identified from a single institutional database. A machine learning model was trained on preoperative variables, including demographics, comorbidity indices, and levels fused. The validation technique was repeated stratified K-Fold cross validation with the area under the receiver operating curve (AUROC) statistic as the performance metric. Shapley Additive Explanation (SHAP) values were calculated to provide further explainability regarding the model's decision making.

RESULTS

The preoperative model performed with an AUROC of 0.83 ± 0.05. SHAP scores revealed the most pertinent risk factors to be age, medicare insurance, and American Society of Anesthesiology (ASA) score. Interaction analysis demonstrated that female patients over 65 with greater fusion levels were more likely to undergo NHD. Likewise, ASA demonstrated positive interaction effects with female sex, levels fused and BMI.

CONCLUSION

We validated an explainable machine learning model for the prediction of NHD using common preoperative variables. Adding transparency is a key step towards clinical application because it demonstrates that our model's "thinking" aligns with clinical reasoning. Interactive analysis demonstrated that those of age over 65, female sex, higher ASA score, and greater fusion levels were more predisposed to NHD. Age and ASA score were similar in their predictive ability. Machine learning may be used to predict NHD, and can assist surgeons with patient counseling or early discharge planning.

Development of a machine learning algorithm to identify total and reverse shoulder arthroplasty implants from X-ray images

Introduction

Demand for total shoulder arthroplasty (TSA) has risen significantly and is projected to continue growing. From 2012 to 2017, the incidence of reverse total shoulder arthroplasty (rTSA) rose from 7.3 cases per 100,000 to 19.3 per 100,000. Anatomical TSA saw a growth from 9.5 cases per 100,000 to 12.5 per 100,000. Failure to identify implants in a timely manner can increase operative time, cost and risk of complications. Several machine learning models have been developed to perform medical image analysis. However, they have not been widely applied in shoulder surgery. The authors developed a machine learning model to identify shoulder implant manufacturers and type from anterior-posterior X-ray images.

Methods

The model deployed was a convolutional neural network (CNN), which has been widely used in computer vision tasks. 696 radiographs were obtained from a single institution. 70% were used to train the model, while evaluation was done on 30%.

Results

On the evaluation set, the model performed with an overall accuracy of 93.9% with positive predictive value, sensitivity and F-1 scores of 94% across 10 different implant types (4 reverse, 6 anatomical). Average identification time was 0.110 s per implant.

Conclusion

This proof of concept study demonstrates that machine learning can assist with preoperative planning and improve cost-efficiency in shoulder surgery.

The Effect of Subsidence on Segmental and Global Lordosis at Long-term Follow-up After Anterior Cervical Discectomy and Fusion

OBJECTIVE

Subsidence following anterior cervical discectomy and fusion (ACDF) may lead to disruptions of cervical alignment and lordosis. The purpose of this study was to evaluate the effect of subsidence on segmental, regional, and global lordosis.

METHODS

This was a retrospective cohort study performed between 2016-2021 at a single institution. All measurements were performed using lateral cervical radiographs at the immediate postoperative period and at final follow-up greater than 6 months after surgery. Associations between subsidence and segmental lordosis, total fused lordosis, C2-7 lordosis, and cervical sagittal vertical alignment change were determined using Pearson correlation and multivariate logistic regression analyses.

RESULTS

One hundred thirty-one patients and 244 levels were included in the study. There were 41 one-level fusions, 67 two-level fusions, and 23 three-level fusions. The median follow-up time was 366 days (interquartile range, 239-566 days). Segmental subsidence was significantly negatively associated with segmental lordosis change in the Pearson (r = -0.154, p = 0.016) and multivariate analyses (beta = -3.78; 95% confidence interval, -7.15 to -0.42; p = 0.028) but no associations between segmental or total fused subsidence and any other measures of cervical alignment were observed.

CONCLUSION

We found that subsidence is associated with segmental lordosis loss 6 months following ACDF. Surgeons should minimize subsidence to prevent long-term clinical symptoms associated with poor cervical alignment.

Applications of Machine Learning Using Electronic Medical Records in Spine Surgery

Developments in machine learning in recent years have precipitated a surge in research on the applications of artificial intelligence within medicine. Machine learning algorithms are beginning to impact medicine broadly, and the field of spine surgery is no exception. Electronic medical records are a key source of medical data that can be leveraged for the creation of clinically valuable machine learning algorithms. This review examines the current state of machine learning using electronic medical records as it applies to spine surgery. Studies across the electronic medical record data domains of imaging, text, and structured data are reviewed. Discussed applications include clinical prognostication, preoperative planning, diagnostics, and dynamic clinical assistance, among others. The limitations and future challenges for machine learning research using electronic medical records are also discussed.