Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores

Importance

Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR.

Objective

Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates.

Methods

Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation.

Results

For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.

Conclusions

In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Biomechanical Comparison of 3 Thumb Ulnar Collateral Ligament Repair Methods

PURPOSE

Differences in range of motion, pinch strength, biomechanical strength, or joint angulation have previously been investigated for various means of treatment of ulnar collateral ligament (UCL) tears. We sought to address a gap in the literature by comparing thumb metacarpophalangeal (MCP) joint angle measurements and biomechanical strength before complete, acute UCL tear and after repair with suture anchors (SA), suture tape (ST) anchor augmentation, or reconstruction with palmaris longus graft (PL).

METHODS

Thumbs and, if present, the PL tendon were harvested from 15 fresh-frozen cadavers. Each thumb specimen was secured into a servohydraulic biomechanical testing frame to evaluate native radiographic MCP joint angles at 0° flexion when loaded with 0, 5, and 13 N of radial force. Subsequently, a single hand surgeon (S.M.K.) performed complete transection and UCL repair via 1 of 3 methods: SA (n = 5), ST (n = 5), or reconstruction with PL (n = 5). Following repair, MCP joint angles were radiographically evaluated. Specimens that did not fail during joint angle testing were transferred to a separate testing frame for load-to-failure testing. Angle measurements and mean load-to-failure were compared between the groups, and angulation was also compared with each group's native control.

RESULTS

Both ST and SA groups demonstrated comparable stiffness to their native controls, whereas the PL group was significantly more lax. The ST repair was significantly stiffer than the other constructs. ST also required higher forces to reach failure compared to both SA and PL. No difference was found between SA and PL groups.

CONCLUSIONS

Although both ST and SA constructs recapitulate native joint stiffness, repair with ST demonstrated the greatest biomechanical strength in stiffness and load-to-failure.

CLINICAL RELEVANCE

For complete, acute tears of the thumb UCL, ST may be superior for maintaining MCP joint stability and strength over SA and PL.

The Impact of Perceived Upper Extremity Severity and Prior Surgery on Patient-Reported Outcomes

Background: The recently validated Hand Questionnaire (HAND-Q) is a multifaceted patient-reported outcome measure (PROM) for hand/upper extremity (UE) pathology and treatment. Here, we conduct a pilot study utilising data collected as a participating site for the Phase II HAND-Q Pilot Multicenter International Validation Study. We hypothesised that self-reported hand functionality, symptom/disease severity, hand appearance, emotional dissatisfaction and treatment satisfaction would be worse in patients who perceived their disease severity to be more severe but would not differ between patients based on prior surgical history. Methods: Patients were prospectively enrolled for HAND-Q participation from September 2018 to August 2019. Patients were included in this analysis if they responded to the following scales of HAND-Q: Hand Functionality Satisfaction, Symptom Severity, Hand Appearance Satisfaction, Emotional Dissatisfaction and Treatment Satisfaction. Composite scores (CS) were created for each section. Surgical versus non-surgical CS and mild versus moderate/severe CS were compared with t-tests. Bi-variate comparisons of responses were performed between surgical and non-surgical groups, and between mild and moderate/severe groups. Results: HAND-Q individual question analysis revealed significant differences in functionality and symptom severity for patients with prior surgery (p < 0.047). CS analysis confirmed greater overall impairment in surgical patients, but no overall impact on symptom severity. Regarding disease severity HAND-Q individual question analysis, moderate/severe patients reported worse outcomes for specific aesthetic qualities and symptoms for almost all items (p < 0.05). CS analysis revealed significantly worse overall hand appearance satisfaction, hand functionality, emotional satisfaction and symptom severity for patients with moderate/severe hand conditions. Conclusions: HAND-Q revealed worsened outcomes for UE patients with self-reported moderate/severe conditions or previous surgical history. Understanding how previous surgery and disease severity may impact clinical outcomes is important for crafting appropriate treatment. Level of Evidence: Level II (Prognostic Study).

Increases in Firework-Related Upper Extremity Injuries Correspond to Increasing Firework Sales: An Analysis of 41,195 Injuries Across 10 Years

INTRODUCTION

Between 2008 and 2017, the American Pyrotechnics Association reported a 41% increase in revenue from firework sales, with 2017 showing $885 million US dollars in consumer sales. We sought to evaluate the epidemiology of firework-related upper extremity injuries during this 10-year period, hypothesizing that hand/upper extremity injuries from fireworks were increasing in the United States.

METHODS

Observational epidemiologic assessment of a weighted cohort of patients via the National Electronic Injury Surveillance System from 2008 to 2017. The National Electronic Injury Surveillance System provides a nationwide probability sample of injuries related to consumer products based on emergency department visits collected from a cohort of about 100 US hospitals.

RESULTS

A total of 1,079 patients representing an estimated 41,195 firework-related upper extremity injuries presented to US emergency departments from 2008 to 2017. The number of injuries increased significantly from 2,576 in 2008 to 5,101 in 2017 (R2 = 0.85, R = 0.92, P < 0.001). A Spearman rank-order correlation determined that there was a strong, positive correlation between the increase in firework sales and the increase in injuries (rs = 0.939, P < 0.01). The overwhelming majority of firework-related injuries were seen in males (77%) aged 11 to 29 years (48%). The hand and fingers accounted for 85.8% of injuries, with the thumb being the most commonly injured body part (51.3%). Burns were the most common injury across all body sites except the wrist, where fractures were most common.

CONCLUSION

Ten-year firework-related upper extremity injuries increased, corresponding to increased consumer sales across the same period. This study provides previously absent population-level data to provide a framework for discussion among policy makers and physicians alike in an attempt to mitigate the use of fireworks and their associated upper extremity injuries.

LEVEL OF EVIDENCE

Level III.

WALANT Hand and Upper Extremity Procedures Performed With Minor Field Sterility Are Associated With Low Infection Rates

Background:

The use of minor field sterility in hand/upper extremity cases has been shown to improve workflow efficiency while maintaining patient safety. As this finding has been limited to specific procedures, we investigated the safety of performing a wide array of hand/upper extremity procedures outside the main operating room using minimal field sterility with Wide-Awake Local Anaesthesia No Tourniquet (WALANT) anaesthesia by evaluating superficial and deep infection rates across a diverse series of cases.

Methods:

This study was a case series conducted between October 2017 and June 2020. Of all, 217 patients underwent hand/upper extremity procedures performed in a minor procedure room via WALANT technique with field sterility. Primary outcome measures include superficial and deep surgical site infections within 14 days post-surgery.

Results:

Of all, 217 patients were included in this study; 265 consecutive hand/upper extremity operations were performed by a single surgeon, with notable case diversity. The majority of patients (n = 215, 99.1%) did not report or present with signs of infection before or after their operation. We report 0% 14-day and 0.37% 30-day surgical site infection rates for such hand/upper extremity procedures performed in a minor procedure room with field sterility.

Conclusion:

Hand/upper extremity procedures performed via WALANT technique with field sterility in a minor procedure room are associated with low surgical site infection rates. These rates are comparable to surgical site infection rates for similar surgeries performed in main operating rooms with standard sterilization procedures. Thus, the implementation of this technique may allow for improved workflow efficiency and reduced waste, all while maintaining patient safety.

Outcomes of an aseptic technique for Kirschner wire percutaneous pinning in the hand and wrist

INTRODUCTION

Kirschner wires (K-wires) have been shown to effectively stabilize fractures of the hand and wrist, but are associated with high infection rates, which may limit its use. Previous literature has attributed the risk of infection to many different aspects of a fixation technique. However, we introduce an approach to percutaneous K-wire pinning to mitigate the risk of infection.

METHODS

Patients undergoing K-wire fixation procedures of the hand and wrist were retrospectively queried. All cases were performed under the same principles of our technique. None of the K-wires were buried, nor bent and were covered with betadine-soaked alcohol pads as pin site dressings. Pins were removed in an outpatient procedure room up to 12 weeks postoperatively and were then assessed for signs of superficial or deep infection.

RESULTS

90 patients were included in this study across a variety of K-wire fixation operations in the hand and wrist. No patients presented with any signs of infection or other complications necessitating postoperative antibiotics, early pin removal, or reoperation.

DISCUSSION/CONCLUSION

The specific guidelines of our technique resulted in no cases of pin site infection, despite a largely underserved patient population. Our low incidence of infection was maintained without the use of prophylactic antibiotics and in patients with long periods of fixation. While the high infection rates in previous literature have often been associated with wires left exposed, the principles of our technique allow for K-wire fixations to be performed percutaneously without burying the wires. This may allow for improved cost and time efficiency, without compromising patient safety.

WALANT Hand Surgery: Do the AORN Guidelines Apply?

This study evaluates current guidelines for patients receiving local anesthesia, set forth by the Association of Perioperative Nurses (AORN), within the context of hand surgery. The study reviewed 217 patients and 265 operations performed under wide-awake local anesthesia no tourniquet (WALANT) technique in an outpatient procedure room with minor field sterility. Only the surgeon, one resident, and one circulating nurse were present. All surgical complications were documented, including any infection at postoperative follow-ups. One female patient developed a deep surgical site infection (SSI) following repair of her flexor digitorum superficialis and flexor digitorum profundus, which resolved after irrigation/debridement. We report 0% intraprocedural complication, 0% superficial SSI, and 0.37% deep SSI (n = 1) incidence across this cohort. Most institutions require two nurses present for local anesthesia, but our low complication and infection incidence suggest a single circulating nurse present during WALANT hand surgeries may improve nurse staffing, drive greater turnover efficiency, and reduce costs. (Journal of Surgical Orthopaedic Advances 30(3):156-160, 2021).

The 100 Most Impactful Papers in Hand and Upper Extremity Surgery over the Last 25 Years: A Bibliometric Analysis of the Orthopaedic Literature

Introduction Despite growth in hand/upper extremity investigation, impactful studies have not been thoroughly identified. Previous studies have been limited in scope. This study sought to identify and characterize the most impactful orthopaedic papers in hand/upper extremity over the past 25 years.

Materials and Methods The top 1,000 hand/upper extremity orthopaedic studies published from 1992 to 2017 were identified with Web of Science. After screening for relevance in order of decreasing citation number, the top 100 articles were identified for bibliometric analysis.

Results The mean number of authors and citations were 4.51 (range, 1–21) and 169.4 (range, 105–863). Common study types included, case series (n = 52), randomized controlled trial (n = 17), and prospective cohort (n = 16), which predominantly covered topics related to shoulder (n = 34), wrist/forearm (n = 21), and hand (n = 17). Among wrist/forearm and hand studies, distal radius fractures (n = 12) and nerve-related topics (n = 10) were most frequently analyzed. Most studies were of level IV (n = 51) and level II (n = 16) evidence. Recent studies had greater impact (mean citations/year: 2011, 82.7/year vs. 1992, 16.1/year).

Conclusion Most of the 100 top orthopaedic articles in hand/upper extremity were of level IV or II evidence, retrospective, and nonrandomized. Despite an observed recent increase in level I studies, a lack of prospective, randomized trials is apparent.