Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores

Importance

Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR.

Objective

Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates.

Methods

Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation.

Results

For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date.

Conclusions

In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study

BACKGROUND

The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating.

OBJECTIVE

To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation.

METHODS

This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters.

RESULTS

In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions.

CONCLUSIONS

A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes.

CLINICALTRIAL

Study of Patients’ Characteristics and Mutual Impact Between Covid-19 and Hyperglycemia at a Community Hospital in Central Brooklyn

Background: Studies have shown that poorly-controlled hyperglycemia worsens the outcomes in patients with COVID-19 (C-19) and C-19 may damage pancreatic islets via ACE2 receptors causing acute hyperglycemia. The major population we serve at Kingsbrook Jewish Medical Center (KJMC) are underprivileged with many of them having multiple comorbidities. Methods: This is a retrospective study wherein patients, admitted from February 2020 to April 2020 with hyperglycemia, were selected and divided into 2 groups based on presence or absence of C-19. Data include demographics, comorbidities, blood glucose level, serum osmolality, serum bicarbonate, anion gap, acute kidney injury (AKI), serum creatinine, ICU admission, length of stay (LOS) and mortality. Data were analyzed using descriptive study and T-test. Results: 100 patients were included in the C-19 group (CG) and 88 patients were included in the Non C-19 group (NCG). Major comorbidities were similar in both groups including HTN, DM, CKD followed by ESRD. Mean age of patients (years) was 65.68 in CG and 61.17 in NCG. 61% were male in CG and 53.41% were male in NCG. 16% and 9% developed DKA and HHS in CG, and 13.64% and 6.82% developed DKA and HHS in NCG respectively. 15% in CG had combined DKA & HHS and 3.41% had same in NCG. Mean blood glucose level (mg/dl) was 541.6 in CG and 460.0 in NCG (p=0.03). Mean serum osmolality (mOsm/kg) was 335.7 (SD±41.01) in CG and 317.1 (SD±30.54) in NCG (p=0.01). Mean serum bicarbonate (mEq/L) was 17.73 (SD±6.31) in CG and 21.46 (SD±5.94) in NCG (p<0.0001). Mean anion gap was 17.93 (SD±7.6) in CG and 13.10 (SD±7.2) in NCG (p<0.0001). 56% in CG and 37% in NCG developed AKI respectively (p=0.01). Mean serum creatinine (mg/dl) was 4.22 in CG and 1.65 in NCG (p=0.004). 55% of CG were admitted to ICU and 34% of NCG were admitted to ICU (p=0.003). Median LOS (days) in discharged patients was 8 in CG and 5 in NCG (p=0.02). Mortality was 40% in CG and 3.41% in NCG (p<0.0001). 12 patients in CG and 2 patients in NCG developed new-onset diabetes. In the subset of DKA, interestingly, mean age (years) was 61.63 (SD±17.73) in CG and 39.67 in NCG (SD±13.39) (p=0.001). Conclusion: In our study, patients in the CG carry worse laboratory parameters, unfavorable clinical outcomes and strikingly higher mortality. We discovered increased incidence of new-onset diabetes and elderly DKA in CG. In an inner city population like ours, the burden of DM with significant social and health care disparities is quite severe. Diabetic patients with concurrent C-19 infection can have particularly negative outcomes and C-19 possibly damages the pancreatic islets resulting in acute hyperglycemic crisis. Further research on larger population is required. References: (1)https://dx.doi.org/10.1016%2Fj.diabres.2020.108142(2) https://doi.org/10.2337/dc20-0723(3)https://www.nejm.org/doi/full/10.1056/NEJMc2018688

Relationship between neutrophil lymphocyte ratio (NLR) and MDSC concentration in localized and metastatic castration resistant prostate cancer (mCRPC) patients

338

Background: Human Myeloid Derived Suppressor Cells (MDSCs) have gained recognition as a significant population of cells that arise in cancer patients to mediate immune tolerance. MDSCs have been validated as a prognostic factor in several tumor types. The Neutrophil-Lymphocyte Ratio (NLR) is an easily obtained marker of inflammation that has also been validated as a prognostic marker in tumors including prostate cancer. We measured and compared MDSC concentrations and NLR in a cohort of patients with prostate cancer and also examined thier relationship with PSA Progression, a validated marker of overall survival. Methods: Blood was collected from patients with localized prostate cancer (n = 14) and mCRPC (n = 44) at 0 and 6 months. The percent CD33+ HLA-DRlow HIF1a+, CD33+ HLADRlow C/EBPb+, CD11b+ HLA-DRlow HIF1a+, and CD11b+ HLA-DRlow C/EBPb+cells as a fraction of PBMC were determined and reported. NLRs were obtained within 4 weeks of MDSC measurements. Patients were followed for a period of 2 years; PSA-Progression (PSA-P) was defined as a PSA doubling time of < 3 months in association with an increase of at least > 5ng/mL from PSA nadir Results: Higher Gleason scores were statistically associated with a NLR of greater than 2 (r = 0.07, p = 0.02, 95% CI [0.01-0.14]). Increasing MDSC% did not correlate with increasing Gleason scores (r = 0.13, p = 0.31) or predict PSA-P at 6 months (p = 0.16). MDSC% in patients with mCRPC (4.25%, 95% CI[3.34-5.17]) were significantly higher than localized prostate cancer (1.7%, 95% CI[1.02-2.37]) . There was a highly statistically significant relationship between the percentage of MDSC in PBMC with the neutrophil lymphocyte ratio (r = 0.68, p = 0.001, 95% CI [0.31-1.05]). Conclusions: MDSC concentrations in metastatic prostate cancer patients are significantly higher than those with localized disease and are associated with an increased neutrophil lymphocyte ratio. Although a trend was detected towards MDSC% predicting PSA-P, in this limited sample it was not statistically significant.

MDSC clinical assay for cancer detection and monitoring in renal cell carcinoma

TPS713

Background: Human Myeloid Derived Suppressor Cells (MDSCs) have gained recognition as a significant population of cells that arise in cancer patients to mediate immune tolerance. Some studies have shown MDSC concentrations to be associated with overall survival and response to therapy in renal cell carcinoma. MDSCs comprise a heterogenous population that are yet to be fully characterized. We have identified a set of biomarkers uniquely expressed by the suppressive human MDSC phenotype and have developed an assay targeted to these markers. MDSCs are potentially a relatively cheap and easily performed screening and monitoring tool in cancers. This study will examine how a proprietary clinical assay works in detecting and monitoring MDSCs in blood and urine samples from patients with localized or metastatic renal cell cancer. Methods: The pilot research trial will enroll 63 subjects in a 1:1:1 ratio in three groups, normal controls over age 40 without evidence of any malignancy, patients with metastatic disease and localized renal cell carcinoma prior to nephrectomy. This sample size provides 80% power to detect a 1SD difference in MDSC concentrations in these groups on an F test. Blood and urine samples will be drawn at baseline and at 4 months for comparison. The percent CD33+ HLA-DRlow HIF1a+, CD33+ HLADRlow C/EBPb+, CD11b+ HLA-DRlow HIF1a+, and CD11b+ HLA-DRlow C/EBPb+cells as a fraction of PBMC will be determined. Primary Outcome Measures will be 1. Change in MDSC levels in patients with known localized renal cell carcinoma who undergo surgical treatment. 2. Change in MDSC level in patients with known metastatic renal cell carcinoma who initiate systemic treatment and 3. The direction and magnitude of the changes compared with radiographically assessed tumor burden. Secondary outcome measures are to assess MDSC level measurements in urine cytology analysis at baseline and after treatment to determine whether the two tests correlate in any of the 3 groups of patients defined in this study. Conclusion: This pilot study will examine if a MDSC clinical assay works in detecting and monitoring MDSCs. Recruitment is open to patients at the USC Cancer center and LAC+USC medical center. Clinical trial information: NCT02664883.