A fast, resource efficient, and reliable rule-based system for COVID-19 symptom identification

A comparative analysis of large language models versus traditional information extraction methods for real-world evidence of patient symptomatology in acute and post-acute sequelae of SARS-CoV-2

BACKGROUND

Patient symptoms, crucial for disease progression and diagnosis, are often captured in unstructured clinical notes. Large language models (LLMs) offer potential advantages in extracting patient symptoms compared to traditional rule-based information extraction (IE) systems.

METHODS

This study compared fine-tuned LLMs (LLaMA2-13B and LLaMA3-8B) against BioMedICUS, a rule-based IE system, for extracting symptoms related to acute and post-acute sequelae of SARS-CoV-2 from clinical notes. The study utilized three corpora: UMN-COVID, UMN-PASC, and N3C-COVID. Prevalence, keyword and fairness analyses were conducted to assess symptom distribution and model equity across demographics.

RESULTS

BioMedICUS outperformed fine-tuned LLMs in most cases. On the UMN PASC dataset, BioMedICUS achieved a macro-averaged F1-score of 0.70 for positive mention detection, compared to 0.66 for LLaMA2-13B and 0.62 for LLaMA3-8B. For the N3C COVID dataset, BioMedICUS scored 0.75, while LLaMA2-13B and LLaMA3-8B scored 0.53 and 0.68, respectively for positive mention detection. However, LLMs performed better in specific instances, such as detecting positive mentions of change in sleep in the UMN PASC dataset, where LLaMA2-13B (0.79) and LLaMA3-8B (0.65) outperformed BioMedICUS (0.60). For fairness analysis, BioMedICUS generally showed stronger performance across patient demographics. Keyword analysis using ANOVA on symptom distributions across all three corpora showed that both corpus (df = 2, p < 0.001) and symptom (df = 79, p < 0.001) have a statistically significant effect on log-transformed term frequency-inverse document frequency (TF-IDF) values such that corpus accounts for 52% of the variance in log_tfidf values and symptom accounts for 35%.

CONCLUSION

While BioMedICUS generally outperformed the LLMs, the latter showed promising results in specific areas, particularly LLaMA3-8B, in identifying negative symptom mentions. However, both LLaMA models faced challenges in demographic fairness and generalizability. These findings underscore the need for diverse, high-quality training datasets and robust annotation processes to enhance LLMs' performance and reliability in clinical applications.

Clinical decision support systems (CDSS) in assistance to COVID-19 diagnosis: A scoping review on types and evaluation methods

Background and Aims

Due to the COVID-19 pandemic, a precise and reliable diagnosis of this disease is critical. The use of clinical decision support systems (CDSS) can help facilitate the diagnosis of COVID-19. This scoping review aimed to investigate the role of CDSS in diagnosing COVID-19.

Methods

We searched four databases (Web of Science, PubMed, Scopus, and Embase) using three groups of keywords related to CDSS, COVID-19, and diagnosis. To collect data from studies, we utilized a data extraction form that consisted of eight fields. Three researchers selected relevant articles and extracted data using a data collection form. To resolve any disagreements, we consulted with a fourth researcher.

Results

A search of the databases retrieved 2199 articles, of which 68 were included in this review after removing duplicates and irrelevant articles. The studies used nonknowledge-based CDSS (n = 52) and knowledge-based CDSS (n = 16). Convolutional Neural Networks (CNN) (n = 33) and Support Vector Machine (SVM) (n = 8) were employed to design the CDSS in most of the studies. Accuracy (n = 43) and sensitivity (n = 35) were the most common metrics for evaluating CDSS.

Conclusion

CDSS for COVID-19 diagnosis have been developed mainly through machine learning (ML) methods. The greater use of these techniques can be due to their availability of public data sets about chest imaging. Although these studies indicate high accuracy for CDSS based on ML, their novelty and data set biases raise questions about replacing these systems as clinician assistants in decision-making. Further studies are needed to improve and compare the robustness and reliability of nonknowledge-based and knowledge-based CDSS in COVID-19 diagnosis.

Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study

Purpose

To conduct a prospective observational study across 12 U.S. hospitals to evaluate real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 on chest radiographs.

Materials and Methods

A total of 95 363 chest radiographs were included in model training, external validation, and real-time validation. The model was deployed as a clinical decision support system, and performance was prospectively evaluated. There were 5335 total real-time predictions and a COVID-19 prevalence of 4.8% (258 of 5335). Model performance was assessed with use of receiver operating characteristic analysis, precision-recall curves, and F1 score. Logistic regression was used to evaluate the association of race and sex with AI model diagnostic accuracy. To compare model accuracy with the performance of board-certified radiologists, a third dataset of 1638 images was read independently by two radiologists.

Results

Participants positive for COVID-19 had higher COVID-19 diagnostic scores than participants negative for COVID-19 (median, 0.1 [IQR, 0.0-0.8] vs 0.0 [IQR, 0.0-0.1], respectively; P < .001). Real-time model performance was unchanged over 19 weeks of implementation (area under the receiver operating characteristic curve, 0.70; 95% CI: 0.66, 0.73). Model sensitivity was higher in men than women (P = .01), whereas model specificity was higher in women (P = .001). Sensitivity was higher for Asian (P = .002) and Black (P = .046) participants compared with White participants. The COVID-19 AI diagnostic system had worse accuracy (63.5% correct) compared with radiologist predictions (radiologist 1 = 67.8% correct, radiologist 2 = 68.6% correct; McNemar P < .001 for both).

Conclusion

AI-based tools have not yet reached full diagnostic potential for COVID-19 and underperform compared with radiologist prediction.Keywords: Diagnosis, Classification, Application Domain, Infection, Lung Supplemental material is available for this article.. © RSNA, 2022.

Predictors of Post-Acute Sequelae of COVID-19 Development and Rehabilitation: A Retrospective Study

Objective

To examine the frequency of postacute sequelae of SARS-CoV-2 (PASC) and the factors associated with rehabilitation utilization in a large adult population with PASC.

Design

Retrospective study.

Setting

Midwest hospital health system.

Participants

19,792 patients with COVID-19 from March 10, 2020, to January 17, 2021.

Intervention

Not applicable.

Main Outcome Measures

Descriptive analyses were conducted across the entire cohort along with an adult subgroup analysis. A logistic regression was performed to assess factors associated with PASC development and rehabilitation utilization.

Results

In an analysis of 19,792 patients, the frequency of PASC was 42.8% in the adult population. Patients with PASC compared with those without had a higher utilization of rehabilitation services (8.6% vs 3.8%, P<.001). Risk factors for rehabilitation utilization in patients with PASC included younger age (odds ratio [OR], 0.99; 95% confidence interval [CI], 0.98-1.00; P=.01). In addition to several comorbidities and demographics factors, risk factors for rehabilitation utilization solely in the inpatient population included male sex (OR, 1.24; 95% CI, 1.02-1.50; P=.03) with patients on angiotensin-converting-enzyme inhibitors or angiotensin-receptor blockers 3 months prior to COVID-19 infections having a decreased risk of needing rehabilitation (OR, 0.80; 95% CI, 0.64-0.99; P=.04).

Conclusions

Patients with PASC had higher rehabilitation utilization. We identified several clinical and demographic factors associated with the development of PASC and rehabilitation utilization.

PASCLex: A comprehensive post-acute sequelae of COVID-19 (PASC) symptom lexicon derived from electronic health record clinical notes

OBJECTIVE

To develop a comprehensive post-acute sequelae of COVID-19 (PASC) symptom lexicon (PASCLex) from clinical notes to support PASC symptom identification and research.

METHODS

We identified 26,117 COVID-19 positive patients from the Mass General Brigham's electronic health records (EHR) and extracted 328,879 clinical notes from their post-acute infection period (day 51-110 from first positive COVID-19 test). PASCLex incorporated Unified Medical Language System® (UMLS) Metathesaurus concepts and synonyms based on selected semantic types. The MTERMS natural language processing (NLP) tool was used to automatically extract symptoms from a development dataset. The lexicon was iteratively revised with manual chart review, keyword search, concept consolidation, and evaluation of NLP output. We assessed the comprehensiveness of PASCLex and the NLP performance using a validation dataset and reported the symptom prevalence across the entire corpus.

RESULTS

PASCLex included 355 symptoms consolidated from 1520 UMLS concepts of 16,466 synonyms. NLP achieved an averaged precision of 0.94 and an estimated recall of 0.84. Symptoms with the highest frequency included pain (43.1%), anxiety (25.8%), depression (24.0%), fatigue (23.4%), joint pain (21.0%), shortness of breath (20.8%), headache (20.0%), nausea and/or vomiting (19.9%), myalgia (19.0%), and gastroesophageal reflux (18.6%).

DISCUSSION AND CONCLUSION

PASC symptoms are diverse. A comprehensive lexicon of PASC symptoms can be derived using an ontology-driven, EHR-guided and NLP-assisted approach. By using unstructured data, this approach may improve identification and analysis of patient symptoms in the EHR, and inform prospective study design, preventative care strategies, and therapeutic interventions for patient care.