Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review

Adapting Generative Large Language Models for Information Extraction from Unstructured Electronic Health Records in Residential Aged Care: A Comparative Analysis of Training Approaches

Information extraction (IE) of unstructured electronic health records is challenging due to the semantic complexity of textual data. Generative large language models (LLMs) offer promising solutions to address this challenge. However, identifying the best training methods to adapt LLMs for IE in residential aged care settings remains underexplored. This research addresses this challenge by evaluating the effects of zero-shot and few-shot learning, both with and without parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) using Llama 3.1-8B. The study performed named entity recognition (NER) to nursing notes from Australian aged care facilities (RACFs), focusing on agitation in dementia and malnutrition risk factors. Performance evaluation includes accuracy, macro-averaged precision, recall, and F1 score. We used non-parametric statistical methods to compare if the differences were statistically significant. Results show that zero-shot and few-shot learning, whether combined with PEFT or RAG, achieve comparable performance across the clinical domains when the same prompting template is used. Few-shot learning significantly outperforms zero-shot learning when neither PEFT nor RAG is applied. Notably, PEFT significantly improves model performance in both zero-shot and few-shot learning; however, RAG significantly improves performance only in few-shot learning. After PEFT, the performance of zero-shot learning reaches a comparable level with few-shot learning. However, few-shot learning with RAG significantly outperforms zero-shot learning with RAG. We also found a similar level of performance between few-shot learning with RAG and zero-shot learning with PEFT. These findings provide valuable insights for researchers, practitioners, and stakeholders to optimize the use of generative LLMs in clinical IE.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41666-025-00190-z.

Utilizing large language models for detecting hospital-acquired conditions: an empirical study on pulmonary embolism

OBJECTIVES

Adverse event detection from Electronic Medical Records (EMRs) is challenging due to the low incidence of the event, variability in clinical documentation, and the complexity of data formats. Pulmonary embolism as an adverse event (PEAE) is particularly difficult to identify using existing approaches. This study aims to develop and evaluate a Large Language Model (LLM)-based framework for detecting PEAE from unstructured narrative data in EMRs.

MATERIALS AND METHODS

We conducted a chart review of adult patients (aged 18-100) admitted to tertiary-care hospitals in Calgary, Alberta, Canada, between 2017-2022. We developed an LLM-based detection framework consisting of three modules: evidence extraction (implementing both keyword-based and semantic similarity-based filtering methods), discharge information extraction (focusing on six key clinical sections), and PEAE detection. Four open-source LLMs (Llama3, Mistral-7B, Gemma, and Phi-3) were evaluated using positive predictive value, sensitivity, specificity, and F1-score. Model performance for population-level surveillance was assessed at yearly, quarterly, and monthly granularities.

RESULTS

The chart review included 10 066 patients, with 40 cases of PEAE identified (0.4% prevalence). All four LLMs demonstrated high sensitivity (87.5-100%) and specificity (94.9-98.9%) across different experimental conditions. Gemma achieved the highest F1-score (28.11%) using keyword-based retrieval with discharge summary inclusion, along with 98.4% specificity, 87.5% sensitivity, and 99.95% negative predictive value. Keyword-based filtering reduced the median chunks per patient from 789 to 310, while semantic filtering further reduced this to 9 chunks. Including discharge summaries improved performance metrics across most models. For population-level surveillance, all models showed strong correlation with actual PEAE trends at yearly granularity (r=0.92-0.99), with Llama3 achieving the highest correlation (0.988).

DISCUSSION

The results of our method for PEAE detection using EMR notes demonstrate high sensitivity and specificity across all four tested LLMs, indicating strong performance in distinguishing PEAE from non-PEAE cases. However, the low incidence rate of PEAE contributed to a lower PPV. The keyword-based chunking approach consistently outperformed semantic similarity-based methods, achieving higher F1 scores and PPV, underscoring the importance of domain knowledge in text segmentation. Including discharge summaries further enhanced performance metrics. Our population-based analysis revealed better performance for yearly trends compared to monthly granularity, suggesting the framework's utility for long-term surveillance despite dataset imbalance. Error analysis identified contextual misinterpretation, terminology confusion, and preprocessing limitations as key challenges for future improvement.

CONCLUSIONS

Our proposed method demonstrates that LLMs can effectively detect PEAE from narrative EMRs with high sensitivity and specificity. While these models serve as effective screening tools to exclude non-PEAE cases, their lower PPV indicates they cannot be relied upon solely for definitive PEAE identification. Further chart review remains necessary for confirmation. Future work should focus on improving contextual understanding, medical terminology interpretation, and exploring advanced prompting techniques to enhance precision in adverse event detection from EMRs.

Exploring immune checkpoint inhibitors: Focus on PD-1/PD-L1 axis and beyond

Immunotherapy emerges as a promising approach, marked by recent substantial progress in elucidating how the host immune response impacts tumor development and its sensitivity to various treatments. Immune checkpoint inhibitors have revolutionized cancer therapy by unleashing the power of the immune system to recognize and eradicate tumor cells. Among these, inhibitors targeting the programmed cell death protein 1 (PD-1) and its ligand (PD-L1) have garnered significant attention due to their remarkable clinical efficacy across various malignancies. This review delves into the mechanisms of action, clinical applications, and emerging therapeutic strategies surrounding PD-1/PD-L1 blockade. We explore the intricate interactions between PD-1/PD-L1 and other immune checkpoints, shedding light on combinatorial approaches to enhance treatment outcomes and overcome resistance mechanisms. Furthermore, we discuss the expanding landscape of immune checkpoint inhibitors beyond PD-1/PD-L1, including novel targets such as CTLA-4, LAG-3, TIM-3, and TIGIT. Through a comprehensive analysis of preclinical and clinical studies, we highlight the promise and challenges of immune checkpoint blockade in cancer immunotherapy, paving the way for future advancements in the field.

Leveraging Electronic Health Records in International Humanitarian Clinics for Population Health Research: Cross-Sectional Study

Background

As more humanitarian relief organizations are beginning to use electronic medical records in their operations, data from clinical encounters can be leveraged for public health planning. Currently, medical data from humanitarian medical workers are infrequently available in a format that can be analyzed, interpreted, and used for public health.

objectives

This study aims to develop and test a methodology by which diagnosis and procedure codes can be derived from free-text medical encounters by medical relief practitioners for the purposes of data analysis.

Methods

We conducted a cross-sectional study of clinical encounters from humanitarian clinics for displaced persons in Mexico between August 3, 2021, and December 5, 2022. We developed and tested a method by which free-text encounters were reviewed by medical billing coders and assigned codes from the International Classification of Diseases, Tenth Revision (ICD-10) and the Current Procedural Terminology (CPT). Each encounter was independently reviewed in duplicate and assigned ICD-10 and CPT codes in a blinded manner. Encounters with discordant codes were reviewed and arbitrated by a more experienced medical coder, whose decision was used to determine the final ICD-10 and CPT codes. We used chi-square tests of independence to compare the ICD-10 codes for concordance across single-diagnosis and multidiagnosis encounters and across patient characteristics, such as age, sex, and country of origin.

Results

We analyzed 8460 encounters representing 5623 unique patients and 2774 unique diagnosis codes. These free-text encounters had a mean of 20.5 words per encounter in the clinical documentation. There were 58.78% (4973/8460) encounters where both coders assigned 1 diagnosis code, 18.56% (1570/8460) encounters where both coders assigned multiple diagnosis codes, and 22.66% (1917/8460) encounters with a mixed number of codes assigned. Of the 4973 encounters with a single code, only 11.82% (n=588) had a unique diagnosis assigned by the arbitrator that was not assigned by either of the initial 2 coders. Of the 1570 encounters with multiple diagnosis codes, only 3.38% (n=53) had unique diagnosis codes assigned by the arbitrator that were not initially assigned by either coder. The frequency of complete concordance across diagnosis codes was similar across sex categories and ranged from 30.43% to 46.05% across age groups and countries of origin.

Conclusions

Free-text electronic medical records from humanitarian relief clinics can be used to develop a database of diagnosis and procedure codes. The method developed in this study, which used multiple independent reviews of clinical encounters, appears to reliably assign diagnosis codes across a diverse patient population in a resource-limited setting.

Low-cost algorithms for clinical notes phenotype classification to enhance epidemiological surveillance: A case study

OBJECTIVE

Our study aims to enhance epidemic intelligence through event-based surveillance in an emerging pandemic context. We classified electronic health records (EHRs) from La Rioja, Argentina, focusing on predicting COVID-19-related categories in a scenario with limited disease knowledge, evolving symptoms, non-standardized coding practices, and restricted training data due to privacy issues.

METHODS

Using natural language processing techniques, we developed rapid, cost-effective methods suitable for implementation with limited resources. We annotated a corpus for training and testing classification models, ranging from simple logistic regression to more complex fine-tuned transformers.

RESULTS

The transformer-based, Spanish-adapted models BETO Clínico and RoBERTa Clínico, further pre-trained with an unannotated portion of our corpus, were the best-performing models (F1= 88.13% and 87.01%). A simple logistic regression (LR) model ranked third (F1=85.09%), outperforming more complex models like XGBoost and BiLSTM. Data classified as COVID-confirmed using LR and BETO Clínico exhibit stronger time-series Pearson correlation with official COVID-19 case counts from the National Health Surveillance System (SNVS 2.0) in La Rioja province compared to the correlations observed between the International Code of Diseases (ICD-10) codes and the SNVS 2.0 data (0.840, 0.873, and 0.663, p-values ≤3×10-7). Both models have a good Pearson correlation with ICD-10 codes assigned to the clinical notes for confirmed (0.940 and 0.902) and for suspected cases (0.960 and 0.954), p-values ≤1.7×10-18.

CONCLUSION

This study shows that simple, resource-efficient methods can achieve results comparable to complex approaches. BETO Clínico and LR strongly correlate with official data, revealing uncoded confirmed cases at the pandemic's onset. Our results suggest that annotating a smaller set of EHRs and training a simple model may be more cost-effective than manual coding. This points to potentially efficient strategies in public health emergencies, particularly in resource-limited settings, and provides valuable insights for future epidemic response efforts.

Neuradicon: Operational representation learning of neuroimaging reports

BACKGROUND AND OBJECTIVE

Radiological reports typically summarize the content and interpretation of imaging studies in unstructured form that precludes quantitative analysis. This limits the monitoring of radiological services to throughput undifferentiated by content, impeding specific, targeted operational optimization. Here we present Neuradicon, a natural language processing (NLP) framework for quantitative analysis of neuroradiological reports.

METHODS

Our framework is a hybrid of rule-based and machine-learning models to represent neurological reports in succinct, quantitative form optimally suited to operational guidance. These include probabilistic models for text classification and tagging tasks, alongside auto-encoders for learning latent representations and statistical mapping of the latent space.

RESULTS

We demonstrate the application of Neuradicon to operational phenotyping of a corpus of 336,569 reports, and report excellent generalizability across time and two independent healthcare institutions. In particular, we report pathology classification metrics with f1-scores of 0.96 on prospective data, and semantic means of interrogating the phenotypes surfaced via latent space representations.

CONCLUSION

Neuradicon allows the segmentation, analysis, classification, representation and interrogation of neuroradiological reports structure and content. It offers a blueprint for the extraction of rich, quantitative, actionable signals from unstructured text data in an operational context.

ChatGPT in Oncology Diagnosis and Treatment: Applications, Legal and Ethical Challenges

PURPOSE OF REVIEW

This study aims to systematically review the trajectory of artificial intelligence (AI) development in the medical field, with a particular emphasis on ChatGPT, a cutting-edge tool that is transforming oncology's diagnosis and treatment practices.

RECENT FINDINGS

Recent advancements have demonstrated that ChatGPT can be effectively utilized in various areas, including collecting medical histories, conducting radiological & pathological diagnoses, generating electronic medical record (EMR), providing nutritional support, participating in Multidisciplinary Team (MDT) and formulating personalized, multidisciplinary treatment plans. However, some significant challenges related to data privacy and legal issues that need to be addressed for the safe and effective integration of ChatGPT into clinical practice. ChatGPT, an emerging AI technology, opens up new avenues and viewpoints for oncology diagnosis and treatment. If current technological and legal challenges can be overcome, ChatGPT is expected to play a more significant role in oncology diagnosis and treatment in the future, providing better treatment options and improving the quality of medical services.

Controversy in Hypertension: Pro-Side of the Argument Using Artificial Intelligence for Hypertension Diagnosis and Management

Hypertension presents the largest modifiable public health challenge due to its high prevalence, its intimate relationship to cardiovascular diseases, and its complex pathogenesis and pathophysiology. Low awareness of blood pressure elevation and suboptimal hypertension diagnosis serve as the major hurdles in effective hypertension management. Advances in artificial intelligence in hypertension have permitted the integrative analysis of large data sets including omics, clinical (with novel sensor and wearable technologies), health-related, social, behavioral, and environmental sources, and hold transformative potential in achieving large-scale, data-driven approaches toward personalized diagnosis, treatment, and long-term management. However, although the emerging artificial intelligence science may advance the concept of precision hypertension in discovery, drug targeting and development, patient care, and management, its clinical adoption at scale today is lacking. Recognizing that clinical implementation of artificial intelligence-based solutions need evidence generation, this opinion statement examines a clinician-centric perspective of the state-of-art in using artificial intelligence in the management of hypertension and puts forward recommendations toward equitable precision hypertension care.

Review of applications of deep learning in veterinary diagnostics and animal health

Deep learning (DL), a subfield of artificial intelligence (AI), involves the development of algorithms and models that simulate the problem-solving capabilities of the human mind. Sophisticated AI technology has garnered significant attention in recent years in the domain of veterinary medicine. This review provides a comprehensive overview of the research dedicated to leveraging DL for diagnostic purposes within veterinary medicine. Our systematic review approach followed PRISMA guidelines, focusing on the intersection of DL and veterinary medicine, and identified 422 relevant research articles. After exporting titles and abstracts for screening, we narrowed our selection to 39 primary research articles directly applying DL to animal disease detection or management, excluding non-primary research, reviews, and unrelated AI studies. Key findings from the current body of research highlight an increase in the utilisation of DL models across various diagnostic areas from 2013 to 2024, including radiography (33% of the studies), cytology (33%), health record analysis (8%), MRI (8%), environmental data analysis (5%), photo/video imaging (5%), and ultrasound (5%). Over the past decade, radiographic imaging has emerged as most impactful. Various studies have demonstrated notable success in the classification of primary thoracic lesions and cardiac disease from radiographs using DL models compared to specialist veterinarian benchmarks. Moreover, the technology has proven adept at recognising, counting, and classifying cell types in microscope slide images, demonstrating its versatility across different veterinary diagnostic modality. While deep learning shows promise in veterinary diagnostics, several challenges remain. These challenges range from the need for large and diverse datasets, the potential for interpretability issues and the importance of consulting with experts throughout model development to ensure validity. A thorough understanding of these considerations for the design and implementation of DL in veterinary medicine is imperative for driving future research and development efforts in the field. In addition, the potential future impacts of DL on veterinary diagnostics are discussed to explore avenues for further refinement and expansion of DL applications in veterinary medicine, ultimately contributing to increased standards of care and improved health outcomes for animals as this technology continues to evolve.

Development of a natural language processing algorithm to extract social determinants of health from clinician notes

Disparities in access to the organ transplant waitlist are well-documented, but research into modifiable factors has been limited due to a lack of access to organized prewaitlisting data. This study aimed to develop a natural language processing (NLP) algorithm to extract social determinants of health (SDOH) from free-text notes and quantify the association of SDOH with access to the transplant waitlist. We collected 261 802 clinician notes from 11 111 adults referred for kidney or liver transplants between 2016 and 2022 at the Duke University Health System. An SDOH ontology and a rule-based NLP algorithm were created to extract and organize terms. Education, transportation, and age were the most frequent terms identified. Negative sentiment and refer were the most negatively associated features with listing in both kidney and liver transplant patients. Income and employment for the kidney, and judgment and positive sentiment for liver were the most positively associated features with the listing. This study suggests that the integration of NLP tools into the transplant clinical workflow could help improve collection and organization of SDOH and inform center-level efforts at resource allocation, potentially improving access to the transplant waitlist and posttransplant outcomes.

Transforming Healthcare in Low-Resource Settings With Artificial Intelligence: Recent Developments and Outcomes

BACKGROUND

Artificial intelligence now encompasses technologies like machine learning, natural language processing, and robotics, allowing machines to undertake complex tasks traditionally done by humans. AI's application in healthcare has led to advancements in diagnostic tools, predictive analytics, and surgical precision.

AIM

This comprehensive review aims to explore the transformative impact of AI across diverse healthcare domains, highlighting its applications, advancements, challenges, and contributions to enhancing patient care.

METHODOLOGY

A comprehensive literature search was conducted across multiple databases, covering publications from 2014 to 2024. Keywords related to AI applications in healthcare were used to gather data, focusing on studies exploring AI's role in medical specialties.

RESULTS

AI has demonstrated substantial benefits across various fields of medicine. In cardiology, it aids in automated image interpretation, risk prediction, and the management of cardiovascular diseases. In oncology, AI enhances cancer detection, treatment planning, and personalized drug selection. Radiology benefits from improved image analysis and diagnostic accuracy, while critical care sees advancements in patient triage and resource optimization. AI's integration into pediatrics, surgery, public health, neurology, pathology, and mental health has similarly shown significant improvements in diagnostic precision, personalized treatment, and overall patient care. The implementation of AI in low-resource settings has been particularly impactful, enhancing access to advanced diagnostic tools and treatments.

CONCLUSION

AI is rapidly changing the healthcare industry by greatly increasing the accuracy of diagnoses, streamlining treatment plans, and improving patient outcomes across a variety of medical specializations. This review underscores AI's transformative potential, from early disease detection to personalized treatment plans, and its ability to augment healthcare delivery, particularly in resource-limited settings.

Natural language processing of electronic health records for early detection of cognitive decline: a systematic review

This systematic review evaluated natural language processing (NLP) approaches for detecting cognitive impairment in electronic health record clinical notes. Following PRISMA guidelines, we analyzed 18 studies (n = 1,064,530) that employed rule-based algorithms (67%), traditional machine learning (28%), and deep learning (17%). NLP models demonstrated robust performance in identifying cognitive decline, with median sensitivity 0.88 (IQR 0.74-0.91) and specificity 0.96 (IQR 0.81-0.99). Deep learning architectures achieved superior results, with area under the receiver operating characteristic curves up to 0.997. Major implementation challenges included incomplete electronic health record data capture, inconsistent clinical documentation practices, and limited external validation. While NLP demonstrates promise, successful clinical translation requires establishing standardized approaches, improving access to annotated datasets, and developing equitable deployment frameworks.

The Role of Artificial Intelligence in Pediatric Intensive Care: A Systematic Review

Pediatric intensive care units (PICUs) could transform due to artificial intelligence (AI), which could improve patient outcomes, increase diagnostic accuracy, and streamline repetitive procedures. The goal of this systematic review was to outline how AI can be used to enhance any health outcomes in pediatric intensive care. We searched four databases (PubMed, Scopus, Web of Science, and IEEE Xplore) for relevant studies using a predefined systematic search. We found 267 studies in these four databases. The studies were first screened to remove the duplicates and then screened by titles to remove irrelevant studies. The studies were further screened based on inclusion and exclusion criteria, in which 32 studies were found suitable for inclusion in this study. The studies were assessed for risk of bias using the Prediction Model Risk Of Bias Assessment Tool (PROBAST) tool. After AI was implemented, almost 22% (n = 7) of studies showed an immediate effect and enhanced health outcomes. A small number of studies involved AI implementation in actual PICUs, while the majority focused on experimental testing. AI models outperformed conventional clinical modalities among the remaining 78% (n = 25) and might have indirectly impacted patient outcomes. Significant variation in metrics and standardization was found when health outcomes were quantitatively assessed using statistical measures, including specificity (38%; n = 12) and area under the receiver operating characteristic curve (AUROC) (56%; n = 18). There are not sufficient studies showing that AI has significantly enhanced pediatric critical care patients' health outcomes. To evaluate AI's impact, more prospective, experimental research is required, utilizing verified outcome measures, defined metrics, and established application frameworks.

Evaluating Use of Generative Artificial Intelligence in Clinical Pathology Practice: Opportunities and the Way Forward

CONTEXT.—

Generative artificial intelligence (GAI) technologies are likely to dramatically impact health care workflows in clinical pathology (CP). Applications in CP include education, data mining, decision support, result summaries, and patient trend assessments.

OBJECTIVE.—

To review use cases of GAI in CP, with a particular focus on large language models. Specific examples are provided for the applications of GAI in the subspecialties of clinical chemistry, microbiology, hematopathology, and molecular diagnostics. Additionally, the review addresses potential pitfalls of GAI paradigms.

DATA SOURCES.—

Current literature on GAI in health care was reviewed broadly. The use case scenarios for each CP subspecialty review common data sources generated in each subspecialty. The potential for utilization of CP data in the GAI context was subsequently assessed, focusing on issues such as future reporting paradigms, impact on quality metrics, and potential for translational research activities.

CONCLUSIONS.—

GAI is a powerful tool with the potential to revolutionize health care for patients and practitioners alike. However, GAI must be implemented with much caution considering various shortcomings of the technology such as biases, hallucinations, practical challenges of implementing GAI in existing CP workflows, and end-user acceptance. Human-in-the-loop models of GAI implementation have the potential to revolutionize CP by delivering deeper, meaningful insights into patient outcomes both at an individual and a population level.

Estimating the Severity of Oral Lesions Via Analysis of Cone Beam Computed Tomography Reports: A Proposed Deep Learning Model

OBJECTIVES

Several factors such as unavailability of specialists, dental phobia, and financial difficulties may lead to a delay between receiving an oral radiology report and consulting a dentist. The primary aim of this study was to distinguish between high-risk and low-risk oral lesions according to the radiologist's reports of cone beam computed tomography (CBCT) images. Such a facility may be employed by dentist or his/her assistant to make the patient aware of the severity and the grade of the oral lesion and referral for immediate treatment or other follow-up care.

METHODS

A total number of 1134 CBCT radiography reports owned by Shiraz University of Medical Sciences were collected. The severity level of each sample was specified by three experts, and an annotation was carried out accordingly. After preprocessing the data, a deep learning model, referred to as CNN-LSTM, was developed, which aims to detect the degree of severity of the problem based on analysis of the radiologist's report. Unlike traditional models which usually use a simple collection of words, the proposed deep model uses words embedded in dense vector representations, which empowers it to effectively capture semantic similarities.

RESULTS

The results indicated that the proposed model outperformed its counterparts in terms of precision, recall, and F1 criteria. This suggests its potential as a reliable tool for early estimation of the severity of oral lesions.

CONCLUSIONS

This study shows the effectiveness of deep learning in the analysis of textual reports and accurately distinguishing between high-risk and low-risk lesions. Employing the proposed model which can Provide timely warnings about the need for follow-up and prompt treatment can shield the patient from the risks associated with delays.

CLINICAL SIGNIFICANCE

Our collaboratively collected and expert-annotated dataset serves as a valuable resource for exploratory research. The results demonstrate the pivotal role of our deep learning model could play in assessing the severity of oral lesions in dental reports.

Safety-netting advice documentation in out-of-hours primary care: a retrospective cohort from 2013 to 2020

BACKGROUND

Providing safety-netting advice (SNA) in out-of-hours (OOH) primary care is a recognised standard of safe care, but it is not known how frequently this occurs in practice.

AIM

Assess the frequency and type of SNA documented in OOH primary care and explore factors associated with its presence.

DESIGN AND SETTING

This was a retrospective cohort study using the Birmingham Out-of-hours general practice Research Database.

METHOD

A stratified sample of 30 adult consultation records per month from July 2013 to February 2020 were assessed using a safety-netting coding tool. Associations were tested using linear and logistic regression.

RESULTS

The overall frequency of SNA per consultation was 78.0% (1472/1886), increasing from 75.7% (224/296) in 2014 to 81.5% (220/270) in 2019. The proportion of specific SNA and the average number of symptoms patients were told to look out for increased with time. The most common symptom to look out for was if the patients' condition worsened followed by if their symptoms persisted, but only one in five consultations included a timeframe to reconsult for persistent symptoms. SNA was more frequently documented in face-to-face treatment-centre encounters compared with telephone consultations (odds ratio [OR] 1.77, 95% confidence interval [CI] = 1.09 to 2.85, P = 0.02), for possible infections (OR 1.53, 95% CI = 1.13 to 2.07, P = 0.006), and less frequently for mental (versus physical) health consultations (OR 0.33, 95% CI = 0.17 to 0.66, P = 0.002) and where follow-up was planned (OR 0.34, 95% CI = 0.25 to 0.46, P<0.001).

CONCLUSION

The frequency of SNA documented in OOH primary care was higher than previously reported during in-hours care. Over time, the frequency of SNA and proportion that contained specific advice increased, however, this study highlights potential consultations where SNA could be improved, such as mental health and telephone consultations.

Development of Artificial Intelligence Systems for Chronic Kidney Disease

Chronic kidney disease (CKD) is a complex disease that is related not only to dialysis but also to the onset of cardiovascular disease and life prognosis. As renal function declines with age and depending on lifestyle, the number of patients with CKD is rapidly increasing in Japan. Accurate prognosis prediction for patients with CKD in clinical settings is important for selecting treatment methods and screening patients with high-risk. In recent years, big databases on CKD and dialysis have been constructed through the use of data science technology, and the pathology of CKD is being elucidated. Therefore, we developed an artificial intelligence (AI) system that can accurately predict the prognosis of CKD such as its progression, the timing of dialysis introduction, and death. Aiming for its social implementation, the prognosis prediction system developed for patients with CKD was released on the website. We then developed a clinical practice guideline creation support system called Doctor K as an AI system. When creating clinical practice guidelines, huge amounts of manpower and time are required to conduct a systematic review of thousands of papers. Therefore, we developed a natural language processing (NLP) AI system to significantly improve work efficiency. Doctor K was used in the preparation of the guidelines of the Japanese Society of Nephrology. Furthermore, by comparing and analyzing the medical word virtual space constructed by the NLP AI system based on patient big data, we proved using the latest mathematical theory (category theory) that this system reflects the pathology of CKD. This suggests the possibility that the NLP AI system can predict patient prognosis. We hope that, through these studies, the use of AI based on big data will lead to the development of new treatments and improvement in patient prognosis.

Identifying people with post-COVID condition using linked, population-based administrative health data from Manitoba, Canada: prevalence and predictors in a cohort of COVID-positive individuals

OBJECTIVE

Many individuals exposed to SARS-CoV-2 experience long-term symptoms as part of a syndrome called post-COVID condition (PCC). Research on PCC is still emerging but is urgently needed to support diagnosis, clinical treatment guidelines and health system resource allocation. In this study, we developed a method to identify PCC cases using administrative health data and report PCC prevalence and predictive factors in Manitoba, Canada.

DESIGN

Cohort study.

SETTING

Manitoba, Canada.

PARTICIPANTS

All Manitobans who tested positive for SARS-CoV-2 during population-wide PCR testing from March 2020 to December 2021 (n=66 365) and were subsequently deemed to have PCC based on International Classification of Disease-9/10 diagnostic codes and prescription drug codes (n=11 316). Additional PCC cases were identified using predictive modelling to assess patterns of health service use, including physician visits, emergency department visits and hospitalisation for any reason (n=4155).

OUTCOMES

We measured PCC prevalence as % PCC cases among Manitobans with positive tests and identified predictive factors associated with PCC by calculating odds ratios with 95% confidence intervals, adjusted for sociodemographic and clinical characteristics (aOR).

RESULTS

Among 66 365 Manitobans with positive tests, we identified 15 471 (23%) as having PCC. Being female (aOR 1.64, 95% CI 1.58 to 1.71), being age 60-79 (aOR 1.33, 95% CI 1.25 to 1.41) or age 80+ (aOR 1.62, 95% CI 1.46 to 1.80), being hospitalised within 14 days of COVID-19 infection (aOR 1.95, 95% CI 1.80 to 2.10) and having a Charlson Comorbidity Index of 1+ (aOR 1.95, 95% CI 1.78 to 2.14) were predictive of PCC. Receiving 1+ doses of the COVID-19 vaccine (one dose, aOR 0.80, 95% CI 0.74 to 0.86; two doses, aOR 0.29, 95% CI 0.22 to 0.31) decreased the odds of PCC.

CONCLUSIONS

This data-driven approach expands our understanding of the prevalence and epidemiology of PCC and may be applied in other jurisdictions with population-based data. The study provides additional insights into risk and protective factors for PCC to inform health system planning and service delivery.

Twenty-Five Years of Evolution and Hurdles in Electronic Health Records and Interoperability in Medical Research: Comprehensive Review

BACKGROUND

Electronic health records (EHRs) facilitate the accessibility and sharing of patient data among various health care providers, contributing to more coordinated and efficient care.

OBJECTIVE

This study aimed to summarize the evolution of secondary use of EHRs and their interoperability in medical research over the past 25 years.

METHODS

We conducted an extensive literature search in the PubMed, Scopus, and Web of Science databases using the keywords Electronic health record and Electronic medical record in the title or abstract and Medical research in all fields from 2000 to 2024. Specific terms were applied to different time periods.

RESULTS

The review yielded 2212 studies, all of which were then screened and processed in a structured manner. Of these 2212 studies, 2102 (93.03%) were included in the review analysis, of which 1079 (51.33%) studies were from 2000 to 2009, 582 (27.69%) were from 2010 to 2019, 251 (11.94%) were from 2020 to 2023, and 190 (9.04%) were from 2024.

CONCLUSIONS

The evolution of EHRs marks an important milestone in health care's journey toward integrating technology and medicine. From early documentation practices to the sophisticated use of artificial intelligence and big data analytics today, EHRs have become central to improving patient care, enhancing public health surveillance, and advancing medical research.

The potential of Generative Pre-trained Transformer 4 (GPT-4) to analyse medical notes in three different languages: a retrospective model-evaluation study

BACKGROUND

Patient notes contain substantial information but are difficult for computers to analyse due to their unstructured format. Large-language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4), have changed our ability to process text, but we do not know how effectively they handle medical notes. We aimed to assess the ability of GPT-4 to answer predefined questions after reading medical notes in three different languages.

METHODS

For this retrospective model-evaluation study, we included eight university hospitals from four countries (ie, the USA, Colombia, Singapore, and Italy). Each site submitted seven de-identified medical notes related to seven separate patients to the coordinating centre between June 1, 2023, and Feb 28, 2024. Medical notes were written between Feb 1, 2020, and June 1, 2023. One site provided medical notes in Spanish, one site provided notes in Italian, and the remaining six sites provided notes in English. We included admission notes, progress notes, and consultation notes. No discharge summaries were included in this study. We advised participating sites to choose medical notes that, at time of hospital admission, were for patients who were male or female, aged 18-65 years, had a diagnosis of obesity, had a diagnosis of COVID-19, and had submitted an admission note. Adherence to these criteria was optional and participating sites randomly chose which medical notes to submit. When entering information into GPT-4, we prepended each medical note with an instruction prompt and a list of 14 questions that had been chosen a priori. Each medical note was individually given to GPT-4 in its original language and in separate sessions; the questions were always given in English. At each site, two physicians independently validated responses by GPT-4 and responded to all 14 questions. Each pair of physicians evaluated responses from GPT-4 to the seven medical notes from their own site only. Physicians were not masked to responses from GPT-4 before providing their own answers, but were masked to responses from the other physician.

FINDINGS

We collected 56 medical notes, of which 42 (75%) were in English, seven (13%) were in Italian, and seven (13%) were in Spanish. For each medical note, GPT-4 responded to 14 questions, resulting in 784 responses. In 622 (79%, 95% CI 76-82) of 784 responses, both physicians agreed with GPT-4. In 82 (11%, 8-13) responses, only one physician agreed with GPT-4. In the remaining 80 (10%, 8-13) responses, neither physician agreed with GPT-4. Both physicians agreed with GPT-4 more often for medical notes written in Spanish (86 [88%, 95% CI 79-93] of 98 responses) and Italian (82 [84%, 75-90] of 98 responses) than in English (454 [77%, 74-80] of 588 responses).

INTERPRETATION

The results of our model-evaluation study suggest that GPT-4 is accurate when analysing medical notes in three different languages. In the future, research should explore how LLMs can be integrated into clinical workflows to maximise their use in health care.

FUNDING

None.

Automated Identification of Heart Failure With Reduced Ejection Fraction Using Deep Learning-Based Natural Language Processing

BACKGROUND

The lack of automated tools for measuring care quality limits the implementation of a national program to assess guideline-directed care in heart failure with reduced ejection fraction (HFrEF).

OBJECTIVES

The authors aimed to automate the identification of patients with HFrEF at hospital discharge, an opportunity to evaluate and improve the quality of care.

METHODS

The authors developed a novel deep-learning language model for identifying patients with HFrEF from discharge summaries of hospitalizations with heart failure at Yale New Haven Hospital during 2015 to 2019. HFrEF was defined by left ventricular ejection fraction <40% on antecedent echocardiography. The authors externally validated the model at Northwestern Medicine, community hospitals of Yale, and the MIMIC-III (Medical Information Mart for Intensive Care III) database.

RESULTS

A total of 13,251 notes from 5,392 unique individuals (age 73 ± 14 years, 48% women), including 2,487 patients with HFrEF (46.1%), were used for model development (train/held-out: 70%/30%). The model achieved an area under receiver-operating characteristic curve (AUROC) of 0.97 and area under precision recall curve (AUPRC) of 0.97 in detecting HFrEF on the held-out set. The model had high performance in identifying HFrEF with AUROC = 0.94 and AUPRC = 0.91 on 19,242 notes from Northwestern Medicine, AUROC = 0.95 and AUPRC = 0.96 on 139 manually abstracted notes from Yale community hospitals, and AUROC = 0.91 and AUPRC = 0.92 on 146 manually reviewed notes from MIMIC-III. Model-based predictions of HFrEF corresponded to a net reclassification improvement of 60.2% ± 1.9% compared with diagnosis codes (P < 0.001).

CONCLUSIONS

The authors developed a language model that identifies HFrEF from clinical notes with high precision and accuracy, representing a key element in automating quality assessment for individuals with HFrEF.

Consensus statements on the current landscape of artificial intelligence applications in endoscopy, addressing roadblocks, and advancing artificial intelligence in gastroenterology

BACKGROUND AND AIMS

The American Society for Gastrointestinal Endoscopy (ASGE) AI Task Force along with experts in endoscopy, technology space, regulatory authorities, and other medical subspecialties initiated a consensus process that analyzed the current literature, highlighted potential areas, and outlined the necessary research in artificial intelligence (AI) to allow a clearer understanding of AI as it pertains to endoscopy currently.

METHODS

A modified Delphi process was used to develop these consensus statements.

RESULTS

Statement 1: Current advances in AI allow for the development of AI-based algorithms that can be applied to endoscopy to augment endoscopist performance in detection and characterization of endoscopic lesions. Statement 2: Computer vision-based algorithms provide opportunities to redefine quality metrics in endoscopy using AI, which can be standardized and can reduce subjectivity in reporting quality metrics. Natural language processing-based algorithms can help with the data abstraction needed for reporting current quality metrics in GI endoscopy effortlessly. Statement 3: AI technologies can support smart endoscopy suites, which may help optimize workflows in the endoscopy suite, including automated documentation. Statement 4: Using AI and machine learning helps in predictive modeling, diagnosis, and prognostication. High-quality data with multidimensionality are needed for risk prediction, prognostication of specific clinical conditions, and their outcomes when using machine learning methods. Statement 5: Big data and cloud-based tools can help advance clinical research in gastroenterology. Multimodal data are key to understanding the maximal extent of the disease state and unlocking treatment options. Statement 6: Understanding how to evaluate AI algorithms in the gastroenterology literature and clinical trials is important for gastroenterologists, trainees, and researchers, and hence education efforts by GI societies are needed. Statement 7: Several challenges regarding integrating AI solutions into the clinical practice of endoscopy exist, including understanding the role of human-AI interaction. Transparency, interpretability, and explainability of AI algorithms play a key role in their clinical adoption in GI endoscopy. Developing appropriate AI governance, data procurement, and tools needed for the AI lifecycle are critical for the successful implementation of AI into clinical practice. Statement 8: For payment of AI in endoscopy, a thorough evaluation of the potential value proposition for AI systems may help guide purchasing decisions in endoscopy. Reliable cost-effectiveness studies to guide reimbursement are needed. Statement 9: Relevant clinical outcomes and performance metrics for AI in gastroenterology are currently not well defined. To improve the quality and interpretability of research in the field, steps need to be taken to define these evidence standards. Statement 10: A balanced view of AI technologies and active collaboration between the medical technology industry, computer scientists, gastroenterologists, and researchers are critical for the meaningful advancement of AI in gastroenterology.

CONCLUSIONS

The consensus process led by the ASGE AI Task Force and experts from various disciplines has shed light on the potential of AI in endoscopy and gastroenterology. AI-based algorithms have shown promise in augmenting endoscopist performance, redefining quality metrics, optimizing workflows, and aiding in predictive modeling and diagnosis. However, challenges remain in evaluating AI algorithms, ensuring transparency and interpretability, addressing governance and data procurement, determining payment models, defining relevant clinical outcomes, and fostering collaboration between stakeholders. Addressing these challenges while maintaining a balanced perspective is crucial for the meaningful advancement of AI in gastroenterology.

Using digital health technologies to optimise antimicrobial use globally

Digital health technology (DHT) describes tools and devices that generate or process health data. The application of DHTs could improve the diagnosis, treatment, and surveillance of bacterial infection and the prevention of antimicrobial resistance (AMR). DHTs to optimise antimicrobial use are rapidly being developed. To support the global adoption of DHTs and the opportunities offered to optimise antimicrobial use consensus is needed on what data are required to support antimicrobial decision making. This Series paper will explore bacterial AMR in humans and the need to optimise antimicrobial use in response to this global threat. It will also describe state-of-the-art DHTs to optimise antimicrobial prescribing in high-income and low-income and middle-income countries, and consider what fundamental data are ideally required for and from such technologies to support optimised antimicrobial use.

Artificial Intelligence in Human Reproduction

The use of artificial intelligence (AI) in human reproduction is a rapidly evolving field with both exciting possibilities and ethical considerations. This technology has the potential to improve success rates and reduce the emotional and financial burden of infertility. However, it also raises ethical and privacy concerns. This paper presents an overview of the current and potential applications of AI in human reproduction. It explores the use of AI in various aspects of reproductive medicine, including fertility tracking, assisted reproductive technologies, management of pregnancy complications, and laboratory automation. In addition, we discuss the need for robust ethical frameworks and regulations to ensure the responsible and equitable use of AI in reproductive medicine.

Assessing the use of unstructured electronic health record data to identify exposure to firearm violence

Objectives

Research on firearm violence is largely limited to people who experienced acute bodily trauma and death which is readily gathered from Inpatient and Emergency Department settings and mortality data. Exposures to firearm violence, such as witnessing firearm violence or losing a loved one to firearm violence, are not routinely collected in health care. As a result, the true public health burden of firearm violence is underestimated. Clinical notes from electronic health records (EHRs) are a promising source of data that may expand our understanding of the impact of firearm violence on health. Pilot work was conducted on a sample of clinical notes to assess how firearm terms present in unstructured clinical notes as part of a larger initiative to develop a natural language processing (NLP) model to identify firearm exposure and injury in ambulatory care data.

Materials and Methods

We used EHR data from 2012 to 2022 from a large multistate network of primary care and behavioral health clinics. A text string search of broad, gun-only, and shooting terms was applied to 9,598 patients with either/both an ICD-10 or an OCHIN-developed structured data field indicating exposure to firearm violence. A sample of clinical notes from 90 patients was reviewed to ascertain the meaning of terms.

Results

Among the 90 clinical patient notes, 13 (14%) notes reflect documentation of exposure to firearm violence or injury from firearms. Results from this study identified refinements that should be considered for NLP text classification.

Conclusion

Unstructured clinical notes from primary and behavioral health clinics have potential to expand understanding of firearm violence.

Artificial Intelligence as a Tool for Creating Patient Visit Summary: A Scoping Review and Guide to Implementation in an Erectile Dysfunction Clinic

PURPOSE OF REVIEW

In modern healthcare, the integration of artificial intelligence (AI) has revolutionized clinical practices, particularly in data management and patient visit summary creation. Manual creation of patient summary is repetitive, time-consuming, prone to errors, and increases clinicians' workload. AI, through voice recognition and Natural Language Processing (NLP), can automate this task more accurately and efficiently. Erectile dysfunction (ED) clinics, which deal with specific pattern of conditions together with an involvement of broader systemic issues, can greatly benefit from AI-driven patient summary. This scoping review examined the evidence on AI-generated patient summary and evaluated their implementation in ED clinics.

RECENT FINDINGS

A total of 381 articles were initially identified, 11 studies were included for the analysis. These studies showcased various methodologies, such as AI-assisted clinical notes and NLP algorithms. Most studies have demonstrated the ability of AI to be used in real life clinical scenarios. Major electronic health record platforms are also integrating AI to their system. However, to date, no studies have specifically addressed AI for patient summary creation in ED clinics.

Enhancing early detection of cognitive decline in the elderly: a comparative study utilizing large language models in clinical notes

BACKGROUND

Large language models (LLMs) have shown promising performance in various healthcare domains, but their effectiveness in identifying specific clinical conditions in real medical records is less explored. This study evaluates LLMs for detecting signs of cognitive decline in real electronic health record (EHR) clinical notes, comparing their error profiles with traditional models. The insights gained will inform strategies for performance enhancement.

METHODS

This study, conducted at Mass General Brigham in Boston, MA, analysed clinical notes from the four years prior to a 2019 diagnosis of mild cognitive impairment in patients aged 50 and older. We developed prompts for two LLMs, Llama 2 and GPT-4, on Health Insurance Portability and Accountability Act (HIPAA)-compliant cloud-computing platforms using multiple approaches (e.g., hard prompting, retrieval augmented generation, and error analysis-based instructions) to select the optimal LLM-based method. Baseline models included a hierarchical attention-based neural network and XGBoost. Subsequently, we constructed an ensemble of the three models using a majority vote approach. Confusion-matrix-based scores were used for model evaluation.

FINDINGS

We used a randomly annotated sample of 4949 note sections from 1969 patients (women: 1046 [53.1%]; age: mean, 76.0 [SD, 13.3] years), filtered with keywords related to cognitive functions, for model development. For testing, a random annotated sample of 1996 note sections from 1161 patients (women: 619 [53.3%]; age: mean, 76.5 [SD, 10.2] years) without keyword filtering was utilised. GPT-4 demonstrated superior accuracy and efficiency compared to Llama 2, but did not outperform traditional models. The ensemble model outperformed the individual models in terms of all evaluation metrics with statistical significance (p < 0.01), achieving a precision of 90.2% [95% CI: 81.9%-96.8%], a recall of 94.2% [95% CI: 87.9%-98.7%], and an F1-score of 92.1% [95% CI: 86.8%-96.4%]. Notably, the ensemble model showed a significant improvement in precision, increasing from a range of 70%-79% to above 90%, compared to the best-performing single model. Error analysis revealed that 63 samples were incorrectly predicted by at least one model; however, only 2 cases (3.2%) were mutual errors across all models, indicating diverse error profiles among them.

INTERPRETATION

LLMs and traditional machine learning models trained using local EHR data exhibited diverse error profiles. The ensemble of these models was found to be complementary, enhancing diagnostic performance. Future research should investigate integrating LLMs with smaller, localised models and incorporating medical data and domain knowledge to enhance performance on specific tasks.

FUNDING

This research was supported by the National Institute on Aging grants (R44AG081006, R01AG080429) and National Library of Medicine grant (R01LM014239).

SEETrials: Leveraging large language models for safety and efficacy extraction in oncology clinical trials

Background

Initial insights into oncology clinical trial outcomes are often gleaned manually from conference abstracts. We aimed to develop an automated system to extract safety and efficacy information from study abstracts with high precision and fine granularity, transforming them into computable data for timely clinical decision-making.

Methods

We collected clinical trial abstracts from key conferences and PubMed (2012-2023). The SEETrials system was developed with three modules: preprocessing, prompt engineering with knowledge ingestion, and postprocessing. We evaluated the system's performance qualitatively and quantitatively and assessed its generalizability across different cancer types- multiple myeloma (MM), breast, lung, lymphoma, and leukemia. Furthermore, the efficacy and safety of innovative therapies, including CAR-T, bispecific antibodies, and antibody-drug conjugates (ADC), in MM were analyzed across a large scale of clinical trial studies.

Results

SEETrials achieved high precision (0.964), recall (sensitivity) (0.988), and F1 score (0.974) across 70 data elements present in the MM trial studies Generalizability tests on four additional cancers yielded precision, recall, and F1 scores within the 0.979-0.992 range. Variation in the distribution of safety and efficacy-related entities was observed across diverse therapies, with certain adverse events more common in specific treatments. Comparative performance analysis using overall response rate (ORR) and complete response (CR) highlighted differences among therapies: CAR-T (ORR: 88 %, 95 % CI: 84-92 %; CR: 95 %, 95 % CI: 53-66 %), bispecific antibodies (ORR: 64 %, 95 % CI: 55-73 %; CR: 27 %, 95 % CI: 16-37 %), and ADC (ORR: 51 %, 95 % CI: 37-65 %; CR: 26 %, 95 % CI: 1-51 %). Notable study heterogeneity was identified (>75 % I 2 heterogeneity index scores) across several outcome entities analyzed within therapy subgroups.

Conclusion

SEETrials demonstrated highly accurate data extraction and versatility across different therapeutics and various cancer domains. Its automated processing of large datasets facilitates nuanced data comparisons, promoting the swift and effective dissemination of clinical insights.

Qualitative Health-Related Quality of Life and Natural Language Processing: Characteristics, Implications, and Challenges

OBJECTIVES

This article focuses on describing the main characteristics of the application of NLP in the qualitative assessment of quality of life, as well as its implications and challenges.

METHODS

The qualitative methodology allows analysing patient comments in unstructured free text and obtaining valuable information through manual analysis of these data. However, large amounts of data are a healthcare challenge since it would require a high number of staff and time resources that are not available in most healthcare organizations.

RESULTS

One potential solution to mitigate the resource constraints of qualitative analysis is the use of machine learning and artificial intelligence, specifically methodologies based on natural language processing.

Scalable incident detection via natural language processing and probabilistic language models

Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risks under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It's based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: (1) suicide attempt; (2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of ~ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR ~ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race differed across phenotypes. Scalable phenotyping models, like most healthcare AI, require algorithmovigilance and debiasing prior to implementation.

Validation of large language models for detecting pathologic complete response in breast cancer using population-based pathology reports

AIMS

The primary goal of this study is to evaluate the capabilities of Large Language Models (LLMs) in understanding and processing complex medical documentation. We chose to focus on the identification of pathologic complete response (pCR) in narrative pathology reports. This approach aims to contribute to the advancement of comprehensive reporting, health research, and public health surveillance, thereby enhancing patient care and breast cancer management strategies.

METHODS

The study utilized two analytical pipelines, developed with open-source LLMs within the healthcare system's computing environment. First, we extracted embeddings from pathology reports using 15 different transformer-based models and then employed logistic regression on these embeddings to classify the presence or absence of pCR. Secondly, we fine-tuned the Generative Pre-trained Transformer-2 (GPT-2) model by attaching a simple feed-forward neural network (FFNN) layer to improve the detection performance of pCR from pathology reports.

RESULTS

In a cohort of 351 female breast cancer patients who underwent neoadjuvant chemotherapy (NAC) and subsequent surgery between 2010 and 2017 in Calgary, the optimized method displayed a sensitivity of 95.3% (95%CI: 84.0-100.0%), a positive predictive value of 90.9% (95%CI: 76.5-100.0%), and an F1 score of 93.0% (95%CI: 83.7-100.0%). The results, achieved through diverse LLM integration, surpassed traditional machine learning models, underscoring the potential of LLMs in clinical pathology information extraction.

CONCLUSIONS

The study successfully demonstrates the efficacy of LLMs in interpreting and processing digital pathology data, particularly for determining pCR in breast cancer patients post-NAC. The superior performance of LLM-based pipelines over traditional models highlights their significant potential in extracting and analyzing key clinical data from narrative reports. While promising, these findings highlight the need for future external validation to confirm the reliability and broader applicability of these methods.

US public health surveillance, reimagined

Introduction

This study presents two novel concepts for standardizing electronic health records (EHR)-based public health surveillance through utilization of existing informatics methods and data platforms.

Methods

Drawing from the collective experience in applied epidemiology, health services research and health informatics, the author presents a vision for an alternative path to public health surveillance by repurposing existing tools and resources, such as (1) computable phenotypes which have already been created and validated for a variety of chronic diseases of interest to public health and (2) large data platforms/collaboratives, such as All of Us Research Program and National COVID Cohort Collaborative. Opportunities and challenges are discussed regarding EHR-based chronic disease surveillance, as well as the concept of phenotype definitions and large data platforms reuse for public health needs.

Results/Framework

Reusing of computable phenotypes for EHR-based public health surveillance would require secure data platforms and nationally representative data. Standardization metrics for reuse of previously developed and validated computable phenotypes are also necessary and are currently being developed by the author. This study presents a reimagined Learning Health System framework by incorporating Public Health and two novel concept sets of solutions into the healthcare ecosystem.

Conclusion/Next Steps

Alternative approaches to limited resources and current infrastructure of the US Public Health System, especially as applied to disease surveillance, are needed and may be possible when repurposing the resources and methodologies across the Learning Health System.

Machine learning applications in preventive healthcare: A systematic literature review on predictive analytics of disease comorbidity from multiple perspectives

Artificial intelligence is constantly revolutionizing biomedical research and healthcare management. Disease comorbidity is a major threat to the quality of life for susceptible groups, especially middle-aged and elderly patients. The presence of multiple chronic diseases makes precision diagnosis challenging to realize and imposes a heavy burden on the healthcare system and economy. Given an enormous amount of accumulated health data, machine learning techniques show their capability in handling this puzzle. The present study conducts a review to uncover current research efforts in applying these methods to understanding comorbidity mechanisms and making clinical predictions considering these complex patterns. A descriptive metadata analysis of 791 unique publications aims to capture the overall research progression between January 2012 and June 2023. To delve into comorbidity-focused research, 61 of these scientific papers are systematically assessed. Four predictive analytics of tasks are detected: disease comorbidity data extraction, clustering, network, and risk prediction. It is observed that some machine learning-driven applications address inherent data deficiencies in healthcare datasets and provide a model interpretation that identifies significant risk factors of comorbidity development. Based on insights, both technical and practical, gained from relevant literature, this study intends to guide future interests in comorbidity research and draw conclusions about chronic disease prevention and diagnosis with managerial implications.

Large language models and rheumatology: are we there yet?

The last 2 years have marked the beginning of a golden age for natural language processing in medicine. The arrival of large language models (LLMs) and multimodal models have raised new opportunities and challenges for research and clinical practice. In rheumatology, a specialty rich in data and requiring complex decision-making, the use of these tools may transform diagnostic procedures, improve patient interaction and simplify data management, leading to more personalized and efficient healthcare outcomes. The objective of this article is to present an overview of the status of LLMs in the field of rheumatology while discussing some of the challenges ahead in this area of great potential.

Text mining method to unravel long COVID's clinical condition in hospitalized patients

Long COVID is characterized by persistent that extends symptoms beyond established timeframes. Its varied presentation across different populations and healthcare systems poses significant challenges in understanding its clinical manifestations and implications. In this study, we present a novel application of text mining technique to automatically extract unstructured data from a long COVID survey conducted at a prominent university hospital in São Paulo, Brazil. Our phonetic text clustering (PTC) method enables the exploration of unstructured Electronic Healthcare Records (EHR) data to unify different written forms of similar terms into a single phonemic representation. We used n-gram text analysis to detect compound words and negated terms in Portuguese-BR, focusing on medical conditions and symptoms related to long COVID. By leveraging text mining, we aim to contribute to a deeper understanding of this chronic condition and its implications for healthcare systems globally. The model developed in this study has the potential for scalability and applicability in other healthcare settings, thereby supporting broader research efforts and informing clinical decision-making for long COVID patients.

Advancements in AI based healthcare techniques with FOCUS ON diagnostic techniques

Since the past decade, the interest towards more precise and efficient healthcare techniques with special emphasis on diagnostic techniques has increased. Artificial Intelligence has proved to be instrumental in development of various such techniques. The various types of AI like ML, NLP, RPA etc. are being used, which have streamlined and organised the Electronic Health Records (EHR) along with aiding the healthcare provider with decision making and sample and data analysis. This article also deals with the 3 major categories of diagnostic techniques - Imaging based, Pathology based and Preventive diagnostic techniques and what all changes and modifications were brought upon them, due to use of AI. Due to such a high demand, the investment in AI based healthcare techniques has increased substantially, with predicted market size of almost 188 billon USD by 2030. In India itself, AI in healthcare is expected to raise the GDP by 25 billion USD by 2028. But there are also several challenges associated with this like unavailability of quality data, black box issue etc. One of the major challenges is the ethical considerations and issues during use of medical records as it is a very sensitive document. Due to this, there is several trust issues associated with adoption of AI by many organizations. These challenges have also been discussed in this article. Need for further development in the AI based diagnostic techniques is also done in the article. Alongside, the production of such techniques and devices which are easy to use and simple to incorporate into the daily workflows have immense scope in the upcoming times. The increasing scope of Clinical Decision Support System, Telemedicine etc. make AI a promising field in the healthcare and diagnostics arena. Concluding the article, it can be said that despite the presence of various challenges to the implementation and usage, the future prospects for AI in healthcare is immense and work needs to be done in order to ensure the availability of resources for same so that high level of accuracy can be achieved and better health outcomes can be provided to patients. Ethical concerns need to be addressed for smooth implementation and to reduce the burden of the developers, which has been discussed in this narrative review article.

Toward Reliable Symptom Coding in Electronic Health Records for Symptom Assessment and Research: Identification and Categorization of International Classification of Diseases, Ninth Revision, Clinical Modification Symptom Codes

To date, symptom documentation has mostly relied on clinical notes in electronic health records or patient-reported outcomes using disease-specific symptom inventories. To provide a common and precise language for symptom recording, assessment, and research, a comprehensive list of symptom codes is needed. The International Classification of Diseases, Ninth Revision or its clinical modification ( International Classification of Diseases, Ninth Revision, Clinical Modification ) has a range of codes designated for symptoms, but it does not contain codes for all possible symptoms, and not all codes in that range are symptom related. This study aimed to identify and categorize the first list of International Classification of Diseases, Ninth Revision, Clinical Modification symptom codes for a general population and demonstrate their use to characterize symptoms of patients with type 2 diabetes mellitus in the Cerner database. A list of potential symptom codes was automatically extracted from the Unified Medical Language System Metathesaurus. Two clinical experts in symptom science and diabetes manually reviewed this list to identify and categorize codes as symptoms. A total of 1888 International Classification of Diseases, Ninth Revision, Clinical Modification symptom codes were identified and categorized into 65 categories. The symptom characterization using the newly obtained symptom codes and categories was found to be more reasonable than that using the previous symptom codes and categories on the same Cerner diabetes cohort.

Using machine learning methods to predict all-cause somatic hospitalizations in adults: A systematic review

AIM

In this review, we investigated how Machine Learning (ML) was utilized to predict all-cause somatic hospital admissions and readmissions in adults.

METHODS

We searched eight databases (PubMed, Embase, Web of Science, CINAHL, ProQuest, OpenGrey, WorldCat, and MedNar) from their inception date to October 2023, and included records that predicted all-cause somatic hospital admissions and readmissions of adults using ML methodology. We used the CHARMS checklist for data extraction, PROBAST for bias and applicability assessment, and TRIPOD for reporting quality.

RESULTS

We screened 7,543 studies of which 163 full-text records were read and 116 met the review inclusion criteria. Among these, 45 predicted admission, 70 predicted readmission, and one study predicted both. There was a substantial variety in the types of datasets, algorithms, features, data preprocessing steps, evaluation, and validation methods. The most used types of features were demographics, diagnoses, vital signs, and laboratory tests. Area Under the ROC curve (AUC) was the most used evaluation metric. Models trained using boosting tree-based algorithms often performed better compared to others. ML algorithms commonly outperformed traditional regression techniques. Sixteen studies used Natural language processing (NLP) of clinical notes for prediction, all studies yielded good results. The overall adherence to reporting quality was poor in the review studies. Only five percent of models were implemented in clinical practice. The most frequently inadequately addressed methodological aspects were: providing model interpretations on the individual patient level, full code availability, performing external validation, calibrating models, and handling class imbalance.

CONCLUSION

This review has identified considerable concerns regarding methodological issues and reporting quality in studies investigating ML to predict hospitalizations. To ensure the acceptability of these models in clinical settings, it is crucial to improve the quality of future studies.

The Use of Deep Learning and Machine Learning on Longitudinal Electronic Health Records for the Early Detection and Prevention of Diseases: Scoping Review

BACKGROUND

Electronic health records (EHRs) contain patients' health information over time, including possible early indicators of disease. However, the increasing amount of data hinders clinicians from using them. There is accumulating evidence suggesting that machine learning (ML) and deep learning (DL) can assist clinicians in analyzing these large-scale EHRs, as algorithms thrive on high volumes of data. Although ML has become well developed, studies mainly focus on engineering but lack medical outcomes.

OBJECTIVE

This study aims for a scoping review of the evidence on how the use of ML on longitudinal EHRs can support the early detection and prevention of disease. The medical insights and clinical benefits that have been generated were investigated by reviewing applications in a variety of diseases.

METHODS

This study was conducted according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A literature search was performed in 2022 in collaboration with a medical information specialist in the following databases: PubMed, Embase, Web of Science Core Collection (Clarivate Analytics), and IEEE Xplore Digital Library and computer science bibliography. Studies were eligible when longitudinal EHRs were used that aimed for the early detection of disease via ML in a prevention context. Studies with a technical focus or using imaging or hospital admission data were beyond the scope of this review. Study screening and selection and data extraction were performed independently by 2 researchers.

RESULTS

In total, 20 studies were included, mainly published between 2018 and 2022. They showed that a variety of diseases could be detected or predicted, particularly diabetes; kidney diseases; diseases of the circulatory system; and mental, behavioral, and neurodevelopmental disorders. Demographics, symptoms, procedures, laboratory test results, diagnoses, medications, and BMI were frequently used EHR data in basic recurrent neural network or long short-term memory techniques. By developing and comparing ML and DL models, medical insights such as a high diagnostic performance, an earlier detection, the most important predictors, and additional health indicators were obtained. A clinical benefit that has been evaluated positively was preliminary screening. If these models are applied in practice, patients might also benefit from personalized health care and prevention, with practical benefits such as workload reduction and policy insights.

CONCLUSIONS

Longitudinal EHRs proved to be helpful for support in health care. Current ML models on EHRs can support the detection of diseases in terms of accuracy and offer preliminary screening benefits. Regarding the prevention of diseases, ML and specifically DL models can accurately predict or detect diseases earlier than current clinical diagnoses. Adding personally responsible factors allows targeted prevention interventions. While ML models based on textual EHRs are still in the developmental stage, they have high potential to support clinicians and the health care system and improve patient outcomes.

Natural Language Processing Accurately Differentiates Cancer Symptom Information in Electronic Health Record Narratives

PURPOSE

Identifying cancer symptoms in electronic health record (EHR) narratives is feasible with natural language processing (NLP). However, more efficient NLP systems are needed to detect various symptoms and distinguish observed symptoms from negated symptoms and medication-related side effects. We evaluated the accuracy of NLP in (1) detecting 14 symptom groups (ie, pain, fatigue, swelling, depressed mood, anxiety, nausea/vomiting, pruritus, headache, shortness of breath, constipation, numbness/tingling, decreased appetite, impaired memory, disturbed sleep) and (2) distinguishing observed symptoms in EHR narratives among patients with cancer.

METHODS

We extracted 902,508 notes for 11,784 unique patients diagnosed with cancer and developed a gold standard corpus of 1,112 notes labeled for presence or absence of 14 symptom groups. We trained an embeddings-augmented NLP system integrating human and machine intelligence and conventional machine learning algorithms. NLP metrics were calculated on a gold standard corpus subset for testing.

RESULTS

The interannotator agreement for labeling the gold standard corpus was excellent at 92%. The embeddings-augmented NLP model achieved the best performance (F1 score = 0.877). The highest NLP accuracy was observed in pruritus (F1 score = 0.937) while the lowest accuracy was in swelling (F1 score = 0.787). After classifying the entire data set with embeddings-augmented NLP, we found that 41% of the notes included symptom documentation. Pain was the most documented symptom (29% of all notes) while impaired memory was the least documented (0.7% of all notes).

CONCLUSION

We illustrated the feasibility of detecting 14 symptom groups in EHR narratives and showed that an embeddings-augmented NLP system outperforms conventional machine learning algorithms in detecting symptom information and differentiating observed symptoms from negated symptoms and medication-related side effects.

A Comprehensive Review of Artificial Intelligence in Prostate Cancer Care: State-of-the-Art Diagnostic Tools and Future Outlook

Prostate cancer remains a significant global health challenge, characterized by high incidence and substantial morbidity and mortality rates. Early detection is critical for improving patient outcomes, yet current diagnostic methods have limitations in accuracy and reliability. Artificial intelligence (AI) has emerged as a promising tool to address these challenges in prostate cancer care. AI technologies, including machine learning algorithms and advanced imaging techniques, offer potential solutions to enhance diagnostic accuracy, optimize treatment strategies, and personalize patient care. This review explores the current landscape of AI applications in prostate cancer diagnostics, highlighting state-of-the-art tools and their clinical implications. By synthesizing recent advancements and discussing future directions, the review underscores the transformative potential of AI in revolutionizing prostate cancer diagnosis and management. Ultimately, integrating AI into clinical practice can potentially improve outcomes and quality of life for patients affected by prostate cancer.

Pre-trained language models in medicine: A survey

With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge achievements in applying PLMs to various medical NLP tasks. Specifically, we first brief PLMS and outline the research of PLMs in medicine. Next, we categorise and discuss the types of tasks in medical NLP, covering text summarisation, question-answering, machine translation, sentiment analysis, named entity recognition, information extraction, medical education, relation extraction, and text mining. For each type of task, we first provide an overview of the basic concepts, the main methodologies, the advantages of applying PLMs, the basic steps of applying PLMs application, the datasets for training and testing, and the metrics for task evaluation. Subsequently, a summary of recent important research findings is presented, analysing their motivations, strengths vs weaknesses, similarities vs differences, and discussing potential limitations. Also, we assess the quality and influence of the research reviewed in this paper by comparing the citation count of the papers reviewed and the reputation and impact of the conferences and journals where they are published. Through these indicators, we further identify the most concerned research topics currently. Finally, we look forward to future research directions, including enhancing models' reliability, explainability, and fairness, to promote the application of PLMs in clinical practice. In addition, this survey also collect some download links of some model codes and the relevant datasets, which are valuable references for researchers applying NLP techniques in medicine and medical professionals seeking to enhance their expertise and healthcare service through AI technology.

Towards the automatic calculation of the EQUAL Candida Score: Extraction of CVC-related information from EMRs of critically ill patients with candidemia in Intensive Care Units

OBJECTIVES

Candidemia is the most frequent invasive fungal disease and the fourth most frequent bloodstream infection in hospitalized patients. Its optimal management is crucial for improving patients' survival. The quality of candidemia management can be assessed with the EQUAL Candida Score. The objective of this work is to support its automatic calculation by extracting central venous catheter-related information from Italian text in clinical notes of electronic medical records.

MATERIALS AND METHODS

The sample includes 4,787 clinical notes of 108 patients hospitalized between January 2018 to December 2020 in the Intensive Care Units of the IRCCS San Martino Polyclinic Hospital in Genoa (Italy). The devised pipeline exploits natural language processing (NLP) to produce numerical representations of clinical notes used as input of machine learning (ML) algorithms to identify CVC presence and removal. It compares the performances of (i) rule-based method, (ii) count-based method together with a ML algorithm, and (iii) a transformers-based model.

RESULTS

Results, obtained with three different approaches, were evaluated in terms of weighted F1 Score. The random forest classifier showed the higher performance in both tasks reaching 82.35%.

CONCLUSION

The present work constitutes a first step towards the automatic calculation of the EQUAL Candida Score from unstructured daily collected data by combining ML and NLP methods. The automatic calculation of the EQUAL Candida Score could provide crucial real-time feedback on the quality of candidemia management, aimed at further improving patients' health.

Applied artificial intelligence in dentistry: emerging data modalities and modeling approaches

Artificial intelligence (AI) is increasingly applied across all disciplines of medicine, including dentistry. Oral health research is experiencing a rapidly increasing use of machine learning (ML), the branch of AI that identifies inherent patterns in data similarly to how humans learn. In contemporary clinical dentistry, ML supports computer-aided diagnostics, risk stratification, individual risk prediction, and decision support to ultimately improve clinical oral health care efficiency, outcomes, and reduce disparities. Further, ML is progressively used in dental and oral health research, from basic and translational science to clinical investigations. With an ML perspective, this review provides a comprehensive overview of how dental medicine leverages AI for diagnostic, prognostic, and generative tasks. The spectrum of available data modalities in dentistry and their compatibility with various methods of applied AI are presented. Finally, current challenges and limitations as well as future possibilities and considerations for AI application in dental medicine are summarized.

LSTM-Based Prediction Model for Tuberculosis Among HIV-Infected Patients Using Structured Electronic Medical Records: A Retrospective Machine Learning Study

Background

Both HIV and TB are chronic infectious diseases requiring long-term treatment and follow-up, resulting in extensive electronic medical records. With the exponential growth of health and medical big data, effectively extracting and analyzing these data has become the research hotspot. As a fundamental aspect of artificial intelligence, machine learning has been extensively applied in medical research, encompassing diagnosis, treatment, patient monitoring, drug development, and epidemiological investigations. This significantly enhances medical information systems and facilitates the interoperability of medical data.

Methods

In our study, we analyzed longitudinal data from the electronic health records of 4540 patients, gathered from the National Clinical Research Center for Infectious Diseases in Shenzhen, China, spanning from 2017 to 2021. Initially, we employed the fine-tuned ChatGLM to structure the electronic medical records. Subsequently, we utilized a multi-layer perceptron to classify each patient and determined the presence of tuberculosis in HIV patients. Using machine learning-based natural language processing, we structured these records to build a specialized database for HIV and TB co-infection. We studied the epidemiological characteristics, focusing on incidence patterns, patient characteristics, and influencing factors, to uncover the transmission characteristics of these diseases in Shenzhen. Additionally, we used Long Short-Term Memory to create a predictive model for TB co-infection among HIV patients, based on their medical records. This model predicted the risk of TB co-infection, providing scientific evidence for clinical decision-making and enabling early detection and precise intervention.

Results

Based on the refined ChatGLM model tailored for structured electronic health records, the accuracy of symptom extraction consistently surpassed 0.95 precision. Key symptoms such as diarrhea and normal showed precision rates exceeding 0.90. High scores were also achieved in recall and F1 scores. Among 4540 HIV patients, 758 were diagnosed with concurrent tuberculosis, indicating a 16.7% co-infection rate, while syphilis co-infection affected 25.1%, underscoring the prevalence of concurrent infections among HIV patients. Utilizing electronic health records, a Multilayer Perceptron classifier was developed as a benchmark against Long Short-Term Memory to predict high-risk groups for HIV and tuberculosis co-infections. The Multilayer Perceptron classifier demonstrated predictive ability with AUROC values ranging from 0.616 to 0.682 on the test set, suggesting opportunities for further optimization and generalization despite its accuracy in identifying HIV-TB co-infections. In tuberculosis intelligent diagnosis based on laboratory results, the Long Short-Term Memory showed consistent performance across 5-fold cross-validation, with AUROC values ranging from 0.827 to 0.850, indicating reliability and consistency in tuberculosis prediction. Furthermore, by optimizing classification thresholds, the model achieved an overall accuracy of 81.18% in distinguishing HIV co-infected tuberculosis from simple HIV infection.

Conclusion

Combining the Multilayer Perceptron classifier with Long Short-Term Memory represented an advanced approach for effectively extracting electronic health records and utilizing it for disease prediction. This underscored the superior performance of deep learning techniques in managing both structured and unstructured medical data. Models leveraging laboratory time-series data demonstrated notably better performance compared to those relying solely on electronic health records for predicting tuberculosis incidence. This emphasized the benefits of deep learning in handling intricate medical data and provided valuable insights for healthcare providers exploring the use of deep learning in disease prediction and management.

The use of natural language processing for the identification of ageing syndromes including sarcopenia, frailty and falls in electronic healthcare records: a systematic review

BACKGROUND

Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear.

METHODS

We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines.

RESULTS

From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms.

CONCLUSIONS

Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported.

Natural Language Processing Algorithm to Extract Multiple Myeloma Stage From Oncology Notes in the Veterans Affairs Healthcare System

PURPOSE

Stage in multiple myeloma (MM) is an essential measure of disease risk, but its measurement in large databases is often lacking. We aimed to develop and validate a natural language processing (NLP) algorithm to extract oncologists' documentation of stage in the national Veterans Affairs (VA) Healthcare System.

METHODS

Using nationwide electronic health record (EHR) and cancer registry data from the VA Corporate Data Warehouse, we developed and validated a rule-based NLP algorithm to extract oncologist-determined MM stage. To that end, a clinician annotated MM stage within over 5,000 short snippets of clinical notes, and annotated MM stage at MM treatment initiation for 200 patients. These were allocated into snippet- and patient-level development and validation sets. We developed MM stage extraction and roll-up algorithms within the development sets. After the algorithms were finalized, we validated them using standard measures in held-out validation sets.

RESULTS

We developed algorithms for three different MM staging systems that have been in widespread use (Revised International Staging System [R-ISS], International Staging System [ISS], and Durie-Salmon [DS]) and for stage reported without a clearly defined system. Precision and recall were uniformly high for MM stage at the snippet level, ranging from 0.92 to 0.99 for the different MM staging systems. Performance in identifying for MM stage at treatment initiation at the patient level was also excellent, with precision of 0.92, 0.96, 0.90, and 0.86 and recall of 0.99, 0.98, 0.94, and 0.92 for R-ISS, ISS, DS, and unclear stage, respectively.

CONCLUSION

Our MM stage extraction algorithm uses rule-based NLP and data aggregation to accurately measure MM stage documented in oncology notes and pathology reports in VA's national EHR system. It may be adapted to other systems where MM stage is recorded in clinical notes.

Using natural language processing in emergency medicine health service research: A systematic review and meta-analysis

OBJECTIVES

Natural language processing (NLP) represents one of the adjunct technologies within artificial intelligence and machine learning, creating structure out of unstructured data. This study aims to assess the performance of employing NLP to identify and categorize unstructured data within the emergency medicine (EM) setting.

METHODS

We systematically searched publications related to EM research and NLP across databases including MEDLINE, Embase, Scopus, CENTRAL, and ProQuest Dissertations & Theses Global. Independent reviewers screened, reviewed, and evaluated article quality and bias. NLP usage was categorized into syndromic surveillance, radiologic interpretation, and identification of specific diseases/events/syndromes, with respective sensitivity analysis reported. Performance metrics for NLP usage were calculated and the overall area under the summary of receiver operating characteristic curve (SROC) was determined.

RESULTS

A total of 27 studies underwent meta-analysis. Findings indicated an overall mean sensitivity (recall) of 82%-87%, specificity of 95%, with the area under the SROC at 0.96 (95% CI 0.94-0.98). Optimal performance using NLP was observed in radiologic interpretation, demonstrating an overall mean sensitivity of 93% and specificity of 96%.

CONCLUSIONS

Our analysis revealed a generally favorable performance accuracy in using NLP within EM research, particularly in the realm of radiologic interpretation. Consequently, we advocate for the adoption of NLP-based research to augment EM health care management.