A survey on clinical natural language processing in the United Kingdom from 2007 to 2022

Low-cost algorithms for clinical notes phenotype classification to enhance epidemiological surveillance: A case study

OBJECTIVE

Our study aims to enhance epidemic intelligence through event-based surveillance in an emerging pandemic context. We classified electronic health records (EHRs) from La Rioja, Argentina, focusing on predicting COVID-19-related categories in a scenario with limited disease knowledge, evolving symptoms, non-standardized coding practices, and restricted training data due to privacy issues.

METHODS

Using natural language processing techniques, we developed rapid, cost-effective methods suitable for implementation with limited resources. We annotated a corpus for training and testing classification models, ranging from simple logistic regression to more complex fine-tuned transformers.

RESULTS

The transformer-based, Spanish-adapted models BETO Clínico and RoBERTa Clínico, further pre-trained with an unannotated portion of our corpus, were the best-performing models (F1= 88.13% and 87.01%). A simple logistic regression (LR) model ranked third (F1=85.09%), outperforming more complex models like XGBoost and BiLSTM. Data classified as COVID-confirmed using LR and BETO Clínico exhibit stronger time-series Pearson correlation with official COVID-19 case counts from the National Health Surveillance System (SNVS 2.0) in La Rioja province compared to the correlations observed between the International Code of Diseases (ICD-10) codes and the SNVS 2.0 data (0.840, 0.873, and 0.663, p-values ≤3×10-7). Both models have a good Pearson correlation with ICD-10 codes assigned to the clinical notes for confirmed (0.940 and 0.902) and for suspected cases (0.960 and 0.954), p-values ≤1.7×10-18.

CONCLUSION

This study shows that simple, resource-efficient methods can achieve results comparable to complex approaches. BETO Clínico and LR strongly correlate with official data, revealing uncoded confirmed cases at the pandemic's onset. Our results suggest that annotating a smaller set of EHRs and training a simple model may be more cost-effective than manual coding. This points to potentially efficient strategies in public health emergencies, particularly in resource-limited settings, and provides valuable insights for future epidemic response efforts.

Natural language processing techniques applied to the electronic health record in clinical research and practice - an introduction to methodologies

Natural Language Processing (NLP) has the potential to revolutionise clinical research utilising Electronic Health Records (EHR) through the automated analysis of unstructured free text. Despite this potential, relatively few applications have entered real-world clinical practice. This paper aims to introduce the whole pipeline of NLP methodologies for EHR analysis to the clinical researcher, with case studies to demonstrate the application of these methods in the existing literature. Essential pre-processing steps are introduced, followed by the two major classes of analytical frameworks: statistical methods and Artificial Neural Networks (ANNs). Case studies which apply statistical and ANN-based methods are then provided and discussed, illustrating information extraction tasks for objective and subjective information, and classification/prediction tasks using supervised and unsupervised approaches. State-of-the-art large language models and future directions for research are then discussed. This educational article aims to bridge the gap between the clinical researcher and the NLP expert, providing clinicians with a background understanding of the NLP techniques relevant to EHR analysis, allowing engagement with this rapidly evolving area of research, which is likely to have a major impact on clinical practice in coming years.

LEVERAGING UNSTRUCTURED DATA IN ELECTRONIC HEALTH RECORDS TO DETECT ADVERSE EVENTS FROM PEDIATRIC DRUG USE - A SCOPING REVIEW

Adverse drug events (ADEs) in pediatric populations pose significant public health challenges, yet research on their detection and monitoring remains limited. This scoping review evaluates the use of unstructured data from electronic health records (EHRs) to identify ADEs in children. We searched six databases, including MEDLINE, Embase and IEEE Xplore, in September 2024. From 984 records, only nine studies met our inclusion criteria, indicating a significant gap in research towards identify ADEs in children. We found that unstructured data in EHRs can indeed be of value and enhance pediatric pharmacovigilance, although its use has been so far very limited. Traditional Natural Language Processing (NLP) methods have been employed to extract ADEs, but the approaches utilized face challenges in generalizability and context interpretation. These challenges could be addressed with recent advances in transformer-based models and large language models (LLMs), unlocking the use of EHR data at scale for pediatric pharmacovigilance.

The foundational capabilities of large language models in predicting postoperative risks using clinical notes

Clinical notes recorded during a patient's perioperative journey holds immense informational value. Advances in large language models (LLMs) offer opportunities for bridging this gap. Using 84,875 preoperative notes and its associated surgical cases from 2018 to 2021, we examine the performance of LLMs in predicting six postoperative risks using various fine-tuning strategies. Pretrained LLMs outperformed traditional word embeddings by an absolute AUROC of 38.3% and AUPRC of 33.2%. Self-supervised fine-tuning further improved performance by 3.2% and 1.5%. Incorporating labels into training further increased AUROC by 1.8% and AUPRC by 2%. The highest performance was achieved with a unified foundation model, with improvements of 3.6% for AUROC and 2.6% for AUPRC compared to self-supervision, highlighting the foundational capabilities of LLMs in predicting postoperative risks, which could be potentially beneficial when deployed for perioperative care.

Clinical entity augmented retrieval for clinical information extraction

Large language models (LLMs) with retrieval-augmented generation (RAG) have improved information extraction over previous methods, yet their reliance on embeddings often leads to inefficient retrieval. We introduce CLinical Entity Augmented Retrieval (CLEAR), a RAG pipeline that retrieves information using entities. We compared CLEAR to embedding RAG and full-note approaches for extracting 18 variables using six LLMs across 20,000 clinical notes. Average F1 scores were 0.90, 0.86, and 0.79; inference times were 4.95, 17.41, and 20.08 s per note; average model queries were 1.68, 4.94, and 4.18 per note; and average input tokens were 1.1k, 3.8k, and 6.1k per note for CLEAR, embedding RAG, and full-note approaches, respectively. In conclusion, CLEAR utilizes clinical entities for information retrieval and achieves >70% reduction in token usage and inference time with improved performance compared to modern methods.

Current Status and Future of Artificial Intelligence in Medicine

Artificial intelligence (AI) has rapidly emerged as a transformative force in medicine, revolutionizing various aspects of healthcare from diagnostics and treatment to public health and patient care. This narrative review synthesizes evidence from diverse study designs, exploring the current and future applications of AI in medicine. We highlight AI's role in improving diagnostic accuracy, optimizing treatment strategies, and enhancing patient care through personalized interventions and remote monitoring, drawing upon recent advancements and landmark studies. Emerging trends such as explainable AI and federated learning are also examined. While acknowledging the tremendous potential of AI in medicine, the review also addresses the barriers and ethical challenges that need to be overcome, including concerns about algorithmic bias, transparency, over-reliance, and the potential impact on the healthcare workforce. We emphasize the importance of establishing regulatory guidelines, fostering collaboration between clinicians and AI developers, and ensuring ongoing education for healthcare professionals. Despite these challenges, the future of AI in medicine holds immense promise, with the potential to significantly improve patient outcomes, transform healthcare delivery, and address healthcare disparities.

Assessing domain adaptation in adverse drug event extraction on real-world breast cancer records

BACKGROUND

Adverse Drug Events (ADE) are key information present in unstructured portions of Electronic Health Records. These pose a significant challenge in healthcare, ranging from mild discomfort to severe complications, and can impact patient safety and treatment outcomes.

METHODS

We explore the influence of domain shift between a set of dummy clinical notes and a real-world hospital corpus of Japanese clinical notes of breast cancer treatment when extracting ADEs from free text. We annotated a subset of the hospital dataset and used it to fine-tune a Named Entity Recognition (NER) model, initially trained with the set of dummy documents. We used increasing amounts of the annotated data and evaluated the impact on the model's performance. Additionally, we examined the extracted information to identify combinations of drugs that are likely to cause ADEs.

RESULTS

We show that domain adaptation can significantly improve model performance in the new domain, as by feeding a small subset of 100 documents for the fine-tuning process we saw a 40% improvement in model performance. However, we also noticed diminishing returns when fine-tuning the model with a larger dataset. For instance, by feeding eight times more data, we only saw further 18% improvement in extraction performance.

CONCLUSION

While variations in writing style and vocabulary in clinical corpora can significantly impact the quality of NER results. We show that domain adaptation can be of great aid in mitigating these discrepancies and achieving better performance. Yet, while providing in-domain data to a model helps, there are diminishing returns when fine-tuning with large amounts of data.

Multifaceted Natural Language Processing Task-Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation

Background

The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model's comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non-English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes.

Objective

In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents.

Methods

Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks.

Results

The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F1-score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F1-score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41.

Conclusions

This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.

A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports

OBJECTIVE

Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations.

MATERIALS AND METHODS

We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model.

RESULTS

Across all 12 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, LSTM-Att (average macro F1-score of 0.86 vs 0.75), with advantage on tasks with high label imbalance. Other LLMs demonstrated poor performance. Frequent GPT-4 error categories included incorrect inferences from multiple samples and from history, and complex task design, and several LSTM-Att errors were related to poor generalization to the test set.

DISCUSSION

On tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of data labeling. However, if the use of LLMs is prohibitive, the use of simpler models with large annotated datasets can provide comparable results.

CONCLUSIONS

GPT-4 demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in clinical studies.

A review of reinforcement learning for natural language processing and applications in healthcare

IMPORTANCE

Reinforcement learning (RL) represents a pivotal avenue within natural language processing (NLP), offering a potent mechanism for acquiring optimal strategies in task completion. This literature review studies various NLP applications where RL has demonstrated efficacy, with notable applications in healthcare settings.

OBJECTIVES

To systematically explore the applications of RL in NLP, focusing on its effectiveness in acquiring optimal strategies, particularly in healthcare settings, and provide a comprehensive understanding of RL's potential in NLP tasks.

MATERIALS AND METHODS

Adhering to the PRISMA guidelines, an exhaustive literature review was conducted to identify instances where RL has exhibited success in NLP applications, encompassing dialogue systems, machine translation, question-answering, text summarization, and information extraction. Our methodological approach involves closely examining the technical aspects of RL methodologies employed in these applications, analyzing algorithms, states, rewards, actions, datasets, and encoder-decoder architectures.

RESULTS

The review of 93 papers yields insights into RL algorithms, prevalent techniques, emergent trends, and the fusion of RL methods in NLP healthcare applications. It clarifies the strategic approaches employed, datasets utilized, and the dynamic terrain of RL-NLP systems, thereby offering a roadmap for research and development in RL and machine learning techniques in healthcare. The review also addresses ethical concerns to ensure equity, transparency, and accountability in the evolution and application of RL-based NLP technologies, particularly within sensitive domains such as healthcare.

DISCUSSION

The findings underscore the promising role of RL in advancing NLP applications, particularly in healthcare, where its potential to optimize decision-making and enhance patient outcomes is significant. However, the ethical challenges and technical complexities associated with RL demand careful consideration and ongoing research to ensure responsible and effective implementation.

CONCLUSIONS

By systematically exploring RL's applications in NLP and providing insights into technical analysis, ethical implications, and potential advancements, this review contributes to a deeper understanding of RL's role for language processing.

Artificial Intelligence in Head and Neck Cancer: Innovations, Applications, and Future Directions

Artificial intelligence (AI) is revolutionizing head and neck cancer (HNC) care by providing innovative tools that enhance diagnostic accuracy and personalize treatment strategies. This review highlights the advancements in AI technologies, including deep learning and natural language processing, and their applications in HNC. The integration of AI with imaging techniques, genomics, and electronic health records is explored, emphasizing its role in early detection, biomarker discovery, and treatment planning. Despite noticeable progress, challenges such as data quality, algorithmic bias, and the need for interdisciplinary collaboration remain. Emerging innovations like explainable AI, AI-powered robotics, and real-time monitoring systems are poised to further advance the field. Addressing these challenges and fostering collaboration among AI experts, clinicians, and researchers is crucial for developing equitable and effective AI applications. The future of AI in HNC holds significant promise, offering potential breakthroughs in diagnostics, personalized therapies, and improved patient outcomes.

From admission to discharge: a systematic review of clinical natural language processing along the patient journey

BACKGROUND

Medical text, as part of an electronic health record, is an essential information source in healthcare. Although natural language processing (NLP) techniques for medical text are developing fast, successful transfer into clinical practice has been rare. Especially the hospital domain offers great potential while facing several challenges including many documents per patient, multiple departments and complex interrelated processes.

METHODS

In this work, we survey relevant literature to identify and classify approaches which exploit NLP in the clinical context. Our contribution involves a systematic mapping of related research onto a prototypical patient journey in the hospital, along which medical documents are created, processed and consumed by hospital staff and patients themselves. Specifically, we reviewed which dataset types, dataset languages, model architectures and tasks are researched in current clinical NLP research. Additionally, we extract and analyze major obstacles during development and implementation. We discuss options to address them and argue for a focus on bias mitigation and model explainability.

RESULTS

While a patient's hospital journey produces a significant amount of structured and unstructured documents, certain steps and documents receive more research attention than others. Diagnosis, Admission and Discharge are clinical patient steps that are researched often across the surveyed paper. In contrast, our findings reveal significant under-researched areas such as Treatment, Billing, After Care, and Smart Home. Leveraging NLP in these stages can greatly enhance clinical decision-making and patient outcomes. Additionally, clinical NLP models are mostly based on radiology reports, discharge letters and admission notes, even though we have shown that many other documents are produced throughout the patient journey. There is a significant opportunity in analyzing a wider range of medical documents produced throughout the patient journey to improve the applicability and impact of NLP in healthcare.

CONCLUSIONS

Our findings suggest that there is a significant opportunity to leverage NLP approaches to advance clinical decision-making systems, as there remains a considerable understudied potential for the analysis of patient journey data.

Narrative review of recent developments and the future of penicillin allergy de-labelling by non-allergists

This article outlines recent developments in non-allergist delivered penicillin allergy de-labelling (PADL), discusses remaining controversies and uncertainties and explores the future for non-allergist delivered PADL. Recent developments include national guidelines for non-allergist delivered PADL and validation of penicillin allergy risk assessment tools. Controversies remain on which penicillin allergy features are low risk of genuine allergy. In the future genetic or immunological tests may facilitate PADL.

The use of natural language processing for the identification of ageing syndromes including sarcopenia, frailty and falls in electronic healthcare records: a systematic review

BACKGROUND

Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear.

METHODS

We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines.

RESULTS

From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms.

CONCLUSIONS

Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported.

A critical assessment of using ChatGPT for extracting structured data from clinical notes

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT's capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral "prompt engineering" process, leveraging OpenAI's API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification

Although supervised machine learning is popular for information extraction from clinical notes, creating large, annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs can reduce the need for large-scale data annotations. We curated a manually labeled dataset of 769 breast cancer pathology reports, labeled with 13 categories, to compare zero-shot classification capability of the GPT-4 model and the GPT-3.5 model with supervised classification performance of three model architectures: random forests classifier, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model. Across all 13 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, the LSTM-Att model (average macro F1 score of 0.83 vs. 0.75). On tasks with a high imbalance between labels, the differences were more prominent. Frequent sources of GPT-4 errors included inferences from multiple samples and complex task design. On complex tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of large-scale data labeling. However, if the use of LLMs is prohibitive, the use of simpler supervised models with large annotated datasets can provide comparable results. LLMs demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for curating large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in observational clinical studies.

A pilot randomised controlled trial exploring the feasibility and efficacy of a human-AI sleep coaching model for improving sleep among university students

Objective

Sleep quality is a crucial concern, particularly among youth. The integration of health coaching with question-answering (QA) systems presents the potential to foster behavioural changes and enhance health outcomes. This study proposes a novel human-AI sleep coaching model, combining health coaching by peers and a QA system, and assesses its feasibility and efficacy in improving university students' sleep quality.

Methods

In a four-week unblinded pilot randomised controlled trial, 59 university students (mean age: 21.9; 64% males) were randomly assigned to the intervention (health coaching and QA system; n = 30) or the control conditions (QA system; n = 29). Outcomes included efficacy of the intervention on sleep quality (Pittsburgh Sleep Quality Index; PSQI), objective and self-reported sleep measures (obtained from Fitbit and sleep diaries) and feasibility of the study procedures and the intervention.

Results

Analysis revealed no significant differences in sleep quality (PSQI) between intervention and control groups (adjusted mean difference = -0.51, 95% CI: [-1.55-0.77], p = 0.40). The intervention group demonstrated significant improvements in Fitbit measures of total sleep time (adjusted mean difference = 32.5, 95% CI: [5.9-59.1], p = 0.02) and time in bed (adjusted mean difference = 32.3, 95% CI: [2.7-61.9], p = 0.03) compared to the control group, although other sleep measures were insignificant. Adherence was high, with the majority of the intervention group attending all health coaching sessions. Most participants completed baseline and post-intervention self-report measures, all diary entries, and consistently wore Fitbits during sleep.

Conclusions

The proposed model showed improvements in specific sleep measures for university students and the feasibility of the study procedures and intervention. Future research may extend the intervention period to see substantive sleep quality improvements.

ChatGPT, GPT-4, and Other Large Language Models: The Next Revolution for Clinical Microbiology?

ChatGPT, GPT-4, and Bard are highly advanced natural language process-based computer programs (chatbots) that simulate and process human conversation in written or spoken form. Recently released by the company OpenAI, ChatGPT was trained on billions of unknown text elements (tokens) and rapidly gained wide attention for its ability to respond to questions in an articulate manner across a wide range of knowledge domains. These potentially disruptive large language model (LLM) technologies have a broad range of conceivable applications in medicine and medical microbiology. In this opinion article, I describe how chatbot technologies work and discuss the strengths and weaknesses of ChatGPT, GPT-4, and other LLMs for applications in the routine diagnostic laboratory, focusing on various use cases for the pre- to post-analytical process.

GERNERMED++: Semantic annotation in German medical NLP through transfer-learning, translation and word alignment

We present a statistical model, GERNERMED++, for German medical natural language processing trained for named entity recognition (NER) as an open, publicly available model. We demonstrate the effectiveness of combining multiple techniques in order to achieve strong results in entity recognition performance by the means of transfer-learning on pre-trained deep language models (LM), word-alignment and neural machine translation, outperforming a pre-existing baseline model on several datasets. Due to the sparse situation of open, public medical entity recognition models for German texts, this work offers benefits to the German research community on medical NLP as a baseline model. The work serves as a refined successor to our first GERNERMED model. Similar to our previous work, our trained model is publicly available to other researchers. The sample code and the statistical model is available at: https://github.com/frankkramer-lab/GERNERMED-pp.

A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study

BACKGROUND

Natural language processing (NLP) models such as bidirectional encoder representations from transformers (BERT) hold promise in revolutionizing disease identification from electronic health records (EHRs) by potentially enhancing efficiency and accuracy. However, their practical application in practice settings demands a comprehensive and multidisciplinary approach to development and validation. The COVID-19 pandemic highlighted challenges in disease identification due to limited testing availability and challenges in handling unstructured data. In the Netherlands, where general practitioners (GPs) serve as the first point of contact for health care, EHRs generated by these primary care providers contain a wealth of potentially valuable information. Nonetheless, the unstructured nature of free-text entries in EHRs poses challenges in identifying trends, detecting disease outbreaks, or accurately pinpointing COVID-19 cases.

OBJECTIVE

This study aims to develop and validate a BERT model for detecting COVID-19 consultations in general practice EHRs in the Netherlands.

METHODS

The BERT model was initially pretrained on Dutch language data and fine-tuned using a comprehensive EHR data set comprising confirmed COVID-19 GP consultations and non-COVID-19-related consultations. The data set was partitioned into a training and development set, and the model's performance was evaluated on an independent test set that served as the primary measure of its effectiveness in COVID-19 detection. To validate the final model, its performance was assessed through 3 approaches. First, external validation was applied on an EHR data set from a different geographic region in the Netherlands. Second, validation was conducted using results of polymerase chain reaction (PCR) test data obtained from municipal health services. Lastly, correlation between predicted outcomes and COVID-19-related hospitalizations in the Netherlands was assessed, encompassing the period around the outbreak of the pandemic in the Netherlands, that is, the period before widespread testing.

RESULTS

The model development used 300,359 GP consultations. We developed a highly accurate model for COVID-19 consultations (accuracy 0.97, F1-score 0.90, precision 0.85, recall 0.85, specificity 0.99). External validations showed comparable high performance. Validation on PCR test data showed high recall but low precision and specificity. Validation using hospital data showed significant correlation between COVID-19 predictions of the model and COVID-19-related hospitalizations (F1-score 96.8; P<.001; R2=0.69). Most importantly, the model was able to predict COVID-19 cases weeks before the first confirmed case in the Netherlands.

CONCLUSIONS

The developed BERT model was able to accurately identify COVID-19 cases among GP consultations even preceding confirmed cases. The validated efficacy of our BERT model highlights the potential of NLP models to identify disease outbreaks early, exemplifying the power of multidisciplinary efforts in harnessing technology for disease identification. Moreover, the implications of this study extend beyond COVID-19 and offer a blueprint for the early recognition of various illnesses, revealing that such models could revolutionize disease surveillance.

Understanding the performance and reliability of NLP tools: a comparison of four NLP tools predicting stroke phenotypes in radiology reports

Background

Natural language processing (NLP) has the potential to automate the reading of radiology reports, but there is a need to demonstrate that NLP methods are adaptable and reliable for use in real-world clinical applications.

Methods

We tested the F1 score, precision, and recall to compare NLP tools on a cohort from a study on delirium using images and radiology reports from NHS Fife and a population-based cohort (Generation Scotland) that spans multiple National Health Service health boards. We compared four off-the-shelf rule-based and neural NLP tools (namely, EdIE-R, ALARM+, ESPRESSO, and Sem-EHR) and reported on their performance for three cerebrovascular phenotypes, namely, ischaemic stroke, small vessel disease (SVD), and atrophy. Clinical experts from the EdIE-R team defined phenotypes using labelling techniques developed in the development of EdIE-R, in conjunction with an expert researcher who read underlying images.

Results

EdIE-R obtained the highest F1 score in both cohorts for ischaemic stroke, ≥93%, followed by ALARM+, ≥87%. The F1 score of ESPRESSO was ≥74%, whilst that of Sem-EHR is ≥66%, although ESPRESSO had the highest precision in both cohorts, 90% and 98%. For F1 scores for SVD, EdIE-R scored ≥98% and ALARM+ ≥90%. ESPRESSO scored lowest with ≥77% and Sem-EHR ≥81%. In NHS Fife, F1 scores for atrophy by EdIE-R and ALARM+ were 99%, dropping in Generation Scotland to 96% for EdIE-R and 91% for ALARM+. Sem-EHR performed lowest for atrophy at 89% in NHS Fife and 73% in Generation Scotland. When comparing NLP tool output with brain image reads using F1 scores, ALARM+ scored 80%, outperforming EdIE-R at 66% in ischaemic stroke. For SVD, EdIE-R performed best, scoring 84%, with Sem-EHR 82%. For atrophy, EdIE-R and both ALARM+ versions were comparable at 80%.

Conclusions

The four NLP tools show varying F1 (and precision/recall) scores across all three phenotypes, although more apparent for ischaemic stroke. If NLP tools are to be used in clinical settings, this cannot be performed "out of the box." It is essential to understand the context of their development to assess whether they are suitable for the task at hand or whether further training, re-training, or modification is required to adapt tools to the target task.

Comprehensive Review of Natural Language Processing (NLP) in Vascular Surgery

Objective

The use of Natural Language Processing (NLP) has attracted increased interest in healthcare with various potential applications including identification and extraction of health information, development of chatbots and virtual assistants. The aim of this comprehensive literature review was to provide an overview of NLP applications in vascular surgery, identify current limitations, and discuss future perspectives in the field.

Data sources

The MEDLINE database was searched on April 2023.

Review methods

The database was searched using a combination of keywords to identify studies reporting the use of NLP and chatbots in three main vascular diseases. Keywords used included Natural Language Processing, chatbot, chatGPT, aortic disease, carotid, peripheral artery disease, vascular, and vascular surgery.

Results

Given the heterogeneity of study design, techniques, and aims, a comprehensive literature review was performed to provide an overview of NLP applications in vascular surgery. By enabling identification and extraction of information on patients with vascular diseases, such technology could help to analyse data from healthcare information systems to provide feedback on current practice and help in optimising patient care. In addition, chatbots and NLP driven techniques have the potential to be used as virtual assistants for both health professionals and patients.

Conclusion

While Artificial Intelligence and NLP technology could be used to enhance care for patients with vascular diseases, many challenges remain including the need to define guidelines and clear consensus on how to evaluate and validate these innovations before their implementation into clinical practice.

Natural Language Processing for Literature Search in Vascular Surgery: A Pilot Study Testing an Artificial Intelligence Based Application

Introduction

The use of natural language processing (NLP) for a literature search has been poorly investigated in vascular surgery so far. The aim of this pilot study was to test the applicability of an artificial intelligence (AI) based mobile application for literature searching in a topic related to vascular surgery.

Technique

A focused scientific question was defined to evaluate the performance of the AI application for a literature search and compare the results with the ground truth provided via a traditional literature search performed by human experts. Using pre-defined keywords, the literature search was performed automatically by the AI application through different steps, including quality assessment based on evaluation of the information available and quality filters using indicators of level of evidence, selection of publications based on relevancy filters using NLP, summarisation, and visualisation of the publications via the mobile app. A traditional literature search performed by human experts required 10 hours to check 154 original articles, among which 26 (16.9%) were truly related to the question, 63 (40.9%) related to the field but not to the specific question, and 65 (42.2%) were unrelated. The AI based search was performed in less than one hour, and, compared with traditional search, the method identified 17 original articles (48.6%) truly related to the question (p < .010), 18 (51.4%) related to the field but not to the specific question (p = .26), and no unrelated publications (p < .001). Fifteen truly related articles (88.2%) were identified jointly by the two methods. No significant difference was observed regarding the median number of citations, year of publications, and impact factor of journals.

Discussion

The AI based method enabled a targeted, focused, and time saving literature search, although the selection of publications was not completely exhaustive. These results suggest that such an AI driven application is a complementary tool to help researchers and clinicians for continuous education and dissemination of knowledge.

Digital Transformation of Cancer Care in the Era of Big Data, Artificial Intelligence and Data-Driven Interventions: Navigating the Field

OBJECTIVES

To navigate the field of digital cancer care and define and discuss key aspects and applications of big data analytics, artificial intelligence (AI), and data-driven interventions.

DATA SOURCES

Peer-reviewed scientific publications and expert opinion.

CONCLUSION

The digital transformation of cancer care, enabled by big data analytics, AI, and data-driven interventions, presents a significant opportunity to revolutionize the field. An increased understanding of the lifecycle and ethics of data-driven interventions will enhance development of innovative and applicable products to advance digital cancer care services.

IMPLICATIONS FOR NURSING PRACTICE

As digital technologies become integrated into cancer care, nurse practitioners and scientists will be required to increase their knowledge and skills to effectively use these tools to the patient's benefit. An enhanced understanding of the core concepts of AI and big data, confident use of digital health platforms, and ability to interpret the outputs of data-driven interventions are key competencies. Nurses in oncology will play a crucial role in patient education around big data and AI, with a focus on addressing any arising questions, concerns, or misconceptions to foster trust in these technologies. Successful integration of data-driven innovations into oncology nursing practice will empower practitioners to deliver more personalized, effective, and evidence-based care.

CoVEffect: interactive system for mining the effects of SARS-CoV-2 mutations and variants based on deep learning

BACKGROUND

Literature about SARS-CoV-2 widely discusses the effects of variations that have spread in the past 3 years. Such information is dispersed in the texts of several research articles, hindering the possibility of practically integrating it with related datasets (e.g., millions of SARS-CoV-2 sequences available to the community). We aim to fill this gap, by mining literature abstracts to extract-for each variant/mutation-its related effects (in epidemiological, immunological, clinical, or viral kinetics terms) with labeled higher/lower levels in relation to the nonmutated virus.

RESULTS

The proposed framework comprises (i) the provisioning of abstracts from a COVID-19-related big data corpus (CORD-19) and (ii) the identification of mutation/variant effects in abstracts using a GPT2-based prediction model. The above techniques enable the prediction of mutations/variants with their effects and levels in 2 distinct scenarios: (i) the batch annotation of the most relevant CORD-19 abstracts and (ii) the on-demand annotation of any user-selected CORD-19 abstract through the CoVEffect web application (http://gmql.eu/coveffect), which assists expert users with semiautomated data labeling. On the interface, users can inspect the predictions and correct them; user inputs can then extend the training dataset used by the prediction model. Our prototype model was trained through a carefully designed process, using a minimal and highly diversified pool of samples.

CONCLUSIONS

The CoVEffect interface serves for the assisted annotation of abstracts, allowing the download of curated datasets for further use in data integration or analysis pipelines. The overall framework can be adapted to resolve similar unstructured-to-structured text translation tasks, which are typical of biomedical domains.