Development and multicenter validation of chest X-ray radiography interpretations based on natural language processing

Knowledge Graph-Based Few-Shot Learning for Label of Medical Imaging Reports

BACKGROUND

The application of artificial intelligence (AI) in the field of automatic imaging report labeling faces the challenge of manually labeling large datasets.

PURPOSE

To propose a data augmentation method by using knowledge graph (KG) and few-shot learning.

METHODS

A KG of lumbar spine X-ray images was constructed, and 2000 data were annotated based on the KG, which were divided into training, validation, and test sets in a ratio of 7:2:1. The training dataset was augmented based on the synonym/replacement attributes of the KG and was the augmented data was input into the BERT (Bidirectional Encoder Representations from Transformers) model for automatic annotation training. The performance of the model under different augmentation ratios (1:10, 1:100, 1:1000) and augmentation methods (synonyms only, replacements only, combination of synonyms and replacements) was evaluated using the precision and F1 scores. In addition, with the augmentation ratio was fixed, iterative experiments were performed by supplementing the data of nodes that perform poorly in the validation set to further improve model's performance.

RESULTS

Prior to data augmentation, the precision was 0.728 and the F1 score was 0.666. By adjusting the augmentation ratio, the precision increased from 0.912 at a 1:10 augmentation ratio to 0.932 at a 1:100 augmentation ratio (P<.05), while F1 score improved from 0.853 at a 1:10 augmentation ratio to 0.881 at a 1:100 augmentation ratio (P<.05). Additionally, the effectiveness of various augmentation methods was compared at a 1:100 augmentation ratio. The augmentation method that combined synonyms and replacements (F1=0.881) was superior to the methods that only used synonyms (F1=0.815) and only used replacements (F1=0.753) (P<.05). For nodes that exhibited suboptimal performance on the validation set, supplementing the training set with target data improved model performance, increasing the average F1 score to 0.979 (P<.05).

CONCLUSION

Based on the KG, this study trained an automatic labeling model of radiology reports using a few-shot data set. This method effectively reduces the workload of manual labeling, improves the efficiency and accuracy of image data labeling, and provides an important research strategy for the application of AI in the domain of automatic labeling of image reports.

NLP-Driven Analysis of Pneumothorax Incidence Following Central Venous Catheter Procedures: A Data-Driven Re-Evaluation of Routine Imaging in Value-Based Medicine

Background: This study presents a systematic approach using a natural language processing (NLP) algorithm to assess the necessity of routine imaging after central venous catheter (CVC) placement and removal. With pneumothorax being a key complication of CVC procedures, this research aims to provide evidence-based recommendations for optimizing imaging protocols and minimizing unnecessary imaging risks. Methods: We analyzed electronic health records from four university hospitals in Salzburg, Austria, focusing on X-rays performed between 2012 and 2021 following CVC procedures. A custom-built NLP algorithm identified cases of pneumothorax from radiologists' reports and clinician requests, while excluding cases with contraindications such as chest injuries, prior pneumothorax, or missing data. Chi-square tests were used to compare pneumothorax rates between CVC insertion and removal, and multivariate logistic regression identified risk factors, with a focus on age and gender. Results: This study analyzed 17,175 cases of patients aged 18 and older, with 95.4% involving CVC insertion and 4.6% involving CVC removal. Pneumothorax was observed in 106 cases post-insertion (1.3%) and in 3 cases post-removal (0.02%), with no statistically significant difference between procedures (p = 0.5025). The NLP algorithm achieved an accuracy of 93%, with a sensitivity of 97.9%, a specificity of 87.9%, and an area under the ROC curve (AUC) of 0.9283. Conclusions: The findings indicate no significant difference in pneumothorax incidence between CVC insertion and removal, supporting existing recommendations against routine imaging post-removal for asymptomatic patients and suggesting that routine imaging after CVC insertion may also be unnecessary in similar cases. This study demonstrates how advanced NLP techniques can support value-based medicine by enhancing clinical decision making and optimizing resources.

Constructing a Large Language Model to Generate Impressions from Findings in Radiology Reports

Background The specialization and complexity of radiology makes the automatic generation of radiologic impressions (ie, a diagnosis with differential diagnosis and management recommendations) challenging. Purpose To develop a large language model (LLM) that generates impressions based on imaging findings and to evaluate its performance in professional and linguistic dimensions. Materials and Methods Six radiologists recorded imaging examination findings from August 2 to 31, 2023, at Shanghai General Hospital and used the developed LLM before routinely writing report impressions for multiple radiologic modalities (CT, MRI, radiography, mammography) and anatomic sites (cranium and face, neck, chest, upper abdomen, lower abdomen, vessels, bone and joint, spine, breast), making necessary corrections and completing the radiologic impression. A subset was defined to investigate cases where the LLM-generated impressions differed from the final radiologist impressions by excluding identical and highly similar cases. An expert panel scored the LLM-generated impressions on a five-point Likert scale (5 = strongly agree) based on scientific terminology, coherence, specific diagnosis, differential diagnosis, management recommendations, correctness, comprehensiveness, harmlessness, and lack of bias. Results In this retrospective study, an LLM was pretrained using 20 GB of medical and general-purpose text data. The fine-tuning data set comprised 1.5 GB of data, including 800 radiology reports with paired instructions (describing the output task in natural language) and outputs. Test set 2 included data from 3988 patients (median age, 56 years [IQR, 40-68 years]; 2159 male). The median recall, precision, and F1 score of LLM-generated impressions were 0.775 (IQR, 0.56-1), 0.84 (IQR, 0.611-1), and 0.772 (IQR, 0.578-0.957), respectively, using the final impressions as the reference standard. In a subset of 1014 patients (median age, 57 years [IQR, 42-69 years]; 528 male), the overall median expert panel score for LLM-generated impressions was 5 (IQR, 5-5), ranging from 4 (IQR, 3-5) to 5 (IQR, 5-5). Conclusion The developed LLM generated radiologic impressions that were professionally and linguistically appropriate for a full spectrum of radiology examinations. © RSNA, 2024 Supplemental material is available for this article.

Fully automated artificial intelligence-based coronary CT angiography image processing: efficiency, diagnostic capability, and risk stratification

OBJECTIVES

To prospectively investigate whether fully automated artificial intelligence (FAAI)-based coronary CT angiography (CCTA) image processing is non-inferior to semi-automated mode in efficiency, diagnostic ability, and risk stratification of coronary artery disease (CAD).

MATERIALS AND METHODS

Adults with indications for CCTA were prospectively and consecutively enrolled at two hospitals and randomly assigned to either FAAI-based or semi-automated image processing using equipment workstations. Outcome measures were workflow efficiency, diagnostic accuracy for obstructive CAD (≥ 50% stenosis), and cardiovascular events at 2-year follow-up. The endpoints included major adverse cardiovascular events, hospitalization for unstable angina, and recurrence of cardiac symptoms. The non-inferiority margin was 3 percentage difference in diagnostic accuracy and C-index.

RESULTS

In total, 1801 subjects (62.7 ± 11.1 years) were included, of whom 893 and 908 were assigned to the FAAI-based and semi-automated modes, respectively. Image processing times were 121.0 ± 18.6 and 433.5 ± 68.4 s, respectively (p <0.001). Scan-to-report release times were 6.4 ± 2.7 and 10.5 ± 3.8 h, respectively (p < 0.001). Of all subjects, 152 and 159 in the FAAI-based and semi-automated modes, respectively, subsequently underwent invasive coronary angiography. The diagnostic accuracies for obstructive CAD were 94.7% (89.9-97.7%) and 94.3% (89.5-97.4%), respectively (difference 0.4%). Of all subjects, 779 and 784 in the FAAI-based and semi-automated modes were followed for 589 ± 182 days, respectively, and the C-statistic for cardiovascular events were 0.75 (0.67 to 0.83) and 0.74 (0.66 to 0.82) (difference 1%).

CONCLUSIONS

FAAI-based CCTA image processing significantly improves workflow efficiency than semi-automated mode, and is non-inferior in diagnosing obstructive CAD and risk stratification for cardiovascular events.

CLINICAL RELEVANCE STATEMENT

Conventional coronary CT angiography image processing is semi-automated. This observation shows that fully automated artificial intelligence-based image processing greatly improves efficiency, and maintains high diagnostic accuracy and the effectiveness in stratifying patients for cardiovascular events.

KEY POINTS

• Coronary CT angiography (CCTA) relies heavily on high-quality and fast image processing. • Full-automation CCTA image processing is clinically non-inferior to the semi-automated mode. • Full automation can facilitate the application of CCTA in early detection of coronary artery disease.

Bidirectional Encoder Representations from Transformers in Radiology: A Systematic Review of Natural Language Processing Applications

INTRODUCTION

Bidirectional Encoder Representations from Transformers (BERT), introduced in 2018, has revolutionized natural language processing. Its bidirectional understanding of word context has enabled innovative applications, notably in radiology. This study aimed to assess BERT's influence and applications within the radiologic domain.

METHODS

Adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a systematic review, searching PubMed for literature on BERT-based models and natural language processing in radiology from January 1, 2018, to February 12, 2023. The search encompassed keywords related to generative models, transformer architecture, and various imaging techniques.

RESULTS

Of 597 results, 30 met our inclusion criteria. The remaining were unrelated to radiology or did not use BERT-based models. The included studies were retrospective, with 14 published in 2022. The primary focus was on classification and information extraction from radiology reports, with x-rays as the prevalent imaging modality. Specific investigations included automatic CT protocol assignment and deep learning applications in chest x-ray interpretation.

CONCLUSION

This review underscores the primary application of BERT in radiology for report classification. It also reveals emerging BERT applications for protocol assignment and report generation. As BERT technology advances, we foresee further innovative applications. Its implementation in radiology holds potential for enhancing diagnostic precision, expediting report generation, and optimizing patient care.

Development and Evaluation of a Natural Language Processing System for Curating a Trans-Thoracic Echocardiogram (TTE) Database

BACKGROUND

Although electronic health records (EHR) provide useful insights into disease patterns and patient treatment optimisation, their reliance on unstructured data presents a difficulty. Echocardiography reports, which provide extensive pathology information for cardiovascular patients, are particularly challenging to extract and analyse, because of their narrative structure. Although natural language processing (NLP) has been utilised successfully in a variety of medical fields, it is not commonly used in echocardiography analysis.

OBJECTIVES

To develop an NLP-based approach for extracting and categorising data from echocardiography reports by accurately converting continuous (e.g., LVOT VTI, AV VTI and TR Vmax) and discrete (e.g., regurgitation severity) outcomes in a semi-structured narrative format into a structured and categorised format, allowing for future research or clinical use.

METHODS

135,062 Trans-Thoracic Echocardiogram (TTE) reports were derived from 146967 baseline echocardiogram reports and split into three cohorts: Training and Validation (n = 1075), Test Dataset (n = 98) and Application Dataset (n = 133,889). The NLP system was developed and was iteratively refined using medical expert knowledge. The system was used to curate a moderate-fidelity database from extractions of 133,889 reports. A hold-out validation set of 98 reports was blindly annotated and extracted by two clinicians for comparison with the NLP extraction. Agreement, discrimination, accuracy and calibration of outcome measure extractions were evaluated.

RESULTS

Continuous outcomes including LVOT VTI, AV VTI and TR Vmax exhibited perfect inter-rater reliability using intra-class correlation scores (ICC = 1.00, p < 0.05) alongside high R2 values, demonstrating an ideal alignment between the NLP system and clinicians. A good level (ICC = 0.75-0.9, p < 0.05) of inter-rater reliability was observed for outcomes such as LVOT Diam, Lateral MAPSE, Peak E Velocity, Lateral E' Velocity, PV Vmax, Sinuses of Valsalva and Ascending Aorta diameters. Furthermore, the accuracy rate for discrete outcome measures was 91.38% in the confusion matrix analysis, indicating effective performance.

CONCLUSIONS

The NLP-based technique yielded good results when it came to extracting and categorising data from echocardiography reports. The system demonstrated a high degree of agreement and concordance with clinician extractions. This study contributes to the effective use of semi-structured data by providing a useful tool for converting semi-structured text to a structured echo report that can be used for data management. Additional validation and implementation in healthcare settings can improve data availability and support research and clinical decision-making.

Automated labelling of radiology reports using natural language processing: Comparison of traditional and newer methods

Automated labelling of radiology reports using natural language processing allows for the labelling of ground truth for large datasets of radiological studies that are required for training of computer vision models. This paper explains the necessary data preprocessing steps, reviews the main methods for automated labelling and compares their performance. There are four main methods of automated labelling, namely: (1) rules-based text-matching algorithms, (2) conventional machine learning models, (3) neural network models and (4) Bidirectional Encoder Representations from Transformers (BERT) models. Rules-based labellers perform a brute force search against manually curated keywords and are able to achieve high F1 scores. However, they require proper handling of negative words. Machine learning models require preprocessing that involves tokenization and vectorization of text into numerical vectors. Multilabel classification approaches are required in labelling radiology reports and conventional models can achieve good performance if they have large enough training sets. Deep learning models make use of connected neural networks, often a long short-term memory network, and are similarly able to achieve good performance if trained on a large data set. BERT is a transformer-based model that utilizes attention. Pretrained BERT models only require fine-tuning with small data sets. In particular, domain-specific BERT models can achieve superior performance compared with the other methods for automated labelling.

Implementation of artificial intelligence in thoracic imaging-a what, how, and why guide from the European Society of Thoracic Imaging (ESTI)

This statement from the European Society of Thoracic imaging (ESTI) explains and summarises the essentials for understanding and implementing Artificial intelligence (AI) in clinical practice in thoracic radiology departments. This document discusses the current AI scientific evidence in thoracic imaging, its potential clinical utility, implementation and costs, training requirements and validation, its' effect on the training of new radiologists, post-implementation issues, and medico-legal and ethical issues. All these issues have to be addressed and overcome, for AI to become implemented clinically in thoracic radiology. KEY POINTS: • Assessing the datasets used for training and validation of the AI system is essential. • A departmental strategy and business plan which includes continuing quality assurance of AI system and a sustainable financial plan is important for successful implementation. • Awareness of the negative effect on training of new radiologists is vital.

Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists

IMPORTANCE

Artificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value.

OBJECTIVE

To prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists.

DESIGN, SETTING, AND PARTICIPANTS

A multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021.

EXPOSURES

A bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved.

MAIN OUTCOMES AND MEASURES

Time to write reports based on different caption generation models.

RESULTS

The training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents' mean (SD) reporting time using the NLP-generated model was 283 (37) seconds-significantly shorter than the normal template (347 [58] seconds; P < .001) and the rule-based model (296 [46] seconds; P < .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)-significantly higher than the normal template (0.37 [0.09]; P < .001) and the rule-based model (0.57 [0.19]; P < .001).

CONCLUSIONS AND RELEVANCE

In this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.

Development and Validation of a Model to Identify Critical Brain Injuries Using Natural Language Processing of Text Computed Tomography Reports

Importance

Clinical text reports from head computed tomography (CT) represent rich, incompletely utilized information regarding acute brain injuries and neurologic outcomes. CT reports are unstructured; thus, extracting information at scale requires automated natural language processing (NLP). However, designing new NLP algorithms for each individual injury category is an unwieldy proposition. An NLP tool that summarizes all injuries in head CT reports would facilitate exploration of large data sets for clinical significance of neuroradiological findings.

Objective

To automatically extract acute brain pathological data and their features from head CT reports.

Design, Setting, and Participants

This diagnostic study developed a 2-part named entity recognition (NER) NLP model to extract and summarize data on acute brain injuries from head CT reports. The model, termed BrainNERD, extracts and summarizes detailed brain injury information for research applications. Model development included building and comparing 2 NER models using a custom dictionary of terms, including lesion type, location, size, and age, then designing a rule-based decoder using NER outputs to evaluate for the presence or absence of injury subtypes. BrainNERD was evaluated against independent test data sets of manually classified reports, including 2 external validation sets. The model was trained on head CT reports from 1152 patients generated by neuroradiologists at the Yale Acute Brain Injury Biorepository. External validation was conducted using reports from 2 outside institutions. Analyses were conducted from May 2020 to December 2021.

Main Outcomes and Measures

Performance of the BrainNERD model was evaluated using precision, recall, and F1 scores based on manually labeled independent test data sets.

Results

A total of 1152 patients (mean [SD] age, 67.6 [16.1] years; 586 [52%] men), were included in the training set. NER training using transformer architecture and bidirectional encoder representations from transformers was significantly faster than spaCy. For all metrics, the 10-fold cross-validation performance was 93% to 99%. The final test performance metrics for the NER test data set were 98.82% (95% CI, 98.37%-98.93%) for precision, 98.81% (95% CI, 98.46%-99.06%) for recall, and 98.81% (95% CI, 98.40%-98.94%) for the F score. The expert review comparison metrics were 99.06% (95% CI, 97.89%-99.13%) for precision, 98.10% (95% CI, 97.93%-98.77%) for recall, and 98.57% (95% CI, 97.78%-99.10%) for the F score. The decoder test set metrics were 96.06% (95% CI, 95.01%-97.16%) for precision, 96.42% (95% CI, 94.50%-97.87%) for recall, and 96.18% (95% CI, 95.151%-97.16%) for the F score. Performance in external institution report validation including 1053 head CR reports was greater than 96%.

Conclusions and Relevance

These findings suggest that the BrainNERD model accurately extracted acute brain injury terms and their properties from head CT text reports. This freely available new tool could advance clinical research by integrating information in easily gathered head CT reports to expand knowledge of acute brain injury radiographic phenotypes.