Early Adoption of a Certainty Scale to Improve Diagnostic Certainty Communication

Multiplane reconstruction modifies the diagnostic performance of CT angiography in carotid webs

INTRODUCTION

Carotid Web (CaW) represents an overlooked stroke etiology and has been associated with high recurrence rates and to be amenable to stenting. We evaluated the diagnostic performance of different computed tomography angiography (CTA) projections in CaW.

METHODS

Consecutive patients <65 years-old with symptomatic CaW (n=31), carotid atherosclerosis (n=27), or normal carotids (n=49) diagnosed with a thin-cut CTA were included. Deidentified CTAs were independently reviewed by three readers, who recorded the diagnosis and level of certainty after evaluating the axial plane alone, after adding sagittal/coronal maximum intensity projection (MIP), then after oblique MPR reformats.

RESULTS

There were 93 total CaW, 81 atherosclerosis, and 147 normal carotid reads. With CTA axial projection alone, less CaW cases (44.1 %) were appropriately diagnosed as compared to atherosclerosis (87.7 %; p<0.001) and normal carotid (83 %; p<0.001) cases. Sagittal/coronal MIPS increased the rate of accurate CaW diagnosis (44.1-76.3 %; p<0.001). Inter-rater agreement in CaW detection increased from k= 0.46 (0.35-0.57) using axial to k= 0.80 (0.69-0.91) with sagittal/coronal planes. The axial projection alone had lower sensitivity (44 % vs. 76 %) but similar specificity (95 % vs. 96 %) in CaW detection compared to axial+ sagittal/coronal MIPS. The accuracy in detecting atherosclerosis or normal carotids did not increase after adding sagittal/coronal MIPS and oblique MPRs. The certainty level for CaW diagnosis was lower when compared to atherosclerosis and normal carotids using axial alone (3.0 [3.0-4.0] vs. 4.0 [3.0-5.0]; p<0.001 and 4.0 [3.0-5.0]; p<0.001) as well as after adding sagittal/coronal MIPS (4.0 [3.0-5.0] vs. 5.0[4.0-5.0]; p=0.01 and 4.0 [4.0-5.0]; p<0.001).

CONCLUSION

CTA axial plane alone was insufficient for CaW detection. CTA sagittal/coronal MIP reconstructions as well as oblique MPR reformats enhanced the accuracy and confidence related to CaW diagnosis.

Degree of Uncertainty in Reporting Imaging Findings for Necrotizing Enterocolitis: A Secondary Analysis from a Pilot Randomized Diagnostic Trial

Diagnosis of necrotizing enterocolitis (NEC) relies heavily on imaging, but uncertainty in the language used in imaging reports can result in ambiguity, miscommunication, and potential diagnostic errors. To determine the degree of uncertainty in reporting imaging findings for NEC, we conducted a secondary analysis of the data from a previously completed pilot diagnostic randomized controlled trial (2019-2020). The study population comprised sixteen preterm infants with suspected NEC randomized to abdominal radiographs (AXRs) or AXR + bowel ultrasound (BUS). The level of uncertainty was determined using a four-point Likert scale. Overall, we reviewed radiology reports of 113 AXR and 24 BUS from sixteen preterm infants with NEC concern. The BUS reports showed less uncertainty for reporting pneumatosis, portal venous gas, and free air compared to AXR reports (pneumatosis: 1 [1-1.75) vs. 3 [2-3], p < 0.0001; portal venous gas: 1 [1-1] vs. 1 [1-1], p = 0.02; free air: 1 [1-1] vs. 2 [1-3], p < 0.0001). In conclusion, we found that BUS reports have a lower degree of uncertainty in reporting imaging findings of NEC compared to AXR reports. Whether the lower degree of uncertainty of BUS reports positively impacts clinical decision making in infants with possible NEC remains unknown.

Applying Natural Language Processing to Single-Report Prediction of Metastatic Disease Response Using the OR-RADS Lexicon

Generating Real World Evidence (RWE) on disease responses from radiological reports is important for understanding cancer treatment effectiveness and developing personalized treatment. A lack of standardization in reporting among radiologists impacts the feasibility of large-scale interpretation of disease response. This study examines the utility of applying natural language processing (NLP) to the large-scale interpretation of disease responses using a standardized oncologic response lexicon (OR-RADS) to facilitate RWE collection. Radiologists annotated 3503 retrospectively collected clinical impressions from radiological reports across several cancer types with one of seven OR-RADS categories. A Bidirectional Encoder Representations from Transformers (BERT) model was trained on this dataset with an 80-20% train/test split to perform multiclass and single-class classification tasks using the OR-RADS. Radiologists also performed the classification to compare human and model performance. The model achieved accuracies from 95 to 99% across all classification tasks, performing better in single-class tasks compared to the multiclass task and producing minimal misclassifications, which pertained mostly to overpredicting the equivocal and mixed OR-RADS labels. Human accuracy ranged from 74 to 93% across all classification tasks, performing better on single-class tasks. This study demonstrates the feasibility of the BERT NLP model in predicting disease response in cancer patients, exceeding human performance, and encourages the use of the standardized OR-RADS lexicon to improve large-scale prediction accuracy.

Lexicon for adrenal terms at CT and MRI: a consensus of the Society of Abdominal Radiology adrenal neoplasm disease-focused panel

PURPOSE

Substantial variation in imaging terms used to describe the adrenal gland and adrenal findings leads to ambiguity and uncertainty in radiology reports and subsequently their understanding by referring clinicians. The purpose of this study was to develop a standardized lexicon to describe adrenal imaging findings at CT and MRI.

METHODS

Fourteen members of the Society of Abdominal Radiology adrenal neoplasm disease-focused panel (SAR-DFP) including one endocrine surgeon participated to develop an adrenal lexicon using a modified Delphi process to reach consensus. Five radiologists prepared a preliminary list of 35 imaging terms that was sent to the full group as an online survey (19 general imaging terms, 9 specific to CT, and 7 specific to MRI). In the first round, members voted on terms to be included and proposed definitions; subsequent two rounds were used to achieve consensus on definitions (defined as ≥ 80% agreement).

RESULTS

Consensus for inclusion was reached on 33/35 terms with two terms excluded (anterior limb and normal adrenal size measurements). Greater than 80% consensus was reached on the definitions for 15 terms following the first round, with subsequent consensus achieved for the definitions of the remaining 18 terms following two additional rounds. No included term had remaining disagreement.

CONCLUSION

Expert consensus produced a standardized lexicon for reporting adrenal findings at CT and MRI. The use of this consensus lexicon should improve radiology report clarity, standardize clinical and research terminology, and reduce uncertainty for referring providers when adrenal findings are present.

Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example

OBJECTIVE

Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models.

METHODS

This institutional review board-approved retrospective study was conducted January 1, 2016, to December 31, 2020, at a large academic institution. A previously trained ML model was trained on 1,000 radiology reports from 2016 (old data). An additional 1,385 randomly selected reports from 2019 to 2020 (new data) were annotated for follow-up recommendations and randomly divided into two sets: training (n = 900) and testing (n = 485). Support vector machine and random forest (RF) algorithms were constructed and trained using 900 new data reports plus old data (augmented data, new models) and using only new data (new data, new models). The 2016 baseline model was used as comparator as is and trained with augmented data. Recall was compared with baseline using McNemar's test.

RESULTS

Follow-up recommendations were contained in 11.3% of reports (157 or 1,385). The baseline model retrained with new data had precision = 0.83 and recall = 0.54; none significantly different from baseline. A new RF model trained with augmented data had significantly better recall versus the baseline model (0.80 versus 0.66, P = .04) and comparable precision (0.90 versus 0.86).

DISCUSSION

ML methods for monitoring follow-up recommendations in radiology reports suffer model drift over time. A newly developed RF model achieved better recall with comparable precision versus simply retraining a previously trained original model with augmented data. Thus, regularly assessing and updating these models is necessary using more recent historical data.

Talking Points: Enhancing Communication Between Radiologists and Patients

Radiologists communicate along multiple pathways, using written, verbal, and non-verbal means. Radiology trainees must gain skills in all forms of communication, with attention to developing effective professional communication in all forms. This manuscript reviews evidence-based strategies for enhancing effective communication between radiologists and patients through direct communication, written means and enhanced reporting. We highlight patient-centered communication efforts, available evidence, and opportunities to engage learners and enhance training and simulation efforts that improve communication with patients at all levels of clinical care.

Adoption of a diagnostic certainty scale in abdominal imaging: 2-year experience at an academic institution

PURPOSE

Assess use of a diagnostic certainty scale (CS) for abdominal imaging reports and identify factors associated with greater adoption.

METHODS

This retrospective, Institutional Review Board-exempt study was conducted at an academic health system. Abdominal radiology reports containing diagnostic certainty phrases (DCPs) generated 4/1/2019-3/31/2021 were identified by a natural language processing tool. Reports containing DCPs were subdivided into those with/without a CS inserted at the end. Primary outcome was monthly CS use rate in reports containing DCPs. Secondary outcomes were assessment of factors associated with CS use, and usage of recommended DCPs over time. Chi-square test was used to compare proportions; univariable and multivariable regression assessed impact of other variables.

RESULTS

DCPs were used in 81,281/124,501 reports (65.3%). One-month post-implementation, 82/2310 (3.6%) of reports with DCPs contained the CS, increasing to 1862/4644 (40.1%) by study completion (p < 0.001). Multivariable analysis demonstrated reports containing recommended DCPs were more likely to have the CS (Odds Ratio [OR] 4.5; p < 0.001). Using CT as a reference, CS use was lower for ultrasound (OR 0.73; p < 0.001) and X-ray (OR 0.38; p < 0.001). There was substantial inter-radiologist variation in CS use (OR 0.01-26.3, multiple p values).

CONCLUSION

DCPs are very common in abdominal imaging reports and can be further clarified with CS use. Although voluntary CS adoption increased 13-fold over 2 years, > 50% of reports with DCPs lacked the CS at study's end. More stringent interventions, including embedding the scale in report templates, are likely needed to reduce inter-radiologist variation and decrease ambiguity in conveying diagnostic certainty to referring providers and patients.

Myeloma Response Assessment and Diagnosis System (MY-RADS): strategies for practice implementation

Structured reporting systems have been developed for many organ systems and disease processes beginning with BI-RADS in 1993. Numerous reports indicate that referring health care providers prefer structured reports. Reducing variability of reports from one radiologist to another helps referring physician and patient confidence. Changing radiologists practice habits from completely free text to structured reports can be met with some resistance, but most radiologists quickly find that structured reports make their job easier. Whole-body MR studies are recommended as first-line imaging, by the International Myeloma Working Group (IMWG), for all patients with suspected diagnosis of asymptomatic myeloma and/or initial diagnosis of solitary plasmacytoma. Whole-body MR imaging (WBMRI) has been shown to have equal or greater sensitivity and specificity compared to PET/CT for detection of bone marrow involvement. Changing to WBMRI from other imaging modalities can be difficult for referring providers. Patient acceptance is high. MY-RADS is for myeloma patients who have WBMRI studies done. The intent of the system is to promote uniformity in MR imaging acquisition, diagnostic criteria, and response assessment and to diminish differences in the subsequent interpretation and reporting. A secondary benefit is a report template that provides a guide for interpretation for radiologists who may not have previously dictated these difficult studies. The characterization of bone marrow abnormalities in myeloma patients usually is fairly straightforward. To date, there is no standardized scoring or risk stratification of abnormalities nor is there an imaging atlas of abnormalities.

Qualifying Certainty in Radiology Reports through Deep Learning-Based Natural Language Processing

BACKGROUND AND PURPOSE

Communication gaps exist between radiologists and referring physicians in conveying diagnostic certainty. We aimed to explore deep learning-based bidirectional contextual language models for automatically assessing diagnostic certainty expressed in the radiology reports to facilitate the precision of communication.

MATERIALS AND METHODS

We randomly sampled 594 head MR imaging reports from an academic medical center. We asked 3 board-certified radiologists to read sentences from the Impression section and assign each sentence 1 of the 4 certainty categories: "Non-Definitive," "Definitive-Mild," "Definitive-Strong," "Other." Using the annotated 2352 sentences, we developed and validated a natural language-processing system based on the start-of-the-art bidirectional encoder representations from transformers (BERT), which can capture contextual uncertainty semantics beyond the lexicon level. Finally, we evaluated 3 BERT variant models and reported standard metrics including sensitivity, specificity, and area under the curve.

RESULTS

A κ score of 0.74 was achieved for interannotator agreement on uncertainty interpretations among 3 radiologists. For the 3 BERT variant models, the biomedical variant (BioBERT) achieved the best macro-average area under the curve of 0.931 (compared with 0.928 for the BERT-base and 0.925 for the clinical variant [ClinicalBERT]) on the validation data. All 3 models yielded high macro-average specificity (93.13%-93.65%), while the BERT-base obtained the highest macro-average sensitivity of 79.46% (compared with 79.08% for BioBERT and 78.52% for ClinicalBERT). The BioBERT model showed great generalizability on the heldout test data with a macro-average sensitivity of 77.29%, specificity of 92.89%, and area under the curve of 0.93.

CONCLUSIONS

A deep transfer learning model can be developed to reliably assess the level of uncertainty communicated in a radiology report.

Radiologist Reporting and Operational Management for Patients With Suspected COVID-19

PURPOSE

The aim of this study was to evaluate the adoption and outcomes of locally designed reporting guidelines for patients with possible coronavirus disease 2019 (COVID-19).

METHODS

A departmental guideline was developed for radiologists that specified reporting terminology and required communication for patients with imaging findings suggestive of COVID-19, on the basis of patient test status and imaging indication. In this retrospective study, radiology reports completed from March 1, 2020, to May 3, 2020, that mentioned COVID-19 were reviewed. Reports were divided into patients with known COVID-19, patients with "suspected" COVID-19 (having an order indication of respiratory or infectious signs or symptoms), and "unsuspected patients" (other order indications, eg, trauma or non-chest pain). The primary outcome was the percentage of COVID-19 reports using recommended terminology; the secondary outcome was percentages of suspected and unsuspected patients diagnosed with COVID-19. Relationships between categorical variables were assessed using the Fisher exact test.

RESULTS

Among 77,400 total reports, 1,083 suggested COVID-19 on the basis of imaging findings; 774 of COVID-19 reports (71%) used recommended terminology. Of 574 patients without known COVID-19 at the time of interpretation, 345 (60%) were eventually diagnosed with COVID-19, including 61% (315 of 516) of suspected and 52% (30 of 58) of unsuspected patients. Nearly all unsuspected patients (46 of 58) were identified on CT.

CONCLUSIONS

Radiologists rapidly adopted recommended reporting terminology for patients with suspected COVID-19. The majority of patients for whom radiologists raised concern for COVID-19 were subsequently diagnosed with the disease, including the majority of clinically unsuspected patients. Using unambiguous terminology and timely notification about previously unsuspected patients will become increasingly critical to facilitate COVID-19 testing and contact tracing as states begin to lift restrictions.