Adoption of a Closed-Loop Communication Tool to Establish and Execute a Collaborative Follow-Up Plan for Incidental Pulmonary Nodules

Cumulative Effect of Targeted Interventions on Radiologist Recommendations for Additional Imaging

Background Ambiguous or unnecessary radiologist recommendations for additional imaging (RAIs) can lead to excessive imaging use and diagnostic errors. Purpose To determine the cumulative impacts of multifaceted technology-enabled interventions aimed at optimizing RAI on RAI rate, actionability, and resolution over an 8-year period. Materials and Methods In this retrospective cohort study, conducted from January 2015 to December 2022, radiology reports from two tertiary hospitals (study site and control site) were analyzed. A series of quality improvement interventions, including radiologist education, electronic communication tools for tracking RAIs, and performance reports, were implemented at the study site but not at the control site. The RAI rate trend over time was compared between the sites using linear regression. Mixed-effects logistic regression was performed to assess the intervention impact on the RAI rate. RAI actionability and resolution were compared between the sites using the Fisher exact test. P values were corrected using the Bonferroni method. Results Among 7 502 521 total radiology reports (1 323 459 patients) (study site, 3 608 977 reports and 660 051 patients; control site, 3 893 544 reports and 690 115 patients), the RAI rate of the study site decreased by 44%, from 10% (8202 of 81 586) to 5.6% (8972 of 159 599), but remained unchanged at the control site, at 10.9% (8757 of 80 030) vs 11% (16 978 of 153 711) (regression coefficient, -0.09; 95% CI: -0.1, -0.09; P < .001). RAI rates declined with each successive intervention at the study site (P < .001), with regression coefficients decreasing progressively from -0.12 (95% CI: -0.14, -0.10) for the initial intervention to -0.81 (95% CI: -0.83, -0.78) for the final intervention. Recommendation actionability at the study site increased 7.6-fold (from 5.6% [19 of 340] to 42.3% [144 of 340]; P < .001) but remained unchanged at the control site (from 15.0% [51 of 340] to 13.8% [47 of 340]; P = .73). Actionable RAIs were more frequently resolved at the study site than at the control site (84.7% [122 of 144] vs 59.6% [28 of 47]; P < .001). Conclusion Multifaceted interventions to optimize RAI improved the rate, actionability, and resolution of RAI. © RSNA, 2025 Supplemental material is available for this article. See also the editorial by Russell in this issue.

Use of ChatGPT Large Language Models to Extract Details of Recommendations for Additional Imaging From Free-Text Impressions of Radiology Reports

BACKGROUND. Automated extraction of actionable details of recommendations for additional imaging (RAIs) from radiology reports could facilitate tracking and timely completion of clinically necessary RAIs and thereby potentially reduce diagnostic delays. OBJECTIVE. The purpose of the study was to assess the performance of large language models (LLMs) in extracting actionable details of RAIs from radiology reports. METHODS. This retrospective single-center study evaluated reports of diagnostic radiology examinations performed across modalities and care settings within five subspecialties (abdominal imaging, musculoskeletal imaging, neuroradiology, nuclear medicine, thoracic imaging) in August 2023. Of reports identified by a previously validated natural language processing algorithm to contain an RAI, 250 were randomly selected; 231 of these reports were confirmed to contain an RAI on manual review and formed the study sample. Twenty-five reports were used to engineer a prompt instructing an LLM, when inputted in a report impression containing an RAI, to extract details about the modality, body part, time frame, and rationale of the RAI; the remaining 206 reports were used for testing the prompt in combination with GPT-3.5 and GPT-4. A 4th-year medical student and radiologist from the relevant subspecialty independently classified the LLM outputs as correct versus incorrect for extracting the four actionable details of RAIs in comparison with the report impressions; a third reviewer assisted in resolving discrepancies. Extraction accuracy was summarized and compared between LLMs using consensus assessments. RESULTS. For GPT-3.5 and GPT-4, the two reviewers agreed about classification of LLM output as correct versus incorrect with respect to report impressions for 95.6% and 94.2% for RAI modality, 89.3% and 88.3% for RAI body part, 96.1% and 95.1% for RAI time frame, and 89.8% and 88.8% for RAI rationale, respectively. GPT-4 was more accurate than GPT-3.5 in extracting RAI modality (94.2% [194/206] vs 85.4% [176/206], p < .001), RAI body part (86.9% [179/206] vs 77.2% [159/206], p = .004), and RAI time frame (99.0% [204/206] vs 95.6% [197/206], p = .02). Both LLMs had accuracy of 91.7% (189/206) for extracting RAI rationale. CONCLUSION. LLMs were used to extract actionable details of RAIs from free-text impression sections of radiology reports; GPT-4 outperformed GPT-3.5. CLINICAL IMPACT. The technique could represent an innovative method to facilitate timely completion of clinically necessary radiologist recommendations.

Improving communication of unexpected findings: The radiology actional findings tracking (RAFT) program

Incidental findings are unexpected, actionable discoveries made on diagnostic imaging that have significant patient care and medicolegal implications if not well managed. Despite their importance, few systems exist to manage incidental findings. The Radiology Actionable Findings Tracking (RAFT) Program was developed to improve communication of incidental findings to radiologists, providers, and their patients. The RAFT template is incorporated into the electronic medical record and discloses important information such as: Finding, Acuity, Communication Status, and General Recommendation for follow-up. This data is automatically compiled into a spreadsheet monitored by a clinical coordinator who is responsible for notifying the primary care physician or referring provider. The alert is resolved once appropriate communication is made and the recommended follow-up measures are documented. Between January 2021 and June 2023, the program has tracked the communication of 2,243 incidental findings, for an average of 75 incidental findings each month. Of those total findings, 270 findings (12 %) triggered additional protocols for provider and patient notification with subsequent follow-up. The program is effective in improving communication of incidental findings and can serve as a valuable tool for radiologists, providers, and the patients they serve.

Clinical Impact of Radiologist's Alert System on Patient Care for High-risk Incidental CT Findings: A Machine Learning-Based Risk Factor Analysis

RATIONALE AND OBJECTIVES

Efficient communication between radiologists and clinicians ordering computed tomography (CT) examinations is crucial for managing high-risk incidental CT findings (ICTFs). Herein, we introduced a Radiologist's Alert and Patient Care Follow-up System (APCFS) for high-risk ICTFs. This study aimed to analyze the ICTFs detected by this system and the factors associated with them.

MATERIALS AND METHODS

This retrospective study was approved by the institutional review board. We analyzed 52,331 CT examinations conducted between 2019 and 2021. In cases where high-risk ICTFs were identified, radiologists utilized APCFS to prompt ordering clinicians for further patient care. We assessed the frequency, affected body organs, presence or absence of therapeutic interventions, and diagnoses of high-risk ICTFs. An automated machine learning platform was employed to analyze the factors associated with high-risk ICTFs.

RESULTS

Among the 52,331 CT examinations, 507 (0.96%) revealed high-risk ICTFs, primarily affecting the lung (18.0%). Of these 507 high-risk ICTFs, 117 (23.1%) underwent therapeutic interventions, while 362 (71.4%) required only follow-up. Of the 117 cases undergoing interventions, 61 (52.1%) required surgery. Of the 219 high-risk ICTFs leading to a confirmed diagnosis, 146 (66.7%) were neoplastic lesions, including 88 (60.3%) malignancies, and 73 (33.3%) were non-neoplastic lesions. The top three risk factors associated with high-risk ICTFs in the regularized logistic regression model were the imaging protocol (especially aortic valve implantation planning protocol), imaging area (especially whole-body imaging), and clinical department (especially cardiology).

CONCLUSION

Utilizing APCFS, high-risk ICTFs were detected in approximately 1% of all CT examinations, likely associated with specific imaging protocols, areas, and clinical departments.

Lung cancer screening in India: Preparing for the future using smart tools & biomarkers to identify highest risk individuals

There is a growing burden of lung cancer cases in India, incidence projected to increase from 63,708 cases (2015) to 81,219 cases (2025). The increasing numbers are attributed to smoking (India currently has nearly 100 million adult smokers) and environmental pollution. Most patients present with advanced disease (80-85% are incurable), causing nearly 60,000 annual deaths from lung cancer. Early detection through lung cancer screening (LCS) can result in curative therapies for earlier stages of lung cancer and improved survival. Annual low-dose computerized tomography (LDCT) is the standard method for LCS. Usually, high-risk populations (age>50 yr and >20 pack-years of smoking) are considered for LCS, but even such focused screening may be challenging in resource-limited countries like India. However, developing a smart LCS programme with high yield may be possible by leveraging demographic and genomic data, use of smart tools, and judicious use of blood-based biomarkers. Developing this model over the next several years will facilitate a structured cancer screening programme for populations at the highest risk of lung cancer. In this paper, we discuss the demographics of lung cancer in India and its relation to smoking patterns. Further, we elaborate on the potential applications and challenges of bringing a smart approach to LCS in high-risk populations in India.

Financial Impact of a Radiology Safety Net Program for Resolution of Clinically Necessary Follow-up Imaging Recommendations

OBJECTIVE

Health care safety net (SN) programs can potentially improve patient safety and decrease risk associated with missed or delayed follow-up care, although they require financial resources. This study aimed to assess whether the revenue generated from completion of clinically necessary recommendations for additional imaging (RAI) made possible by an IT-enabled SN program could fund the required additional labor resources.

METHODS

Clinically necessary RAI generated October 21, 2019, to September 24, 2021, were tracked to resolution as of April 13, 2023. A new radiology SN team worked with existing schedulers and care coordinators, performing chart review and patient and provider outreach to ensure RAI resolution. We applied relevant Current Procedural Terminology, version 4 codes of the completed imaging examinations to estimate total revenue. Coprimary outcomes included revenue generated by total performed examinations and estimated revenue attributed to SN involvement. We used Student's t test to compare the secondary outcome, RAI time interval, for higher versus lower revenue-generating modalities.

RESULTS

In all, 24% (3,243) of eligible follow-up recommendations (13,670) required SN involvement. Total estimated revenue generated by performed recommended examinations was $6,116,871, with $980,628 attributed to SN. Net SN-generated revenue per 1.0 full-time equivalent was an estimated $349,768. Greatest proportion of performed examinations were cross-sectional modalities (CT, MRI, PET/CT), which were higher revenue-generating than non-cross-sectional modalities (x-ray, ultrasound, mammography), and had shorter recommendation time frames (153 versus 180 days, P < .001).

DISCUSSION

The revenue generated from completion of RAI facilitated by an IT-enabled quality and safety program supplemented by an SN team can fund the required additional labor resources to improve patient safety. Realizing early revenue may require 5 to 6 months postimplementation.

Actionability of Recommendations for Additional Imaging in Head and Neck Radiology

PURPOSE

The aims of this study were to measure the actionability of recommendations for additional imaging (RAIs) in head and neck CT and MRI, for which there is a near complete absence of best practices or guidelines; to identify the most common recommendations; and to assess radiologist factors associated with actionability.

METHODS

All head and neck CT and MRI radiology reports across a multi-institution, multipractice health care system from June 1, 2021, to May 31, 2022, were retrospectively reviewed. The actionability of RAIs was scored using a validated taxonomy. The most common RAIs were identified. Actionability association with radiologist factors (gender, years out of training, fellowship training, practice type) and with trainees was measured using a mixed-effects model.

RESULTS

Two hundred nine radiologists generated 60,543 reports, of which 7.2% (n = 4,382) contained RAIs. Only 3.9% of RAIs (170 of 4,382) were actionable. More than 60% of RAIs were for eight examinations: thyroid ultrasound (14.1%), neck CT (12.6%), brain MRI (6.9%), chest CT (6.5%), neck CT angiography (5.5%), temporal bone CT (5.3%), temporal bone MRI (5.2%), and pituitary MRI (4.6%). Radiologists >23 years out of training (odds ratio, 0.39; 95% confidence interval, 0.15-1.02; P = .05) and community radiologists (odds ratio, 0.53; 95% confidence interval, 0.22-1.31; P = .17) had substantially lower estimated odds of making actionable RAIs than radiologists <7 years out of training and academic radiologists, respectively.

CONCLUSIONS

The studied radiologists rarely made actionable RAIs, which makes it difficult to identify and track clinically necessary RAIs to timely performance. Multifaceted quality improvement initiatives including peer comparisons, clinical decision support at the time of reporting, and the development of evidence-based best practices, may help improve tracking and timely performance of clinically necessary RAIs.

Incidental pulmonary nodule management in Canada: exploring current state through a narrative literature review and expert interviews

Background and Objective

Incidental pulmonary nodules (IPNs) are common and increasingly detected with the overall rise of radiologic imaging. Effective IPN management is necessary to ensure lung cancer is not missed. This study aims to describe the current landscape of IPN management in Canada, understand barriers to optimal IPN management, and identify opportunities for improvement.

Methods

We performed a narrative literature review by searching biomedical electronic databases for relevant articles published between January 1, 2010, and November 22, 2023. To validate and complement the identified literature, we conducted structured interviews with multidisciplinary experts involved in the pathway of patients with IPNs across Canada. Interviews between December 2021 and May 2022 were audiovisual recorded, transcribed, and thematically analyzed.

Key Content and Findings

A total of 1,299 records were identified, of which 37 studies were included for analysis. Most studies were conducted in Canada and the United States and highlighted variability in radiology reporting of IPNs and patient management, and limited adherence to recommended follow-up imaging. Twenty experts were interviewed, including radiologists, respirologists, thoracic surgeons, primary care physicians, medical oncologists, and an epidemiologist. Three themes emerged from the interviews, supported by the literature, including: variability in radiology reporting of IPNs, suboptimal communication, and variability in guideline adherence and patient management.

Conclusions

Despite general awareness of guidelines, there is inconsistency and lack of standardization in the management of patients with IPNs in Canada. Multidisciplinary expert consensus is recommended to help overcome the communication and operational barriers to a safe and cost-effective approach to this common clinical issue.

Impact of an Automated Closed-Loop Communication and Tracking Tool on the Rate of Recommendations for Additional Imaging in Thoracic Radiology Reports

OBJECTIVE

Assess the effects of feedback reports and implementing a closed-loop communication system on rates of recommendations for additional imaging (RAIs) in thoracic radiology reports.

METHODS

In this retrospective, institutional review board-approved study at an academic quaternary care hospital, we analyzed 176,498 thoracic radiology reports during a pre-intervention (baseline) period from April 1, 2018, to November 30, 2018; a feedback report only period from December 1, 2018, to September 30, 2019; and a closed-loop communication system plus feedback report (IT intervention) period from October 1, 2019, to December 31, 2020, promoting explicit documentation of rationale, time frame, and imaging modality for RAI, defined as complete RAI. A previously validated natural language processing tool was used to classify reports with an RAI. Primary outcome of rate of RAI was compared using a control chart. Multivariable logistic regression determined factors associated with likelihood of RAI. We also estimated the completeness of RAI in reports comparing IT intervention to baseline using χ2 statistic.

RESULTS

The natural language processing tool classified 3.2% (5,682 of 176,498) reports as having an RAI; 3.5% (1,783 of 51,323) during the pre-intervention period, 3.8% (2,147 of 56,722) during the feedback report only period (odds ratio: 1.1, P = .03), and 2.6% (1,752 of 68,453) during the IT intervention period (odds ratio: 0.60, P < .001). In subanalysis, the proportion of incomplete RAI decreased from 84.0% (79 of 94) during the pre-intervention period to 48.5% (47 of 97) during the IT intervention period (P < .001).

DISCUSSION

Feedback reports alone increased RAI rates, and an IT intervention promoting documentation of complete RAI in addition to feedback reports led to significant reductions in RAI rate, incomplete RAI, and improved overall completeness of the radiology recommendations.

The FIND Program: Improving Follow-up of Incidental Imaging Findings

Incidental findings are findings identified on imaging which are unrelated to the original reason for examination and require follow-up. The Radiology Finding Incidental Disease (FIND) Program was designed to track and improve follow-up of incidental imaging findings. The purpose of this study was to determine the frequency of incidental findings on cross-sectional imaging and the adherence to suggested follow-up of incidental findings prior to and after implementation of a structured reporting and tracking system. A retrospective analysis of 2000 patients with computed tomographic cross-sectional imaging was performed: 1000 patients prior to implementation of the FIND Program and 1000 patients 1 year after establishment of the program. Data collected included the frequency of incidental findings, inclusion of follow-up recommendations in the radiology report, and adherence to suggested follow-up. There was a higher rate of completion of recommended follow-up imaging in the post-implementation group (34/67, 50.7%) compared to the pre-implementation (16/52, 30.8%) (p = 0.03). Implementation of an incidental findings tracking program resulted in improved follow-up of incidental imaging findings. This has the potential to reduce the burden of clinically significant incidental findings possibly resulting in later presentation of advanced disease.

Comparing thoracic and abdominal subspecialists' follow-up recommendations for abdominal findings identified on chest CT

PURPOSE

To compare thoracic and abdominal radiologists' follow-up recommendations for abdominal findings identified on chest CT.

METHODS

This Institutional Review Board-exempt, retrospective study was performed at a large academic medical center with subspecialty radiology divisions. We used a combination of natural language processing and manual reviews to identify chest CT reports with and without abdominal findings that were interpreted by thoracic radiologists in 2019. Three random samples of reports were reviewed by two subspecialty trained abdominal radiologists for their agreement with thoracic radiologists' reporting: abdominal findings with follow-up recommendation (Group 1), abdominal findings without follow-up recommendation (Group 2), and no abdominal findings reported (Group 3). Primary outcome was agreement between thoracic and abdominal radiologists for the need for follow-up of abdominal findings. Secondary outcomes were agreement between subspecialists for the presence of abdominal findings and referring clinician adherence to recommendations. Fischer's exact test was used to compare proportions.

RESULTS

Abdominal radiologists agreed with need for follow-up in 48.5% (16/33) of Group 1 cases and agreed follow-up was not necessary for 100% (34/34) of Group 2 cases (p < 0.001). Abdominal radiologists identified abdominal findings in 31.4% (11/35) of Group 3 cases, none of which required follow-up. Referring clinician adherence to thoracic radiologist follow-up recommendations for abdominal findings was 13/33 (39.4%).

CONCLUSION

Abdominal radiologists frequently disagreed with thoracic radiologist recommendations for follow-up of abdominal findings on chest CT. Chest radiologists may consider abdominal subspecialty consultation or clinical decision support to reduce unnecessary imaging.

Incidental Pulmonary Nodules - What Do We Know in 2022

Lung cancer (LC) is the leading cause of cancer-related mortality worldwide, and early LC diagnosis can significantly improve outcomes and survival rates in affected patients. Implementation of LC screening programs using low-dose computed tomography CT in high-risk subjects aims to detect LC as early as possible, but so far, adoption of screening programs into routine clinical care has been very slow. In recent years, the use of CT has significantly increased the rate of incidentally detected pulmonary nodules. Although most of those incidental pulmonary nodules (IPNs) are benign, some of them represent early-stage LC. Given the large number of IPNs detected in the range of several millions each year, this represents an additional, maybe even larger, opportunity to drive stage shift in LC diagnosis, next to LC screening programs. Comprehensive evaluation and targeted work-up of IPNs are mandatory to identify the malignant nodules from the crowd, and several guidelines provide radiologists and physicians' guidance on IPN assessment and management. However, IPNs still seem to be inadequately processed due to various reasons including insufficient reporting in the radiological report, missing communication between stakeholders, absence of patient tracking systems, and uncertainty regarding responsibilities for the IPN management. In recent years, several approaches such as lung nodule programs, patient tracking software, artificial intelligence, and communication software were introduced into clinical practice to address those shortcomings. This review evaluates the current situation of IPN management and highlights recent developments in process improvement to achieve first steps toward stage shift in LC diagnosis.

Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example

OBJECTIVE

Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models.

METHODS

This institutional review board-approved retrospective study was conducted January 1, 2016, to December 31, 2020, at a large academic institution. A previously trained ML model was trained on 1,000 radiology reports from 2016 (old data). An additional 1,385 randomly selected reports from 2019 to 2020 (new data) were annotated for follow-up recommendations and randomly divided into two sets: training (n = 900) and testing (n = 485). Support vector machine and random forest (RF) algorithms were constructed and trained using 900 new data reports plus old data (augmented data, new models) and using only new data (new data, new models). The 2016 baseline model was used as comparator as is and trained with augmented data. Recall was compared with baseline using McNemar's test.

RESULTS

Follow-up recommendations were contained in 11.3% of reports (157 or 1,385). The baseline model retrained with new data had precision = 0.83 and recall = 0.54; none significantly different from baseline. A new RF model trained with augmented data had significantly better recall versus the baseline model (0.80 versus 0.66, P = .04) and comparable precision (0.90 versus 0.86).

DISCUSSION

ML methods for monitoring follow-up recommendations in radiology reports suffer model drift over time. A newly developed RF model achieved better recall with comparable precision versus simply retraining a previously trained original model with augmented data. Thus, regularly assessing and updating these models is necessary using more recent historical data.

Overview of Noninterpretive Artificial Intelligence Models for Safety, Quality, Workflow, and Education Applications in Radiology Practice

Artificial intelligence has become a ubiquitous term in radiology over the past several years, and much attention has been given to applications that aid radiologists in the detection of abnormalities and diagnosis of diseases. However, there are many potential applications related to radiologic image quality, safety, and workflow improvements that present equal, if not greater, value propositions to radiology practices, insurance companies, and hospital systems. This review focuses on six major categories for artificial intelligence applications: study selection and protocoling, image acquisition, worklist prioritization, study reporting, business applications, and resident education. All of these categories can substantially affect different aspects of radiology practices and workflows. Each of these categories has different value propositions in terms of whether they could be used to increase efficiency, improve patient safety, increase revenue, or save costs. Each application is covered in depth in the context of both current and future areas of work. Keywords: Use of AI in Education, Application Domain, Supervised Learning, Safety © RSNA, 2022.

Improving Results Management Processes in an Acute Hospital Using a Multi-Faceted Approach

BACKGROUND

Radiological examinations and laboratory tests are routinely ordered by hospital physicians as part of the care plan to diagnose and treat patients. However, the failure to actively review and follow-up on these results pose a significant problem to patient safety. A study team was formed to mitigate the clinical risks of poor results management, which was identified as a top clinical risk in our organisation, in order to make improvements to the results management process and to ensure the timely review, acknowledgement, and follow-up of test results.

METHODS

The institutional expectations of results management were set and published as a hospital policy, which was communicated to all clinical departments for compliance. Improvements to the electronic medical records system were made to facilitate the results acknowledgement process, and physicians were engaged to educate them on the importance of results management.

RESULTS

The study team observed a decrease in unacknowledged results from approximately 16,000 in March 2017 to 2673 in December 2020. The compliance rate for acknowledgement results increased from a monthly average of 83.7% (from March to December 2017) to a monthly average of 99.3% (in 2020). The risk score for results management decreased from 16 to 6.5, and was excluded from the organisation's top clinical risks.

CONCLUSION

This study showed the importance of both system improvements and culture changes that are required to improve the process of results management, and provides a step forward for the hospital to safeguard patient safety and mitigate clinical risk.

Use of a PACS Embedded System for Communicating Radiologist to Technologist Learning Opportunities and Patient Callbacks

OBJECTIVE

This study aimed to determine effect of modality, care setting, and radiology subspecialty on frequency of diagnostic image quality issues identified by radiologists during image interpretation.

METHODS

This Institutional Review Board-exempt retrospective study was performed 10/1/18-6/30/20 at an academic radiology practice performing 700,000+ examinations annually. A closed-loop communication tool integrated in PACS workflow enabled radiologists to alert technologists to image quality issues. Radiologists categorized communications as requiring patient callback, or as technologist learning opportunities if image quality was adequate to generate a diagnostic report. Fisher's exact test assessed impact of imaging modality, radiology subspecialty, and care setting on radiologist-identified image quality issues.

RESULTS

976,915 imaging examinations were performed during the study period. Radiologists generated 1,935 technologist learning opportunities (0.20%) and 208 callbacks (0.02%). Learning opportunity rates were highest for MRI (0.60%) when compared to CT (0.26%) and radiography (0.08%) (p<0.0001). The same was true for patient callbacks (MRI 0.13%, CT 0.02%, radiography 0.0006%; p<0.0001). Outpatient examinations generated more learning opportunities (1479/637,092; 0.23%) vs. inpatient (305/200,206; 0.15%) and Emergency Department (151/139,617; 0.11%) (p<0.0001). Abdominal subspecialists were most likely to generate learning opportunities when compared to other subspecialists and cardiovascular imagers were most likely to call a patient back.

CONCLUSIONS

Image quality issues identified by radiologists during the interpretation process were rare and 10 times more commonly categorized as learning opportunities not interfering with a clinically adequate report than as requiring patient callback. Further work is necessary to determine if creating learning opportunities leads to fewer patients requiring repeat examinations.

Integrity of clinical information in radiology reports documenting pulmonary nodules

OBJECTIVE

Quantify the integrity, measured as completeness and concordance with a thoracic radiologist, of documenting pulmonary nodule characteristics in CT reports and assess impact on making follow-up recommendations.

MATERIALS AND METHODS

This Institutional Review Board-approved, retrospective cohort study was performed at an academic medical center. Natural language processing was performed on radiology reports of CT scans of chest, abdomen, or spine completed in 2016 to assess presence of pulmonary nodules, excluding patients with lung cancer, of which 300 reports were randomly sampled to form the study cohort. Documentation of nodule characteristics were manually extracted from reports by 2 authors with 20% overlap. CT images corresponding to 60 randomly selected reports were further reviewed by a thoracic radiologist to record nodule characteristics. Documentation completeness for all characteristics were reported in percentage and compared using χ2 analysis. Concordance with a thoracic radiologist was reported as percentage agreement; impact on making follow-up recommendations was assessed using kappa.

RESULTS

Documentation completeness for pulmonary nodule characteristics differed across variables (range = 2%-90%, P < .001). Concordance with a thoracic radiologist was 75% for documenting nodule laterality and 29% for size. Follow-up recommendations were in agreement in 67% and 49% of reports when there was lack of completeness and concordance in documenting nodule size, respectively.

DISCUSSION

Essential pulmonary nodule characteristics were under-reported, potentially impacting recommendations for pulmonary nodule follow-up.

CONCLUSION

Lack of documentation of pulmonary nodule characteristics in radiology reports is common, with potential for compromising patient care and clinical decision support tools.

Improvement in Follow-up Imaging With a Patient Tracking System and Computerized Registry for Lung Nodule Management

PURPOSE

Despite established guidelines, radiologists' recommendations and timely follow-up of incidental lung nodules remain variable. To improve follow-up of nodules, a system using standardized language (tracker phrases) recommending time-based follow-up in chest CT reports, coupled with a computerized registry, was created.

MATERIALS AND METHODS

Data were obtained from the electronic health record and a facility-built electronic lung nodule registry. We evaluated two randomly selected patient cohorts with incidental nodules on chest CT reports: before intervention (September 2008 to March 2011) and after intervention (August 2011 to December 2016). Multivariable logistic regression was used to compare the cohorts for the main outcome of timely follow-up, defined as a subsequent report within 13 months of the initial report.

RESULTS

In all, 410 patients were included in the pretracker cohort versus 626 in the tracker cohort. Before system inception, 30% of CT reports lacked an explicit time-based recommendation for nodule follow-up. The proportion of patients with timely follow-up increased from 46% to 55%, and the proportion of those with no documented follow-up or follow-up beyond 24 months decreased from 48% to 31%. The likelihood of timely follow-up increased 41%, adjusted for high risk for lung cancer and age 65 years or older. After system inception, reports missing a tracker phrase for nodule recommendation averaged 6%, without significant interyear variation.

CONCLUSIONS

Standardized language added to CT reports combined with a computerized registry designed to identify and track patients with incidental lung nodules was associated with improved likelihood of follow-up imaging.

Radiologist Variation in the Rates of Follow-up Imaging Recommendations Made for Pulmonary Nodules

OBJECTIVE

Determine whether differences exist in rates of follow-up recommendations made for pulmonary nodules after accounting for multiple patient and radiologist factors.

METHODS

This Institutional Review Board-approved, retrospective study was performed at an urban academic quaternary care hospital. We analyzed 142,001 chest and abdominal CT reports from January 1, 2016, to December 31, 2018, from abdominal, thoracic, and emergency radiology subspecialty divisions. A previously validated natural language processing (NLP) tool identified 24,512 reports documenting pulmonary nodule(s), excluding reports NLP-positive for lung cancer. A second validated NLP tool identified reports with follow-up recommendations specifically for pulmonary nodules. Multivariable logistic regression was used to determine the likelihood of pulmonary nodule follow-up recommendation. Interradiologist variability was quantified within subspecialty divisions.

RESULTS

NLP classified 4,939 of 24,512 (20.1%) reports as having a follow-up recommendation for pulmonary nodule. Male patients comprised 45.3% (11,097) of the patient cohort; average patient age was 61.4 years (±14.1 years). The majority of reports were from outpatient studies (62.7%, 15,376 of 24,512), were chest CTs (75.9%, 18,615 of 24,512), and were interpreted by thoracic radiologists (63.7%, 15,614 of 24,512). In multivariable analysis, studies for male patients (odds ratio [OR]: 0.9 [0.8-0.9]) and abdominal CTs (OR: 0.6 [0.6-0.7] compared with chest CT) were less likely to have a pulmonary nodule follow-up recommendation. Older patients had higher rates of follow-up recommendation (OR: 1.01 for each additional year). Division-level analysis showed up to 4.3-fold difference between radiologists in the probability of making a follow-up recommendation for a pulmonary nodule.

DISCUSSION

Significant differences exist in the probability of making a follow-up recommendation for pulmonary nodules among radiologists within the same subspecialty division.

Closing the Loop-A Pilot in Health System Improvement

A significant number of patients are reported to not receive timely completion of their recommended follow-up intervention following the interpretation of their imaging studies, contributing to patient deaths resulting from inaccurate or delayed diagnosis. Though automated critical test notification systems and computerized communication mechanisms currently exist, many institutions are discovering that there continue to be gaps in the completion of follow-up recommendations. Herein, we describe how we developed and implemented a closed-loop program dedicated to identifying such gaps and ensuring patients were aware of and received appropriate follow-up.

Ambulatory Safety Nets to Reduce Missed and Delayed Diagnoses of Cancer

BACKGROUND

An ambulatory safety net (ASN) is an innovative organizational intervention for addressing patient safety related to missed and delayed diagnoses of abnormal test results. ASNs consist of a set of tools, reports and registries, and associated work flows to create a high-reliability system for abnormal test result management.

METHODS

Two ASNs implemented at an academic medical center are described, one focusing on colon cancer and the other on lung cancer. Data from electronic registries and chart reviews were used to evaluate the effectiveness of the ASNs, which were defined as follows: colon cancer-the proportion of patients who were scheduled for or completed a colonoscopy following safety net team outreach to the patient; lung cancer-the proportion of patients for whom the safety net was able to identify and implement appropriate follow-up, as defined by scheduled or completed chest CT.

RESULTS

The effectiveness of the colon cancer ASN was 44.0%, and the effectiveness of the lung cancer ASN was 56.9%. The ASNs led to the development of registries to address patient safety, fostered collaboration among interdisciplinary teams of clinicians and administrative staff, and created new work flows for patient outreach and tracking.

CONCLUSION

Two ASNs were successfully implemented at an academic medical center to address missed and delayed recognition of abnormal test results related to colon cancer and lung cancer. The ASNs are providing a framework for development of additional safety nets in the organization.

Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports

PURPOSE

The aims of this study were to assess follow-up recommendations in radiology reports, develop and assess traditional machine learning (TML) and deep learning (DL) models in identifying follow-up, and benchmark them against a natural language processing (NLP) system.

METHODS

This HIPAA-compliant, institutional review board-approved study was performed at an academic medical center generating >500,000 radiology reports annually. One thousand randomly selected ultrasound, radiography, CT, and MRI reports generated in 2016 were manually reviewed and annotated for follow-up recommendations. TML (support vector machines, random forest, logistic regression) and DL (recurrent neural nets) algorithms were constructed and trained on 850 reports (training data), with subsequent optimization of model architectures and parameters. Precision, recall, and F1 score were calculated on the remaining 150 reports (test data). A previously developed and validated NLP system (iSCOUT) was also applied to the test data, with equivalent metrics calculated.

RESULTS

Follow-up recommendations were present in 12.7% of reports. The TML algorithms achieved F1 scores of 0.75 (random forest), 0.83 (logistic regression), and 0.85 (support vector machine) on the test data. DL recurrent neural nets had an F1 score of 0.71; iSCOUT also had an F1 score of 0.71. Performance of both TML and DL methods by F1 scores appeared to plateau after 500 to 700 samples while training.

CONCLUSIONS

TML and DL are feasible methods to identify follow-up recommendations. These methods have great potential for near real-time monitoring of follow-up recommendations in radiology reports.