Clinical decision support system recommendations: how often do radiologists and clinicians accept them?

Comparison of CT referral justification using clinical decision support and large language models in a large European cohort

BACKGROUND

Ensuring appropriate use of CT scans is critical for patient safety and resource optimization. Decision support tools and artificial intelligence (AI), such as large language models (LLMs), have the potential to improve CT referral justification, yet require rigorous evaluation against established standards and expert assessments.

AIM

To evaluate the performance of LLMs (Generation Pre-trained Transformer 4 (GPT-4) and Claude-3 Haiku) and independent experts in justifying CT referrals compared to the ESR iGuide clinical decision support system as the reference standard.

METHODS

CT referral data from 6356 patients were retrospectively analyzed. Recommendations were generated by the ESR iGuide, LLMs, and independent experts, and evaluated for accuracy, precision, recall, F1 score, and Cohen's kappa across medical test, organ, and contrast predictions. Statistical analysis included demographic stratification, confidence intervals, and p-values to ensure robust comparisons.

RESULTS

Independent experts achieved the highest accuracy (92.4%) for medical test justification, surpassing GPT-4 (88.8%) and Claude-3 Haiku (85.2%). For organ predictions, LLMs performed comparably to experts, achieving accuracies of 75.3-77.8% versus 82.6%. For contrast predictions, GPT-4 showed the highest accuracy (57.4%) among models, while Claude demonstrated poor agreement with guidelines (kappa = 0.006).

CONCLUSION

Independent experts remain the most reliable, but LLMs show potential for optimization, particularly in organ prediction. A hybrid human-AI approach could enhance CT referral appropriateness and utilization. Further research should focus on improving LLM performance and exploring their integration into clinical workflows.

KEY POINTS

Question Can GPT-4 and Claude-3 Haiku justify CT referrals as accurately as independent experts, using the ESR iGuide as the gold standard? Findings Independent experts outperformed large language models in test justification. GPT-4 and Claude-3 showed comparable organ prediction but struggled with contrast selection, limiting full automation. Clinical relevance While independent experts remain most reliable, integrating AI with expert oversight may improve CT referral appropriateness, optimizing resource allocation and enhancing clinical decision-making.

Evaluation of a Clinical Decision Support System for Imaging Requests: A Cluster Randomized Clinical Trial

Importance

Given the widespread use of medical imaging, evaluating the effectiveness of interventions to improve appropriateness is crucial for optimizing health care resources and patient outcomes.

Objective

To assess the effects of implementing a clinical decision support system (CDSS), the European Society of Radiology iGuide, on the appropriateness of the medical imaging ordering behavior of physicians.

Design and Setting

A cluster randomized clinical trial with 26 departments at 3 German university hospitals acting as clusters, incorporating a before and after discontinued design. All imaging requests originating from physicians in the participating departments over a 2.5-year period were included (between December 2021 and June 2024).

Interventions

All departments started without a CDSS and required structured clinical indication data entry and tracking of requested imaging. After randomization, 13 clusters (departments at hospitals) received the CDSS intervention (intervention clusters) and 13 clusters did not (control clusters). The CDSS intervention provided ordering physicians with information as to whether their imaging requests were appropriate, appropriate under certain conditions, or inappropriate; in addition, alternative diagnostic tests, including the corresponding appropriateness score, were suggested by the CDSS, after which physicians could choose to modify their imaging requests.

Main Outcomes and Measures

The primary outcome measure was the proportion of inappropriate imaging requests made per department. A difference-in-differences analysis was used to investigate changes in the proportion of inappropriate imaging requests between departments with vs those without the CDSS.

Results

A total of 65 764 imaging requests were scored using the CDSS; 50.1% of imaging requests were for female patients and the mean patient age was 64 years (SD, 17.1 years). Prior to implementation of the CDSS, there were 21 625 imaging requests from the control clusters, 1367 (6.3%) of which were categorized as inappropriate; and there were 13 338 imaging requests from the intervention clusters, 1007 (7.6%) of which were categorized as inappropriate. After implementation of the CDSS, there were 10 055 imaging requests from the control clusters, 518 (5.2%) of which were categorized as inappropriate; and there were 7206 imaging requests from the intervention clusters, 461 (6.4%) of which were categorized as inappropriate. The intervention clusters showed a similar reduction (mean difference, -0.5% [99% CI, -2.4% to 0.4%]) in inappropriate imaging requests compared with the control clusters (mean difference, -1.8% [99% CI, -4.3% to -0.4%]) and there was a difference-in-differences value of 1.3 percentage points (99% CI, -2.0 to 1.8 percentage points; P = .69), which was not statistically significant.

Conclusions and Relevance

The CDSS did not reduce the number of inappropriate imaging requests ordered by physicians in academic hospital settings.

Trial Registration

ClinicalTrials.gov Identifier: NCT05490290.

Computed tomography referral guidelines adherence in Europe: insights from a seven-country audit

BACKGROUND

Ensuring appropriate computed tomography (CT) utilization optimizes patient care while minimizing radiation exposure. Decision support tools show promise for standardizing appropriateness.

OBJECTIVES

In the current study, we aimed to assess CT appropriateness rates using the European Society of Radiology (ESR) iGuide criteria across seven European countries. Additional objectives were to identify factors associated with appropriateness variability and examine recommended alternative exams.

METHODS

As part of the European Commission-funded EU-JUST-CT project, 6734 anonymized CT referrals were audited across 125 centers in Belgium, Denmark, Estonia, Finland, Greece, Hungary, and Slovenia. In each country, two blinded radiologists independently scored each exam's appropriateness using the ESR iGuide and noted any recommended alternatives based on presented indications. Arbitration was used in case auditors disagreed. Associations between appropriateness rate and institution type, patient's age and sex, inpatient/outpatient patient status, anatomical area, and referring physician's specialty were statistically examined within each country.

RESULTS

The average appropriateness rate was 75%, ranging from 58% in Greece to 86% in Denmark. Higher rates were associated with public hospitals, inpatient settings, and referrals from specialists. Variability in appropriateness existed by country and anatomical area, patient age, and gender. Common alternative exam recommendations included magnetic resonance imaging, X-ray, and ultrasound.

CONCLUSION

This multi-country evaluation found that even when using a standardized imaging guideline, significant variations in CT appropriateness persist, ranging from 58% to 86% across the participating countries. The study provided valuable insights into real-world utilization patterns and identified opportunities to optimize practices and reduce clinical and demographic disparities in CT use.

KEY POINTS

Question Largest multinational study (7 EU countries, 6734 CT referrals) assessed real-world CT appropriateness using ESR iGuide, enabling cross-system comparisons. Findings Significant variability in appropriateness rates across institution type, patient status, age, gender, exam area, and physician specialty, highlighted the opportunities to optimize practices based on local factors. Clinical relevance International collaboration on imaging guidelines and decision support can maximize CT benefits while optimizing radiation exposure; ongoing research is crucial for refining evidence-based guidelines globally.

Evaluating the Accuracy and Impact of the ESR-iGuide Decision Support Tool in Optimizing CT Imaging Referral Appropriateness

Radiology referral quality impacts patient care, yet factors influencing quality are poorly understood. This study assessed the quality of computed tomography (CT) referrals, identified associated characteristics, and evaluated the ESR-iGuide clinical decision support tool's ability to optimize referrals. A retrospective review analyzed 300 consecutive CT referrals from an acute care hospital. Referral quality was evaluated on a 5-point scale by three expert reviewers (inter-rater reliability κ = 0.763-0.97). The ESR-iGuide tool provided appropriateness scores and estimated radiation exposure levels for the actual referred exams and recommended exams. Scores were compared between actual and recommended exams. Associations between ESR-iGuide scores and referral characteristics, including the specialty of the ordering physician (surgical vs. non-surgical), were explored. Of the referrals, 67.1% were rated as appropriate. The most common exams were head and abdomen/pelvis CTs. The ESR-iGuide deemed 70% of the actual referrals "usually appropriate" and found that the recommended exams had lower estimated radiation exposure compared to the actual exams. Logistic regression analysis showed that non-surgical physicians were more likely to order inappropriate exams compared to surgical physicians. Over one-third of the referrals showed suboptimal quality in the unstructured system. The ESR-iGuide clinical decision support tool identified opportunities to optimize appropriateness and reduce radiation exposure. Implementation of such a tool warrants consideration to improve communication and maximize patient care quality.

Insights from a web-based questionnaire: examining diagnostic procedures prior to magnetic resonance imaging

The appropriate use of diagnostic imaging, particularly MRI, is a critical concern in modern healthcare. This paper examines the current state of MRI utilization in Israel, drawing on a recent study by Kaim et al. that surveyed 557 Israeli adults who underwent MRI in the public health system. The study revealed that 60% of participants had undergone other imaging tests before their MRI, with 23% having more than one prior examination. While these findings highlight potential inefficiencies in the diagnostic pathway, they also underscore the complexity of medical decision-making in imaging.The paper discusses various factors influencing MRI utilization, including regulatory pressures, healthcare system structure, and the need for evidence-based guidelines. It explores potential strategies for optimizing MRI justification and scheduling, such as implementing clinical decision support systems, enhancing interdisciplinary communication, and leveraging artificial intelligence (AI) for predictive analytics and resource optimization.The need for comprehensive research into MRI justification and scheduling optimization is presented. Key areas for investigation include the effectiveness of decision support tools, patient outcomes, economic analyses, and the application of quality improvement methodologies.

Exploring pre-MRI imaging tests: patient survey reveals potential implications for healthcare efficiency in Israel

BACKGROUND

Medical imaging tests are vital in healthcare but can be costly, impacting national health expenditures. Magnetic resonance imaging (MRI) is a crucial diagnostic tool for assessing medical conditions. However, the rising demand for MRI scans has frequently strained available resources. This study aimed to estimate the prevalence of different imaging tests in individuals who eventually had an MRI, in the Israeli public health system.

METHODS

An online survey of patient experience of scheduling an MRI was conducted in January-February 2023, among 557 Israeli adults, representing all four health maintenance organizations (HMOs). All participants had undergone an MRI in the public health system within the past year.

RESULTS

Results showed that 60% of participants underwent other imaging tests before their MRI scan. Of those, computed tomography (CT) scans (43%), X-rays (39%), and ultrasounds (32%) were the most common additional imaging procedures. In addition, of the 60% of participants, 23% had undergone more than one prior imaging examination.

CONCLUSIONS

These findings highlight the high prevalence of preliminary imaging tests prior to MRI, with many patients undergoing multiple tests for the same problem. The health system may need to evaluate whether current clinical guidelines defining the use of various imaging tests are cost-effective.

A content-aware chatbot based on GPT 4 provides trustworthy recommendations for Cone-Beam CT guidelines in dental imaging

OBJECTIVES

To develop a content-aware chatbot based on GPT-3.5-Turbo and GPT-4 with specialized knowledge on the German S2 Cone-Beam CT (CBCT) dental imaging guideline and to compare the performance against humans.

METHODS

The LlamaIndex software library was used to integrate the guideline context into the chatbots. Based on the CBCT S2 guideline, 40 questions were posed to content-aware chatbots and early career and senior practitioners with different levels of experience served as reference. The chatbots' performance was compared in terms of recommendation accuracy and explanation quality. Chi-square test and one-tailed Wilcoxon signed rank test evaluated accuracy and explanation quality, respectively.

RESULTS

The GPT-4 based chatbot provided 100% correct recommendations and superior explanation quality compared to the one based on GPT3.5-Turbo (87.5% vs. 57.5% for GPT-3.5-Turbo; P = .003). Moreover, it outperformed early career practitioners in correct answers (P = .002 and P = .032) and earned higher trust than the chatbot using GPT-3.5-Turbo (P = 0.006).

CONCLUSIONS

A content-aware chatbot using GPT-4 reliably provided recommendations according to current consensus guidelines. The responses were deemed trustworthy and transparent, and therefore facilitate the integration of artificial intelligence into clinical decision-making.

Advancing acceptance: assessing acceptance of the ESR iGuide clinical decision support system for improved computed tomography test justification

Background

A clinical decision support system (CDSS), the European Society of Radiologists (ESR) iGuide, was developed to address gaps in the availability and use of effective imaging referral guidelines.

Aim

This study aimed to assess the appropriateness of computed tomography (CT) exams with and without ESR iGuide use, as well as the usability and acceptance of the physician systems.

Methods

A retrospective single-center study was conducted in which data from 278 consecutive CT tests referred by physicians were collected in the first phase (T1), and physicians used the ESR iGuide system for imaging referrals in the second phase (T2; n = 85). The appropriateness of imaging referrals in each phase was assessed by two experts, and physicians completed the System Usability Scale.

Results

The mean appropriateness level on a scale of 0-9 was 6.62 ± 2.69 at T1 and 7.88 ± 1.4 at T2. When using a binary variable (0-6 = non-appropriate; 7-9 = appropriate), 70.14% of cases were found appropriate at T1 and 96.47% at T2. Surgery physician specialty and post-intervention phase showed a higher likelihood of ordering an appropriate test (p = 0.0045 and p = 0.0003, respectively). However, the questionnaire results indicated low system trust and minimal clinical value, with all physicians indicating they would not recommend collegial use (100%).

Conclusion

The study suggests that ESR iGuide can effectively guide the selection of appropriate imaging tests. However, physicians showed low system trust and use, indicating a need for further understanding of CDSS acceptance properties. Maximizing CDSS potential could result in crucial decision-support compliance and promotion of appropriate imaging.

Inappropriate CT examinations: how much, who and where? Insights from a clinical decision support system (CDSS) analysis

OBJECTIVE

To assess the appropriateness of Computed Tomography (CT) examinations, using the ESR-iGuide.

MATERIAL AND METHODS

A retrospective study was conducted in 2022 in a medium-sized acute care teaching hospital. A total of 278 consecutive cases of CT referral were included. For each imaging referral, the ESR-iGuide provided an appropriateness score using a scale of 1-9 and the Relative Radiation Level using a scale of 0-5. These were then compared with the appropriateness score and the radiation level of the recommended ESR-iGuide exam.

DATA ANALYSIS

Pearson's chi-square test or Fisher exact test was used to explore the correlation between ESR-iGuide appropriateness level and physician, patients, and shift characteristics. A stepwise logistic regression model was used to capture the contribution of each of these factors.

RESULTS

Most of exams performed were CT head (63.67%) or CT abdominal pelvis (23.74%). Seventy percent of the actual imaging referrals resulted in an ESR-iGuide score corresponding to "usually appropriate." The mean radiation level for actual exam was 3.2 ± 0.45 compared with 2.16 ± 1.56 for the recommended exam. When using a stepwise logistic regression for modeling the probability of non-appropriate score, both physician specialty and status were significant (p = 0.0011, p = 0.0192 respectively). Non-surgical and specialist physicians were more likely to order inappropriate exams than surgical physicians.

CONCLUSIONS

ESR-iGuide software indicates a substantial rate of inappropriate exams of CT head and CT abdominal-pelvis and unnecessary radiation exposure mainly in the ED department. Inappropriate exams were found to be related to physicians' specialty and seniority.

CLINICAL RELEVANCE STATEMENT

These findings underscore the urgent need for improved imaging referral practices to ensure appropriate healthcare delivery and effective resource management. Additionally, they highlight the potential benefits and necessity of integrating CDSS as a standard medical practice. By implementing CDSS, healthcare providers can make more informed decisions, leading to enhanced patient care, optimized resource allocation, and improved overall healthcare outcomes.

KEY POINTS

• The overall mean of appropriateness for the actual exam according to the ESR-iGuide was 6.62 ± 2.69 on a scale of 0-9. • Seventy percent of the actual imaging referrals resulted in an ESR-iGuide score corresponding to "usually appropriate." • Inappropriate examination is related to both the specialty of the physician who requested the exam and the seniority status of the physician.

American College of Radiology (ACR) Appropriateness Criteria and EURO-2000 Guidelines Offer Limited Guidance for MRI Imaging of Pediatric Patients

PURPOSE

This study aims to evaluate the clinical scenarios addressed by the EURO-2000 guidelines and the ACR appropriateness criteria for referring children to MRI and assessing the referring physician compliance with both guidelines.

METHODS

In January 2022, clinical scenarios reported in the last 1000 consecutive MRI requests for children (0-16 years) in one university children's hospital and two general university hospitals in Belgium, Europe, were retrospectively matched to the EURO-2000 guidelines and ACR appropriateness criteria. The number of clinical scenarios addressed and the justification for MRI referral were independently assessed for both guidelines. Pooled data from the three centers were evaluated and then analyzed by center, body area and prescriber using McNemar's test for paired proportions and χ2-tests unpaired proportions.

RESULTS

After excluding incomplete or missing MRI requests, 2932 of 3000 requests were included in the analysis. Overall, out of 2932 clinical scenarios, 1229 (37.99%) were addressed by EURO-2000 and 1081 (36.37%) were addressed by the ACR appropriateness criteria (McNemar test, p = 0.12). The proportions of clinical scenarios covered by the two guidelines were statistically similar when comparing centers, but varied across body regions (p < 0.001) and referring physician specialty (p between 0.75 and 0.001). EURO-2000 guidelines provided better coverage for head and spine (p < 0.05), while the ACR appropriateness criteria provided broader coverage for abdomen, pelvis, and musculoskeletal system (p < 0.0001). For addressed clinical scenarios, prescriber compliance for both guidelines was excellent with > 94% of justified MRI examinations in all the centers.

CONCLUSIONS

Both the EURO-2000 guidelines and the ACR appropriateness criteria did not address two-thirds of clinical scenarios in children. Head and neck, chest and abdominal-pelvic examinations are the anatomic regions which should receive a specific attention for the future implementation of evidence-based clinical decision support tools for all referring specialists.