Advancing acceptance: assessing acceptance of the ESR iGuide clinical decision support system for improved computed tomography test justification

Background

A clinical decision support system (CDSS), the European Society of Radiologists (ESR) iGuide, was developed to address gaps in the availability and use of effective imaging referral guidelines.

Aim

This study aimed to assess the appropriateness of computed tomography (CT) exams with and without ESR iGuide use, as well as the usability and acceptance of the physician systems.

Methods

A retrospective single-center study was conducted in which data from 278 consecutive CT tests referred by physicians were collected in the first phase (T1), and physicians used the ESR iGuide system for imaging referrals in the second phase (T2; n = 85). The appropriateness of imaging referrals in each phase was assessed by two experts, and physicians completed the System Usability Scale.

Results

The mean appropriateness level on a scale of 0-9 was 6.62 ± 2.69 at T1 and 7.88 ± 1.4 at T2. When using a binary variable (0-6 = non-appropriate; 7-9 = appropriate), 70.14% of cases were found appropriate at T1 and 96.47% at T2. Surgery physician specialty and post-intervention phase showed a higher likelihood of ordering an appropriate test (p = 0.0045 and p = 0.0003, respectively). However, the questionnaire results indicated low system trust and minimal clinical value, with all physicians indicating they would not recommend collegial use (100%).

Conclusion

The study suggests that ESR iGuide can effectively guide the selection of appropriate imaging tests. However, physicians showed low system trust and use, indicating a need for further understanding of CDSS acceptance properties. Maximizing CDSS potential could result in crucial decision-support compliance and promotion of appropriate imaging.

Evaluating the reliability of ChatGPT as a tool for imaging test referral: a comparative study with a clinical decision support system

OBJECTIVES

As the technology continues to evolve and advance, we can expect to see artificial intelligence (AI) being used in increasingly sophisticated ways to make a diagnosis and decisions such as suggesting the most appropriate imaging referrals. We aim to explore whether Chat Generative Pretrained Transformer (ChatGPT) can provide accurate imaging referrals for clinical use that are at least as good as the ESR iGuide.

METHODS

A comparative study was conducted in a tertiary hospital. Data was collected from 97 consecutive cases that were admitted to the emergency department with abdominal complaints. We compared the imaging test referral recommendations suggested by the ESR iGuide and the ChatGPT and analyzed cases of disagreement. In addition, we selected cases where ChatGPT recommended a chest abdominal pelvis (CAP) CT (n = 66), and asked four specialists to grade the appropriateness of the referral.

RESULTS

ChatGPT recommendations were consistent with the recommendations provided by the ESR iGuide. No statistical differences were found between the appropriateness of referrals by age or gender. Using a sub-analysis of CAP cases, a high agreement between ChatGPT and the specialists was found. Cases of disagreement (12.4%) were further analyzed and presented themes of vague recommendations such as "it would be advisable" and "this would help to rule out."

CONCLUSIONS

ChatGPT's ability to guide the selection of appropriate tests may be comparable to some degree with the ESR iGuide. Features such as the clinical, ethical, and regulatory implications are still warranted and need to be addressed prior to clinical implementation. Further studies are needed to confirm these findings.

CLINICAL RELEVANCE STATEMENT

The article explores the potential of using advanced language models, such as ChatGPT, in healthcare as a CDS for selecting appropriate imaging tests. Using ChatGPT can improve the efficiency of the decision-making process KEY POINTS: • ChatGPT recommendations were highly consistent with the recommendations provided by the ESR iGuide. • ChatGPT's ability in guiding the selection of appropriate tests may be comparable to some degree with ESR iGuide's.

Does clinical decision support system promote expert consensus for appropriate imaging referrals? Chest-abdominal-pelvis CT as a case study

OBJECTIVES

We assessed the appropriateness of chest-abdominal-pelvis (CAP) CT scan use in the Emergency Department (ED), based on expert physicians and the ESR iGuide, a clinical decision support system (CDSS).

METHODS

A retrospective cross-study was conducted. We included 100 cases of CAP-CT scans ordered at the ED. Four experts rated the appropriateness of the cases on a 7-point scale, before and after using the decision support tool.

RESULTS

Before using the ESR iGuide the overall mean rating of the experts was 5.2 ± 1.066, and it increased slightly after using the system (5.85 ± 0.911 (p < 0.01)). Using a threshold of 5 (on a 7-level scale), the experts considered only 63% of the tests appropriate before using the ESR iGuide. The number increased to 89% after consultation with the system. The degree of overall agreement among the experts was 0.388 before ESR iGuide consultation and 0.572 after consultation. According to the ESR iGuide, for 85% of the cases, CAP CT was not a recommended option (score 0). Abdominal-Pelvis CT was "usually appropriate" for 65 out of the 85 (76%) cases (score 7-9). 9% of the cases did not require CT as first exam modality.

CONCLUSIONS

According to both the experts and the ESR iGuide, inappropriate testing was prevalent, in terms of both frequency of the scans and also inappropriately chosen body regions. These findings raise the need for unified workflows that might be achieved using a CDSS. Further studies are needed to investigate the CDSS contribution to the informed decision-making and increased uniformity among different expert physicians when ordering the appropriate test.

European Society of Radiology (ESR). Summary of the proceedings of the international forum 2016: "Imaging referral guidelines and clinical decision support - how can radiologists implement imaging referral guidelines in clinical routine?"

The International Forum is held once a year by the ESR and its international radiological partner societies with the aim to address and discuss selected subjects of global relevance in radiology. In 2016, the issue of implementing imaging referral guidelines in clinical routine was analysed. The legal environment in the USA requires that after January 1, 2017, physicians must consult government-approved, evidence-based appropriate-use criteria through a clinical decision support system when ordering advanced diagnostic imaging exams. The ESR and the National Decision Support Company are developing "ESR iGuide", a clinical decision support system for European imaging referral guidelines using ESR imaging referral guidelines based on ACR Appropriateness Criteria. In many regions of the world, the situation is different and quite diverse, depending on the specific features of health care systems in different countries, but there are, unlike in the USA and EU, no legal obligations to implement imaging referral guidelines into the clinical practice. Imaging referral guidelines and clinical decision support implementation is a complex issue everywhere and the legal environment surrounding it even more so; how they will be implemented into the clinical practice in different areas of the world needs yet to be decided.

MAIN MESSAGES

• Implementation of imaging referral guidelines in clinical routine is needed. • Potential benefits are improved appropriateness in referrals and reduction of unnecessary radiation exposure. • The educational benefits include new insights through data collection and reporting. • The system will potentially highlight the lack in quality or availability of equipment.