Monitoring performance of clinical artificial intelligence in health care: a scoping review

Enhancing professional communication training in higher education through artificial intelligence(AI)-integrated exercises: study protocol for a randomised controlled trial

BACKGROUND

Effective communication skills are fundamental for health care professionals, yet conventional training methods face challenges in scalability and accessibility due to resource constraints. The emergence of artificial intelligence (AI), particularly generative AI, offers innovative ways for enhancing communication skills training by simulating realistic conversational scenarios and providing personalised, adaptive feedback. This manuscript is presented as study protocol for a cluster-randomised controlled trial that aims at evaluating the efficacy of an AI-supported higher education training protocol incorporating generative AI exercises to enhance communication competencies among psychology students.

METHODS

In this cluster-randomised controlled trial, psychology students enrolled in communication skill seminars at a medium sized university in a medium sized German city will participate. Classes will be assigned within a parallel group design to the AI condition (AI-enhanced exercises alongside teaching-as-usual, TAU, that includes classical exercises) or the control condition (TAU only). Additional non-randomised comparison classes will comprise students with TAU only, but not be part of main analyses. The primary outcome is the change in communication skills from baseline, assessed through questions reflecting the communication techniques emphasised in the training. Secondary outcomes include communication skills, self-efficacy and self-concept, motivation, attitudes toward AI, user experience with the AI tool, student evaluations of course quality, and feasibility aspects such as uptake and usability. Data will be collected via online surveys and the university's teaching platform. Statistical analyses will employ mixed models to evaluate the intervention's impact.

DISCUSSION

This study will provide empirical evidence on the effectiveness and feasibility of integrating AI into higher education communication skills training. Successful integration of AI-enhanced training could revolutionise educational practices by offering scalable, accessible, and personalised learning experiences. The findings may have broader implications for incorporating AI tools in various educational and professional training contexts, while addressing ethical considerations and promoting responsible use of AI in education.

TRIAL REGISTRATION

This trial has been pre-registered on the Open Science Framework (OSF) under identifier 'th6f4'.

How Do Radiologists Currently Monitor AI in Radiology and What Challenges Do They Face? An Interview Study and Qualitative Analysis

Artificial intelligence (AI) in radiology is becoming increasingly prevalent; however, there is not a clear picture of how AI is being monitored today and how this should practically be done given the inherent risk of AI model performance degradation over time. This research investigates current practices and what difficulties radiologists face in monitoring AI. Semi-structured virtual interviews were conducted with 6 USA and 10 Europe-based radiologists. The interviews were automatically transcribed and underwent thematic analysis. The findings suggest that AI monitoring in radiology is still relatively nascent as most of the AI projects had not yet progressed into a fully live clinical deployment. The most common method of monitoring involved a manual process of retrospectively comparing the AI results against the radiology report. Automated and statistical methods of monitoring were much less common. The biggest challenges are a lack of resources to support AI monitoring and uncertainty about how to create a robust and scalable process of monitoring the breadth and variety of radiology AI applications available. There is currently a lack of practical guidelines on how to monitor AI which has led to a variety of approaches being proposed from both healthcare providers and vendors. An ensemble of mixed methods is recommended to monitor AI across multiple domains and metrics. This will be enabled by appropriate allocation of resources and the formation of robust and diverse multidisciplinary AI governance groups.