Using artificial intelligence to assess clinicians' communication skills

Artificial intelligence and pain medicine education: Benefits and pitfalls for the medical trainee

OBJECTIVES

Artificial intelligence (AI) represents an exciting and evolving technology that is increasingly being utilized across pain medicine. Large language models (LLMs) are one type of AI that has become particularly popular. Currently, there is a paucity of literature analyzing the impact that AI may have on trainee education. As such, we sought to assess the benefits and pitfalls that AI may have on pain medicine trainee education. Given the rapidly increasing popularity of LLMs, we particularly assessed how these LLMs may promote and hinder trainee education through a pilot quality improvement project.

MATERIALS AND METHODS

A comprehensive search of the existing literature regarding AI within medicine was performed to identify its potential benefits and pitfalls within pain medicine. The pilot project was approved by UPMC Quality Improvement Review Committee (#4547). Three of the most commonly utilized LLMs at the initiation of this pilot study - ChatGPT Plus, Google Bard, and Bing AI - were asked a series of multiple choice questions to evaluate their ability to assist in learner education within pain medicine.

RESULTS

Potential benefits of AI within pain medicine trainee education include ease of use, imaging interpretation, procedural/surgical skills training, learner assessment, personalized learning experiences, ability to summarize vast amounts of knowledge, and preparation for the future of pain medicine. Potential pitfalls include discrepancies between AI devices and associated cost-differences, correlating radiographic findings to clinical significance, interpersonal/communication skills, educational disparities, bias/plagiarism/cheating concerns, lack of incorporation of private domain literature, and absence of training specifically for pain medicine education. Regarding the quality improvement project, ChatGPT Plus answered the highest percentage of all questions correctly (16/17). Lowest correctness scores by LLMs were in answering first-order questions, with Google Bard and Bing AI answering 4/9 and 3/9 first-order questions correctly, respectively. Qualitative evaluation of these LLM-provided explanations in answering second- and third-order questions revealed some reasoning inconsistencies (e.g., providing flawed information in selecting the correct answer).

CONCLUSIONS

AI represents a continually evolving and promising modality to assist trainees pursuing a career in pain medicine. Still, limitations currently exist that may hinder their independent use in this setting. Future research exploring how AI may overcome these challenges is thus required. Until then, AI should be utilized as supplementary tool within pain medicine trainee education and with caution.

Assessment of communication skills in health professions education; Ottawa 2024 consensus statement

Despite the increasing inclusion of communication skills in accreditation standards and an increase in time dedicated to teaching these skills, communication is often regarded as a separate skill and is therefore, not consistently represented in overall systems of assessment in Health Professions Education (HPE). The ascendence of competency-based medical education, programmatic assessment, artificial intelligence, and widespread use of telehealth, alongside changing patient expectations warrant an update in thinking about the assessment of communication skills in health professions education. This consensus statement draws on existing literature, expert pinion, and emerging challenges to situate the assessment of communication skills in the contemporary health professions education context. The statement builds on previous work to offer an update on the topic and include new developments related to assessment, particularly: the challenges and opportunities associated with systems of assessment; patient and peer perspectives in assessment; assessment of interprofessional communication, cross-cultural communication, digital communication; and assessment using digital technologies. Consensus was reached through extensive discussion among the authors and other experts in HPE, exploration of the literature, and discussion during an Ottawa 2024 conference workshop. The statement puts forward a summary of available evidence with suggestions for what educators and curriculum developers should consider in their planning and design of the assessment of communication.

The Potential of Artificial Intelligence Tools for Reducing Uncertainty in Medicine and Directions for Medical Education

Unlabelled

In the field of medicine, uncertainty is inherent. Physicians are asked to make decisions on a daily basis without complete certainty, whether it is in understanding the patient's problem, performing the physical examination, interpreting the findings of diagnostic tests, or proposing a management plan. The reasons for this uncertainty are widespread, including the lack of knowledge about the patient, individual physician limitations, and the limited predictive power of objective diagnostic tools. This uncertainty poses significant problems in providing competent patient care. Research efforts and teaching are attempts to reduce uncertainty that have now become inherent to medicine. Despite this, uncertainty is rampant. Artificial intelligence (AI) tools, which are being rapidly developed and integrated into practice, may change the way we navigate uncertainty. In their strongest forms, AI tools may have the ability to improve data collection on diseases, patient beliefs, values, and preferences, thereby allowing more time for physician-patient communication. By using methods not previously considered, these tools hold the potential to reduce the uncertainty in medicine, such as those arising due to the lack of clinical information and provider skill and bias. Despite this possibility, there has been considerable resistance to the implementation of AI tools in medical practice. In this viewpoint article, we discuss the impact of AI on medical uncertainty and discuss practical approaches to teaching the use of AI tools in medical schools and residency training programs, including AI ethics, real-world skills, and technological aptitude.

CUSCO: An Unobtrusive Custom Secure Audio-Visual Recording System for Ambient Assisted Living

The ubiquity of digital technology has facilitated detailed recording of human behaviour. Ambient technology has been used to capture behaviours in a broad range of applications ranging from healthcare and monitoring to assessment of cooperative work. However, existing systems often face challenges in terms of autonomy, usability, and privacy. This paper presents a portable, easy-to-use and privacy-preserving system for capturing behavioural signals unobtrusively in home or in office settings. The system focuses on the capture of audio, video, and depth imaging. It is based on a device built on a small-factor platform that incorporates ambient sensors which can be integrated with the audio and depth video hardware for multimodal behaviour tracking. The system can be accessed remotely and integrated into a network of sensors. Data are encrypted in real time to ensure safety and privacy. We illustrate uses of the device in two different settings, namely, a healthy-ageing IoT application, where the device is used in conjunction with a range of IoT sensors to monitor an older person's mental well-being at home, and a healthcare communication quality assessment application, where the device is used to capture a patient-clinician interaction for consultation quality appraisal. CUSCO can automatically detect active speakers, extract acoustic features, record video and depth streams, and recognise emotions and cognitive impairment with promising accuracy.

Chatbots for future docs: exploring medical students' attitudes and knowledge towards artificial intelligence and medical chatbots

Artificial intelligence (AI) in medicine and digital assistance systems such as chatbots will play an increasingly important role in future doctor - patient communication. To benefit from the potential of this technical innovation and ensure optimal patient care, future physicians should be equipped with the appropriate skills. Accordingly, a suitable place for the management and adaptation of digital assistance systems must be found in the medical education curriculum. To determine the existing levels of knowledge of medical students about AI chatbots in particular in the healthcare setting, this study surveyed medical students of the University of Luebeck and the University Hospital of Tuebingen. Using standardized quantitative questionnaires and qualitative analysis of group discussions, the attitudes of medical students toward AI and chatbots in medicine were investigated. From this, relevant requirements for the future integration of AI into the medical curriculum could be identified. The aim was to establish a basic understanding of the opportunities, limitations, and risks, as well as potential areas of application of the technology. The participants (N = 12) were able to develop an understanding of how AI and chatbots will affect their future daily work. Although basic attitudes toward the use of AI were positive, the students also expressed concerns. There were high levels of agreement regarding the use of AI in administrative settings (83.3%) and research with health-related data (91.7%). However, participants expressed concerns that data protection may be insufficiently guaranteed (33.3%) and that they might be increasingly monitored at work in the future (58.3%). The evaluations indicated that future physicians want to engage more intensively with AI in medicine. In view of future developments, AI and data competencies should be taught in a structured way during the medical curriculum and integrated into curricular teaching.

Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review

BACKGROUND

Communication is a crucial element of every health care profession, rendering communication skills training in all health care professions as being of great importance. Technological advances such as artificial intelligence (AI) and particularly machine learning (ML) may support this cause: it may provide students with an opportunity for easily accessible and readily available communication training.

OBJECTIVE

This scoping review aimed to summarize the status quo regarding the use of AI or ML in the acquisition of communication skills in academic health care professions.

METHODS

We conducted a comprehensive literature search across the PubMed, Scopus, Cochrane Library, Web of Science Core Collection, and CINAHL databases to identify articles that covered the use of AI or ML in communication skills training of undergraduate students pursuing health care profession education. Using an inductive approach, the included studies were organized into distinct categories. The specific characteristics of the studies, methods and techniques used by AI or ML applications, and main outcomes of the studies were evaluated. Furthermore, supporting and hindering factors in the use of AI and ML for communication skills training of health care professionals were outlined.

RESULTS

The titles and abstracts of 385 studies were identified, of which 29 (7.5%) underwent full-text review. Of the 29 studies, based on the inclusion and exclusion criteria, 12 (3.1%) were included. The studies were organized into 3 distinct categories: studies using AI and ML for text analysis and information extraction, studies using AI and ML and virtual reality, and studies using AI and ML and the simulation of virtual patients, each within the academic training of the communication skills of health care professionals. Within these thematic domains, AI was also used for the provision of feedback. The motivation of the involved agents played a major role in the implementation process. Reported barriers to the use of AI and ML in communication skills training revolved around the lack of authenticity and limited natural flow of language exhibited by the AI- and ML-based virtual patient systems. Furthermore, the use of educational AI- and ML-based systems in communication skills training for health care professionals is currently limited to only a few cases, topics, and clinical domains.

CONCLUSIONS

The use of AI and ML in communication skills training for health care professionals is clearly a growing and promising field with a potential to render training more cost-effective and less time-consuming. Furthermore, it may serve learners as an individualized and readily available exercise method. However, in most cases, the outlined applications and technical solutions are limited in terms of access, possible scenarios, the natural flow of a conversation, and authenticity. These issues still stand in the way of any widespread implementation ambitions.

Proof of Concept: Using ChatGPT to Teach Emergency Physicians How to Break Bad News

Background Breaking bad news is an essential skill for practicing physicians, particularly in the field of emergency medicine (EM). Patient-physician communication teaching has previously relied on standardized patient scenarios and objective structured clinical examination formats. The novel use of artificial intelligence (AI) chatbot technology, such as Chat Generative Pre-trained Transformer (ChatGPT), may provide an alternative role in graduate medical education in this area. As a proof of concept, the author demonstrates how providing detailed prompts to the AI chatbot can facilitate the design of a realistic clinical scenario, enable active roleplay, and deliver effective feedback to physician trainees. Methods ChatGPT-3.5 language model was utilized to assist in the roleplay of breaking bad news. A detailed input prompt was designed to outline rules of play and grading assessment via a standardized scale. User inputs (physician role), chatbot outputs (patient role) and ChatGPT-generated feedback were recorded. Results ChatGPT set up a realistic training scenario on breaking bad news based on the initial prompt. Active roleplay as a patient in an emergency department setting was accomplished, and clear feedback was provided to the user through the application of the Setting up, Perception, Invitation, Knowledge, Emotions with Empathy, and Strategy or Summary (SPIKES) framework for breaking bad news. Conclusion The novel use of AI chatbot technology to assist educators is abundant with potential. ChatGPT was able to design an appropriate scenario, provide a means for simulated patient-physician roleplay, and deliver real-time feedback to the physician user. Future studies are required to expand use to a targeted group of EM physician trainees and provide best practice guidelines for AI use in graduate medical education.

An Evolution of Reporting: Identifying the Missing Link

In recent years, radiologic imaging has undergone tremendous technological advances and is now a pillar of diagnostic and treatment algorithms in clinical medicine. The increased complexity and volume of medical imaging has led clinicians to become ever more reliant on radiologists to both identify and interpret patient studies. A radiologist’s report provides key insights into a patient’s immediate state of health, information that is vital when choosing the most appropriate next steps in management. As errors in imaging interpretation or miscommunication of results can greatly impair patient care, identifying common error sources is vital to minimizing their occurrence. Although mistakes in medical imaging are practically inevitable, changes to the delivery of imaging reporting and the addition of artificial intelligence algorithms to analyze clinicians’ communication skills can minimize the impact of these errors, keep up with the continuously evolving landscape of medical imaging, and ultimately close the communication gap.

Toward a basic science of communication in serious illness

High-quality communication can mitigate suffering during serious illness. Innovations in theory and technology present the opportunity to advance serious illness communication research, moving beyond inquiry that links broad communication constructs to health outcomes toward operationalizing and understanding the impact of discrete communication functions on human experience. Given the high stakes of communication during serious illness, we see a critical need to develop a basic science approach to serious illness communication research. Such an approach seeks to link "what actually happens during a conversation" - the lexical and non-lexical communication content elements, as well as contextual factors - with the emotional and cognitive experiences of patients, caregivers, and clinicians. This paper defines and justifies a basic science approach to serious illness communication research and outlines investigative and methodological opportunities in this area. A systematic understanding of the building blocks of serious illness communication can help identify evidence-informed communication strategies that promote positive patient outcomes, shape more targeted communication skills training for clinicians, and lead to more tailored and meaningful serious illness care.

Communication in Health Care: Impact of Language and Accent on Health Care Safety, Quality, and Patient Experience

Permanent or temporary migration results in communication issues related to language barriers. The migrant's mother tongue is often different from that of the host country. Even when the same language is spoken, communication barriers arise because of differences in accent. These communication barriers have a significant negative impact on migrants accessing health care and their ability to understand instructions and seek follow-up care. A multidisciplinary team often has professionals from various countries. These migrant health care professionals find it difficult to communicate with patients of the host country and with their colleagues. Communication barriers, therefore, result in miscommunication or no communication between health care professionals and between health care professionals and patients. This increases the risk of medical errors and impacts quality of care and patient safety. This review looks at the impact of communication barriers in health care and endeavors to find effective solutions.

Multiple approaches to enhancing cancer communication in the next decade: translating research into practice and policy

Communicating risk and other health information in a clear, understandable, and actionable manner is critical for the prevention and control of cancer, as well as the care of affected individuals and their family members. However, the swift pace of development in communication technologies has dramatically changed the health communication landscape. This digital era presents new opportunities and challenges for cancer communication research and its impact on practice and policy. In this article, we examine the science of health communication focused on cancer and highlight important areas of research for the coming decade. Specifically, we discuss three domains in which cancer communication may occur: (a) among patients and their healthcare providers; (b) within and among families and social networks; and (c) across communities, populations, and the public more broadly. We underscore findings from the prior decade of cancer communication research, provide illustrative examples of future directions for cancer communication science, and conclude with considerations for diverse populations. Health informatics studies will be necessary to fully understand the growing and complex communication settings related to cancer: such works have the potential to change the face of information exchanges about cancer and elevate our collective discourse about this area as newer clinical and public health priorities emerge. Researchers from a wide array of specialties are interested in examining and improving cancer communication. These interdisciplinary perspectives can rapidly advance and help translate findings of cancer communication in the field of behavioral medicine.

A general model of conversational dynamics and an example application in serious illness communication

Conversation has been a primary means for the exchange of information since ancient times. Understanding patterns of information flow in conversations is a critical step in assessing and improving communication quality. In this paper, we describe COnversational DYnamics Model (CODYM) analysis, a novel approach for studying patterns of information flow in conversations. CODYMs are Markov Models that capture sequential dependencies in the lengths of speaker turns. The proposed method is automated and scalable, and preserves the privacy of the conversational participants. The primary function of CODYM analysis is to quantify and visualize patterns of information flow, concisely summarized over sequential turns from one or more conversations. Our approach is general and complements existing methods, providing a new tool for use in the analysis of any type of conversation. As an important first application, we demonstrate the model on transcribed conversations between palliative care clinicians and seriously ill patients. These conversations are dynamic and complex, taking place amidst heavy emotions, and include difficult topics such as end-of-life preferences and patient values. We use CODYMs to identify normative patterns of information flow in serious illness conversations, show how these normative patterns change over the course of the conversations, and show how they differ in conversations where the patient does or doesn’t audibly express anger or fear. Potential applications of CODYMs range from assessment and training of effective healthcare communication to comparing conversational dynamics across languages, cultures, and contexts with the prospect of identifying universal similarities and unique “fingerprints” of information flow.

Informatics opportunities to involve patients in hospital safety: a conceptual model

Objective

Inpatients could play an important role in identifying, preventing, and reporting problems in the quality and safety of their care. To support them effectively in that role, informatics solutions must align with their experiences. Thus, we set out to understand how inpatients experience undesirable events (UEs) and to surface opportunities for those informatics solutions.

Materials and Methods

We conducted a survey with 242 patients and caregivers during their hospital stay, asking open-ended questions about their experiences with UEs. Based on our qualitative analysis, we developed a conceptual model representing their experiences and identified informatics opportunities to support patients.

Results

Our 4-stage conceptual model illustrates inpatient experiences, from when they first encounter UEs, when they could intervene, when harms emerge, what types of harms they experience, and what they do in response to harms.

Discussion

Existing informatics solutions address the first stage of inpatients’ experiences by increasing their awareness of potential UEs. However, future researchers can explore new opportunities to fill gaps in support that patients experience in subsequent stages, especially at critical decision points such as intervening in UEs and responding to harms that occur.

Conclusions

Our conceptual model reveals the complex inpatient experiences with UEs, and opportunities for new informatics solutions to support them at all stages of their experience. Investigating these new opportunities could promote inpatients’ participation and engagement in the quality and safety of their care, help healthcare systems learn from inpatients’ experience, and reduce these harmful events.

Artificial Intelligence: Help or Hindrance for Family Physicians?

The use of Artificial Intelligence (AI) and related technologies is rapidly increasing and its application in clinical practice is a promising area of development. Artificial Intelligence can be a solution in the future as a physician’s new assistant; AI-physician combinations can act like models of ‘peaceful co-existence’. While it has the potential to mold many dimensions of patient care and can augment quality improvement, it cannot replace a family physician’s diagnostic intelligence, empathy and relationships. Physicians need to strike a balance between these combinations for better health outcomes without increasing patients’ frustration.

Machine learning to extract communication and history-taking skills in OSCE transcripts

OBJECTIVES

Observed Structured Clinical Exams (OSCEs) allow assessment of, and provide feedback to, medical students. Clinical examiners and standardised patients (SP) typically complete itemised checklists and global scoring scales, which have known shortcomings. In this study, we applied machine learning (ML) to label some communication skills and interview content information in OSCE transcripts and to compare several ML methodologies by performance and transferability.

METHODS

One-hundred and twenty-one transcripts of two OSCE scenarios were manually annotated per utterance across 19 communication skills and content areas. Utterances were converted to two types of numeric sentence vector representations and were paired with three types of ML algorithms. First, ML models (MLMs) were evaluated using a five K-fold cross-validation technique on all transcripts in one scenario to generate precision and recall, and their harmonic mean, F1 scores. Second, ML models were trained on all 101 transcripts from scenario 1 and tested for transferability on 20 scenario 2 transcripts.

RESULTS

Performance testing in the K-fold cross-validation demonstrated relatively high mean F1 scores: median 0.87 and range 0.53-0.98 across all 19 labels. Transferability testing demonstrated success: F1 median 0.76 and range 0.46-0.97. The combination of a bi-directional long short-term memory neural network (biLSTM) algorithm with GenSen numeric sentence vector representations was associated with greater F1 scores across both performance and transferability (P < .005).

CONCLUSIONS

We report the first application of ML in the context of student-SP OSCEs. We demonstrated that several MLMs automatically labelled OSCE transcripts for a range of interview content and some clinical communications skills. Some MLMs achieved greater performance and transferability. Optimised MLMs could provide automated and accurate assessment of OSCEs with potential to track student progress and identify areas for further practice.

Improving Mental Health Services: A 50-Year Journey from Randomized Experiments to Artificial Intelligence and Precision Mental Health

This conceptual paper describes the current state of mental health services, identifies critical problems, and suggests how to solve them. I focus on the potential contributions of artificial intelligence and precision mental health to improving mental health services. Toward that end, I draw upon my own research, which has changed over the last half century, to highlight the need to transform the way we conduct mental health services research. I identify exemplars from the emerging literature on artificial intelligence and precision approaches to treatment in which there is an attempt to personalize or fit the treatment to the client in order to produce more effective interventions.

Supporting Innovative Person-Centred Care in Financially Constrained Environments: The WE CARE Exploratory Health Laboratory Evaluation Strategy

The COST CARES project aims to support healthcare cost containment and improve healthcare quality across Europe by developing the research and development necessary for person-centred care (PCC) and health promotion. This paper presents an overview evaluation strategy for testing 'Exploratory Health Laboratories' to deliver these aims. Our strategy is theory driven and evidence based, and developed through a multi-disciplinary and European-wide team. Specifically, we define the key approach and essential criteria necessary to evaluate initial testing, and on-going large-scale implementation with a core set of accompanying methods (metrics, models, and measurements). This paper also outlines the enabling mechanisms that support the development of the "Health Labs" towards innovative models of ethically grounded and evidenced-based PCC.

Story Arcs in Serious Illness: Natural Language Processing features of Palliative Care Conversations

OBJECTIVE

Serious illness conversations are complex clinical narratives that remain poorly understood. Natural Language Processing (NLP) offers new approaches for identifying hidden patterns within the lexicon of stories that may reveal insights about the taxonomy of serious illness conversations.

METHODS

We analyzed verbatim transcripts from 354 consultations involving 231 patients and 45 palliative care clinicians from the Palliative Care Communication Research Initiative. We stratified each conversation into deciles of "narrative time" based on word counts. We used standard NLP analyses to examine the frequency and distribution of words and phrases indicating temporal reference, illness terminology, sentiment and modal verbs (indicating possibility/desirability).

RESULTS

Temporal references shifted steadily from talking about the past to talking about the future over deciles of narrative time. Conversations progressed incrementally from "sadder" to "happier" lexicon; reduction in illness terminology accounted substantially for this pattern. We observed the following sequence in peak frequency over narrative time: symptom terms, treatment terms, prognosis terms and modal verbs indicating possibility.

CONCLUSIONS

NLP methods can identify narrative arcs in serious illness conversations.

PRACTICE IMPLICATIONS

Fully automating NLP methods will allow for efficient, large scale and real time measurement of serious illness conversations for research, education and system re-design.

Using artificial intelligence to analyse and teach communication in healthcare

Communication is a core component of effective healthcare that impacts many patient and doctor outcomes, yet is complex and challenging to both analyse and teach. Human-based coding and audit systems are time-intensive and costly; thus, there is considerable interest in the application of artificial intelligence to this topic, through machine learning using both supervised and unsupervised learning algorithms. In this article we introduce health communication, its importance for patient and health professional outcomes, and the need for rigorous empirical data to support this field. We then discuss historical interaction coding systems and recent developments in applying artificial intelligence (AI) to automate such coding in the health setting. Finally, we discuss available evidence for the reliability and validity of AI coding, application of AI in training and audit of communication, as well as limitations and future directions in this field. In summary, recent advances in machine learning have allowed accurate textual transcription, and analysis of prosody, pauses, energy, intonation, emotion and communication style. Studies have established moderate to good reliability of machine learning algorithms, comparable with human coding (or better), and have identified some expected and unexpected associations between communication variables and patient satisfaction. Finally, application of artificial intelligence to communication skills training has been attempted, to provide audit and feedback, and through the use of avatars. This looks promising to provide confidential and easily accessible training, but may be best used as an adjunct to human-based training.

•

Artificial intelligence (AI) applied to health professional-patient communication enables efficient audit and feedback.

•

Very recent advances have increased the ability of AI to encode the complexity in human interaction.

•

AI can now encode words as well as a person does, as well as emotion and non-verbal aspects of communication.

•

AI coding has been shown to be moderately to substantially reliable.

•

Translation into the real world has yet to be demonstrated.