Emotion recognition in doctor-patient interactions from real-world clinical video database: Initial development of artificial empathy

Validating nonverbal cues for assessing physician empathy in telemedicine: a Delphi study

Nonverbal communication is essential in physician-patient interaction, especially in telemedicine where verbal cues may be limited. This study aimed to identify and validate key nonverbal cues for assessing physician empathy in telemedicine consultations through a Delphi method. A three-round Delphi study was conducted from June to November 2022, involving various experts, including academics, healthcare professionals, AI/telemedicine researchers, industry professionals, and patients. Experts evaluated the importance, validity, and reliability of potential nonverbal cues. Consensus was determined based on median responses and expert scoring percentages, with statistical agreement and stability assessed using Kendall's coefficient of concordance (Kendall's W) and Wilcoxon signed-rank test. Analyses were conducted using SPSS, version 23.0 with significance set at p < 0.05. Of the 72 experts invited, 37 (51%) agreed to participate, with 35 completing the first round (95% completion rate). Eight significant nonverbal cues were identified in the first round, though one did not reach consensus. The second round obtained an 89% response rate (31/35), with three new cues introduced; one did not reach consensus. Round 3 achieved a 94% response rate (29/31), finalizing nine key cues: facial expression, eye contact, tone of voice, smiling, head nodding, body posture, hand gesture, distance, and environmental cues. Among these, facial expression, eye contact, and tone of voice were identified as the most crucial. Inter-expert agreement was statistically significant across all items with strong agreement on the importance (W = 0.739, p < 0.001), good agreement on their validity (W = 0.689, p < 0.001), and moderate agreement on their reliability (W = 0.452, p < 0.001). This study highlights the importance of specific nonverbal cues in telemedicine, particularly facial expression, eye contact, and tone of voice. It provides a validated foundation for developing tools to enhance physician-patient interactions and potentially improve health outcomes in telemedicine.

HGF-MiLaG: Hierarchical Graph Fusion for Emotion Recognition in Conversation with Mid-Late Gender-Aware Strategy

Emotion recognition in conversation (ERC) is an important research direction in the field of human-computer interaction (HCI), which recognizes emotions by analyzing utterance signals to enhance user experience and plays an important role in several domains. However, existing research on ERC mainly focuses on constructing graph networks by directly modeling interactions on multimodal fused features, which cannot adequately capture the complex dialog dependency based on time, speaker, modalities, etc. In addition, existing multi-task learning frameworks for ERC do not systematically investigate how and where gender information is injected into the model to optimize ERC performance. To address the above problems, this paper proposes a Hierarchical Graph Fusion for ERC with Mid-Late Gender-aware Strategy (HGF-MiLaG). HGF-MiLaG uses hierarchical fusion graph to adequately capture intra-modal and inter-modal speaker dependency and temporal dependency. In addition, HGF-MiLaG explores the effect of the location of gender information injections on ERC performance, and ultimately employs a Mid-Late multilevel gender-aware strategy in order to allow the hierarchical graph network to determine the proportion of emotion and gender information in the classifier. Empirical results on two public multimodal datasets(i.e.,IEMOCAP and MELD), demonstrate that HGF-MiLaG outperforms existing methods.

Instruments evaluating the duration and pace of clinical encounters: A scoping review

OBJECTIVE

Hurried encounters in clinical settings contribute to dissatisfaction among both patients and clinicians and may indicate and contribute to low-quality care. We sought to identify patient- or clinician-reported instruments concerning this experience of time in clinical encounters.

METHODS

We searched multiple databases from inception through July 2023. Working in duplicate without restrictions by language or clinical context, we identified published instruments or single items measuring perceptions of time adequacy in clinical encounters. We characterized these by time domain (perceived duration or pace of the encounter), responder (patient or clinician), and reference (experience of care in general or of a particular encounter).

RESULTS

Of the 96 instruments found, none focused exclusively on perception of time adequacy in clinical encounters. Nonetheless, these instruments contained 107 time-related items. Of these, 81 items (77 %) measured the perception of the encounter duration, assessing whether there was adequate consultation time overall or for specific tasks (e.g., listening to the patient, exploring psychosocial issues, formulating the care plan). Another 19 (18 %) assessed encounter pace, and 7 (7 %) assessed both duration and pace. Pace items captured actions perceived as rushed or hurried or the perception that patients and clinicians felt pressed for time or rushed. Patients were the respondents for 76 (71 %) and clinicians for 24 (22 %) items. Most patient-reported items (48 of 76) referred to the patient's general care experience.

CONCLUSION

There are existing items to capture patient and clinician perceptions of the duration and/or pace of clinical encounters. Further work should ascertain their ability to identify hurried consultations and to detect the effect of interventions to foster unhurried encounters.

PRACTICE IMPLICATIONS

The available items assessing patient and clinician perceptions of duration and pace can illuminate the experience of time adequacy in clinical encounters as a target for quality improvement interventions. These items may capture unintended consequences on perceived time for care of interventions to improve healthcare access and efficiency.

Evaluating the Potential and Pitfalls of AI-Powered Conversational Agents as Humanlike Virtual Health Carers in the Remote Management of Noncommunicable Diseases: Scoping Review

BACKGROUND

The rising prevalence of noncommunicable diseases (NCDs) worldwide and the high recent mortality rates (74.4%) associated with them, especially in low- and middle-income countries, is causing a substantial global burden of disease, necessitating innovative and sustainable long-term care solutions.

OBJECTIVE

This scoping review aims to investigate the impact of artificial intelligence (AI)-based conversational agents (CAs)-including chatbots, voicebots, and anthropomorphic digital avatars-as human-like health caregivers in the remote management of NCDs as well as identify critical areas for future research and provide insights into how these technologies might be used effectively in health care to personalize NCD management strategies.

METHODS

A broad literature search was conducted in July 2023 in 6 electronic databases-Ovid MEDLINE, Embase, PsycINFO, PubMed, CINAHL, and Web of Science-using the search terms "conversational agents," "artificial intelligence," and "noncommunicable diseases," including their associated synonyms. We also manually searched gray literature using sources such as ProQuest Central, ResearchGate, ACM Digital Library, and Google Scholar. We included empirical studies published in English from January 2010 to July 2023 focusing solely on health care-oriented applications of CAs used for remote management of NCDs. The narrative synthesis approach was used to collate and summarize the relevant information extracted from the included studies.

RESULTS

The literature search yielded a total of 43 studies that matched the inclusion criteria. Our review unveiled four significant findings: (1) higher user acceptance and compliance with anthropomorphic and avatar-based CAs for remote care; (2) an existing gap in the development of personalized, empathetic, and contextually aware CAs for effective emotional and social interaction with users, along with limited consideration of ethical concerns such as data privacy and patient safety; (3) inadequate evidence of the efficacy of CAs in NCD self-management despite a moderate to high level of optimism among health care professionals regarding CAs' potential in remote health care; and (4) CAs primarily being used for supporting nonpharmacological interventions such as behavioral or lifestyle modifications and patient education for the self-management of NCDs.

CONCLUSIONS

This review makes a unique contribution to the field by not only providing a quantifiable impact analysis but also identifying the areas requiring imminent scholarly attention for the ethical, empathetic, and efficacious implementation of AI in NCD care. This serves as an academic cornerstone for future research in AI-assisted health care for NCD management.

TRIAL REGISTRATION

Open Science Framework; https://doi.org/10.17605/OSF.IO/GU5PX.

Beyond words: analyzing non-verbal communication techniques in a medical communication skills course via synchronous online platform

Background

Effective doctor-patient relationships hinge on robust communication skills, with non-verbal communication techniques (NVC) often overlooked, particularly in online synchronous interactions. This study delves into the exploration of NVC types during online feedback sessions for communication skill activities in a medical education module.

Methods

A cohort of 100 first-year medical students and 10 lecturers at the Faculty of Medicine, Universiti Kebangsaan Malaysia (UKM), engaged in communication skills activities via Microsoft Teams. Sessions were recorded, and lecturer NVC, encompassing body position, facial expressions, voice intonation, body movements, eye contact, and paralinguistics, were meticulously observed. Following these sessions, students provided reflective writings highlighting their perceptions of the feedback, specifically focusing on observed NVC.

Results

The study identified consistent non-verbal communication patterns during feedback sessions. Lecturers predominantly leaned forward and toward the camera, maintained direct eye contact, and exhibited dynamic voice intonation. They frequently engaged in tactile gestures and paused to formulate thoughts, often accompanied by filler sounds like "um" and "okay." This consistency suggests proficient use of NVC in providing synchronous online feedback. Less observed NVC included body touching and certain paralinguistic cues like long sighs. Initial student apprehension, rooted in feelings of poor performance during activities, transformed positively upon observing the lecturer's facial expressions and cheerful intonation. This transformation fostered an open reception of feedback, motivating students to address communication skill deficiencies. Additionally, students expressed a preference for comfortable learning environments to alleviate uncertainties during feedback reception. Potential contrivances in non-verbal communication (NVC) due to lecturer awareness of being recorded, a small sample size of 10 lecturers limiting generalizability, a focus solely on preclinical lecturers, and the need for future research to address these constraints and explore diverse educational contexts.

Conclusion

Medical schools globally should prioritize integrating NVC training into their curricula to equip students with essential communication skills for diverse healthcare settings. The study's findings serve as a valuable reference for lecturers, emphasizing the importance of employing effective NVC during online feedback sessions. This is crucial as NVC, though occurring online synchronously, remains pivotal in conveying nuanced information. Additionally, educators require ongoing professional development to enhance proficiency in utilizing NVC techniques in virtual learning environments. Potential research directions stemming from the study's findings include longitudinal investigations into the evolution of NVC patterns, comparative analyses across disciplines, cross-cultural examinations, interventions to improve NVC skills, exploration of technology's role in NVC enhancement, qualitative studies on student perceptions, and interdisciplinary collaborations to deepen understanding of NVC in virtual learning environments.

Text-Mining and Video Analytics of COVID-19 Narratives Shared by Patients on YouTube

This study explores how individuals who have experienced COVID-19 share their stories on YouTube, focusing on the nature of information disclosure, public engagement, and emotional impact pertaining to consumer health. Using a dataset of 186 YouTube videos, we used text mining and video analytics techniques to analyze textual transcripts and visual frames to identify themes, emotions, and their relationship with viewer engagement metrics. Findings reveal eight key themes: infection origins, symptoms, treatment, mental well-being, isolation, prevention, government directives, and vaccination. While viewers engaged most with videos about infection origins, treatment, and vaccination, fear and sadness in the text consistently drove views, likes, and comments. Visuals primarily conveyed happiness and sadness, but their influence on engagement varied. This research highlights the crucial role YouTube plays in disseminating COVID-19 patient narratives and suggests its potential for improving health communication strategies. By understanding how emotions and content influence viewer engagement, healthcare professionals and public health officials can tailor their messaging to better connect with the public and address pandemic-related anxieties.

Exploring factors affecting patient satisfaction in online healthcare: A machine learning approach grounded in empathy theory

Objective

Empathy between doctors and patients is crucial in enhancing patient satisfaction with medical consultations. This study, grounded in empathy theory, employs natural language processing and machine learning algorithms to explore the factors influencing patient satisfaction in online healthcare services, particularly the impact of doctor-patient empathy.

Methods

Utilizing the three dimensions of the Jefferson Scale of Physician Empathy, seven variables were extracted from patient-doctor dialogs as independent variables, with patient satisfaction as the dependent variable. Employing machine learning algorithms, a classification model was constructed to identify the best-fitting model for exploring the pivotal factors influencing patient satisfaction in online medical services. The optimal model was then chosen to investigate the essential factors impacting patients' satisfaction with online healthcare.

Results

A total of 7586 data points were collected, with 5447 consultation dialogs (71.8%) receiving a satisfactory rating from patients. LightGBM emerged as the best-performing model, achieving an F1 score of 0.78 and an area under the curve value of 0.81. Factors within the Standing in Patient's Shoes and Perspective Taking dimensions were identified as key determinants of patient satisfaction in online healthcare services.

Conclusion

This study broadens the conventional scope of applying empathy theory, signifying its crucial role in cultivating doctor-patient empathy within the realm of online healthcare and elevating the overall quality of medical services. The findings indicate that two pivotal factors influencing patients' satisfaction with online healthcare are doctors' perceived competence and ability to empathize, understanding patients' perspectives, and offering assistance.

Artificial intelligence and obesity management: An Obesity Medicine Association (OMA) Clinical Practice Statement (CPS) 2023

Background

This Obesity Medicine Association (OMA) Clinical Practice Statement (CPS) provides clinicians an overview of Artificial Intelligence, focused on the management of patients with obesity.

Methods

The perspectives of the authors were augmented by scientific support from published citations and integrated with information derived from search engines (i.e., Chrome by Google, Inc) and chatbots (i.e., Chat Generative Pretrained Transformer or Chat GPT).

Results

Artificial Intelligence (AI) is the technologic acquisition of knowledge and skill by a nonhuman device, that after being initially programmed, has varying degrees of operations autonomous from direct human control, and that performs adaptive output tasks based upon data input learnings. AI has applications regarding medical research, medical practice, and applications relevant to the management of patients with obesity. Chatbots may be useful to obesity medicine clinicians as a source of clinical/scientific information, helpful in writings and publications, as well as beneficial in drafting office or institutional Policies and Procedures and Standard Operating Procedures. AI may facilitate interactive programming related to analyses of body composition imaging, behavior coaching, personal nutritional intervention & physical activity recommendations, predictive modeling to identify patients at risk for obesity-related complications, and aid clinicians in precision medicine. AI can enhance educational programming, such as personalized learning, virtual reality, and intelligent tutoring systems. AI may help augment in-person office operations and telemedicine (e.g., scheduling and remote monitoring of patients). Finally, AI may help identify patterns in datasets related to a medical practice or institution that may be used to assess population health and value-based care delivery (i.e., analytics related to electronic health records).

Conclusions

AI is contributing to both an evolution and revolution in medical care, including the management of patients with obesity. Challenges of Artificial Intelligence include ethical and legal concerns (e.g., privacy and security), accuracy and reliability, and the potential perpetuation of pervasive systemic biases.